Skip to content

Deferred Product Decisions

This page records deliberate deferred product decisions and their revisit criteria. It is not a release roadmap or a substitute for issue tracking. An entry is removed when it is implemented or no longer applies.

Every entry must include an explicit revisit criterion. Ideas without one belong in issue tracking, not here.

The Changelog records what happened. This page records what was deliberately deferred and the observable condition that would justify reconsidering it.

Parquet Input

Status: Deferred.

Reason: CSV addresses the current adoption barrier without adding pyarrow or optional-format complexity.

Revisit criterion: Demonstrated user demand for Parquet, or observed dataset scale or type-preservation limitations that CSV cannot adequately address.

Direction if reopened: Prefer optional dependency support rather than adding pyarrow to the base runtime installation.

In-Memory DataFrame Datasets

Status: Deferred.

Reason: Pathless or in-memory datasets belong to a programmatic workflow rather than being added as an exception to the current file-oriented registry.

Revisit criterion: Design of a stable high-level programmatic evaluation workflow.

Direction if reopened: Design DataFrame support together with that workflow and its artifact and feature semantics.

paths.input Legacy Compatibility

Status: Retained deliberately.

Reason: It remains part of the historical path contract even though modern registered CSV datasets are located through [datasets.*].path.

Revisit criterion: Explicit removal or redesign of the legacy reduction-based configuration path.

reduction_type Output Field

Status: Retained deliberately.

Reason: It preserves historical compatibility and is normally empty for modern registered datasets.

Revisit criterion: An explicit breaking cleanup of legacy reduction compatibility. It must not be removed piecemeal.

Independent Machine-Readable schema_version

Status: Deferred.

Reason: Current output schemas can be tied to the MELITE package version.

Revisit criterion: A demonstrated need for output schemas to evolve independently of package releases or for external consumers to negotiate schema versions.

Optimization Engine and Policy

Status: Fixed runtime architecture. The classification policy was established for v0.3.0; v0.3.2 applies the corresponding task-specific contract to regression. Optuna is the single optimization engine for tunable models; coexistence with GridSearchCV is not part of the current design. The only optimization-specific user setting is n_trials, with a normal default of 100. The sampler seed derives from the canonical RANDOM_STATE; MELITE does not expose a separate optimization seed. The smoke budget is an internal, non-configurable 5 trials.

Classification policy: Tunable classifiers maximize mean inner-CV F1-macro. Inner folds use shuffled StratifiedKFold, and outer evidence uses RepeatedStratifiedKFold. The classifier with the highest mean outer-CV F1-macro is selected; accuracy and ROC-AUC are additional evidence metrics. Targets with two labels use binary ROC-AUC, while targets with three or more labels use probability-based One-vs-One macro ROC-AUC. Original label identities are retained. Stacking has no separate Optuna search.

Regression policy: Ridge and Lasso minimize mean inner-CV RMSE over their alpha search spaces. Inner folds use shuffled KFold, and outer evidence uses RepeatedKFold. The regressor with the lowest mean outer-CV RMSE is selected; MAE and R² are additional evidence metrics. LinearRegression and PolynomialRegression are fitted without Optuna.

Fixed method: Seeded, sequential, independent TPE with n_startup_trials=20, multivariate=False, group=False, constant_liar=False, no pruning, in-memory studies, and study n_jobs=1. Each sampler receives RANDOM_STATE explicitly. Conditional branches receive adaptive allocation without quotas or an exhaustive coverage guarantee. Normal runs whose effective n_trials does not exceed n_startup_trials remain valid but warn that they remain within startup sampling and do not reach model-based TPE. Smoke mode is intentionally exempt.

Failure and evidence boundary: Failed candidate evaluations may fail their trial without aborting the study; contract, programming, and final refit failures propagate. For tunable models, completed outer and final optimization search summaries are persisted. Direct-fit-only runs may produce header-only search artifacts. Full per-trial traces are not persisted.

Execution boundary: Sklearn cross_validate remains the outer orchestrator, with task-specific tunable searches adapted through sklearn-compatible Optuna estimators. Classification and regression outer evaluation use error_score="raise" for tunable and direct models. Optuna startup accounting includes COMPLETE and PRUNED trials. MELITE's no-op pruner makes PRUNED a contract violation, while FAIL consumes budget without advancing startup accounting.

For either task, optimization within an outer split uses only its training portion. Exact aggregate-metric ties preserve configured model order. After selection, a tunable winner receives a fresh full-data optimization and fit; non-tunable winners fit directly. Export later reconstructs the selected model from the recorded final parameters and does not repeat optimization.

Compatibility: MELITE v0.3.0 validates Optuna 4.x. Optuna 5 may be adopted only after it is stable and MELITE explicitly verifies compatible sampler behavior, trial/error semantics, and optimization policy.

Optimization Evidence and Provenance

Status: Fixed task-specific artifact contracts. The classification contract was established for v0.3.0; the regression artifact family was added for v0.3.2.

For classification, MELITE persists one row per completed optimization search, not per trial, in optimization_searches.csv. Outer and final searches are distinguished by search_scope; selected applies only to outer rows and is not applicable to final rows. A Stack-only run legitimately produces a header-only search artifact. evaluations.csv and evaluation_folds.csv preserve aggregate and per-outer-split F1-macro, accuracy, and ROC-AUC evidence. results.csv contains the selected classifier and its final parameters. The F1-macro figure is derived from existing outer evidence without further model fitting.

For regression, the corresponding files are regression_optimization_searches.csv, regression_evaluations.csv, regression_evaluation_folds.csv, and regression_results.csv. Search rows exist only for Ridge and Lasso; a run containing only LinearRegression and/or PolynomialRegression produces a header-only regression search artifact. RMSE, MAE, and R² are preserved as aggregate and per-outer-split evidence, and the selected result is determined by the lowest mean outer RMSE. Regression does not currently produce an evaluation figure.

For both tasks, exact selection ties retain the first model in configured order, and full trial traces are deliberately not persisted. Outer-CV evidence describes the configured evaluation design; it is not evidence of external scientific validation or generalizability.

optimization_provenance.json records the classification cross-validation design, canonical RANDOM_STATE, effective budget, fixed policy, backend identity and runtime version, and active classifier search spaces; Stack is represented as null. regression_optimization_provenance.json records the task-specific outer and inner strategies, RMSE minimization contract, active regressors, and their search spaces; LinearRegression and PolynomialRegression are represented as null. Filesystem, data, and environment provenance remain outside these artifacts.

Classification additionally persists evaluation_class_folds.csv, containing held-out precision, recall, F1-score, and support for every original class in every outer fold, and evaluation_class_summary.csv, containing fold-level means and population standard deviations (ddof=0). Undefined per-class metrics use zero_division=0. Support summaries remain per-fold; predictions are not pooled across repeats. classification_metric_provenance.json records the binary or multiclass metric definitions and each dataset's ordered original class identities without changing optimization_provenance.json.

The MELITE package version governs these artifact schemas; no independent schema_version is added. Optimization evidence is not an operational model input: results.csv is the persisted classification parameter source used by melite export, while regression export uses regression_results.csv.

Public Classifier Extensibility

Status: Public classifier registration remains deferred.

Resolved internal decision: v0.3.0 work establishes a durable internal search-space contract that can represent discrete, integer, continuous, and conditional search policy without depending on one optimization backend.

Reason: The internal contract does not itself define a stable public API for registering user classifiers and their estimator or artifact semantics.

Revisit criterion: Actual work begins on public classifier registration.

v0.3.0 Optimization Characterization

Status: Characterization completed and passed for v0.3.0.

The calibration stage executes only the historical v0.2.5 GridSearchCV engine; no v0.3.0 candidate metrics are inspected. The later comparison is defined as the v0.2.5 engine versus the v0.3.0 engine on one shared current pinned scientific stack, not as reproduction of the historical v0.2.5 environment.

The fixed synthetic generator uses 240 samples, 20 features, 12 informative features, 4 redundant features, balanced binary classes, flip_y=0.05, and random_state=42. Stage 1 evaluates SVC only at ordered class_sep candidates 0.60, 0.70, 0.80, 0.90, and 1.00. Eligible SVC means lie in [0.65, 0.95]; the selected profile is closest to 0.80, with lower class_sep breaking an exact tie.

Stage 2 evaluates SVC, RandomForest, and XGBoost exactly once on the selected profile. Every classifier mean must lie in [0.65, 0.95], and the selected classifier mean must lie in [0.70, 0.90]. Failure at either stage stops the protocol without candidate execution or automatic fallback. If B5-0 is explicitly reopened after review, the predefined next axis is n_informative with ordered candidates (10, 14, 8, 16); additional outer repeats are not the reopen axis.

Calibration is not smoke mode: outer CV is 5 folds × 1 repeat with 3 inner folds. Later characterization uses only SVC, RandomForest, and XGBoost. fit_count estimates estimator.fit() invocations and includes the best-model refit performed by GridSearchCV. end_to_end_wall_seconds measures the complete MELITE subprocess and is descriptive, not optimization-only timing. Candidate comparison may proceed only after verifying the candidate Python interpreter and common pinned dependencies against the baseline environment recorded during the v0.3.0 calibration procedure, and after verifying the frozen selected dataset bytes against the exact recorded SHA-256.

Candidate characterization reuses the committed baseline; v0.2.5 is not rerun. The orchestrator Python version and all seven common dependency versions must match the baseline evidence exactly. During v0.3.0 characterization, the experimental input dataset was frozen byte-for-byte and protected by its recorded SHA-256 before MELITE executes. The candidate uses a fresh output directory, SVC, RandomForest, and XGBoost, 100 trials per search, 5 folds × 1 repeat outside, and 3 folds inside.

Every optimization row must account for its complete 100-trial budget (n_trials_complete + n_trials_failed == n_trials_requested) and must have no failed trials. Scientific acceptance requires each candidate classifier's mean outer F1-macro delta from baseline to be at least -0.05; a winner change is informational only, and the worst classifier delta is reported. Estimated fit count and end-to-end wall-clock remain separate evidence: wall-clock is descriptive and is not a runtime gate. The full verbose candidate console log is retained locally for diagnostics.

The completed characterization produced the following mean outer-CV F1-macro comparison:

Classifier v0.2.5 baseline v0.3.0 candidate Delta Result
SVC 0.8153189729660317 0.8285693852008889 +0.013250412234857123 Pass
RandomForest 0.7660740105672047 0.7701989893283668 +0.004124978761162135 Pass
XGBoost 0.7534739915895242 0.7366799005213484 -0.016794091068175865 Pass

All three classifiers passed the predefined -0.05 acceptance margin. SVC was the baseline and candidate winner, so the winner did not change. Under the static fit-count definition, the estimated workload fell from 42,208 fits for v0.2.5 to 4,816 for v0.3.0: approximately 88.6% fewer estimator fits, or approximately 8.8× fewer fits.

End-to-end wall-clock was 2981.05904740002 s for the baseline and 3005.0019774999237 s for the candidate, a ratio of 1.0080316859609963. Wall-clock was descriptive and not an acceptance gate. The fit-count reduction demonstrates a smaller declared optimization workload, not a general wall-clock speedup claim: the historical GridSearchCV baseline allowed parallel execution with n_jobs=-1, while the v0.3.0 Optuna policy is sequential with n_jobs=1.

Under this frozen B5 characterization condition, v0.3.0 preserved classifier selection and predictive performance within the predefined scientific margin. These results must not be generalized to every dataset or search budget. Parallel Optuna trials or searches may be reconsidered in a future version only with separate characterization; v0.3.0 retains n_jobs=1 and constant_liar=False.