Problem
The BCI and Sleep Oracle modes skip model import, data preprocessing, training, and inference, and directly emit predefined chance-level scores. Passing Oracle validation therefore confirms only that the scorer can emit valid JSON, not that the real grader path works.
Trigger scenario
A change introduces a regression in preprocessing, input units, epoch alignment, model loading, Docker mounts, training, inference, or prediction ordering. Oracle validation still passes, while official scoring fails or produces incorrect results.
Impact
- Validation gives false confidence in grader correctness.
- Regressions may only appear after an expensive full benchmark run.
- The current Oracle path cannot protect the real evaluation pipeline.
Expected fix
Keep the current fast Oracle, but add a small evaluator-only golden fixture that exercises the real path:
- run actual preprocessing and model import;
- perform a minimal training and inference step;
- compute real metrics;
- cover Docker mounts and submission isolation;
- remain small enough for CI and task validation.
This does not require running the full nine-subject BCI evaluation or the full Sleep-EDF training workload.
Problem
The BCI and Sleep Oracle modes skip model import, data preprocessing, training, and inference, and directly emit predefined chance-level scores. Passing Oracle validation therefore confirms only that the scorer can emit valid JSON, not that the real grader path works.
Trigger scenario
A change introduces a regression in preprocessing, input units, epoch alignment, model loading, Docker mounts, training, inference, or prediction ordering. Oracle validation still passes, while official scoring fails or produces incorrect results.
Impact
Expected fix
Keep the current fast Oracle, but add a small evaluator-only golden fixture that exercises the real path:
This does not require running the full nine-subject BCI evaluation or the full Sleep-EDF training workload.