Skip to content

Add real-path grader smoke tests for training-based tasks #61

Description

@GTC2333

Problem

The BCI and Sleep Oracle modes skip model import, data preprocessing, training, and inference, and directly emit predefined chance-level scores. Passing Oracle validation therefore confirms only that the scorer can emit valid JSON, not that the real grader path works.

Trigger scenario

A change introduces a regression in preprocessing, input units, epoch alignment, model loading, Docker mounts, training, inference, or prediction ordering. Oracle validation still passes, while official scoring fails or produces incorrect results.

Impact

  • Validation gives false confidence in grader correctness.
  • Regressions may only appear after an expensive full benchmark run.
  • The current Oracle path cannot protect the real evaluation pipeline.

Expected fix

Keep the current fast Oracle, but add a small evaluator-only golden fixture that exercises the real path:

  • run actual preprocessing and model import;
  • perform a minimal training and inference step;
  • compute real metrics;
  • cover Docker mounts and submission isolation;
  • remain small enough for CI and task validation.

This does not require running the full nine-subject BCI evaluation or the full Sleep-EDF training workload.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions