Evaluate Sleep-EDF across three predefined train-validation subject splits

[GTC2333]

Problem

Sleep-EDF currently uses one fixed public split: subjects 0-13 for training and 14-15 for validation. The selected checkpoint and final private-test score may depend on whether subjects 14-15 are representative validation subjects.

Trigger scenario

A model stops early or selects a poor checkpoint on validation subjects 14-15, while the same architecture behaves differently when another public subject pair is used for validation.

Impact

• A single train-validation subject split may overestimate or underestimate a model.
• Validation-subject sensitivity is not measured.
• Small ranking differences may reflect the split rather than architecture quality.

Expected fix

Predefine and freeze three train-validation splits over public subjects 0-15. Each split must contain:

• 14 training subjects;
• 2 validation subjects.

For every split, use the same fixed training seed, select the checkpoint by validation loss, and evaluate on the unchanged private test subjects 16-19.

Report each split result, the mean test Kappa, and the split-level standard deviation. The three splits must be fixed before evaluation and must never be selected or changed using private-test performance.

[HaoxuanLiTHUAI]

Thanks for raising this. We agree that validation-subject sensitivity should be measured, and we will keep this issue open. We are intentionally not changing the current main split or scoring protocol yet: evaluating three train/validation splits roughly triples the training cost and requires a new frozen task version, split manifest, metric schema, timeout budget, and baseline. A dedicated experimental branch may be used to implement and calibrate the predefined splits without affecting main. We plan to revisit and finalize this for the next BrainPilotBench release; all splits will be selected from public information and frozen before any private-test evaluation, while existing v0.2 results remain immutable.