Problem
The BCI IV-2a grader currently trains each subject once with seed 42. Model initialization, mini-batch order, Dropout, and BatchNorm can affect the selected validation checkpoint and the final Session-E Kappa.
Trigger scenario
The same submitted architecture produces noticeably different Session-E results across training seeds, making a single-seed score unusually high or low.
Impact
- Model rankings may depend on one initialization.
- Stable architectures cannot be distinguished from lucky runs.
- Small score differences between agents or harnesses may not be meaningful.
Expected fix
Keep the current within-subject cross-session protocol and the same fixed 230/58 train-validation split, but train every subject with three predefined seeds, for example [42, 123, 2026].
For each seed, continue selecting the checkpoint by validation loss and evaluating it on that subject's private Session E. Report:
- mean Kappa across seeds as the primary score;
- Kappa standard deviation across seeds;
- per-seed results;
- per-subject mean and standard deviation.
This issue does not propose K-fold cross-validation or changing the existing train-validation split.
Problem
The BCI IV-2a grader currently trains each subject once with seed
42. Model initialization, mini-batch order, Dropout, and BatchNorm can affect the selected validation checkpoint and the final Session-E Kappa.Trigger scenario
The same submitted architecture produces noticeably different Session-E results across training seeds, making a single-seed score unusually high or low.
Impact
Expected fix
Keep the current within-subject cross-session protocol and the same fixed 230/58 train-validation split, but train every subject with three predefined seeds, for example
[42, 123, 2026].For each seed, continue selecting the checkpoint by validation loss and evaluating it on that subject's private Session E. Report:
This issue does not propose K-fold cross-validation or changing the existing train-validation split.