Fix scoring lifecycle and evaluator validation gaps

[HaoxuanLiTHUAI]

Summary

• decouple submission validity, Agent lifecycle, grader state, and official leaderboard eligibility
• grade valid artifacts from incomplete runs while excluding those runs from official tables by default
• add real-path BCI and Sleep preprocessing, model import, training, inference, metric, and Docker isolation smoke coverage
• chance-center TOPS-fMRI Study5 AUC scoring and freeze the v0.2 formula baseline
• clarify that the Sleep-EDF single-epoch interface cannot model cross-epoch stage transitions and bump it to v0.3

Validation

• npm run typecheck
• npm run build
• npm test — 216 tests, zero failures
• npm run test:docs
• task canary validation
• bp-bench validate all — all five task definitions and Oracle/NOP gates pass
• 19 Python metric, preprocessing, real-path, and live Docker checks pass
• live generic, TOPS, BCI, and Sleep Docker isolation checks pass
• workflow YAML and registry verification pass

Compatibility

• TOPS-fMRI v0.2 uses max(0, 2*AUC-1) for Study5 subgroup aggregation; raw AUC diagnostics are unchanged and v0.1 scores remain historical
• Sleep-EDF v0.3 changes prompt guidance only; its scorer remains unchanged from v0.2
• Evaluate BCI IV-2a with three fixed training seeds #63 and Evaluate Sleep-EDF across three predefined train-validation subject splits #65 remain open for a future Bench release and are not implemented here

Closes #60
Closes #61
Closes #62
Closes #64

[bondingelectron]

Thorough review with local verification — approving. All four fixes are correct, well-tested, and conservative with respect to existing behavior.

#60 — lifecycle decoupling (src/)

The three-axis model (artifactState / runState / graderState) is clean, and the edge semantics are right:

• lifecycleFromSignals gives reason precedence over a contradictory completed: true (fail-closed), tolerates external bundles without signals (unknown / null), and unknown lifecycle stays leaderboard-eligible — preserving backward compatibility for externally produced scores.json.
• classifyRunScores requires graderState === "scored" for eligibility; scoring_failed / unscored runs remain in the leaderboard coverage denominator exactly as before (verified by the new leaderboard test).
• buildLeaderboard filters only runCompleted === false; legacy scores.json (field absent) is unaffected. The legacy aggregate state field is preserved for existing consumers.
• CLI tests cover both branches end-to-end (valid artifacts from a timed-out run still graded + marked ineligible; missing artifacts stay submission_invalid with graderState: "not_run").

#61 — real-path smoke tests (EEG evaluators)

I was the author of the original v0.1/v0.2 scorers, so I checked the refactoring against the pinned real data, not just the synthetic fixtures:

• bciciv-2a: refactored load_epochs → preprocess_raw on the real A01T.gdf/A01E.gdf from the content-addressed release reproduces v0.2 exactly: (288, 22, 512) float32, std 6.867 µV, 72 trials/class, E-session labels None. The expected_trials parameterization (with the divisible-by-4 check) is strictly additive.
• sleep-edf: refactored load_subject_recording → preprocess_recording on the real SC4001E0 pair reproduces v0.2 exactly: (841, 1, 3000) float32, std 22.883 µV, label distribution 188/58/250/220/125. fit_model_from_arrays/predict_array preserve the exact training math (Adam lr=1e-3, weighted CE, early-stop threshold 1e-4, batched val loss) — confirmed line by line.
• The smoke fixtures genuinely exercise the production path (filter, resample, epoching, µV conversion, model import, a real optimizer step, inference, real metric computation) rather than a copy of it, embed no real EEG or private labels, and the Docker isolation tests are correctly gated behind BPB_DOCKER_TEST/BPB_TEST_INFERENCE_IMAGE with use_gpu=False so CPU-only CI runners work.
• The sys.modules restore in both contract tests fixes a real pollution bug from the original versions — thank you.
• CI additions are sane: CPU-only torch wheel for the smoke job, and eeg-real-path-smoke correctly joins the required gate.

#62 — TOPS-fMRI AUC chance-centering (v0.2)

• chance_centered_auc / aggregate_scores implement max(0, 2·AUC−1) correctly; study4 clipping semantics unchanged; raw r/AUC diagnostics preserved untouched.
• I independently recomputed the frozen v0.2 baseline from the preserved raw metrics: study4 = 0.14077015583589433, study5 = 0.514801999231065, score = 0.32778607753347966 — matches test_metrics.py to the last digit.
• output_schema.json, both READMEs, and task.yaml are consistent, and the updated Oracle note (chance submission now headlines 0.0 instead of ~0.25) is accurate. Local test_metrics.py: 4/4 pass.

#64 — sleep-edf prompt (v0.3)

The rewritten hint is scientifically precise: it keeps the useful part (dilated convs / TCN expand the receptive field within the 30 s waveform) while explicitly stating the single-epoch interface cannot encode cross-epoch stage transitions. Prompt-only change with the scorer untouched, so v0.2 score comparability is preserved, and task.yaml + README version notes are consistent.

Local verification on this branch

• tsc build, 216 Node tests (0 fail), docs smoke, bp-bench validate all → 5/5 pass (Oracle+NOP gates)
• Both contract suites (5+4), both real-path smoke suites, tops-fmri/checks/test_metrics.py — all pass
• Real-data regression on both EEG loaders as described above

Two non-blocking nits

1. src/cli.ts: execution.result!.reason prints run=undefined if a bundle ever lacks a reason; consider ?? "unknown" for consistent display. Cosmetic only.
2. Operator note (not this PR's bug): the tops-fmri Oracle runs the evaluator's apply_signature.py child with PYTHONNOUSERSITE=1, so a machine whose numpy lives only in user site-packages fails the Oracle with ModuleNotFoundError. Putting the task venv first on PATH (as env/setup-python.sh intends) fixes it. May be worth one line in the docs for maintainers running bp-bench validate outside CI.