Problem
BrainPilotBench currently marks a submission as submission_invalid and skips the grader whenever signals.completed is false, even if all expected artifacts were produced and passed submission verification.
Trigger scenario
A long-running task produces a complete model, inference script, and report, but the Agent is still auditing or wrapping up when the run ends because of a timeout, stream end, or another non-completed terminal state.
Impact
- Artifact quality cannot be distinguished from run-lifecycle failure.
- Complete and potentially valid artifacts receive no score.
- Harness and lifecycle regressions are harder to diagnose.
- Re-evaluation may require manually changing run state or repeating an expensive experiment.
Expected fix
Track artifact validity, Agent completion, and grader status separately.
For a submission with valid artifacts but an incomplete run:
- still execute the grader and save the artifact score;
- mark the run as timed out or incomplete;
- exclude it from the official leaderboard by default.
Submissions with missing or invalid artifacts should remain submission_invalid and should not run the grader.
Problem
BrainPilotBench currently marks a submission as
submission_invalidand skips the grader wheneversignals.completedisfalse, even if all expected artifacts were produced and passed submission verification.Trigger scenario
A long-running task produces a complete model, inference script, and report, but the Agent is still auditing or wrapping up when the run ends because of a timeout, stream end, or another non-completed terminal state.
Impact
Expected fix
Track artifact validity, Agent completion, and grader status separately.
For a submission with valid artifacts but an incomplete run:
Submissions with missing or invalid artifacts should remain
submission_invalidand should not run the grader.