Skip to content

Decouple run completion status from artifact scoring #60

Description

@GTC2333

Problem

BrainPilotBench currently marks a submission as submission_invalid and skips the grader whenever signals.completed is false, even if all expected artifacts were produced and passed submission verification.

Trigger scenario

A long-running task produces a complete model, inference script, and report, but the Agent is still auditing or wrapping up when the run ends because of a timeout, stream end, or another non-completed terminal state.

Impact

  • Artifact quality cannot be distinguished from run-lifecycle failure.
  • Complete and potentially valid artifacts receive no score.
  • Harness and lifecycle regressions are harder to diagnose.
  • Re-evaluation may require manually changing run state or repeating an expensive experiment.

Expected fix

Track artifact validity, Agent completion, and grader status separately.

For a submission with valid artifacts but an incomplete run:

  • still execute the grader and save the artifact score;
  • mark the run as timed out or incomplete;
  • exclude it from the official leaderboard by default.

Submissions with missing or invalid artifacts should remain submission_invalid and should not run the grader.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions