Apply the shared eval contract across existing harnesses

[miguelg719]

Applies the shared eval contract to Claude Code, Codex, Deepagents, Eve, FX, Mastra, and Pi. The autonomy policy uses an additive native system channel where available and one task prefix otherwise, preserving each harness's stock instructions.

Forwards session loss, retained evidence, usage-presence flags, and the actual budget unit into the shared runner. Pi reasoning follows explicit override, model suffix, then SDK default; Codex forwards the requested reasoning effort even with a caller-owned SDK. Pi aliases use the canonical model for token-price computation, and console diagnostics survive normal log persistence. Eve rejects malformed provider/model IDs before execution. FX totals count cached input once: 1,000 input tokens, including 600 cached, plus 200 output and 50 reasoning tokens total 1,250. Native Anthropic thinking blocks are retained as reasoning without appearing in visible Deepagents output.

Validation: 88 focused Codex/Eve/Pi TypeScript tests across six files and 32 Deepagents Python tests pass. Eight FX runner tests also pass, including the cached-input total regression. A separate 12-test Eve run passes with both budget overrides set in the invoking environment. The final consolidated build and workspace typechecks pass with the FX correction, along with all 1,006 eval tests and 769 runtime tests. Existing Pi loader regressions cover suffix/override precedence. Provider responses are local fixtures; this does not establish live model performance or disconnect recovery.

Uses @openai/codex-sdk 0.153.4 with its bundled CLI, which supports max, ultra, and persistent reasoning effort. The allowlist is checked against the SDK type, and invalid effort is rejected before browser startup. No EVAL_CODEX_PATH override is needed for these values.

Validation for the SDK update: 30 Codex runner/adapter/bridge tests and 26 SDK session tests pass, with eval and SDK typechecks. A live smoke run at max completed in 14 seconds using the bundled CLI and local Chrome: the agent used the mounted facade, returned the Example Domain heading and IANA link, and produced browser evidence.

Stack position: 13/17. Base: evals/consolidation-12-shared-runner-contract. Review the diff against the immediate predecessor.

Reviewer entry points:

• packages/evals/framework/codexRunner.ts
• packages/evals/framework/piRunner.ts
• packages/integrations/deepagents/runner/run_eval.py
• packages/evals/framework/fxRunner.ts

[changeset-bot]

⚠️ No Changeset found

Latest commit: 2c5f43b

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

[cubic-dev-ai]

All reported issues were addressed across 51 files

Architecture diagram

sequenceDiagram
    participant O as Operator (CI/Dev)
    participant R as Eval Runner (externalRunner)
    participant B as Step Budget Resolver
    participant E as Env Config
    participant A as Tool Adapter (per-harness)
    participant S as Agent SDK (per-harness)
    participant T as Trajectory Adapter
    participant V as Verifier

    Note over R: Shared Eval Contract - Budget, System Prompt, Reasoning, Evidence

    O->>R: runHarness(plan, model, toolSurface)
    R->>E: Read EVAL_<HARNESS>_MAX_STEPS
    E-->>R: raw budget
    R->>B: resolveStepBudget(dataset, harnessDefault)
    B-->>R: stepBudget + unit (turns/tool_calls/agent_steps)

    Note over R: System Prompt Policy
    alt Native channel available (claude, deepagents, mata, pi)
        R->>S: runSession(prompt, nativeSystemPrompt)
        S-->>R: System prompt appended/preset preserved
    else Caller-owned SDK (codex)
        R->>S: runSession(prompt + EVAL_SYSTEM_PROMPT prefix)
    end

    Note over R: Tool Surface Mounting
    R->>A: prepareAdapter(toolSurface)
    alt stagehand_facade (default)
        A->>S: Mount MCP/bridge runtime
        S-->>A: browser Session info + captureEvidence + sessionLoss API
    else stagehand_facade_legacy (explicit override)
        A-->>R: Explicit historical surface
    end
    A-->>R: Adapter (browserSession, evidence)

    Note over R: Budget Enforcement
    R->>S: Execute session (maxSteps, budgetUnit)
    S-->>R: Streamed events (tool_calls, reasoning, usage)
    alt Step budget exhausted
        S-->>R: max_turns / step_budget status
        R-->>V: No final answer (gate fires)
    else Session completes
        S-->>R: terminal message (committed turn only)
    end

    Note over R: Evidence & Session Loss
    R->>A: captureEvidence() / browserSessionLoss()
    alt CDP/browser disconnected
        A-->>R: SessionLoss info
        R-->>V: Record browser_session_lost, fail closed
    else Success continues
        A-->>R: ProbeEvidence
    end

    Note over R: Reasoning and Trajectory
    S-->>R: Events (reasoning blocks, text, tool calls)
    R->>T: Adapt to normalized trajectory
    alt codex: rollout usage recovery when aborted
        R->>E: Read isolated CODEX_HOME rollout JSONL
        E-->>R: Recovered usage (reported=true)
    end
    alt Eve reasoning (step-index pairing)
        T->>T: Pair reasoning.completed with stepIndex
    else Pi thinking precedence
        alt EVAL_PI_THINKING set
            R->>S: thinkingLevel = explicit override
        else Model suffix present
            R->>S: thinkingLevel = suffix (e.g. :high)
        else SDK default
            R->>S: thinkingLevel = medium
        end
    end
    T-->>R: Trajectory (steps + reasoning + finalAnswer)

    Note over R: Verification and Result
    R->>V: Verify(trajectory, rubric)
    V-->>R: VerifierError or outcome
    R-->>O: TaskResult (budget unit, usage presence, evidence, finalAnswer)

Loading

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>

[socket-security]

Review the following changes in direct dependencies. Learn more about Socket for GitHub.

Diff	Package	Supply Chain Security	Vulnerability	Quality	Maintenance	License
	npm/​@​openai/​codex-sdk@​0.147.0 ⏵ 0.153.4	+26		+1	+1

View full report

[socket-security]

Review the following changes in direct dependencies. Learn more about Socket for GitHub.

Diff	Package	Supply Chain Security	Vulnerability	Quality	Maintenance	License
	npm/​@​openai/​codex-sdk@​0.147.0 ⏵ 0.153.4	+26		+1	+1

View full report