Standardize eval policy, budgets, session metadata, traces, and cost accounting

[miguelg719]

Defines one eval-owned autonomy policy and a shared result contract for external harnesses. Records policy channel/version, effective budget and unit, requested configuration, browser identity, execution reason, normalized usage and trace/timing metadata.

Unknown usage and subscription costs remain unavailable. Astra estimates also remain unavailable until per-request context-tier accounting exists. Other supported direct-provider estimates include the dated catalog source. Generic task JSON is preserved in results and verifier trajectories. Failed facade calls remain failures regardless of tool-output text; runner-owned telemetry determines run-level session loss. PR13 wires browser identity and budget helpers into existing adapters.

Validation: 195 focused tests across 11 files; eval typecheck, 13 dependency build tasks, scoped formatting and diff checks passed. Tests cover thrown-run session metadata, JSON deliverables, trace sanitization, pricing refresh rejection, normalized usage and the CLI verifiability gate.

Stack position: 12/17. Base: evals/consolidation-11-verifier-results. Each PR contains the change relative to its immediate predecessor.

Reviewer entry points:

• packages/evals/framework/evalSystemPrompt.ts
• packages/evals/framework/harnesses/externalRunner.ts
• packages/evals/framework/usageNormalization.ts

Readiness: local validation is described above; GitHub CI and automated review feedback are pending. This PR is open as non-draft so automated reviewers can run.

[changeset-bot]

⚠️ No Changeset found

Latest commit: 8f03f33

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

[cubic-dev-ai]

All reported issues were addressed across 33 files

Architecture diagram

sequenceDiagram
    participant CLI as TUI Run Command
    participant Planner as Bench Planner
    participant Runner as Eval Runner
    participant Harness as External Harness
    participant Adapter as Tool Adapter
    participant Session as Browser Session
    participant Normalizer as Usage Normalizer
    participant Cost as Cost Estimator
    participant Trace as Trace Logger
    participant DB as Braintrust

    Note over CLI,DB: Standardized Eval Execution Pipeline

    CLI->>Planner: resolveToolSurface()
    alt stagehand_facade supported
        Planner->>Planner: Default to stagehand_facade
    else fallback
        Planner->>Planner: Use first supported surface
    end

    Planner->>Runner: Build testcases with metadata
    Runner->>Runner: buildExperimentMetadata()
    Note over Runner: Extracts tool surface, model, provider, dataset from planned rows

    Runner->>Harness: execute() with row config
    Harness->>Harness: resolveStepBudget()
    Note over Harness: Env key > AGENT_EVAL_MAX_STEPS > dataset budget > default

    Harness->>Adapter: prepareToolAdapter()
    Adapter-->>Harness: browserSession info
    Harness->>Session: browserSessionFromMetadata()
    Session-->>Harness: Normalized session (id, url, debug)

    Harness->>Harness: logBrowserSession()
    Note over Harness: Level-0 line, survives all log filters

    Harness->>Harness: runExternalHarnessTask()
    Harness->>Harness: Build prompt with EVAL_SYSTEM_PROMPT
    alt systemPromptMode = native
        Harness->>Harness: Dispatch policy via system channel
    else task_prefix
        Harness->>Harness: Prepend policy to task prompt
    end

    Harness->>Harness: runSession(prompt, systemPrompt)
    Note over Harness: OpenAI models get reasoningSummary (unless EVAL_REASONING_SUMMARY=off)

    alt browserSessionLoss detected
        Harness->>Harness: Override status to sdk_error
        Harness->>Harness: Set stopReason = browser_session_lost
    end

    Harness->>Normalizer: normalizeUsage(harness, raw)
    Normalizer-->>Harness: NormalizedUsage (convention-based)

    Harness->>Cost: resolveBilledCost(harness, model, usage)
    alt reportedCostUsd present
        Cost-->>Harness: Use reported dollars
    else direct-provider harness + priced model
        Cost->>Cost: computeListCost() from pricing.json
        Cost-->>Harness: Computed cost + provenance (as_of, source)
    else subscription/unpriced/unreported
        Cost-->>Harness: cost_source = unavailable
    end

    Harness->>Trace: emitTrajectoryTrace()
    Note over Trace: Per-step lines (think, run, screenshot) + summary + answer + result

    Harness->>Harness: deriveTerminationReason()
    Note over Harness: completed | step_budget | browser_session_lost | aborted | sdk_error

    Harness->>Harness: resolveFinalAnswer()
    alt report has finalAnswer
        Harness->>Harness: Use report's finalAnswer
    else last message contains eval report
        Harness->>Harness: stripEmbeddedEvalReports()
    else plain JSON deliverable
        Harness->>Harness: Preserve as-is
    end

    Harness-->>Runner: TaskResult with browser session, usage, cost, trace

    Runner->>Runner: toSummaryResult()
    Note over Runner: Handles rows with undefined output (counts as failed)

    Runner->>DB: Report row
    Note over DB: Metadata includes browser_provider, session info, normalized usage, cost

    Runner-->>CLI: Run summary with verifiability gate
    CLI->>CLI: Check passesWithoutBrowserUse
    Note over CLI: Non-zero browser-free passes fails the gate

    alt verifier trace enabled
        Harness->>Harness: Build verifier with verbose=2
    else disabled
        Harness->>Harness: Build verifier with verbose=0
    end

Loading

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>