evals: fixture recorder to turn bug reports into deterministic repros

[yawbtng]

Is your feature request related to a problem? Please describe.

Now that #2826 has restored regression evals on pull requests with a score gate, eval determinism and eval cost are per-PR concerns rather than occasional ones. Against that baseline, one gap stands out: there is no on-ramp from a bug report to a fixture.

The fixture pattern itself is well established here. 63 of the 77 task files under packages/evals/tasks/ already point at the pinned mirror (browserbase.github.io/stagehand-eval-sites/...); only 13 still hit live sites, and 10 of those are archives.gov. Tasks built on it assert exact post-conditions rather than rubrics — act/amazon_add_to_cart passes only on an exact final URL — which is what makes a failure attributable to Stagehand instead of to page drift.

But a fixture exists only when someone hand-builds a mirror. When an issue arrives with "here's my script and the site it broke on," nothing turns that into something replayable. So the fix can't be verified cheaply, and it leaves no regression test behind. The bug is closed on the strength of a manual check against a live site that may look different next week.

That gap is load-bearing for everything automated around this repo. claude.yml runs Claude Code on @claude mentions, [Claimed #...] PRs come from the actions bot, and outside bots (octo-patch, kernel-systems-bot) open unsolicited fix PRs. All of that produces candidate diffs faster than anyone can confirm they're correct — and with evals now gating PRs, finding out costs eval budget each time.

Prior art / where this came from

This started from Vercel's "software factory" for the AI SDK — seven single-purpose agents (classifier, analyzer, implementer, reviewer, bug-repro, bug-fix, backporter) that took their backlog from 1,022 open issues to 844 in four weeks and authored 25–35% of weekly merged PRs.

I don't think a direct port fits Stagehand, and I'd rather say why than pitch it:

• The volume isn't comparable. AI SDK was taking 100+ new issues/month against a ~1,000-issue backlog. Stagehand sits near break-even, with 198 PRs merged in the last 30 days. There's no throughput crisis to automate away.
• The load-bearing assumption doesn't transfer. Every agent in that pipeline works because an AI SDK bug reproduces in a sandbox with pnpm test — deterministic, seconds, free. Stagehand's correctness surface is "does an LLM driving a real browser against a live site do the right thing," which is none of those.

Dropped onto a nondeterministic substrate, a factory mostly manufactures plausible, unverifiable PRs. The piece worth borrowing isn't the agent fleet — it's the prerequisite the fleet quietly assumes.

Describe the solution you'd like

A fixture recorder for packages/evals: tooling that captures a live page interaction into a replayable static fixture in the same shape the pinned mirror already serves.

Worth noting Stagehand is unusually well-placed to build this. A general web archiver has to guess what matters on a page; Stagehand already computes the exact representation it reasons over — the accessibility tree and DOM extraction. A fixture only needs to preserve what Stagehand actually consumes, not a pixel-perfect site clone. That keeps it far smaller than proxy/HAR replay, and it fails informatively: if a recorded fixture can't reproduce the bug, the observation layer is what changed.

Smallest useful sequencing:

1. Recorder CLI in packages/evals — point it at a URL plus an interaction, get a self-contained fixture directory out.
2. Migrate a few of the remaining 13 live-site tasks. Validates the tool against tasks that already exist with known-good expectations, and introduces no new eval semantics.
3. Then wire it to bug reports: turn "script + site" into a committed fixture and a failing test — the artifact that makes a fix verifiable, whether a human or an agent wrote it.

Steps 1–2 stand on their own even if nothing agentic is ever built on top.

Describe alternatives you've considered

• Port the full factory. Rejected above — wrong prerequisite, and it would add volume to a PR queue that already has ~96 PRs older than 90 days.
• Fix flaky tasks as they break. Works, but unbounded, and produces nothing reusable for bug reproduction.
• Full HAR / proxy replay. Heavier and more brittle than needed when static mirrors in the existing shape already work.

Known limitations, stated up front

• Not every bug is fixturizable — auth walls, anti-bot, personalized or time-sensitive content. Live-site tasks should stay for those.
• Fixtures drift. They catch regressions in Stagehand, not changes in the web, and need a refresh story or they quietly stop testing anything real. Some tasks probably should stay live for exactly that reason.
• Recorded mirrors of third-party sites need the same care re: size and licensing that the existing mirrors presumably already got.

Questions for maintainers

1. How is the pinned mirror currently produced — is there an internal script I'd be duplicating?
2. With ci: restore regression evals on pull requests #2826 now gating PRs on eval score, is reducing eval variance and cost per PR something you actively want help on, or is the current budget fine?
3. Are the 13 remaining live-site tasks deliberate — archives.gov in particular looks like a stability-chosen target — or just not migrated yet?

Happy to be told the premise is wrong on any of these before writing code.

Are you willing to contribute to implementing this feature or fix?

• Yes, I can submit a PR
• Yes, but I need guidance
• No, I cannot contribute at this time