Skip to content

docs: explain benchmark-backed experiment workflows - #352

Draft
joshblack wants to merge 1 commit into
feat/benchmark-experiment-uifrom
docs/benchmark-experiment-workflow
Draft

joshblack wants to merge 1 commit into
feat/benchmark-experiment-uifrom
docs/benchmark-experiment-workflow

Conversation

@joshblack

Copy link
Copy Markdown
Member

Closes #329.

This PR completes the benchmark-backed experiment workflow guidance for the stack. It adds configuration and CLI examples and explains the reference treatments, setup isolation, trial matrix, saved hierarchy, and capability-first results.

Changelog

New

  • Add complete benchmark-backed experiment examples.
  • Add guidance for comparing capability results and drilling into scenarios.

Changed

  • Update benchmark, experiment, plan, treatment, and result references for the new workflow.
  • Update trial-count guidance for the two automatic reference treatments.
  • Clarify that grouping snapshots do not freeze setup code or scenario contents.

Removed

None.

Validation

  • pnpm exec agent-eval experiment plan create benchmark-comparison --experiments .agents/tmp/doc-example/experiments --benchmarks .agents/tmp/doc-example/benchmarks --scenarios .agents/tmp/doc-example/scenarios --output-path .agents/tmp/doc-example/plan.json: passed; produced the expected three reference/custom treatment trials.
  • pnpm run format:diff: passed.
  • pnpm exec turbo run lint lint:npm: passed.
  • pnpm exec turbo run test -- --run: passed.
  • pnpm exec turbo run type-check --filter='!@primer/agent-eval-website': passed.

The website build was not repeated for this documentation-only entry. No live agent evaluations were run.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: a259adf3-3caa-433f-a9d7-7ac4dbc5101c
@joshblack
joshblack added this pull request to stack #353 October 6, 2026 21:22

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant