feat: run experiments across benchmark scenarios

[Copilot]

Experiments currently require callers to list scenarios manually, even when those scenarios already belong to a benchmark. This adds a benchmark reference as an alternative.

Changelog

New

• Set benchmark to a benchmark filename ID instead of scenarios:

  export const experiment = defineConfig({
    name: 'Treatment comparison',
    description: 'Compare treatments across the design system benchmark',
    models: ['gpt-5.5'],
    benchmark: 'design-system',
    treatments: [{name: 'Skill'}],
  })

• Use --benchmarks <directory> with experiment run and plan commands when benchmarks are outside ./benchmarks.

Changed

• Expand the benchmark’s capabilities into distinct experiment scenarios. Experiment models and setup remain independent of benchmark models and setup.
• Require exactly one of scenarios or benchmark in experiment configuration.

• Fixes Create a way to run an experiment on a benchmark #329

[joshblack]

@copilot I think we need to find a good way to show result for experiments based on benchmarks that is distinct from how we currently compare scenarios. I think we'll want the taxonomy to include if the capability improved or not alongside the scenario diff. It'd be great if this could be its own path separate from the current experiment stuff so that we don't have to branch each time (instead its just a different call chain) but don't hold close to that if it makes things worse.

[Copilot]

@copilot I think we need to find a good way to show result for experiments based on benchmarks that is distinct from how we currently compare scenarios. I think we'll want the taxonomy to include if t...

The current experiment report has no capability membership or control-vs-treatment deltas, so this needs a benchmark-specific results path. Before I change code, could you confirm the intended scope and meaning of “improved”? My proposal is a separate CLI report showing capability → scenario comparisons against Control, with improvement based on directional check results (and N/A when checks are missing or incomparable). Should this also be represented in saved output/the website, and how should a capability with mixed improving and regressing checks be classified?