simulate: --samples runs every scenario k times for pass@k / pass^k

[u9g]

Depends on livekit/protocol#1799. livekit/agents-private#457 (server fan-out) is merged and needs a follow-up for Sampling.

One sample per scenario cannot tell a flaky scenario from a broken one — and sampling only helps if it can also change the scenario's verdict.

lk agent simulate text --scenarios scenarios.yaml --samples 3 --pass-rate 0.75

• --samples K (-k) asks the server for K samples of every scenario in one run. Requires --scenarios.
• --pass-rate RATE is the share of a scenario's samples that must pass for that scenario to pass, defaulting to 0.75. It rounds to whole samples and is floored at one, so 0.75 of 3 needs 2 and a scenario may flake once in three. Requires --samples.
• A scenario is no longer all-or-nothing. 2/3 is a pass at the default rate. The run fails on scenarios that finished below the bar rather than on any failed sample, so the exit code changes for sampled runs and is unchanged without --samples.
• The TUI lists one row per scenario with its pass count, and the samples branch under it:

  ✓   2. Ask about nut allergies  2/3
       ├─ ✓ SRJ_02 sample 1
       ├─ ✗ SRJ_06 sample 2
       └─ ✓ SRJ_10 sample 3
  

  A scenario's icon settles as soon as the remaining samples cannot change the verdict, so one needing 2 of 3 goes green on its second pass rather than waiting for the third. ENTER on the scenario row opens the scenario as a whole; ENTER on a sample opens that job as before.
• The counts header gains N/M scenarios failed when a sampled run has any, because the sample counts beside it no longer imply the verdict.
• The CI report groups samples under their scenario in scenarios.yaml order and labels each one (sample 2/3).
• The counts header, the CI summary line, and the report summary carry pass@3 3/4 (0.75), pass^3 1/4 (0.25): share of scenarios that passed at least once, and that passed every time, over the scenarios whose samples have all finished. Both only report; neither gates.
• export picks up the new sampling, scenario_id and sample fields via the verbatim protojson passthrough.

On the default

At three samples every rate in [0.5, 0.833] rounds to the same two samples, so 0.75 is indistinguishable from 0.6 or 0.8. It is a three-position dial, not a percentage — worth knowing before tuning it. A scenario passing at 2/3 also displays a rate below the configured one, which is the rounding working as intended.

[u9g]

Demo: --repeats 3 against a stub server (agents-private#457 is not deployed yet). The guard without --scenarios, the pass@3 / pass^3 summary line, then the TUI: one row per scenario with attempts nested under it, cursor onto a scenario, ENTER, back, cursor onto one attempt, ENTER.

simulate-repeats.mp4

The detail views print into the terminal's own scrollback, which the recorder does not capture. ENTER on the scenario row prints this:

  ✗ Ask about nut allergies  2/3 attempts passed

  Instructions:
    Say you have a severe nut allergy and ask which dishes are safe.

  Expected:
    The agent names nut-free dishes or offers to check with the kitchen.

  ✓ Attempt 1 SRJ_02
    The agent met the expectation.

  ✗ Attempt 2 SRJ_06
    The agent did not meet the expectation.

  ✓ Attempt 3 SRJ_10
    The agent met the expectation.

ENTER on an attempt prints that job as before, titled Ask about nut allergies (attempt 2/3).

[u9g]

limit at 3 serverside

[u9g]

enforce an odd number of samples serverside