test(bench): falsification fixtures for oracles + typed runner outcomes

[thymikee]

Summary

Implements #1481 — the two surviving deterministic PR-time quality items carved out of #1406 for the help-conformance bench (the repo's single non-gating small-model planning oracle):

1. Falsification fixtures for benchmark oracles

• Every named expectation in EXPECTATION_SCORERS (scripts/help-conformance-case-checks.mjs, exported as KNOWN_EXPECTATIONS) now has a fixture entry in the new scripts/__tests__/help-conformance-expectation-fixtures.ts: a minimal passing witness, at least one known-bad counterexample, and a metamorphic variant where useful (a second passing plan built from different domain nouns, proving the scorer keys on plan shape rather than memorizing content).
• scripts/__tests__/help-conformance-expectation-falsification.test.ts is the completeness gate (same "what enumerates N" shape as the existing topic-coverage/error-recovery-coverage gates): a new named expectation with no fixture fails it, and a stale fixture for a removed expectation also fails.
• Fixtures run through the real validatePlanCommands/scoreExpectations production pipeline, not a reimplementation, so they can't drift from what the bench actually does with a model's plan.
• Regressions included per the issue: swallowed lifecycle command prefixes (bare open/press instead of agent-device open/agent-device press), unsupported flags (wait --selector) and selector combinations (is role=... label=...), pseudo refs (@<search-ref>), shell operators (| tee), and invalid positional ordering (get @e5 text instead of get text @e5).

2. Typed runner outcomes

• scripts/help-conformance-runner-output.mjs now exports classifyRunnerOutput, returning a discriminated RunnerOutcome (declared in the new companion scripts/help-conformance-runner-output.d.mts, following the same .mjs+.d.mts pairing already used for help-conformance-sample-outputs.mjs):

  type RunnerOutcome =
    | { kind: 'success'; raw: string; commands: string[] }
    | { kind: 'runner-error'; raw: string; message: string; reason: RunnerErrorReason };

  instead of the old raw-string detectRunnerError(): string | undefined inference.
• runCase in help-conformance-bench.mjs now only calls validatePlanCommands/scoreExpectations for a 'success' outcome; a 'runner-error' result carries runnerError/runnerErrorReason and nothing else — a result can no longer carry both runnerError and model-validation checks.
• summarizeResults (help-conformance-summary.mjs) now reports passRate: null for a group with zero evaluated trials (every trial was a runner error), and the CLI's aggregate-stability print renders that as N/A instead of the misleading 0/0 (0%).
• Single-trial mode is unchanged (the Aggregate stability summary already only prints when --repeat > 1), so it still isn't mislabeled as stability evidence.

No paid model calls in either gate — everything runs through the local command-validator subprocess deterministically.

Test plan

• pnpm typecheck — clean
• pnpm lint (oxlint --deny-warnings) — clean
• pnpm check:fallow (fallow audit) — clean (0 issues in changed files)
• pnpm format:check — clean
• npx vitest run scripts/__tests__/help-conformance-* — 100/100 passing (bench, sample-outputs, topic-coverage, error-recovery-coverage, and the new expectation-falsification suite)
• Full pnpm test:unit (unit-core + subprocess-stub, ~7200 tests) — all green; two pre-existing &&-chain plan-validator tests hit the 5s timeout only under full-suite CPU contention and pass cleanly in isolation (unrelated to this change)
• node scripts/help-conformance-bench.mjs --dry-run smoke-tested end to end with an override doc

[github-actions]

Size Report

Metric	Base	Current	Diff
JS raw	2.33 MB	2.33 MB	0 B
JS gzip	765.3 kB	765.3 kB	0 B
npm tarball	890.0 kB	890.0 kB	0 B
npm unpacked	3.10 MB	3.10 MB	0 B

Startup median (7 runs, lower is better):

Scenario	Base	Current	Diff
CLI --version	23.8 ms	22.2 ms	-1.6 ms
CLI --help	54.5 ms	55.1 ms	+0.7 ms

Top changed chunks: no changes in the largest emitted chunks.

[thymikee]

Reviewed exact head 4e4fe2517d64901b2124b98e69f697c52d1a9396: code-clean. The falsification fixtures are non-vacuous and exercise the real validator/scorer pipeline; the registry-derived completeness check catches missing and stale fixtures, and the typed outcome branch keeps infrastructure errors out of model scoring. Exact-head CI is green and size is unchanged.

Before merge, please retitle the body section to ## Validation, note the 10-file/no-expansion scope, and add Closes #1481 so the PR metadata matches the repository template.

[github-actions]

PR Preview Action v1.8.1
Preview removed because the pull request was closed.
2026-08-20 10:52 UTC