The gap
A 2119 verdict answers a judgment question: does a competent fresh reviewer believe this test proves this requirement?
The thing we actually want to know is a factual one: if the property vanished, would anything go red?
The reviewer agent already names the right question. .claude/agents/2119-reviewer.md step 2 says:
judge whether they would genuinely FAIL if the requirement were violated
But it asks the reviewer to answer it by reading. On 2026-08-04, across a six-PR merge train, reading got it wrong repeatedly and executing got it right every time.
Evidence from one day
Seven mutations run by hand during adversarial reviews. Every one converted an opinion into a fact, and they cut both ways.
Caught a passing verdict resting on evidence that could not fail. A coordinator PR restored a throw that had been weakened to return true. It had a passing 2119 verdict. Reverting the fix left the suite bit-identical — 324 pass either way — and instrumenting the catch recorded zero hits across the entire suite. The test named for that path rewrote the credential file to valid JSON, so the branch never executed. A human reviewer had read it and approved. The mutation took thirty seconds.
Cleared a fix I had wrongly suspected. A different PR replaced an fs.watch assertion with readdir. I suspected it had gone vacuous. Mutating the source two ways — auto-create the parent directory, drop a file in it — made each surviving assertion fail. Not vacuous. Exoneration is as valuable as condemnation, and reading could not settle it either way.
Found two holes behind a passing verdict and two prior approving reviews. A PWA board PR: 7 mutations, 5 killed, 2 survived.
if (r.ok) → if (true) survived, because the 401/500 test cases supplied invalid JSON bodies — so they fell back because r.json() threw, not because the status was checked. The requirement's own word was "successful authenticated fetch"; nothing enforced the "successful".- Appending
?token=… to the request URL survived. "The token never appears in a URL" had been verified by inspection in two prior reviews. Nothing enforced it.
Proved guards were load-bearing — deleting a service-worker SSE early-return, and deleting a package's test script, each turned CI red exactly as claimed.
Seven for seven on producing information that reading did not.
Proposal: the reviewer executes, the tool records
1. Reviewer agent (.claude/agents/2119-reviewer.md) — change the verb.
Step 2 currently says judge whether they would fail. Make it: make them fail. Break the specific property the requirement names, in a scratch copy, run the evidence tests, and observe.
This requires relaxing one constraint. The agent is currently "read-only apart from the verdict commands" — which forbids exactly the edit a mutation needs. Scope the relaxation precisely: mutation is permitted only in a throwaway copy outside the working tree, never in place, and the working tree must be verifiably unchanged at exit. The agent already has Bash.
2. Verdict schema — add a field for the mutation and its outcome.
Today a verdict is reviewId, requirementId, hash, verdict, summary, timestamp. There is no field recording how the reviewer established the verdict. Add one: what was broken, and what failed as a result.
This is the part that makes the practice durable rather than a habit. A prose summary saying "the tests genuinely verify this" is unfalsifiable. "broke X at file:line → tests A and B failed" is checkable by a human months later, and a missing mutation record becomes visible in audit.
3. check validates the field's presence and specificity — it must never run mutations.
Keep the checker static. It runs in ~1.5s today; that property is why it gets run rather than routed around. Teaching it to execute arbitrary suites in every language, deterministically, in CI, is a different product. The reviewer runs; the checker records.
4. review instruction generation (src/review.ts) — emit a mutation prompt naming the requirement's specific criterion, so the reviewer is asked to break that property rather than something incidental.
Two honest limits
Killing a mutant proves a test can fail, not that it fails for the right reason. An unrelated assertion tripping still counts as a kill. This raises the floor; it does not certify.
Targeted beats exhaustive. What worked was breaking the specific property the requirement names. Stryker/mutmut-style tools making thousands of random AST edits are a different, far more expensive thing that mostly produces noise — and they create an incentive to write mutation-resistant tests rather than meaningful ones. Do not adopt a mutation framework; adopt one targeted mutation per review.
The independence point
The mutation must be chosen by the reviewer, never the author. An author-chosen mutation is self-supplied evidence — the exact failure class 2119 exists to catch. The reviewer picking what to break is where the whole value lives, and it is why this belongs in the review step rather than in a test-authoring guideline.
The gap
A 2119 verdict answers a judgment question: does a competent fresh reviewer believe this test proves this requirement?
The thing we actually want to know is a factual one: if the property vanished, would anything go red?
The reviewer agent already names the right question.
.claude/agents/2119-reviewer.mdstep 2 says:But it asks the reviewer to answer it by reading. On 2026-08-04, across a six-PR merge train, reading got it wrong repeatedly and executing got it right every time.
Evidence from one day
Seven mutations run by hand during adversarial reviews. Every one converted an opinion into a fact, and they cut both ways.
Caught a passing verdict resting on evidence that could not fail. A coordinator PR restored a
throwthat had been weakened toreturn true. It had a passing 2119 verdict. Reverting the fix left the suite bit-identical — 324 pass either way — and instrumenting thecatchrecorded zero hits across the entire suite. The test named for that path rewrote the credential file to valid JSON, so the branch never executed. A human reviewer had read it and approved. The mutation took thirty seconds.Cleared a fix I had wrongly suspected. A different PR replaced an
fs.watchassertion withreaddir. I suspected it had gone vacuous. Mutating the source two ways — auto-create the parent directory, drop a file in it — made each surviving assertion fail. Not vacuous. Exoneration is as valuable as condemnation, and reading could not settle it either way.Found two holes behind a passing verdict and two prior approving reviews. A PWA board PR: 7 mutations, 5 killed, 2 survived.
if (r.ok)→if (true)survived, because the 401/500 test cases supplied invalid JSON bodies — so they fell back becauser.json()threw, not because the status was checked. The requirement's own word was "successful authenticated fetch"; nothing enforced the "successful".?token=…to the request URL survived. "The token never appears in a URL" had been verified by inspection in two prior reviews. Nothing enforced it.Proved guards were load-bearing — deleting a service-worker SSE early-return, and deleting a package's
testscript, each turned CI red exactly as claimed.Seven for seven on producing information that reading did not.
Proposal: the reviewer executes, the tool records
1. Reviewer agent (
.claude/agents/2119-reviewer.md) — change the verb.Step 2 currently says judge whether they would fail. Make it: make them fail. Break the specific property the requirement names, in a scratch copy, run the evidence tests, and observe.
This requires relaxing one constraint. The agent is currently "read-only apart from the verdict commands" — which forbids exactly the edit a mutation needs. Scope the relaxation precisely: mutation is permitted only in a throwaway copy outside the working tree, never in place, and the working tree must be verifiably unchanged at exit. The agent already has
Bash.2. Verdict schema — add a field for the mutation and its outcome.
Today a verdict is
reviewId, requirementId, hash, verdict, summary, timestamp. There is no field recording how the reviewer established the verdict. Add one: what was broken, and what failed as a result.This is the part that makes the practice durable rather than a habit. A prose summary saying "the tests genuinely verify this" is unfalsifiable.
"broke X at file:line → tests A and B failed"is checkable by a human months later, and a missing mutation record becomes visible in audit.3.
checkvalidates the field's presence and specificity — it must never run mutations.Keep the checker static. It runs in ~1.5s today; that property is why it gets run rather than routed around. Teaching it to execute arbitrary suites in every language, deterministically, in CI, is a different product. The reviewer runs; the checker records.
4.
reviewinstruction generation (src/review.ts) — emit a mutation prompt naming the requirement's specific criterion, so the reviewer is asked to break that property rather than something incidental.Two honest limits
Killing a mutant proves a test can fail, not that it fails for the right reason. An unrelated assertion tripping still counts as a kill. This raises the floor; it does not certify.
Targeted beats exhaustive. What worked was breaking the specific property the requirement names. Stryker/mutmut-style tools making thousands of random AST edits are a different, far more expensive thing that mostly produces noise — and they create an incentive to write mutation-resistant tests rather than meaningful ones. Do not adopt a mutation framework; adopt one targeted mutation per review.
The independence point
The mutation must be chosen by the reviewer, never the author. An author-chosen mutation is self-supplied evidence — the exact failure class 2119 exists to catch. The reviewer picking what to break is where the whole value lives, and it is why this belongs in the review step rather than in a test-authoring guideline.