Review cost is requirements × rounds: shared preludes mass-stale verdicts, adjacent annotations produce empty evidence blocks

[tylerwillis]

Measured on a real PR today

A single PR writing an enforced-auth cutover runbook:

requirements in the spec:        65
verdict files recorded:         224   (~3.4 per requirement)
review rounds:                    6
one agent turn:              4h 13m   (and still going)
diff:            12 files, 2,370 insertions

Nothing was hung. That is the honest cost of getting 65 requirements to a clean check.

Why it compounds

src/annotations.ts:65-67 scopes an annotation's evidence block from its own line to the line before the file's next annotation, plus a prelude containing the file's shared imports and mocks. The prelude is deliberate and correct — the docstring names the reason:

shared imports and mocks — the classic test-neutering vector — stay under it

But it means the prelude is inside every annotation's hash in that file. Touch a shared import, a fixture, or a mock, and every requirement annotated in that file goes stale at once.

With 65 requirements sharing a handful of test files, one fix in response to one finding can stale twenty requirements. Six rounds of that is 224 verdicts.

Review cost is not linear in requirement count. It is requirements × rounds, where rounds are driven by findings — and each stale requirement is a fresh-context LLM call.

The same rule produces the opposite pathology

Also seen today, in a different PR: a requirement whose annotation was immediately followed by another annotation got an evidence block of exactly one line. Its hash covered nothing meaningful, so:

• the gate reported PASS while the covering test was failing, and
• the verdict attested to a red test, and
• fixing the test never staled that verdict, so it was never re-reviewed.

Independently confirmed: editing an assertion 80 lines down staled only the following annotation, never the one that nominally owned the test.

Both pathologies fall out of one rule. Neither is a bug in the rule — the intent is right in both directions — but the extremes are unguarded.

Three suggestions, roughly in value order

1. Lint adjacent annotations that yield an empty or near-empty block. Cheap, and it would have caught the false attestation above. If an annotation's block is a single line — no assertions between it and the next annotation — that requirement's evidence is definitionally not there. Warn at lint time, before a verdict is ever recorded against it.

2. Make review --dispatch parallelism explicit and safe. These reviews are independent by construction, which is the entire premise of fresh-context review. Serializing them is pure latency. Worth noting the failure mode this pressure produces in practice: an agent, facing dozens of stale requirements, batched six judgment reviews into a single codex exec call — so reviewer six had read reviews one through five. That is a rational response to sequential cost and it silently erodes the independence property. Better to make real parallel dispatch the easy path than to leave batching as the tempting one.

3. Report staleness causality. When check reports N stale reviews, say why — "23 requirements staled because tests/foo.py's prelude changed" is actionable; 23 separate "run 2119 review" lines are not. It would also make the shared-prelude cost visible to spec authors while they can still do something about it, which is the point at which granularity is chosen.

Not proposed

Not loosening the prelude rule. It is guarding the right thing, and cheapening it to save review time would trade a real security property for latency.

Not fewer requirements as guidance. Fine granularity buys precise coverage and precise failure attribution; the problem is that its cost is invisible until you are six rounds deep.

Related: #26 (mutation step in judgment reviews) — adjacent, since a required mutation adds per-review cost and makes parallel dispatch worth more.