fix(clean): editing a canonical silently lost its duplicate's content

[sturlese]

Eighth iteration of the autonomous bughunt sweep, moving to the pipeline packages. Silent, permanent content loss with no error and no route back.

Bug

test_classify_pending_promotes_orphaned_duplicate already states the invariant:

When the canonical of a duplicate leaves the inventory, the duplicate must be reprocessed so the content it still holds gets its own page (otherwise it is lost forever).

That was enforced for only one of the two ways a canonical stops holding the shared content — leaving the inventory — and not for its bytes changing. Two loci, one pass apart.

state.py — orphaned_dup tested duplicateOf not in inventory. Edit the canonical and the duplicate is never reconsidered:

pending after A changed: [('A', 'changed')]      <- B never reconsidered

B still holds the old bytes, so that document has no page anywhere. And there is no route back: B's own bytes never change again, and its canonical stays in the inventory forever, so no later pass revisits it. Silent, permanent, no error.

main.py — dedup_pending seeded the canonical index from the recorded rawHash of any processed file still in the inventory, including a file that is in pending precisely because its hash changed — i.e. one whose recorded hash it no longer serves:

state: A processed rawHash=hOLD;  pending: A(hNEW, changed), B(hOLD, new)
-> kept ['A'], B filed as duplicate of A

So a new document carrying those old bytes is deduped onto a page that does not have them. Same loss, reached from the other direction.

Fix

• dedup_pending skips any pending file's recorded hash when seeding the index. A pending doc carrying a rawHash is by definition one whose recorded hash is stale, so it cannot vouch for that content.
• classify_pending hashes the whole present inventory up front, so it can compare the canonical's current hash rather than only its presence.

hashes.get(canon, raw_hash) is what keeps the pinned second half of the existing test green: a canonical that is in the inventory but missing on disk stays deduped, exactly as before, with no churn. Hashing up front is the same total work the loop already did file-by-file — nothing is hashed that was not hashed before.

Test

All three confirmed to fail with the source changes stashed.

• test_dedup_never_points_a_duplicate_at_changed_content — end to end through run_once: pass 1 dedups B onto A, then a.md is edited; B must be processed and get its own page, and it must settle with nothing pending on the next pass. Today: assert ['A'] == ['A', 'B'].

• test_dedup_does_not_seed_the_index_with_a_changed_docs_old_hash — the other direction: C carries A's original content while A has moved on; C must be processed, not deduped. Today: assert 'duplicate' == 'processed'.

• test_classify_pending_promotes_a_duplicate_whose_canonical_changed — the unit-level half, extending the file that already pins the canonical-gone case.

• test_dedup_converges_when_a_canonical_of_several_duplicates_changes — convergence, which is the property that makes re-pending safe at all: with three identical files, editing the canonical promotes exactly one duplicate and re-dedups the other onto it, the next pass is zero-pending, and reverting the bytes settles too.

Mutation-tested: reverting either locus fails 3 tests, and dropping the missing-on-disk default fails the pinned contract — so the fix cannot drift into re-pending duplicates whose canonical is merely absent from disk.

clean 256 (+4) · corpus 48 · graph 40 · slack 13 · fetch 33 · answer 66 · benchmark 3 — bare pytest (CI mode). ruff clean; golden scorecard 25/25, including the curation: dedup + allowlist metric.

Why this was safe to fix now

It sat on the deferred backlog through several earlier reviews, and the reason was churn: re-pending duplicates that should stay deduped costs LLM calls on every 30-minute pass, forever. So convergence was verified before anything was claimed — by hand across seven passes, and then by the gate with a differential fuzz of 60 seeds × 6 mutations against main, with three idle passes after each step as a convergence invariant. Zero non-convergence on either revision, and the total price of the fix is 20 one-time LLM calls out of 234 across 360 mutation steps. Every behavioural difference from main is a document that used to have no page getting one.

Keying the decision off a computed hash rather than mere presence is what satisfies both halves of the existing test at once.

Deliberately still open

hashes.get(canon, raw_hash) leaves one hole: a duplicate whose canonical is in the inventory but absent from disk still holds content no page serves. That is exactly what test_classify_pending_promotes_orphaned_duplicate's second assertion pins, so preserving it is correct for this change — but it means one of two remaining holes is closed here, not both. The open question is whether in-inventory-but-missing-on-disk is transient (a mid-sync raw dir) or permanent; the answer decides whether that default is right or is the next bug. Recorded on the backlog.

Minor, unmentioned before: promotion adds the doc to touched, so the versions and dossiers post-pass phases fire on it once. Bounded, and neither writes anything classify_pending reads, so it cannot cause churn.

[sturlese]

Gate verdict

VERDICT: SURVIVES — the first clean verdict in eight iterations. Worth recording what it took to earn it, because the reviewer attacked the thing that actually mattered.

Churn was the risk, and it was attacked properly

Dedup exists to avoid LLM calls, so a fix that re-pends duplicates it shouldn't costs money on every 30-minute pass, forever. That is why this bug sat on the deferred list through several earlier reviews. The reviewer ran nine hand-built scenarios plus a differential fuzz: 60 seeds × 6 random mutations over 4 files, 3 passes per step plus 3 further idle passes as a convergence invariant, replayed identically against main and the branch:

NOT CONVERGED main:   0
NOT CONVERGED branch: 0
branch: 1080 passes recorded, total llm calls=234, calls on pass1/pass2 (should be 0): 0
main:   1080 passes recorded, total llm calls=214, calls on pass1/pass2 (should be 0): 0
diff branch vs main: 354 changed lines

Every one of those 354 diff lines has the same shape: main leaves a duplicate pointing at a canonical whose content moved on and no page serves its bytes; the branch gives it a page. Total cost: 20 extra LLM calls out of 234 across 360 mutation steps, all one-time. Not one pass anywhere re-issued a call for a settled file.

It also confirmed the three-identical-files case resolves in a single pass (B promoted, C re-dedups onto B), that flip-flopping the canonical six times costs at most one call per flip, and that max_docs truncation — the one way a duplicate can be pointed at a canonical the pass then drops — self-heals on the next pass.

The regression I was most worried about does not exist

I'd checked the requeued-with-unchanged-bytes case myself; the reviewer proved why it can't bite, which is better than my empirical answer: a requeued/error file never seeded the index on main either, because the seed requires status == "processed". So restale cannot have taken anything away. Verified across three orderings, output byte-identical to main.

Hashing cost claim verified

                          branch    main
total file_sha256 calls:    50       50
distinct paths hashed:      50       50
paths hashed more than once: []      []

2000 files × 200 KB: branch 543 ms vs main 584 ms — within noise. Nothing hashed that wasn't hashed before.

Third locus checked: promotion restores page and facts

_state_of dumped verbatim across the transition — B regains lastResult, its page and its facts rows, and duplicateOf is gone because the success path assigns a fresh dict rather than merging. The reverse demotion removes A's page and facts.

Fixed from the review

• My PR body's test count was wrong — I wrote "clean 255 (+3)"; main is 252 and the branch is 256, so +4. Corrected. That was the only inaccuracy the reviewer found and it was mine.
• A comment claimed a tighter invariant than the expression enforces. restale is every pending file with a hash, including new docs and duplicates; what makes it correct is the overlap with the seeding candidates. The comment now describes that mechanism instead of asserting the conclusion. The reviewer built the tighter variant and proved it a semantically equivalent mutant rather than a coverage gap — byte-identical output across the whole fuzz — so the code is unchanged.

Recorded, not fixed

hashes.get(canon, raw_hash) deliberately leaves one hole open: a duplicate whose canonical is in the inventory but absent from disk still holds content no page serves. That is pinned by test_classify_pending_promotes_orphaned_duplicate, so preserving it is right for this change — but it means this PR closes one of two remaining holes. The open question is whether in-inventory-but-missing-on-disk is transient (mid-sync) or permanent; the answer decides whether that default is correct or is the next bug. On the backlog.

Also noted: promotion adds the doc to touched, so the versions and dossiers post-pass phases fire on it once. Bounded, and neither writes anything classify_pending reads, so no churn.

Mutation set: 4 of 5 killed, the survivor proven equivalent. clean 256 (+4), coverage 94%; corpus 48 · graph 40 · slack 13 · fetch 33 · answer 66 · benchmark 3; ruff clean; scorecard 25/25 including the curation: dedup + allowlist metric.