fix(fetch): changing GOOGLE_DOCS_FORMAT was a silent, permanent no-op

[sturlese]

Twelfth iteration of the autonomous bughunt sweep.

Bug

The skip decision compared the remote fingerprint and "some file is at the recorded path" — never whether that path is the name the current config would write. fingerprint is modifiedTime|size|md5, purely remote, so a local export-format change is invisible to it:

GOOGLE_DOCS_FORMAT=md   -> {'added': 1}   on disk D.md   localPath D.md
GOOGLE_DOCS_FORMAT=pdf  -> {'skipped': 1} on disk D.md   localPath D.md
                           downloads: []

Switching md → pdf did nothing, forever. No download, no log line, no error — the operator believes PDFs are flowing while clean keeps converting the old Markdown, and no recipe in docs/operations/runbook.md tells you to wipe raw/ for it.

This is a documented knob (docs/pipeline/fetch.md, pipeline/.env.example), and fetch/index.md even lists "New export format / MIME mapping" as a supported task.

Fix

The skip now also requires prev_local == expected_local, computed with the same content_name() the download uses.

Two details worth stating:

• That comparison subsumes the old truthiness check on prev_local: None == "D.pdf" is False, so a missing recorded path still falls through to download.

• It is case-insensitive, like content_name and _clobbered_by_sidecar. On a case-folding filesystem (macOS APFS, Windows, Docker Desktop bind mounts) D.md and D.MD are one file, so a case-only difference is not a format change — and treating it as one is worse than missing it.

  I shipped the exact comparison first and the gate refuted it. With GOOGLE_DOCS_FORMAT=MD — a typo in the very knob this PR fixes — the download writes D.MD, then the pre-existing rename cleanup unlinks D.md, the same inode, so the mirror loses the document for a full poll interval:

  pass 2: downloads=['D'] skipped=0  disk=['D.json']       <- content GONE
  pass 3: downloads=['D'] skipped=0  disk=['D.MD', 'D.json']
  

  On main that config is a silent no-op; my first version made it destructive. My stated reason for removing the guard — "the mutation set showed nothing distinguished it" — was itself the bug: the mutation set was blind to case-folding filesystems, which two tests in this file already cover.

Test

test_changing_the_export_format_re_downloads — fails today with assert 'D' in []. It asserts the whole cycle, not just the download: the new D.pdf exists, the manifest records it, the stale D.md is gone (the existing rename cleanup handles it), and the pass after the switch re-downloads nothing — so the fix converges rather than re-exporting every pass.

test_a_case_only_name_difference_is_not_a_format_change — zero downloads on a case-only difference, and the document is still in the mirror. This is the one my first version failed.

test_a_recorded_path_missing_from_disk_re_downloads — the .exists() half, which nothing covered. It is what limits any name-comparison mistake to a single cycle rather than permanent silent absence, so it is load-bearing.

Mutation-tested with hash-verified restores, all four caught: dropping the expected-name check, the case-sensitive comparison (the version I had shipped), dropping .exists(), and computing expected_local from a hardcoded format.

fetch 36 (+3) · clean 259 · corpus 48 · graph 51 · slack 13 · answer 66 · benchmark 3 — bare pytest (CI mode). ruff clean; golden scorecard 25/25.

What the gate caught, and why it matters

The failure mode here is the inverse of #43's, and worse. There I documented a rule no test pinned. Here I had a rule that mattered, my tests couldn't see it, and I resolved that by deleting the rule instead of extending the tests — while citing #43 as the justification. Same root cause (treating the test set as the definition of what matters), opposite and more damaging outcome.

The reviewer also investigated and dismissed the charges I most expected to land: no unwanted re-downloads across an 11-shape corpus over six passes; content_name is deterministic (no guess_type, and skip and write call the identical function on the identical item); the clobber-heal path still converges; every GOOGLE_DOCS_FORMAT change necessarily changes the extension so none is undetectable; and this is not one instance of a class, since content_name is the single source of local naming.

Process note

I lost time this iteration to "3 failed" runs that turned out to be my own mutation harness leaving a mutated file in place — not a defect. Worth recording because the wrong conclusion was available and tempting: every individual condition tested True in isolation while the suite reported failures, which looks exactly like a subtle logic bug. What resolved it was checking whether main was also failing (it was, on a different test) and then re-running the shipped version five times consecutively: 34/34 each time. The mutation harness now verifies the restore by hash before reporting.

[sturlese]

Gate verdict

VERDICT: WEAKENED — and this one caught a mistake I should not have made, because I made it by misapplying a lesson from two PRs ago.

I deleted a guard I needed, on a false premise

My first version compared case-insensitively. I removed the .lower() and wrote in the PR body that "the mutation set showed nothing distinguished it from the exact form." The premise was false. The mutation set was blind to case-folding filesystems — macOS APFS, Windows, Docker Desktop bind mounts — which is exactly the environment content_name's docstring and _clobbered_by_sidecar's samefile() dance exist for, and which two tests in this very file already cover.

On such a filesystem D.md and D.MD are one file. Measured on this machine (verified case-folding) with GOOGLE_DOCS_FORMAT=MD — a typo in the very knob this PR fixes:

pass 2: downloads=['D'] skipped=0  disk=['D.json']            <- content GONE
pass 3: downloads=['D'] skipped=0  disk=['D.MD', 'D.json']
pass 4: downloads=[]    skipped=1

The download writes D.MD, then the pre-existing rename cleanup unlinks D.md — the same inode — so the mirror loses the document for a full poll interval (1800s default) and downloads twice. On main that config is a silent no-op; my branch made it destructive.

The guard is restored, with a comment that reasons about the filesystem rather than asserting "any difference is a real change" — and this time it is pinned: the test asserts zero downloads and that the document is still in the mirror.

The mistake is the inverse of #43's, and worse

In #43 I documented a rule no test pinned. Here I had a rule that mattered, my tests couldn't see it, and I resolved that by deleting the rule rather than extending the tests. Same root cause — treating the test set as the definition of what matters — opposite and more damaging outcome. Worth stating plainly since I cited #43 as the justification for the deletion.

Also pinned: .exists()

Mutant (c) — dropping the .exists() guard — survived, a pre-existing gap that my change made load-bearing: it is what limits any name-comparison mistake to a single cycle instead of permanent silent absence. With it removed, the case-typo hole becomes forever, reported as skipped=1. Now covered by a test that deletes a valid recorded file and asserts the re-download.

Mutation set, all four caught (each restore verified by hash): dropping the expected-name check, the case-sensitive form, dropping .exists(), and computing expected_local from a hardcoded format.

Charges the reviewer investigated and dismissed

Recording these because they were the ones I most expected to land:

• No unwanted re-downloads. An 11-file corpus — binary PDF, native Doc/Sheet/Slides, x.json, SHOUT.PDF, y.JSON, no-extension, name-with-spaces-no-dot, report.2026.tar.gz, weird. (trailing dot) — skips 11/11 for six consecutive passes, zero downloads.
• content_name is deterministic: no mimetypes.guess_type, extension comes from splitext(remote name), and the skip and write sides call the identical function on the identical item within one loop iteration, so expected_local cannot disagree with what was written.
• The clobber-heal path still converges: prev_local = None → falls through → one download, sidecar intact, then skips.
• The "narrower scope" charge fails: export_for returns (fmt, "." + fmt) for Docs, so every GOOGLE_DOCS_FORMAT change necessarily changes the extension — no undetectable format change exists. Changing the hardcoded Sheets entry in code is detected too.
• Not one instance of a class: content_name is the single source of local content naming, folder/all_drives affect listing and lineage rather than layout, and the existing backfill branch handles those with zero downloads.

Recorded, not fixed

The "unlink your own download on a case-folding filesystem" bug this exposed is pre-existing on main when the fingerprint also changes, and is already on the bughunt backlog as its own lead. This change no longer triggers it.

fetch 36 (+3) · clean 259 · corpus 48 · graph 51 · slack 13 · answer 66 · benchmark 3, bare pytest (CI mode) · ruff clean · scorecard 25/25.