improving lobbying data pipeline reliability - #2221
Open
nesanders wants to merge 1 commit into
Open
Conversation
… limit
The live weekly cursor (scrapers/lobbying) stored the entire processed-URL
history and per-registrant summary cache as two fields on a single Firestore
document. Firestore documents have a hard 1MB size limit; once run against
the full corpus, this document grew past it partway through, and every write
after that point failed silently — meaning every registrant processed for
the rest of that run (and any future run building on that cursor) was
silently skipped rather than captured.
Moved to subcollections — scrapers/lobbying/processedUrls/{hash} and
scrapers/lobbying/summaryCache/{hash}, one small doc per URL — mirroring the
pattern the backfill cursor (scrapers/lobbyingBackfill/processedUrls)
already used elsewhere in this same file. Point lookups replace the
in-memory set/dict that used to get loaded and saved as one blob on every
write.
Tested with a small in-memory Firestore fake (tests/test_scrape.py) real
enough to simulate write-then-read-back across calls, including a
regression test asserting the parent cursor doc never grows beyond a
handful of small fields. Confirmed this test fails against the pre-fix code
by checking it out and rerunning the suite, then passes again with the fix
restored. Also validated live against a project with existing data.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
nesanders
requested review from
Mephistic,
alexjball,
kiminkim724,
mertbagt,
mvictor55,
sashamaryl and
timblais
as code owners
August 23, 2026 22:33
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
This was referenced Aug 23, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The weekly lobbying scraper's incremental cursor (
scrapers/lobbying) stores which disclosure URLs it's already processed as two fields — a growing URL list and a per-registrant cache — on a single Firestore document. Firestore caps a document at 1MB. Run against the full corpus, that document grows past the limit partway through, and every write after that point fails silently: the scraper thinks it succeeded, but every registrant processed for the rest of the run (and beyond, since the cursor itself stops updating) never actually gets recorded as done. In practice this meant the weekly scraper silently stopped making progress once enough disclosures accumulated.Fix
Moved the cursor to subcollections — one small document per URL — mirroring the pattern the backfill cursor (
scrapers/lobbyingBackfill/processedUrls) already uses elsewhere in this same file. No document ever grows large enough to hit the limit, regardless of how much history accumulates.Testing
tests/test_scrape.py(new): unit tests against a small in-memory Firestore fake, including a regression test asserting the parent cursor document stays small. Confirmed this actually catches the bug by checking out the pre-fix code and rerunning the suite against it (fails), then restoring the fix (passes).pytest tests/) still green — 49 tests total.--mode weekly --dry-run).