improving lobbying data pipeline reliability (part 2, depends on #2182) - #2222
Draft
nesanders wants to merge 20 commits into
Draft
improving lobbying data pipeline reliability (part 2, depends on #2182)#2222nesanders wants to merge 20 commits into
nesanders wants to merge 20 commits into
Conversation
- Extend LobbyingFiling type with registrantId; add LobbyingStats, LobbyingClientSummary, LobbyingPositionCounts aggregate types using Dictionary for string-keyed maps (runtypes v5 compat) - Add lobbyingMeta/stats singleton collection constants; add CLIENTS_COLLECTION - Add Firestore composite indexes for registrantId+year and clientNameNorm+year (DESCENDING) on lobbyingFilings to support firm and client detail queries - Add data hooks (useLobbyingFilingsForBill, useLobbyingRegistrant, etc.) in components/db/lobbying.ts following existing useAsync pattern - Add lobbying.json locale namespace; add navigation.lobbying to common.json - Add NavbarLinkLobbying component; wire into Navbar behind lobbyingTable flag - Add LobbyingPositionChip (support/oppose/neutral/none pill), LobbyingFilingsTable (configurable columns), LobbyingBillCard (sidebar card with position stacked bar, truncated table, explorer link) - Replace dummy LobbyingTable in BillDetails with LobbyingBillCard - Enable lobbyingTable feature flag in development env - Install recharts (v3); update chartTheme.tsx for v3 TooltipContentProps API Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Add /lobbying/bills browse page: session dropdown, position filter, bill ID search, table with per-bill filing counts and mini position bar - Add stub pages for /lobbying, /lobbying/clients, /lobbying/firms and dynamic routes for client/firm detail - Add useLobbyingFilingsForCourt hook for court-level data loading - Fix LobbyingBillCard: surface errors in dev, handle not-requested state, inline collection constants to avoid pulling firebase-admin into browser bundle (was causing net module build error on bill detail page) - Add lobbying namespace to bill detail page getServerSideProps so LobbyingBillCard translations load correctly - Add titles.lobbying key to common.json for page title Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Implements four new routes for the lobbying explorer: - /lobbying/firms — browse all registrant firms, filter by type/name - /lobbying/firms/[registrantId] — firm detail with filings table, client list, disclosure links - /lobbying/clients — browse unique clients derived from registrant docs - /lobbying/clients/[clientSlug] — client detail with filings table, linked firms, position summary Firm and client slugs use URL-encoded entityNameNorm/clientNameNorm. Both detail pages use getStaticPaths with fallback:blocking. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Skips clientNameNorm values that are empty or equal to _total_salary_, which is used for pre-2013 legacy filings where compensation is reported as a single total rather than per client. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Build out /lobbying overview: stat cards (bills, clients, sessions, spend), dual-axis spend+filings chart, entry cards linking to browse pages - Extract LobbyingAttribution component with correct SoS link - Fix broken attribution on bills page (was passing Trans interpolation syntax to t() which rendered literal <0> tags) - Stats and chart degrade gracefully when the pipeline stats doc is absent Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Add LobbyingAttribution to firms browse/detail and clients browse/detail - Add seedLobbyingStats admin script: reads all filings and registrants, computes totals and per-year breakdowns, writes to lobbyingMeta/stats (run with: yarn firebase-admin run-script seedLobbyingStats --env dev) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Add allow read: if true for lobbyingMeta collection so the stats doc is readable by the frontend (was silently returning undefined) - Move Lobbying nav item after Testimony in both mobile and desktop nav Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…il to lobbying explorer - Add LobbyingSubnav (Overview / Bills / Clients / Lobbying Firms) to all lobbying pages - Add usePagination hook and LobbyingPaginationBar component (50/page on all browse indexes) - Add sortable column headers (SortTh) to bills, clients, and firms index pages - Move bills.tsx → bills/index.tsx; add bills/[court]/[billId].tsx paginated filings detail - Filter _total_salary_ sentinel by both clientName and clientNameNorm in clients browse - Fix duplicate "View all lobbying activity" link in LobbyingBillCard - Fix undefined–undefined year range on firm detail when registrant query fails - Add composite Firestore indexes for entityNameNorm+year and clientNameNorm+year queries - Add showActivity prop to LobbyingFilingsTable Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…onsive column hiding - Wire t() through all lobbying components: LobbyingSubnav, LobbyingPaginationBar, LobbyingFilingsTable, and all browse/detail pages - Add missing translation keys: subnav, pagination, misc, registrantType, and new fields (bill, positions, activity, sessions, type, firms, total) - Add lobbying.module.css with media query to hide table columns on mobile - Bills index: hide Session and position count columns on mobile (<768px), leaving Bill | Filings | Positions mini-bar - Bill detail: hide Activity column on mobile to reduce table width - Client and firm detail pages: paginate filings table at 25/page (year desc) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…duleNotFoundError at runtime Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
On timeout/resume, the backfill was re-fetching every Summary.aspx page for completed years (~5 hours of HTTP requests just to re-verify nothing was new). Now writes completedYears[] to the cursor doc on year finish; subsequent runs skip those years entirely with zero network requests. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Fix seedLobbyingStats to sum compensation from registrant client records instead of per-bill filing amounts; fixes spendByYear being wildly understated - Add compute_stats() to writer.py and wire it into scrape.py so stats are recomputed automatically after any run that writes new data - Add chart legend and per-axis labels (Compensation / Filings) with correct tooltip formatting (filings shown as count, not currency) - Add data notes below chart explaining portal discontinuities in 2010, 2014, and 2019 with citations to MGL Chapter 3 and Acts of 2009 Ch.28 Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Show bill description (activityTitle) once in the header instead of repeating it verbatim in every table row - Show General Court session (e.g. "194th General Court") below bill ID - Remove redundant Activity column from the filings table Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…bility - Align position colors to MAPLE bill palette: support=green, neutral=blue, oppose=orange (was red/gray); update chip, bar, legend, and inline styles - Sticky subnav now measures main navbar height dynamically to avoid hiding under it on scroll - Add usePrefersReducedMotion hook; disable Bar/Line animations and suppress recharts tooltip transition for prefers-reduced-motion users - Make bills and clients hero stat cards clickable links to their list pages - Bill detail page: remove compensation column (usually $0); fix dead import - Firm detail page: show total compensation in header - Client list: filter out "total salaries received" pseudo-client rows - LobbyingFilingsTable: bill links now point to lobbying explorer detail page; show "Showing X of Y" when truncated via maxRows Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…e restructure - Add sortable Filings column to firms and clients list pages; backed by new lobbyingMeta/entityFilingCounts and clientFilingCounts Firestore docs written by compute_stats() and seedLobbyingStats - Hide "View bill on MAPLE" link for sessions before the 192nd General Court (earliest session in the MAPLE bill database) - Replace overview entry-card row with a Firms hero stat card; sessions count moved into the data-notes paragraph below the chart - Add one-sentence explainer text to all four hero stat cards and to the top of each listing page (bills, clients, firms) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Keep NavbarLinkLobbying alongside upstream's new NavbarLinkAiTools - Keep lobbying i18n key alongside upstream's new learn_hub / testimony keys - Accept upstream deletion of tracked .pyc files Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
… summaries
The bills index page was loading every filing document for the selected
session (~30K Firestore reads) just to group them by bill ID client-side.
compute_stats() and seedLobbyingStats now build lobbyingMeta/billSummaries_{court}
docs (one per General Court) in the same filings pass, storing a
billId→{total,support,oppose,neutral,none} map (~200KB per court).
The frontend replaces useLobbyingFilingsForCourt with useLobbyingBillSummaries,
reducing the bills page load from ~30K document reads to a single doc fetch.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…nts/firms cols, pct-sort, orange scheme - Bills index: session selector is now a multiselect dropdown with year annotations (e.g. "Session 194 · 2025–26") and an "All sessions" toggle; Session column appears only when multiple courts are selected - Add Title column (sourced from activityTitle on first filing per bill) included in search alongside bill ID - Add Clients and Firms (lobbyist) columns with sortable unique-entity counts - Positions column header cycles through % Support → % Oppose → % Neutral on repeated clicks - Fix position color scheme: all three count columns and mini-bar now use POSITION_COLORS (green/orange/blue) — oppose was previously rendering as red (#c71e32 / dark brownish #7a3800) in two separate code paths - Remove LobbyingSubnav sticky positioning (it leaked above the viewport when the main navbar scrolled away); subnav now scrolls with content - Seed script and writer.py: bill summaries now include title, clients, and lobbyists counts; stored as JSON blob to avoid Firestore index limit Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…ata scale
Three related Firestore document/field size-limit bugs surfaced when the
weekly incremental scraper was run for the first time against the full
production-scale dataset (300K+ filings):
1. The live weekly cursor (scrapers/lobbying) stored the entire processed-URL
history and summary cache as two fields on one document. That document
exceeded Firestore's 1MB limit partway through a run, silently failing
(and thus skipping) every registrant processed afterward. Moved to
subcollections — one small doc per URL — mirroring the pattern the
backfill cursor already used, with point lookups instead of an in-memory
set/dict.
2. compute_stats() streamed the full lobbyingFilings/lobbyingRegistrants
collections (300K+ docs) in one unbounded query, which timed out
server-side; the client library's automatic stream-retry then crashed on
an internal AttributeError instead of recovering. Replaced with
cursor-paginated batches (50K docs/request) and a manual retry that
re-issues a fresh query rather than resuming a broken stream.
3. Once (2) was fixed, compute_stats() reached a third limit: the
billSummaries_{court} JSON blob itself exceeded Firestore's 1MB
field-size limit for the current session (1,057KB for court 194's ~5,600
bills), with courts 192/193 close behind. Restructured to one small doc
per bill in a bills subcollection instead of one JSON blob per court —
same fix pattern as (1), applied to writer.py, seedLobbyingStats.ts, and
the frontend fetcher in components/db/lobbying.ts.
All three fixes validated end-to-end against dev Firestore at current scale
(373K filings, 25.6K registrants, 11 courts including the previously-failing
194th).
Also includes scripts/firebase-admin/checkLobbyingFreshness.ts, a read-only
diagnostic for checking scraper cursor state and data recency.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
run_backfill() marked a year "complete" after one pass and skipped it forever on every future run. That's wrong for the current (still-accruing) year: a run partway through the year would mark it complete based on whatever existed at that moment, silently missing every disclosure filed afterward — no future backfill run would ever see it again. This is exactly what happened to 2026 in production: marked complete in July with 0 disclosures captured. run_backfill already has a fully correct, granular completeness check — _is_backfill_processed, a per-URL subcollection lookup. The year-level flag only ever bought a coarse fast-path (skip re-listing a year's registrants entirely) and it's what caused the bug. Removed it: every run now always re-lists every requested year (one cheap HTTP request per year) and relies solely on the per-URL cursor for correctness, so no year can ever be skipped wholesale again. Added tests/test_scrape.py with a small in-memory Firestore fake (real enough to simulate write-then-read-back across calls, unlike a plain mock) covering both cursor systems: - Regression test reproducing the exact bug scenario (empty pass, then real data appears for the same year) — fails against the old code with 4/4 backfill tests red, passes with the fix, confirmed by checking out the pre-fix scrape.py and rerunning the suite against it. - Backfill always re-lists every year, per-URL dedup still works, dry-run never touches Firestore, completedYears is never written anywhere. - Weekly-mode cursor sanity checks (prior-year caching, current-year always live, parent doc stays small — all state in subcollections). Also validated live against dev: --mode backfill --year 2005 --limit 2 run twice confirms the year is re-listed both times while already-processed disclosures are correctly not reprocessed (0 new on both runs, as expected since 2005 was already backfilled in July). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Two more reliability fixes for the lobbying data pipeline, found while running the weekly scraper for the first time against the full production-scale dataset (see #2221 for the first, already-mergeable fix). Both are variants of the same root cause: a single Firestore document/field has a hard size ceiling, and this pipeline had a couple more places that stored unboundedly-growing state in one doc.
compute_stats()crashed on the full-collection stream and the summary blob it wrotelobbyingFilings/lobbyingRegistrants(300K+ docs) via one unbounded query, which timed out server-side, and the installed Firestore client's retry-on-timeout path hit an internal bug instead of recovering. Fixed with cursor-paginated batches and a manual retry that re-issues a fresh query.completedYearspermanently skipped a year based on a partial scanTesting
tests/test_scrape.py) against a small in-memory Firestore fake, including a regression test for thecompletedYearsbug — confirmed it fails against the pre-fix code and passes with the fix.pytest tests/).compute_stats()completing end-to-end including the previously-failing session, and backfill correctly re-listing a year while still deduping already-processed disclosures.Full details and validation steps are in the individual commit messages.