Repository navigation
Upstream-delta register: migrate what the lectures use, review data updates after the migration #39
Description
Activity
- added 2 commits that reference this issue
on Aug 6, 2026 The register now has a schema field, not just prose
Following the question of whether #39 needed schema support: it did, and the gap was live rather than theoretical.
mpd2020.xlsxwas recordingintegrity.upstream.status: verifiedwhile differing from upstream — because the vocabulary (verified | spot-checked | unverifiable | unverified | failing) had no value for the outcome that actually occurred. The check ran in full and the file did not match.failingwould imply something is broken, andverifiedis a claim a reader would act on: someone refreshing that file from the source would find it silently changes lecture output.Added in #38: a
divergedstatus, plus the four fields this register needs.Field Purpose delta_kindlocal-editorupstream-moved— the two kinds this issue distinguishes, now machine-readable rather than buried in prosedeltawhat differs, precisely enough that resolving it later needs no re-investigation delta_evidencehow delta_kindwas established rather than assumedregisterpoints back here delta_evidenceearns its place for a reason worth recording. Ruling out the other kind is the whole work, and it determines the file's class, not just its status: if Maddison had renamed its own headers after we took the file, our copy would be a faithful older vintage andverbatim; because upstream was byte-stable throughout, our copy was edited and it isconstructed. Same three-cell difference, opposite classification. Adelta_kindasserted without evidence would silently misclassify the dataset.divergedis deliberately not a defect state, which matters for how this register should be read. A migration moves the copy the lectures already consume, so a known delta against today's upstream is an expected result to record — not a problem to fix mid-repoint. The catalog gives it its own mark (⇄) so it cannot be misread as either a clean pass or a failure.The broader schema catch-up stays out of scope here and belongs with #14. The sketch is behind the manifests in ways that predate this work:
consumers[].repo/.fileis used by 10 manifests and documented in none, as aresource.doi,source.note,source.versionandsource.file_url, plus the multi-sheet vocabulary. One sweep against a decided convention beats piecemeal additions.Second entry:
life-expectancy-vs-gdp-per-capita.csv— upstream movedFound while scoping Track A step 4 in QuantEcon/workspace-lectures#23. It is the first register entry that would break a build rather than quietly change a figure, so the ordering matters more here than it did for
mpd2020.xlsx.The delta. Our committed copy (OWID, 2,059,709 B, in
lecture-python-introatlectures/_static/lecture_specific/simple_linear_regression/) has the headerEntity,Code,Year,Life expectancy at birth (historical),GDP per capita,417485-annotations,Population (historical estimates),Continent. Today's export from the same OWID grapher differs in two columns.Column Our copy OWID today 4 Life expectancy at birth (historical)Life expectancy at birth6 417485-annotations(dropped) Why it fails closed.
lectures/simple_linear_regression.md:433hardcodes the old label and passes it asusecols:cols = ['Code', 'Year', 'Life expectancy at birth (historical)', 'GDP per capita'] df = pd.read_csv(data_url, usecols=cols)
Against today's real header, pandas 2.3.3 raises
ValueError: Usecols do not match columns. This is the mirror image ofmpd2020.xlsx: there the renames are ours and load-bearing, and a pristine upstream copy would change the plot with no error; here the rename is upstream's and a refreshed copy stops the build on the spot. Both cases argue the same thing — the file and the lecture move together — but this one is the benign failure mode, and only if it is on the record before anyone reaches for a refresh.Classification.
delta_kind: upstream-moved,integrity.upstream.status: diverged, per rule 4's second kind. Resolution therefore has a settled shape already: a newer vintage gets a new filename, so the frozen copy stays valid andsimple_linear_regression.md:433keeps working untouched. Nothing needs deciding now.delta_evidenceis weaker than mpd2020's and should say so. What is measured is the two-column difference between our bytes and today's export; the direction rests on417485-annotationsbeing an OWID grapher export artefact keyed by variable id, which an edited copy would not acquire. That is inference, not the Internet Archive digest chain that settled mpd2020. The honest entry records it as such, and the Archive check on the grapher CSV endpoint is the work that would upgrade it.Register it before the step-4 repoint PR, not after. This file is not yet manifested, so its manifest is authored inside the repoint PR — which means
integrity.upstreamgets its first commit there. Rule 4 step 2 wants manifest and register landing together; if the entry trails the repoint, the manifest ships asserting nothing about the divergence, and the obvious next action on a fresh manifest — verify against upstream — is exactly what breaks. This is the first entry filed ahead of its own manifest, and that is the correct order for any delta discovered during scoping rather than during verification."Nothing else, yet" now reads: one more, on a file still in the remaining 23.
Second entry added to the register:
life-expectancy-vs-gdp-per-capita.csv, the firstupstream-movedcase (the first entry,mpd2020.xlsx, islocal-edit). Paired with its manifest in #74, per this issue's own rule that a register entry with no manifest note is invisible to anyone reading the dataset.Three things worth flagging beyond the entry itself.
One widely repeated description of this delta was wrong, and it pointed at the wrong failure. The work plan recorded that today's export "drops
417485-annotations". It does not — the column is renamed toGDP per capita (Annotations)and moved from position 6 to 8. TheValueErrora refresh raises names only the life-expectancy column, because the annotations column is not inusecolsat all. Had the entry been written from the plan's text, the public record would have been materially wrong about which change breaks the read.The delta is four renames, not one, plus halved coverage. 62,156 → 29,912 rows,
Year-10000..2021 → 1..2023, 317 → 276 entities. OnlyEntity,Code,YearandGDP per capitasurvive unchanged.This entry is the clearest case yet for why the register exists. The loud failure —
ValueError: Usecols do not match columns— is the harmless one, because it stops the build. The dangerous one is thatyear == 2018returns 166 rows in both vintages, so someone who fixed theValueErrorand rebuilt would get a scatter that looks right, an OLS line that looks right, and every underlying life-expectancy value silently different. That is rule 4 stated as a measurement rather than a principle.Also worth noting for whoever eventually reviews this register:
mpd2020.xlsxand this file are opposite kinds and want opposite handling.mpd2020islocal-editwhere the edits are load-bearing, so file and lecture must move together. This one isupstream-moved, so perAGENTS.md"Corrections vs vintages" adopting the new vintage means a new filename and consumers opting in — plus editingusecolsand two hardcoded row counts in four repos, and re-reviewing five figures. Neither is urgent; both are now documented precisely enough that resolving them needs no re-investigation.- added 6 commits that reference this issue
on Aug 11, 2026 Register review, 2026-09-01 — the migration is complete, so this is the review step 3 promised
The static migration finished on 2026-08-18 (40 of 40 datasets
repointed, strict audit green on the 2026-08-31 run), which is the trigger this issue set for reviewing the register. Two entries, opposite kinds, and the recommendation is the same for both: keep the frozen copy, deliberately, and record that here. Neither delta is a defect; both are documented precisely enough in their manifests that the decision needs no re-investigation.mpd2020.xlsx(local-edit) — keep the edited copy as the intended stateThe data is the current Maddison release — the Internet Archive digest chain shows upstream has been byte-stable since 2021, so there is no vintage ambiguity and no newer data to gain. The only delta is three header labels that
long_run_growthreads withheader=(0,1,2), and those labels are load-bearing. Restoring the pristine upstream file would mean changing the column handling in all four consuming repos (lecture-python-intro,lecture-wasm,lecture-intro.zh-cn, and the canarytest-actions-lecture-intro— corrected from "three" after Copilot's review on #108) for a lecture whose output would not change by a single value. The class is alreadyconstructedand the manifest already says the state is intended. Decision: keep. No manifest change needed, beyond optionally one line inintegrity.upstream.notesaying "reviewed 2026-09-01, kept deliberately" so a reader of the catalog sees the delta was adjudicated and not merely recorded.life-expectancy-vs-gdp-per-capita.csv(upstream-moved) — keep the frozen copy; adopting the new vintage is a lecture decision, not an infrastructure oneAdopting OWID's current export is a content change with the profile rule 4 warns about: four renamed columns, coverage cut from 62,156 to 29,912 rows, and a
year == 2018scatter that would look identical while every life-expectancy value shifted. Doing it properly means a new filename (perAGENTS.md, so consumers opt in), edits tousecolsand two hardcoded row counts in four repos, and five figures re-reviewed. There is no reader-facing benefit to set against that: the lecture already embeds the live OWID grapher as an iframe beside the frozen data, so a reader sees the current vintage next to the figure today. Decision: keep the frozen copy. If the lecture authors ever want the 2023 coverage for pedagogical reasons, that request should arrive as alecture-python-introissue and be executed as a new-filename vintage landing here — the register entry and manifest already describe exactly what that would take.What this means for the register
Rule 4's three steps are now all exercised at least once: migrate unchanged, record the delta, review after the migration. The issue stays open as the standing register — entries are added by any future verification that finds a delta — and the upstream-freshness dashboard (#40) remains the proposal for finding those deltas routinely rather than by accident. Both entries above are marked as reviewed by this comment; the next review is due when a new entry lands or the #40 dashboard reports drift, not on a calendar.
- added a commit that references this issue
on Aug 31, 2026 Third entry:
business_cycle_data.csv— upstream moved, and for the first time that is the file's normal stateAdded with its manifest in #109 (per this issue's rule that the register entry and the manifest's
integrity.upstreamland together). This one is different in kind from the first two, and the difference is worth stating because it is what every future dynamic snapshot will look like.The delta. The builder was re-run in full against live WDI (
lastupdated2026-07-13) on 2026-09-01, wbgapi 1.0.12 / pandas 2.3.3. Same five rows, same layout, same single structural null (YR1960). Two new year columns, YR2024 and YR2025, populated for all five economies. In the 1960–2023 overlap, 63 of the 64 year columns carry at least one revised cell — 236 of 320 cells in all — with median |change| 0.0 pp, 90th percentile 0.29, 99th percentile 1.00, maximum 1.50 pp.upstream-moved, and the evidence is structural rather than archival. The committed file is the verbatim output ofwb.data.DataFrame(...).to_csv()with no post-processing, and the builder that wrote it is the one committed beside it, so a local edit has nowhere to have happened. The shape of the delta — small revisions spread across the whole history, largest in recent years, plus appended years — is the signature of national-accounts revision and rebasing, not of anything on this side.Why this entry does not want a decision. The first two entries were frozen extracts where the delta was a surprise and "keep or adopt" was a real question. This file is
class: dynamic-snapshotwithcadence: annual: being behind the source is its steady state between refreshes, and the resolution is the refresh itself — a deliberate, reviewed in-place update via the Phase 5 refresh-as-PR workflow, not a new filename (it is not a new vintage in the AGENTS.md sense; it is the same tracking snapshot moved forward) and not a correction. Nothing consumes the file today, so there is no figure at stake in the meantime.What the measurement did settle. The builder's
validate()cannot assert overlap-window equality for a revised aggregate — the very first refresh would fail. It now bounds any single-cell revision at 5 pp (three times the largest routine revision observed), refuses a populated cell going empty or a column disappearing, and prints the revision summary on every run as the review surface. That is the overlap-window policy the Phase 5 template inherits for dynamic snapshots, and this entry is the measurement it rests on.Next review of this entry: the first refresh PR, whose diff summary either confirms the 5 pp bound or teaches us it was wrong.
- added a commit that references this issue
on Sep 1, 2026 Closing the loop on the third entry: the first refresh PR it pointed forward to has happened — #112, opened by the new
refresh-snapshotsworkflow on its first dispatch and merged 2026-09-01.business_cycle_data.csvis now the 2025 vintage; its manifest readsintegrity.upstream.status: verifiedwith the delta block dropped by the stamp, andsnapshots.py duereports it not due until 2027-09.The 5 pp bound held with room to spare: the largest revision in the 1960–2023 overlap was 1.50 pp, exactly the measurement the bound was set from, and no populated cell went empty. So the overlap-window policy this entry recorded stands as written; the next data point is the 2027 refresh.
One thing the PR review turned up that belongs in the register's rules rather than in this entry: a dynamic snapshot's manifest must not carry prose that embeds a vintage fact (an end year, an observed range, a row count), because the stamp rewrites fields, not sentences — the first refresh shipped a column description still saying "YR2023 in the committed bytes". Recorded in AGENTS.md and
builders/_template.pyvia #112.This entry is therefore resolved in the way a dynamic snapshot's entries always will be — by the refresh — and needs no further review. The register stays open for the next delta.
Closing as completed, ruled by @mmcky on 2026-09-07. The register did its job: the static migration completed on 2026-08-18 (40 of 40 repointed, strict audit green 2026-08-31), the review step ran on 2026-09-01 with both frozen copies kept deliberately, and the third entry was refreshed by #112 the same day. The manifests'
integrity.upstreamfield and the generated catalog are the register of record from here — #125 records that in AGENTS.md and cites this issue's history as the worked example of the three steps. New deltas are set in the manifest in the PR that finds them; #40 remains the proposal for detecting them routinely.- added a commit that references this issue
on Sep 7, 2026 - added a commit that references this issue
on Oct 6, 2026
Register of datasets whose committed copy differs from what upstream publishes today, so the migration can proceed without stopping to reconcile each one, and the deltas get reviewed together once it completes.
The policy this implements
A migration moves bytes; it does not update them. The copy that lands in this repo is the copy the lectures already consume, validated byte-identical in the repoint PR. That is what makes a repoint safe to merge: it provably cannot change a single figure.
Adopting a newer upstream vintage is a different change with a different risk profile — it does change lecture output, needs figures re-reviewed, and is an author-facing decision rather than an infrastructure one. Conflating the two would turn every repoint into a content review and stall the programme.
So when a migration discovers a delta against upstream:
Recording the delta in the dataset's manifest (
integrity.upstream) is what makes this honest rather than a deferral — the gap is visible in the generated catalog from the day it is found, not filed away in an issue nobody reads.Two kinds of delta
They look similar and need opposite responses, so the register distinguishes them.
Upstream moved. A newer vintage exists; our copy is an older one. Resolving means deciding whether to adopt it — and per
AGENTS.md, a new vintage gets a new filename rather than replacing the old one, so consumers opt in and existing figures stay valid.Our copy diverges. Upstream is unchanged but the committed file was modified, or was constructed by a process we cannot reproduce. Resolving means reconciling the edit — usually by recording it precisely enough to reapply, or by rebuilding from upstream and accepting the output change.
Register
mpd2020.xlsx— our copy diverges (three header labels)Found while writing its manifest in #38.
Regional datagdppc_2011GDP pc 2011 pricespopPopulationgdppc_2011Every data value matches, including all 21,683 rows of
Full data. Only these three labels differ.The edits are ours, established rather than assumed. The Internet Archive holds the upstream file with one unchanging content digest (
4OWZWHTE5HXBBF4XLCCQTGDXY4HXOGOK) across every snapshot from 2021-01-10 to 2026-01-02, and a copy fetched 2026-08-06 matches it exactly — so upstream was byte-stable two years before these bytes were committed tolecture-python-introon 2023-03-23.Why this cannot simply be "fixed".
long_run_growthreads that sheet withheader=(0,1,2). The renames are load-bearing: replacing the file with a pristine upstream copy would silently change what the lecture plots, with no error raised. Any resolution has to move the lecture and the file together.Options when reviewed: keep the edited copy and document it as the intended state (cheapest, and what the manifest records today); or restore the upstream file and adapt the lecture's column handling, which makes the file genuinely
verbatimand removes a permanent local patch. There is no urgency either way — the current state is correct and documented.life-expectancy-vs-gdp-per-capita.csv— upstream moved (a new vintage, and it is not a drop-in)Found while writing its manifest in #74. Measured against the live grapher endpoint on 2026-08-12:
HTTP 200, 1,414,428 B against our 2,059,709 B.Four of eight columns are renamed. Only
Entity,Code,YearandGDP per capitaare stable.Life expectancy at birth (historical)Life expectancy at birthPopulation (historical estimates)PopulationContinentWorld region according to OWID417485-annotations— position 6GDP per capita (Annotations)— position 8The annotations column is renamed and moved, not dropped. That correction matters, because "drops
417485-annotations" is how this delta had been described in the work plan, and it points at the wrong failure. Coverage changes too: 62,156 → 29,912 rows,Year-10000..2021 → 1..2023, entities 317 → 276. Today's metadata cites Maddison Project Database 2023, HMD 2025 and UN WPP 2024, all of which postdate this export.Why a refresh is a breaking change, in two different ways. The loud one: all four consuming lectures hardcode the old label and pass it as
usecols, so today's export raisesValueError: Usecols do not match columns, columns expected but not found: ['Life expectancy at birth (historical)']. Note it names only the life-expectancy column — the annotations column is not inusecols, so the annotations change is not what breaks the read.The quiet one is the reason this belongs in a register rather than a bug tracker. After relabelling
usecols,dropna()yields 13,239 rows against our 12,445, so the sentence "dropped the number of rows in our DataFrame from 62156 to 12445" — hardcoded in the prose of all four repos — becomes wrong. Andyear == 2018returns 166 rows in both vintages, so the final scatter and OLS fit would look superficially identical while every life-expectancy value shifted. A refresh that fixed only theValueErrorwould ship a silently different figure.upstream-moved, established rather than assumed. The committed file's column names are OWID's own older schema — the(historical)suffixes and the numeric417485-annotationsvariable id are not names anyone here would invent — and itsYearceiling of 2021 is consistent with an export taken before the source releases the current metadata cites. Our 62,156 rows are a superset of today's 29,912 in coverage, which rules out a local truncation. No QuantEcon edit is implicated.Options when reviewed. Per
AGENTS.md"Corrections vs vintages", adopting the new vintage means a new filename so consumers opt in — never a replacement of this file. That would also mean editingusecolsand the two hardcoded row counts in four repos, and re-reviewing five figures. Keeping the frozen copy is the status quo and is what the manifest records today. There is no urgency: the current state is correct, documented, and the lecture embeds the live grapher as an iframe beside the frozen data anyway, so a reader already sees the current OWID vintage next to the figure.Nothing else, yet
The other twenty-four manifested datasets have no known upstream delta. Twelve carry
integrity.upstream.status: unverifiable, which is a different condition: not "we know it differs" but "we cannot check". Those are a reproducibility gap tracked throughbuilder_statusand PLAN Phase 9, not a drift item — they only enter this register if a check is later performed and finds a difference.How entries get added
Any migration or verification that finds a delta adds a section here, and sets the dataset's
integrity.upstreamfields in the same PR so the catalog reflects it. The two must not diverge — a register entry with no manifest note is invisible to anyone reading the dataset.Automating the detection side is proposed separately; see the upstream-freshness dashboard issue. This issue is the human register and stays useful regardless of whether that lands.
Part of #8. Related: #38 (the manifests that surfaced the first entry), #35 (licensing, the same record-and-track shape applied to a different question), and PLAN Phase 7.