Update LCFS, WAS, and ETB prerequisites#413
Merged
Merged
Conversation
Contributor
Author
|
CI status: lint, changelog, and release-manifest checks pass. The Test job fails during |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
lcfs_2023_24.zip,was_2006_22.zip, andetb_1977_24.zip..pklfiles are not reused.year == 2023for FYE 2024.Data inputs
The current restricted UKDS TAB zips have been fetched through the UKDS catalogue download endpoint and uploaded to the private HF repo
policyengine/policyengine-uk-data-privateunder the filenames this PR expects:lcfs_2023_24.zip: UKDS SN 9468 TAB package9468tab_A03FF22348E5E7D12FFD971D315D8E54BDC3CE7F6395D0108D85DEEBDF6BE8E3_V1.zip, SHA256a03ff22348e5e7d12ffd971d315d8e54bdc3ce7f6395d0108d85deebdf6be8e3.was_2006_22.zip: UKDS SN 7215 TAB package7215tab_DF5E8BE49E51AA70F4BF686B98AB44EF11EE1EB260CBAF51308F89DC62449AE1_V1.zip, SHA256df5e8be49e51aa70f4bf686b98ab44ef11ee1eb260cbaf51308f89dc62449ae1.etb_1977_24.zip: UKDS SN 8856 TAB package8856tab_96FFF4868745A2A9BB1169DCECDAF9958C1179D81F9EF67494C1F29622C7D405_V1.zip, SHA25696fff4868745a2a9bb1169dcecdaf9958c1179d81f9ef67494c1f29622c7d405.The downloader now flattens each source package's
UKDA-*-tab/tabfolder, because the current UKDS TAB zips are nested and the data filenames themselves are not prefixed with the study number.Refs #411
Tests
uv run ruff format --check .uv run ruff check policyengine_uk_data/datasets/private_releases.py policyengine_uk_data/storage/download_private_prerequisites.py policyengine_uk_data/datasets/imputations/wealth.py policyengine_uk_data/datasets/imputations/consumption.py policyengine_uk_data/datasets/imputations/vat.py policyengine_uk_data/datasets/imputations/services/etb.py policyengine_uk_data/tests/test_frs_prerequisites.py policyengine_uk_data/tests/test_student_loan_balance.py policyengine_uk_data/tests/test_road_fuel_volume_uprating.py policyengine_uk_data/tests/test_vat_parameters.py policyengine_uk_data/tests/test_private_releases.pyuv run pytest policyengine_uk_data/tests/test_frs_prerequisites.py policyengine_uk_data/tests/test_private_releases.py policyengine_uk_data/tests/test_student_loan_balance.py policyengine_uk_data/tests/test_road_fuel_volume_uprating.py policyengine_uk_data/tests/test_vat_parameters.py -qlcfs_2023_24.zip,was_2006_22.zip, andetb_1977_24.zip: downloaded frompolicyengine/policyengine-uk-data-private, verified SHA256, and verified extractor output contains the expected household/person tab files.