[smart-meter-analysis] Convert comed meter data from csv to parquet#59

Open

griffinsharps wants to merge 28 commits into

mainfrom

43-convert-coned-meter-data-from-csv-to-parquet

Commits on Jan 26, 2026

WIP Wide-to-long transformation code and accompanying runner and validation.
Griffin Sharps
committed

Commits on Feb 4, 2026

WIP on transform CSV scripts

Griffin Sharps
and
cursoragent
committed

Commits on Feb 5, 2026

Add duplicate detection, row counts, and run artifact validation to month validator

Griffin Sharps
and
claude
committed
Fix Polars 1.38 compatibility: Expr.is_sorted() removed, use Series.is_sorted()

Griffin Sharps
and
claude
committed
Fix sortedness bug: sink_parquet ignores sort, use collect+write_parquet
Griffin Sharps
committed

Commits on Feb 6, 2026

Add migrate-month recipe for EC2 CSV-to-Parquet production runs

Griffin Sharps
and
claude
committed
Add .txt and archive dirs to .gitignore, add secrets pre-commit guards

Griffin Sharps
and
claude
committed
Refactor validate_month_output.py for O(batch_size) memory streaming

Griffin Sharps
and
claude
committed
Fix false sortedness failures in sample mode due to overlapping slices

Griffin Sharps
and
claude
committed
Add migrate-month recipe to Justfile (based on main branch)

Griffin Sharps
and
claude
committed
Add professional annotations to CSV-to-Parquet pipeline files

Griffin Sharps
and
claude
committed
Merge remote-tracking branch 'origin/main' into 43-convert-coned-meter-data-from-csv-to-parquet
Griffin Sharps
committed
Fix trailing whitespace in README files from main merge

Griffin Sharps
and
claude
committed

Commits on Feb 9, 2026

Add multi-month orchestration recipes to Justfile

Griffin Sharps
and
claude
committed

Commits on Feb 17, 2026

Add deterministic month-level compaction stage with atomic swap and validation

Griffin Sharps
and
cursoragent
committed
Filter migrate-month inputs by filename month to tolerate mixed-month S3 prefixes
Griffin Sharps
committed
Run migrate_month_runner via uv-managed environment
Griffin Sharps
committed
Add --compact-no-swap mode; perform full compaction+validation without atomic swap
Griffin Sharps
committed

Commits on Feb 18, 2026

Fix adjacent-key validation to avoid to_list() memory blowup
Griffin Sharps
committed
update .gitignore to explicitly avoid committing test parquets
Griffin Sharps
committed

Commits on Feb 24, 2026

Replace _validate_adjacent_keys with streaming PyArrow iter_batches validator

Griffin Sharps
and
claude
committed

Commits on Feb 26, 2026

Cap rows_per_chunk at 50M to prevent OOM on dense months

Griffin Sharps
and
claude
committed
Refactor _stream_write_chunks to write multi-row-group files targeting 1 GiB

Griffin Sharps
and
claude
committed

Commits on Feb 27, 2026

Switch to plain YYYY/MM partition dirs, Spark part- naming, and add export restructure script

Griffin Sharps
and
claude
committed

Commits on Mar 2, 2026

Commit AGENTS.md and ignore .cursor/ in .gitignore
Griffin Sharps
committed
Move edit_geojson.py from root to scripts/
Griffin Sharps
committed
Restore code quality targets to Justfile from main
Griffin Sharps
committed
Archive deprecated Chicago visualization scripts

Griffin Sharps
and
claude
committed