60 smart meter analysis pricing simulation#61

Open

griffinsharps wants to merge 149 commits into

mainfrom

60-smart-meter-analysis-pricing-simulation

Commits on Jan 26, 2026

WIP Wide-to-long transformation code and accompanying runner and validation.
Griffin Sharps
committed

Commits on Feb 4, 2026

WIP on transform CSV scripts

Griffin Sharps
and
cursoragent
committed

Commits on Feb 5, 2026

Add duplicate detection, row counts, and run artifact validation to month validator

Griffin Sharps
and
claude
committed
Fix Polars 1.38 compatibility: Expr.is_sorted() removed, use Series.is_sorted()

Griffin Sharps
and
claude
committed
Fix sortedness bug: sink_parquet ignores sort, use collect+write_parquet
Griffin Sharps
committed

Commits on Feb 6, 2026

Add migrate-month recipe for EC2 CSV-to-Parquet production runs

Griffin Sharps
and
claude
committed
Add .txt and archive dirs to .gitignore, add secrets pre-commit guards

Griffin Sharps
and
claude
committed
Refactor validate_month_output.py for O(batch_size) memory streaming

Griffin Sharps
and
claude
committed
Fix false sortedness failures in sample mode due to overlapping slices

Griffin Sharps
and
claude
committed
Add migrate-month recipe to Justfile (based on main branch)

Griffin Sharps
and
claude
committed
Add professional annotations to CSV-to-Parquet pipeline files

Griffin Sharps
and
claude
committed
Merge remote-tracking branch 'origin/main' into 43-convert-coned-meter-data-from-csv-to-parquet
Griffin Sharps
committed
Fix trailing whitespace in README files from main merge

Griffin Sharps
and
claude
committed

Commits on Feb 9, 2026

Add multi-month orchestration recipes to Justfile

Griffin Sharps
and
claude
committed

Commits on Feb 10, 2026

Add rate_structures directory for tariff definitions
Griffin Sharps
committed
Added STOU YAML file to begin simulation testing
Griffin Sharps
committed
Generalize billing to compare two generic hourly tariffs (A vs B)

Griffin Sharps
and
claude
committed
Add hourly tariff price calendar builder from YAML rate structure

Griffin Sharps
and
claude
committed
Add builder for hourly flat-rate price calendar
Griffin Sharps
committed

Commits on Feb 11, 2026

Add end-to-end DST roll-in validator for RTP billing pipeline

Griffin Sharps
and
claude
committed
Add BG regression dataset builder and multi-month billing orchestrator

Griffin Sharps
and
claude
committed
Add pricing simulation E2E tests and reviewer test plan; allow tracking tests/test_*.py
Griffin Sharps
committed
Add reviewer "why" annotations to pricing simulation pipeline

Griffin Sharps
and
claude
committed

Commits on Feb 16, 2026

Update E2E tests for month-scoped bills and expanded regression outputs

Griffin Sharps
and
claude
committed

Commits on Feb 17, 2026

Add deterministic month-level compaction stage with atomic swap and validation

Griffin Sharps
and
cursoragent
committed
Filter migrate-month inputs by filename month to tolerate mixed-month S3 prefixes
Griffin Sharps
committed
Run migrate_month_runner via uv-managed environment
Griffin Sharps
committed
Add --compact-no-swap mode; perform full compaction+validation without atomic swap
Griffin Sharps
committed

Commits on Feb 18, 2026

update .gitignore to explicitly avoid committing test parquets
Griffin Sharps
committed
Fix adjacent-key validation to avoid to_list() memory blowup
Griffin Sharps
committed
update .gitignore to explicitly avoid committing test parquets
Griffin Sharps
committed
rtp: support glob/directory input and multi-file parquet scan in compute_hourly_loads

Griffin Sharps
and
cursoragent
committed

Commits on Feb 19, 2026

add BG correlation check, class-aware GeoJSON export, and account chain-of-custody audit

Griffin Sharps
and
cursoragent
committed
fix code-review issues in pricing_pilot analysis scripts

Griffin Sharps
and
claude
committed
fix B007 lint error in validate_mf_esh_bg_count.py

Griffin Sharps
and
claude
committed
add wide_to_long unit tests; sync ruff to v0.14.4

Griffin Sharps
and
claude
committed

Commits on Feb 20, 2026

add build_account_bg_map.py: reproducible account→BG map builder

Griffin Sharps
and
claude
committed
add TOU window audit script and alignment unit tests

Griffin Sharps
and
claude
committed
enforce STOU/DTOU TOU window alignment at YAML and CI level

Griffin Sharps
and
claude
committed
add per-class scatterplots and fix axis labeling in pricing pilot regression script

Griffin Sharps
and
claude
committed
cosmetic fixes to pricing pilot scatterplots

Griffin Sharps
and
claude
committed
fix stats box redundancy and Times New Roman font in scatterplots

Griffin Sharps
and
claude
committed

Commits on Feb 21, 2026

include all block groups in GeoJSON export (left join, null-fill for no-data BGs)

Griffin Sharps
and
claude
committed
add --pct flag for percentage-change y-axis in pricing pilot regressions

Griffin Sharps
and
claude
committed

Commits on Feb 23, 2026

GeoJSON export: global symmetric cap + fail-loud validation
Griffin Sharps
committed
fix GeoJSON exporter sign convention: prefer bill_b - bill_a over bill_diff_dollars

Griffin Sharps
and
claude
committed
filter GeoJSON export to Chicago ZIP-based BG universe

Griffin Sharps
and
claude
committed
fix CI: restore wide_to_long module and update regression x-axis scale

Griffin Sharps
and
claude
committed

Commits on Feb 24, 2026

add 95% CI band to regression scatterplots; rename Rate BEST -> STOU

Griffin Sharps
and
claude
committed
Replace _validate_adjacent_keys with streaming PyArrow iter_batches validator

Griffin Sharps
and
claude
committed

Commits on Feb 26, 2026

Cap rows_per_chunk at 50M to prevent OOM on dense months

Griffin Sharps
and
claude
committed
Refactor _stream_write_chunks to write multi-row-group files targeting 1 GiB

Griffin Sharps
and
claude
committed

Commits on Feb 27, 2026

Switch to plain YYYY/MM partition dirs, Spark part- naming, and add export restructure script

Griffin Sharps
and
claude
committed

Commits on Mar 2, 2026

Commit AGENTS.md and ignore .cursor/ in .gitignore
Griffin Sharps
committed
Move edit_geojson.py from root to scripts/
Griffin Sharps
committed
Restore code quality targets to Justfile from main
Griffin Sharps
committed
Archive deprecated Chicago visualization scripts

Griffin Sharps
and
claude
committed
Merge into csv/parquet to get new agent code.
Griffin Sharps
committed
Fix sign convention docs in geojson export
Griffin Sharps
committed
Fix delta fallback to match Flat − Alt convention
Griffin Sharps
committed
Add HC1 robust standard errors to BG OLS fit
Griffin Sharps
committed
Use lazy scan for bills parquet load
Griffin Sharps
committed
Use lazy scan for aggregate BG bill loading
Griffin Sharps
committed

Commits on Mar 3, 2026

Fix pre-existing ruff lint warnings
Griffin Sharps
committed
Make ZIP filter optional in account-BG mapper
Griffin Sharps
committed
Parameterize geo scope and add domain anchors in GeoJSON exporter
Griffin Sharps
committed
Archive superseded GeoJSON exporter
Griffin Sharps
committed
Fix memory safety and rename paths in chain-of-custody script
Griffin Sharps
committed
Sync lockfile with pyproject.toml
Griffin Sharps
committed
Fix sign convention docs and minor review findings
Griffin Sharps
committed
Remove redundant unique before n_unique in chain-of-custody
Griffin Sharps
committed

Commits on Mar 5, 2026

Allow glob patterns in billing pipeline interval-pattern
Griffin Sharps
committed
Add chunked processing to compute_hourly_loads for OOM safety
Griffin Sharps
committed
Add sub-file row chunking to compute_hourly_loads
Griffin Sharps
committed
Add multi-pass merge to re-aggregation step
Griffin Sharps
committed
Reduce batch size and merge fan-in for OOM safety
Griffin Sharps
committed
Fix OOM by limiting Polars to single thread
Griffin Sharps
committed
Use streaming sink_parquet for merge passes
Griffin Sharps
committed
Reduce batch size to 100k and fan-in to 8
Griffin Sharps
committed

Commits on Mar 6, 2026

Replace final merge pass with dict-based aggregation to avoid OOM
Griffin Sharps
committed
Shard final merge by account hash to fit in memory
Griffin Sharps
committed
fix: restore hourly aggregation in sharded merge and shard bill computation for memory safety
Griffin Sharps
committed
fix: correct STOU midday_peak prices, carry delivery_service_class through pipeline
Griffin Sharps
committed
fix: preserve Categorical dtype for delivery_service_class in hourly loads merge pipeline
Griffin Sharps
committed
revert: remove delivery_service_class pipeline changes, keep corrected STOU prices
Griffin Sharps
committed
Add output/ to .gitignore
Griffin Sharps
committed
Remove dead files and clean up lint config
Griffin Sharps
committed
Rewrite infra teardown scripts with state drift recovery
Griffin Sharps
committed
Add pricing simulation report scaffold
Griffin Sharps
committed
Fix mkdocs and deptry CI failures
Griffin Sharps
committed
Fix OOM in delivery class lookup with streaming PyArrow reader
Griffin Sharps
committed

Commits on Mar 7, 2026

Add STOU delivery delta post-processing script
Griffin Sharps
committed

Commits on Mar 9, 2026

Fix inverted sign convention in GeoJSON export and bill stats scripts
Griffin Sharps
committed
Fix hourly aggregation grain in sharded merge passes
Griffin Sharps
committed
Reduce peak memory in delivery class lookup with incremental dedup
Griffin Sharps
committed
Cast delivery_service_class to Utf8 before dedup in lookup builder
Griffin Sharps
committed
Standardize pricing pilot scripts against repo conventions
Griffin Sharps
committed
Fix OOM in delivery class lookup with set-based dedup
Griffin Sharps
committed
Remove Polars lazy-scan path that OOMs before Python can catch
Griffin Sharps
committed
Add DTOU results packaging script
Griffin Sharps
committed

Commits on Mar 10, 2026

Add statewide account→BG map builder from combined Parquets
Griffin Sharps
committed
Add statewide BG-level pricing regression analysis script
Griffin Sharps
committed
Fix log-income regression bug in statewide pricing analysis
Griffin Sharps
committed
Add statewide GeoJSON export for pricing simulation maps
Griffin Sharps
committed
Add phase 0 repo audit script
Griffin Sharps
committed

Commits on Mar 11, 2026

Update 2026 STOU rates and compute supply deltas inline
Griffin Sharps
committed

Commits on Mar 12, 2026

Add income-vs-savings scatterplot generator from combined parquets
Griffin Sharps
committed
Add end-to-end pipeline audit with synthetic data verification
Griffin Sharps
committed
Write per-class BG-level CSV in statewide analysis regression loop
Griffin Sharps
committed
Add 16 per-class scenarios to GeoJSON export
Griffin Sharps
committed
Add cost_change field to GeoJSON for Felt-friendly coloring
Griffin Sharps
committed

Commits on Mar 19, 2026

Add income quintile analysis to statewide pipeline
Griffin Sharps
committed
Draft 1 of memo in index.qmd
Griffin Sharps
committed

Commits on Mar 20, 2026

Updated index.qmd to full draft
Griffin Sharps
committed

Commits on Mar 23, 2026

Wire index.qmd setup block to pickle-backed report variables
Griffin Sharps
committed
devcontainer: switch from features to Dockerfile, raise fd limit
Griffin Sharps
committed
devcontainer: fix stale yarn repo blocking chromium install
Griffin Sharps
committed
devcontainer: set ulimit via runArgs
Griffin Sharps
committed
devcontainer: create /workspaces dir
Griffin Sharps
committed
fix: devcontainer startup — move venv to /opt/venv, add WORKDIR, add plotnine
Griffin Sharps
committed
Replace analysis notebook with CUB TOU equity analysis
Griffin Sharps
committed
updated memo draft with correct python format to plug into notebooks/analysis.qmd
Griffin Sharps
committed
Fix deptry dependency issues in CI
Griffin Sharps
committed
Fix deptry dependency issues in CI

Griffin Sharps
and
claude
committed

Commits on Mar 24, 2026

[smart-meter-analysis] Fix column names in analysis notebook
Griffin Sharps
committed
[smart-meter-analysis] Add Quarto project config, references, and report targets
Griffin Sharps
committed
Fix month filter comparisons to use integers in assign-regression-vars
Griffin Sharps
committed
Add Felt map iframes for DTOU and Rate BEST bill change maps
Griffin Sharps
committed
add report_variables.pkl to cache, ignore .quarto build dir
Griffin Sharps
committed
WIP: proposed edits to report

Max Shron (Switchbox)
authored and
Max Shron (Switchbox)
committed
Merge branch '60-smart-meter-analysis-pricing-simulation' of https://github.com/switchbox-data/smart-meter-analysis into 60-smart-meter-analysis-pricing-simulation

Max Shron (Switchbox)
authored and
Max Shron (Switchbox)
committed
Add Switchbox shared lib (plotnine theme, quarto helpers) from reports2
Griffin Sharps
committed
Add build dependencies and uncommitted project files
Griffin Sharps
committed
Merge branch '60-smart-meter-analysis-pricing-simulation' of https://github.com/switchbox-data/smart-meter-analysis into 60-smart-meter-analysis-pricing-simulation

Max Shron (Switchbox)
authored and
Max Shron (Switchbox)
committed
Merge branch '60-smart-meter-analysis-pricing-simulation' of https://github.com/switchbox-data/smart-meter-analysis into 60-smart-meter-analysis-pricing-simulation

Max Shron (Switchbox)
authored and
Max Shron (Switchbox)
committed

Commits on Mar 26, 2026

Memo draft 2 for review/comment
Griffin Sharps
committed

Commits on Mar 27, 2026

Proposed edits to final draft
Max Shron
committed
Line edits for memo
Griffin Sharps
committed
Updated visualization titles
Griffin Sharps
committed

Commits on Mar 30, 2026

Add phase1 streaming audit script for production parquet outputs
Griffin Sharps
committed
Vectorize inner loop: numpy comparisons + pyarrow group_by for acct-day counts
Griffin Sharps
committed

Commits on Mar 31, 2026

Split phase1 audit into --mode integrity and --mode acct-day
Griffin Sharps
committed
Optimize integrity audit to boundary-only checks (~300 comparisons vs ~20M)
Griffin Sharps
committed
Add phase4 S3 publish script for versioned parquet releases
Griffin Sharps
committed
Add compact_all_months wrapper to batch-compact production months
Griffin Sharps
committed
Fix needs_compaction path: use YYYY/MM not year=/month= Hive layout
Griffin Sharps
committed
Broaden parquet glob to *.parquet to pick up part-* files after compaction
Griffin Sharps
committed

Commits on Apr 1, 2026

Add BESTEC off-peak sensitivity script for CUB presentation
Griffin Sharps
committed

Commits on Apr 8, 2026

full draft after revenue neutrality adjustment
Griffin Sharps
committed