2026 demand data changes

[dylanjmcconnell]

The newly released 2026 ISP demand data has some small changes to format / naming conventions.

Changes

• New region field: CSV filenames now include a region as well as a sub-region. For example:

  • 2024: CNSW_RefYear_2023_HYDROGEN_EXPORT_POE50_PV_TOT.csv
  • 2026: NSW__CNSW_RefYear_2011_STEP_CHANGE_POE10_PV_TOT.csv

• Scenario names: Filename mappings (between csv filenames and scenario names) has changed.

  • 2024:

    HYDROGEN_EXPORT: Green Energy Exports
    STEP_CHANGE: Step Change
    PROGRESSIVE_CHANGE: Progressive Change

  • 2026:

    STEP_CHANGE: Step Change
    PROGRESSIVE_CHANGE: Slow Growth
    GREEN_ENERGY_INDUSTRIES: Accelerated Transition

• Missing data: Use of NA rather than 0 for non-solar hours in PV_TOT csvs.

Issues and potential solutions

(for discussion)

1. New region field: One solution is including an optional group for region in the regex filename matching should maintain backwards compatibility with the 2024 traces. I don't think it is necessary to store this extra information in the processed files, as it is redundant as far as I can tell
   • (An alternative is to have explicit regex pattern for different data versions - though might be good to avoid)
2. Scenario names: Scenario name mappings are slightly more annoying - as there is a name clash ("PROGRESSIVE_CHANGE" maps to different scenario names across the version. In this case, having a generic map won't work - will likely (?) need to pass an year version number/flag to get the right mapping (.. and there is an argument if we are doing that for this, perhaps should do that for the regex in 1?)
3. Missing data: Relatively easy - just add a null=["NA"] top read_csv function. Question is what to do with the na values (fill to zero is probably easiest / likely avoids any downstream issues).
   • (That said, na is strictly not zero - and also maybe there isn't any downstream issues with storing a na in this case anyway - given the nature of PV_TOT field?)

[dylanjmcconnell]

@nick-gorman raised question or concept of keeping up to date with AEMO structure (rather than "locking in" particular historic data structure), which has some merit / logic (and maybe not too hard to implement). Would mean that NA's are stored as null, and would retain the region metadata in the output parquet tables too (as we do with other metadata).

Will consider changes to VRE traces before making decisions on best approach.