[WIP] Prototype for new technology data

.claude/settings.json

@@ -0,0 +1,13 @@
+{
+  "permissions": {
+    "allow": [
+      "Bash(uv sync:*)",
+      "Bash(uv run pytest:*)",
+      "Bash(uv run ruff:*)",
+      "Bash(uv run mypy:*)",
+      "Bash(uv run pre-commit:*)",
+      "Bash(uv run reuse:*)",
+      "Bash(uv run mkdocs build:*)"
+    ]
+  }
+}

.claude/settings.json.license

@@ -0,0 +1,3 @@
+SPDX-FileCopyrightText: technologydata contributors
+
+SPDX-License-Identifier: MIT

.claude/skills/dataset-factsheet/SKILL.md

@@ -0,0 +1,97 @@
+---
+name: dataset-factsheet
+description: Create or update the fact sheet in docs/datasets/<key>.md for a dataset shipped with technologydata, once its parser and parsed JSON exist. Use when a data source or a new version of one is added, or when a fact sheet's doctests fail after the data changed.
+---
+
+# Dataset fact sheet
+
+<!--
+SPDX-FileCopyrightText: technologydata contributors
+
+SPDX-License-Identifier: MIT
+
+-->
+
+The template, section order and writing guidelines are in `docs/contributing/adding_a_dataset.md`; read it first and follow it exactly.
+`docs/datasets/dea_energy_storage.md` and `docs/datasets/manual_input_usa.md` are finished examples.
+This skill covers how to find each fact and how to check it.
+
+Readers are experienced modellers and AI agents: tables and short bullets, no description of the parser code.
+Only state what you verified in the code or the data; if something cannot be verified (e.g. the upstream license), write what is known and tell the user.
+
+## Inputs
+
+Determine, or ask the user if unclear:
+
+- `<key>`: the `DataSourceName` value in `src/technologydata/data_accessor.py`.
+- `<version>`: the directory in `src/technologydata/parsers/<key>/`.
+- The parser module `src/technologydata/parsers/<key>/`, the raw file in `src/technologydata/parsers/raw/`, and `sources.json` of the version.
+
+For a new version of an existing dataset, update the existing page instead (see the end of the checklist in `adding_a_dataset.md`).
+
+## Where each fact comes from
+
+| Section | Source |
+|---|---|
+| Front matter, At a glance | `sources.json` (title, authors, `url`, `url_archive`, `url_date`), the license in `REUSE.toml` and upstream, the parser (region, currency) |
+| Years, cases, currency | the loaded data, see the snippet below |
+| Available versions | `get_supported_versions()` of the parser, the parse options that reproduce the shipped JSON |
+| Field mapping | the raw file's columns and the parser; trace one real row from raw value to parsed value |
+| Naming conventions | the parser's cleaning functions (regexes, renames, case maps, unit replacements); one real example each |
+| Assumptions and deviations | hard-coded values in the parser: fixed region, parameter filters, filled-in units, dropped rows, merged cases, row-level sources not kept |
+| Known limitations | checks below |
+
+```python
+from technologydata import DataAccessor
+
+df = DataAccessor(data_source="<key>", version="<version>").load().technologies.to_dataframe()
+print(df[["region", "case"]].value_counts(), df["year"].min(), df["year"].max())
+```
+
+Run all Python from the repository root: `DataAccessor.load()` resolves the data relative to the working directory.
+
+## Generate the code block outputs
+
+Copy the `Accessing the data` and `Contents` blocks from the template, change only `data_source` and `version`, and pick a representative technology and parameter for `Accessing the data`.
+Put a placeholder line as the expected output, then run:
+
+```bash
+uv run pytest test/test_docs.py --test-docs -k <key>
+```
+
+The failure report shows `Got:` for each block; paste that output into the page and run the test again until it passes.
+Do not edit the output by hand; the tests compare it with whitespace normalised only.
+
+## Checks for known limitations
+
+Look for these and quantify them where cheap ("7 of about 76 parameters"):
+
+- **Overwritten values**: parameters are stored in a dict per technology, so raw rows mapping to the same technology, case, year and parameter key overwrite each other.
+  Count duplicates in the raw data after applying the parser's name cleaning.
+- **Dropped data**: raw columns not mapped, rows removed by filters or failed parsing (compare raw row count with parsed parameter count).
+- **Heterogeneous units**: visible in the `Parameters` table when one parameter has several reference units.
+- **Order of operations** that loses precision, such as rounding before unit scaling.
+- **Provenance**: whether row-level sources are kept, whether an archived copy of the source exists.
+
+## Verify Reproduce
+
+The documented `parse()` call must reproduce the shipped files.
+It overwrites them, so check the working tree is clean for that directory first:
+
+```bash
+git status --short src/technologydata/parsers/<key>/
+uv run python -c 'from technologydata import DataAccessor; DataAccessor(data_source="<key>", version="<version>").parse(input_file_name="<file>", num_digits=3)'
+git diff --stat src/technologydata/parsers/<key>/
+git checkout -- src/technologydata/parsers/<key>/
+```
+
+Only a trailing-newline difference is acceptable; otherwise adjust the documented options until the output matches.
+Never use `archive_source=True` here: it calls the Wayback Machine and rewrites `sources.json`.
+
+## Wire up and check
+
+1. Add the page to the `Datasets` nav in `mkdocs.yaml` and a row to the table in `docs/datasets/index.md`.
+2. If a user guide page `docs/user_guide/<key>_parser.md` exists, keep it to structure and usage and link to the fact sheet's `#reproduce`.
+3. Run `uv run pytest test/test_docs.py --test-docs` and `uv run pre-commit run --all-files`.
+   Codespell may flag domain abbreviations; add real terms to `.codespell.ignore`.
+4. Report to the user: anything not verified, and parser problems found (such as overwritten values) as candidates for issues.

.codespell.ignore

@@ -0,0 +1,5 @@
+# SPDX-FileCopyrightText: technologydata contributors
+#
+# SPDX-License-Identifier: MIT
+CAF
+FOM

.github/CODEOWNERS

@@ -0,0 +1,23 @@
+# CODEOWNERS file for technology-data
+# These owners will be the default owners for everything in the repo
+# and will be requested for review when someone opens a pull request.
+
+# Default owners for the entire repository
+* @euronion @finozzifa
+
+# Documentation
+/docs/ @euronion @finozzifa
+
+# Source code
+/src/ @euronion @finozzifa
+
+# Tests
+/tests/ @euronion @finozzifa
+
+# CI/CD and workflows
+/.github/ @euronion @finozzifa
+
+# Configuration files
+*.toml @euronion @finozzifa
+*.yaml @euronion @finozzifa
+*.yml @euronion @finozzifa

.github/PULL_REQUEST_TEMPLATE.md

@@ -0,0 +1,18 @@
+<!--
+SPDX-FileCopyrightText: technologydata contributors
+
+SPDX-License-Identifier: MIT
+
+-->
+
+# Summary
+
+<!-- What does this PR change, and why? Link related issues with "Closes #ISSUE_NUMBER". -->
+
+## Checklist
+
+- [ ] Tests added/updated for the change and `uv run pytest` passes
+- [ ] New files carry SPDX license info (checked by REUSE)
+- [ ] `uv run pre-commit run --all-files` passes (lint, format, mypy, REUSE)
+- [ ] Documentation/docstrings has been updated
+- [ ] (If AI-assisted) Changes were reviewed by a human before opening this PR

.github/dependabot.yml

@@ -1,3 +1,7 @@
+# SPDX-FileCopyrightText: technologydata contributors
+#
+# SPDX-License-Identifier: MIT
+
 # dependabot
 # Ref: https://docs.github.com/en/code-security/dependabot/dependabot-version-updates/configuration-options-for-the-dependabot.yml-file
 # ------------------------------------------------------------------------------
@@ -6,7 +10,7 @@ updates:
 - package-ecosystem: github-actions
   directory: /
   schedule:
-    interval: monthly
+    interval: weekly
   groups:
     # open a single pull-request for all GitHub actions updates
     github-actions:

.github/release.yml

@@ -0,0 +1,20 @@
+changelog:
+  categories:
+  - title: Breaking Changes
+    labels:
+    - breaking change
+  - title: Features
+    labels:
+    - feature
+  - title: Bug Fixes
+    labels:
+    - bug
+  - title: Data
+    labels:
+    - data
+  - title: Documentation
+    labels:
+    - documentation
+  - title: Other Changes
+    labels:
+    - "*"

.github/workflows/ci.yaml

@@ -1,64 +1,43 @@
-name: CI
+# SPDX-FileCopyrightText: technologydata contributors
+#
+# SPDX-License-Identifier: MIT
+name: Technologydata prototype CI
+
+permissions: read-all
 
 on:
   push:
-    branches: [master]
+    branches: [prototype-2]
   pull_request:
-    branches: [master]
-  schedule:
-  - cron: "0 5 * * 0"
-
-
-# Cancel any in-progress runs when a new run is triggered
-concurrency:
-  group: ${{ github.workflow }}-${{ github.ref }}
-  cancel-in-progress: true
+    branches: [prototype-2]
 
 jobs:
-  build:
-    name: CI
+  test:
     runs-on: ubuntu-latest
 
-    defaults:
-      run:
-        shell: bash -l {0}
-
     steps:
-    - name: Checkout code
-      uses: actions/checkout@v4
+    - name: Checkout repository
+      uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1   # v7.0.1
+
 
-    - name: Setup micromamba
-      uses: mamba-org/setup-micromamba@v2
+    - name: Set up Python
+      uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97  # v7.0.0
       with:
-        micromamba-version: latest
-        environment-file: environment.yaml
-        log-level: debug
-        init-shell: bash
-        cache-environment: true
-        cache-downloads: true
+        python-version: "3.13"
 
-    - name: Run unit tests
+    - name: Cache Python dependencies
+      uses: actions/cache@55cc8345863c7cc4c66a329aec7e433d2d1c52a9  # v 6.1.0
+      with:
+        path: ~/.cache/pip
+        key: ${{ runner.os }}-pip-${{ hashFiles('**/uv.lock') }}
+        restore-keys: |
+          ${{ runner.os }}-pip-
+    - name: Install package and dependencies
       run: |
-        make unit-test
+        python -m pip install uv
+        uv lock
+        uv sync
 
-    - name: Test snakemake workflows
+    - name: Run unit tests
       run: |
-        make test
-
-    - name: Upload artifacts
-      uses: actions/upload-artifact@v4
-      with:
-        name: technology-data
-        path: outputs/
-
-    - name: Check for differences in generated outputs
-      run: |-
-        git config --global --add safe.directory $GITHUB_WORKSPACE
-        git add outputs/
-        if ! git diff --cached --quiet; then
-          echo "Generated outputs differ from the branch. Please commit rerun `snakemake` and commit the changes to the `outputs/*.csv` files."
-          git diff --cached
-          exit 1
-        else
-          echo "No differences found in generated outputs."
-        fi
+        uv run pytest