Merge branch 'main' into reduce-size-test-cli

Author: barneydobson

.github/workflows/publish.yml

@@ -61,7 +61,7 @@ jobs:
         path: dist
 
     - name: Generate artifact attestation for sdist and wheel
-      uses: actions/attest-build-provenance@v1
+      uses: actions/attest-build-provenance@v2
       with:
         subject-path: dist
 
@@ -84,7 +84,7 @@ jobs:
         path: dist
 
     - name: Generate artifact attestation for sdist and wheel
-      uses: actions/attest-build-provenance@v1
+      uses: actions/attest-build-provenance@v2
       with:
         subject-path: dist
 

.pre-commit-config.yaml

@@ -5,33 +5,33 @@ repos:
       - id: check-merge-conflict
       - id: debug-statements
   - repo: https://github.com/astral-sh/ruff-pre-commit
-    rev: "v0.8.3"
+    rev: "v0.9.10"
     hooks:
       - id: ruff
         args: [--fix, --exit-non-zero-on-fix]
       - id: ruff-format
   - repo: https://github.com/pre-commit/mirrors-mypy
-    rev: "v1.13.0"
+    rev: "v1.15.0"
     hooks:
       - id: mypy
         additional_dependencies:
           [types-requests, types-PyYAML]
   - repo: https://github.com/igorshubovych/markdownlint-cli
-    rev: v0.43.0
+    rev: v0.44.0
     hooks:
       - id: markdownlint
         args: ["--disable", "MD013", "--disable", "MD025", "--"]
 
   - repo: https://github.com/codespell-project/codespell
-    rev: v2.3.0
+    rev: v2.4.1
     hooks:
     - id: codespell
       name: Check common misspellings in text files with codespell.
       additional_dependencies:
       - tomli
 
   - repo: https://github.com/tox-dev/pyproject-fmt
-    rev: v2.5.0
+    rev: v2.5.1
     hooks:
     - id: pyproject-fmt
       name: Apply a consistent format to pyproject.toml

docs/config_guide.md

@@ -49,9 +49,11 @@ parameter_overrides:
     max_street_length: 40
 ```
 
-Note that we must provide the parameter category for the parameter that we are
+Note that we must provide the parameter group for the parameter that we are
 changing (`subcatchment_derivation` above).
 
+If you want to understand how parameters are implemented in more detail, and in particular if you are creating new behaviours and need to add your own parameters, see our [parameter guide](parameters_guide.md).
+
 As our SWMManywhere paper [link preprint](https://doi.org/10.1016/j.envsoft.2025.106358) demonstrates, you can capture an enormously wide range of UDM behaviours through changing parameters. However, if your system is particularly unusual, or you are testing out new behaviours then you may need to adopt a more elaborate approach.
 
 ### Customise `graphfcns`

docs/index.md

@@ -6,11 +6,12 @@ derive a synthetic urban drainage network anywhere in the world.
 ## Table of contents
 <!-- markdownlint-disable MD007 -->
 - [Home](index.md)
-- [About](./paper/paper.pdf)
+- [About](./paper/paper.md)
 - [Quickstart](quickstart.md)
 - Guides:
     - [Configuration file](config_guide.md)
     - [Extended demo](./notebooks/extended_demo.py)
+    - [Parameters guide](parameters_guide.md)
     - [Graph functions](graphfcns_guide.md)
     - [Metrics guide](metrics_guide.md)
 - [Contributing](CONTRIBUTING.md)

docs/parameters_guide.md

@@ -0,0 +1,109 @@
+# Parameters guide
+
+SWMManywhere is a deliberately highly parameterised workflow, with the goal of enabling users to create a diverse range of UDMs. This guide is to explain the logic of the implemented parameters and how to customise them, as what each parameter does is highly specific to the [`graphfcn`](graphfcns_guide.md) that uses it. Instead, to understand specific parameter purposes, you can view all available parameters at the [API](reference-parameters.md).
+
+## Using parameters
+
+Let's look at a [parameter group](reference-parameters.md#swmmanywhere.parameters.OutfallDerivation), which is a group of parameters related to identifying outfall locations.
+
+:::swmmanywhere.parameters.OutfallDerivation
+    handler: python
+    options:
+      members: no
+      show_root_heading: false
+      show_bases: false
+      show_source: true
+      show_root_toc_entry: false
+      show_docstring_attributes: false
+      show_docstring_description: false
+      show_docstring_examples: false
+      show_docstring_parameters: false
+      show_docstring_returns: false
+      show_docstring_raises: false
+
+We can see here three related parameters and relevant metadata, grouped together in a [`pydantic.BaseModel`](https://docs.pydantic.dev/latest/api/base_model/) object. Parameters in SWMManywhere are grouped together because `graphfcns` that need one of them tend to need the others. Let's look at [`identify_outfalls`](reference-graph-utilities.md#swmmanywhere.graphfcns.outfall_graphfcns.identify_outfalls), which needs these parameters.
+
+:::swmmanywhere.graphfcns.outfall_graphfcns.identify_outfalls
+    handler: python
+    options:
+      members: no
+      show_root_heading: false
+      show_bases: false
+      show_source: true
+      show_root_toc_entry: false
+      show_docstring_attributes: false
+      show_docstring_description: false
+      show_docstring_examples: false
+      show_docstring_parameters: false
+      show_docstring_returns: false
+      show_docstring_raises: false
+
+When calling [`iterate_graphfcns`](reference-graph-utilities.md#swmmanywhere.graph_utilities.iterate_graphfcns), for more information see [here](graphfcns_guide.md#lists-of-graph-functions), SWMManywhere will automatically provide any parameters that have been registered to any graphfcn.
+
+## Registering parameters
+
+When you create a new parameter, it will need to belong to an existing or new parameter group.
+
+### Creating a new parameter group(s)
+
+You create a new module(s) that can contain multiple parameter groups. See below as a template of such amodule.
+
+```python
+{%
+    include-markdown "../tests/test_data/custom_parameters.py"
+    comments=false
+%}
+```
+
+### Adjust config file
+
+We will add the required lines to the
+[minimum viable config](config_guide.md#minimum-viable-configuration) template.
+
+```yml
+{%
+    include-markdown "snippets/minimum_viable_template.yml"
+    comments=false
+%}
+custom_parameter_modules: 
+  - /path/to/custom_parameters.py
+```
+
+Now when we run our `config` file, these parameters will be registered and any [custom graphfcns](graphfcns_guide.md#add-a-new-graph-function) will have access to them.
+
+### Changing existing parameter groups
+
+There may be cases where you want to change existing parameter groups, such as introducing new weights to the [`calculate_weights`](reference-graph-utilities.md#swmmanywhere.graphfcns.topology_graphfcns.calculate_weights) step so that they are minimized during the shortest path optimization. In this example, we want the [`TopologyDerivation`](reference-parameters.md#swmmanywhere.parameters.TopologyDerviation) group to include some new parameters. We can do this in a similar way to [above](#creating-a-new-parameter-groups), but being mindful to inherit from `TopologyDerivation` rather than `BaseModel`:
+
+```python
+from swmmanywhere.parameters import register_parameter_group, TopologyDerivation, Field
+
+@register_parameter_group("topology_derivation")
+class NewTopologyDerivation(TopologyDerivation):
+    new_weight_scaling: float = Field(
+        default=1,
+        le=1,
+        ge=0,
+    )
+    new_weight_exponent: float = Field(
+        default=1,
+        le=2,
+        ge=0,
+    )
+```
+
+Now the `calculate_weights` function will have access to these new weighting parameters, as well as existing ones.
+
+Note, in this specific example of adding custom weights, you will also have to:
+
+- Update the `weights` parameter in your `config` file, for example:
+
+```yaml
+parameter_overrides:
+  topology_derviation:
+    weights:
+      - new_weight
+      - length
+```
+
+- [Create and register a `graphfcn`](graphfcns_guide.md#add-a-new-graph-function) that adds the `new_weight` parameter to the graph.

mkdocs.yml

@@ -41,11 +41,12 @@ extra_javascript:
 
 nav:
   - Home: index.md
-  - About: ./paper/paper.pdf
+  - About: ./paper/paper.md
   - Quickstart: quickstart.md
   - Guides:
     - Configuration guide: config_guide.md
     - Extended demo: ./notebooks/extended_demo.py
+    - Parameters guide: parameters_guide.md
     - Graph functions: graphfcns_guide.md
     - Metrics guide: metrics_guide.md
   - Contributing: CONTRIBUTING.md

pyproject.toml

@@ -114,7 +114,7 @@ lint.pydocstyle.convention = "google"
 
 [tool.codespell]
 skip = "src/swmmanywhere/defs/iso_converter.yml,*.inp,docs/paper/*"
-ignore-words-list = "gage,gages,Carrer"
+ignore-words-list = "gage,gages,Carrer,anc"
 
 [tool.pytest.ini_options]
 addopts = "-v --cov=src/swmmanywhere --cov-report=xml --doctest-modules --ignore=src/swmmanywhere/logging.py"

src/netcomp/distance/exact.py

@@ -192,8 +192,7 @@ def lambda_dist(A1, A2, k=None, p=2, kind="laplacian"):
         evals1, evals2 = [evals[::-1] for evals in [evals1, evals2]]
     else:
         raise InputError(
-            "Invalid type, choose from 'laplacian', "
-            "'laplacian_norm', and 'adjacency'."
+            "Invalid type, choose from 'laplacian', 'laplacian_norm', and 'adjacency'."
         )
     dist = la.norm(evals1[:k] - evals2[:k], ord=p)
     return dist

src/netcomp/linalg/resistance.py

@@ -71,7 +71,7 @@ def resistance_matrix(A, check_connected=True):
             G = nx.from_numpy_array(A)
         if not nx.is_connected(G):
             raise UndefinedException(
-                "Graph is not connected. " "Resistance matrix is undefined."
+                "Graph is not connected. Resistance matrix is undefined."
             )
     L = laplacian_matrix(A)
     with suppress(Exception):

src/swmmanywhere/defs/schema.yml

@@ -33,4 +33,5 @@ properties:
   parameter_overrides: {type: ['object', 'null']}
   custom_metric_modules: {type: array, items: {type: string}}
   custom_graphfcn_modules: {type: array, items: {type: string}}
+  custom_parameters_modules: {type: array, items: {type: string}}
 required: [base_dir, project, bbox]

src/swmmanywhere/parameters.py

@@ -2,19 +2,36 @@
 
 from __future__ import annotations
 
+from typing import Callable
+
 import numpy as np
 from pydantic import BaseModel, Field, model_validator
 
+from swmmanywhere.logging import logger
+
+parameter_register = {}
+
+
+def register_parameter_group(name: str) -> Callable:
+    """Register a parameter group.
+
+    Args:
+        name (str): Name of the parameter group that it will be keyed to in
+            parameter_register.
+    """
+
+    def wrapper(cls: BaseModel) -> BaseModel:
+        if name in parameter_register:
+            logger.warning(f"{name} already in parameter register, overwriting.")
+        parameter_register[name] = cls()
+        return cls
+
+    return wrapper
+
 
 def get_full_parameters():
     """Get the full set of parameters."""
-    return {
-        "subcatchment_derivation": SubcatchmentDerivation(),
-        "outfall_derivation": OutfallDerivation(),
-        "topology_derivation": TopologyDerivation(),
-        "hydraulic_design": HydraulicDesign(),
-        "metric_evaluation": MetricEvaluation(),
-    }
+    return parameter_register
 
 
 def get_full_parameters_flat():
@@ -32,6 +49,7 @@ def get_full_parameters_flat():
     return parameters_flat
 
 
+@register_parameter_group(name="subcatchment_derivation")
 class SubcatchmentDerivation(BaseModel):
     """Parameters for subcatchment derivation."""
 
@@ -86,6 +104,7 @@ class SubcatchmentDerivation(BaseModel):
     )
 
 
+@register_parameter_group(name="outfall_derivation")
 class OutfallDerivation(BaseModel):
     """Parameters for outfall derivation."""
 
@@ -113,6 +132,7 @@ class OutfallDerivation(BaseModel):
     )
 
 
+@register_parameter_group(name="topology_derivation")
 class TopologyDerivation(BaseModel):
     """Parameters for topology derivation."""
 
@@ -212,6 +232,7 @@ def check_weights(cls, values):
         return values
 
 
+@register_parameter_group("hydraulic_design")
 class HydraulicDesign(BaseModel):
     """Parameters for hydraulic design."""
 
@@ -273,6 +294,7 @@ class HydraulicDesign(BaseModel):
     )
 
 
+@register_parameter_group(name="metric_evaluation")
 class MetricEvaluation(BaseModel):
     """Parameters for metric evaluation."""