Merge branch 'main' into pre-commit-ci-update-config

Author: barneydobson

.github/workflows/publish.yml

@@ -61,7 +61,7 @@ jobs:
         path: dist
 
     - name: Generate artifact attestation for sdist and wheel
-      uses: actions/attest-build-provenance@v1
+      uses: actions/attest-build-provenance@v2
       with:
         subject-path: dist
 
@@ -84,7 +84,7 @@ jobs:
         path: dist
 
     - name: Generate artifact attestation for sdist and wheel
-      uses: actions/attest-build-provenance@v1
+      uses: actions/attest-build-provenance@v2
       with:
         subject-path: dist
 

docs/config_guide.md

@@ -49,9 +49,11 @@ parameter_overrides:
     max_street_length: 40
 ```
 
-Note that we must provide the parameter category for the parameter that we are
+Note that we must provide the parameter group for the parameter that we are
 changing (`subcatchment_derivation` above).
 
+If you want to understand how parameters are implemented in more detail, and in particular if you are creating new behaviours and need to add your own parameters, see our [parameter guide](parameters_guide.md).
+
 As our SWMManywhere paper [link preprint](https://doi.org/10.1016/j.envsoft.2025.106358) demonstrates, you can capture an enormously wide range of UDM behaviours through changing parameters. However, if your system is particularly unusual, or you are testing out new behaviours then you may need to adopt a more elaborate approach.
 
 ### Customise `graphfcns`

docs/index.md

@@ -6,11 +6,12 @@ derive a synthetic urban drainage network anywhere in the world.
 ## Table of contents
 <!-- markdownlint-disable MD007 -->
 - [Home](index.md)
-- [About](./paper/paper.pdf)
+- [About](./paper/paper.md)
 - [Quickstart](quickstart.md)
 - Guides:
     - [Configuration file](config_guide.md)
     - [Extended demo](./notebooks/extended_demo.py)
+    - [Parameters guide](parameters_guide.md)
     - [Graph functions](graphfcns_guide.md)
     - [Metrics guide](metrics_guide.md)
 - [Contributing](CONTRIBUTING.md)

docs/parameters_guide.md

@@ -0,0 +1,109 @@
+# Parameters guide
+
+SWMManywhere is a deliberately highly parameterised workflow, with the goal of enabling users to create a diverse range of UDMs. This guide is to explain the logic of the implemented parameters and how to customise them, as what each parameter does is highly specific to the [`graphfcn`](graphfcns_guide.md) that uses it. Instead, to understand specific parameter purposes, you can view all available parameters at the [API](reference-parameters.md).
+
+## Using parameters
+
+Let's look at a [parameter group](reference-parameters.md#swmmanywhere.parameters.OutfallDerivation), which is a group of parameters related to identifying outfall locations.
+
+:::swmmanywhere.parameters.OutfallDerivation
+    handler: python
+    options:
+      members: no
+      show_root_heading: false
+      show_bases: false
+      show_source: true
+      show_root_toc_entry: false
+      show_docstring_attributes: false
+      show_docstring_description: false
+      show_docstring_examples: false
+      show_docstring_parameters: false
+      show_docstring_returns: false
+      show_docstring_raises: false
+
+We can see here three related parameters and relevant metadata, grouped together in a [`pydantic.BaseModel`](https://docs.pydantic.dev/latest/api/base_model/) object. Parameters in SWMManywhere are grouped together because `graphfcns` that need one of them tend to need the others. Let's look at [`identify_outfalls`](reference-graph-utilities.md#swmmanywhere.graphfcns.outfall_graphfcns.identify_outfalls), which needs these parameters.
+
+:::swmmanywhere.graphfcns.outfall_graphfcns.identify_outfalls
+    handler: python
+    options:
+      members: no
+      show_root_heading: false
+      show_bases: false
+      show_source: true
+      show_root_toc_entry: false
+      show_docstring_attributes: false
+      show_docstring_description: false
+      show_docstring_examples: false
+      show_docstring_parameters: false
+      show_docstring_returns: false
+      show_docstring_raises: false
+
+When calling [`iterate_graphfcns`](reference-graph-utilities.md#swmmanywhere.graph_utilities.iterate_graphfcns), for more information see [here](graphfcns_guide.md#lists-of-graph-functions), SWMManywhere will automatically provide any parameters that have been registered to any graphfcn.
+
+## Registering parameters
+
+When you create a new parameter, it will need to belong to an existing or new parameter group.
+
+### Creating a new parameter group(s)
+
+You create a new module(s) that can contain multiple parameter groups. See below as a template of such amodule.
+
+```python
+{%
+    include-markdown "../tests/test_data/custom_parameters.py"
+    comments=false
+%}
+```
+
+### Adjust config file
+
+We will add the required lines to the
+[minimum viable config](config_guide.md#minimum-viable-configuration) template.
+
+```yml
+{%
+    include-markdown "snippets/minimum_viable_template.yml"
+    comments=false
+%}
+custom_parameter_modules: 
+  - /path/to/custom_parameters.py
+```
+
+Now when we run our `config` file, these parameters will be registered and any [custom graphfcns](graphfcns_guide.md#add-a-new-graph-function) will have access to them.
+
+### Changing existing parameter groups
+
+There may be cases where you want to change existing parameter groups, such as introducing new weights to the [`calculate_weights`](reference-graph-utilities.md#swmmanywhere.graphfcns.topology_graphfcns.calculate_weights) step so that they are minimized during the shortest path optimization. In this example, we want the [`TopologyDerivation`](reference-parameters.md#swmmanywhere.parameters.TopologyDerviation) group to include some new parameters. We can do this in a similar way to [above](#creating-a-new-parameter-groups), but being mindful to inherit from `TopologyDerivation` rather than `BaseModel`:
+
+```python
+from swmmanywhere.parameters import register_parameter_group, TopologyDerivation, Field
+
+@register_parameter_group("topology_derivation")
+class NewTopologyDerivation(TopologyDerivation):
+    new_weight_scaling: float = Field(
+        default=1,
+        le=1,
+        ge=0,
+    )
+    new_weight_exponent: float = Field(
+        default=1,
+        le=2,
+        ge=0,
+    )
+```
+
+Now the `calculate_weights` function will have access to these new weighting parameters, as well as existing ones.
+
+Note, in this specific example of adding custom weights, you will also have to:
+
+- Update the `weights` parameter in your `config` file, for example:
+
+```yaml
+parameter_overrides:
+  topology_derviation:
+    weights:
+      - new_weight
+      - length
+```
+
+- [Create and register a `graphfcn`](graphfcns_guide.md#add-a-new-graph-function) that adds the `new_weight` parameter to the graph.

mkdocs.yml

@@ -41,11 +41,12 @@ extra_javascript:
 
 nav:
   - Home: index.md
-  - About: ./paper/paper.pdf
+  - About: ./paper/paper.md
   - Quickstart: quickstart.md
   - Guides:
     - Configuration guide: config_guide.md
     - Extended demo: ./notebooks/extended_demo.py
+    - Parameters guide: parameters_guide.md
     - Graph functions: graphfcns_guide.md
     - Metrics guide: metrics_guide.md
   - Contributing: CONTRIBUTING.md

src/swmmanywhere/defs/schema.yml

@@ -33,4 +33,5 @@ properties:
   parameter_overrides: {type: ['object', 'null']}
   custom_metric_modules: {type: array, items: {type: string}}
   custom_graphfcn_modules: {type: array, items: {type: string}}
+  custom_parameters_modules: {type: array, items: {type: string}}
 required: [base_dir, project, bbox]

src/swmmanywhere/parameters.py

@@ -2,19 +2,36 @@
 
 from __future__ import annotations
 
+from typing import Callable
+
 import numpy as np
 from pydantic import BaseModel, Field, model_validator
 
+from swmmanywhere.logging import logger
+
+parameter_register = {}
+
+
+def register_parameter_group(name: str) -> Callable:
+    """Register a parameter group.
+
+    Args:
+        name (str): Name of the parameter group that it will be keyed to in
+            parameter_register.
+    """
+
+    def wrapper(cls: BaseModel) -> BaseModel:
+        if name in parameter_register:
+            logger.warning(f"{name} already in parameter register, overwriting.")
+        parameter_register[name] = cls()
+        return cls
+
+    return wrapper
+
 
 def get_full_parameters():
     """Get the full set of parameters."""
-    return {
-        "subcatchment_derivation": SubcatchmentDerivation(),
-        "outfall_derivation": OutfallDerivation(),
-        "topology_derivation": TopologyDerivation(),
-        "hydraulic_design": HydraulicDesign(),
-        "metric_evaluation": MetricEvaluation(),
-    }
+    return parameter_register
 
 
 def get_full_parameters_flat():
@@ -32,6 +49,7 @@ def get_full_parameters_flat():
     return parameters_flat
 
 
+@register_parameter_group(name="subcatchment_derivation")
 class SubcatchmentDerivation(BaseModel):
     """Parameters for subcatchment derivation."""
 
@@ -86,6 +104,7 @@ class SubcatchmentDerivation(BaseModel):
     )
 
 
+@register_parameter_group(name="outfall_derivation")
 class OutfallDerivation(BaseModel):
     """Parameters for outfall derivation."""
 
@@ -113,6 +132,7 @@ class OutfallDerivation(BaseModel):
     )
 
 
+@register_parameter_group(name="topology_derivation")
 class TopologyDerivation(BaseModel):
     """Parameters for topology derivation."""
 
@@ -212,6 +232,7 @@ def check_weights(cls, values):
         return values
 
 
+@register_parameter_group("hydraulic_design")
 class HydraulicDesign(BaseModel):
     """Parameters for hydraulic design."""
 
@@ -273,6 +294,7 @@ class HydraulicDesign(BaseModel):
     )
 
 
+@register_parameter_group(name="metric_evaluation")
 class MetricEvaluation(BaseModel):
     """Parameters for metric evaluation."""
 

src/swmmanywhere/swmmanywhere.py

@@ -314,6 +314,38 @@ def check_parameter_overrides(config: dict):
     return config
 
 
+def import_module(module: Path):
+    """Import module with importlib.
+
+    Args:
+        module (Path): path to module.
+    """
+    # Import the module
+    spec = importlib.util.spec_from_file_location(  # type: ignore[attr-defined]
+        module.stem, module
+    )
+    module = importlib.util.module_from_spec(spec)  # type: ignore[attr-defined]
+    spec.loader.exec_module(module)
+
+
+def import_modules(modules: list[str | Path]):
+    """Import modules specified in list of files.
+
+    Args:
+        modules (list): List of files
+
+    Raises:
+        ValueError: If a module does not exist.
+    """
+    for module in modules:
+        module = Path(module)
+
+        # Check that module exists
+        if not module.exists():
+            raise FileNotFoundError(f"Module not found at {module}")
+        import_module(module)
+
+
 def check_and_register_custom_graphfcns(config: dict):
     """Check, register and validate custom graphfcns in the config.
 
@@ -324,21 +356,7 @@ def check_and_register_custom_graphfcns(config: dict):
         ValueError: If a graphfcn module does not exist.
         ValueError: If a custom graphfcn is not successfully registered.
     """
-    for custom_graphfcn_module in config.get("custom_graphfcn_modules", []):
-        custom_graphfcn_module = Path(custom_graphfcn_module)
-
-        # Check that the custom graphfcn exists
-        if not custom_graphfcn_module.exists():
-            raise FileNotFoundError(
-                f"Custom graphfcn not found at {custom_graphfcn_module}"
-            )
-
-        # Import the custom graphfcn module
-        spec = importlib.util.spec_from_file_location(  # type: ignore[attr-defined]
-            custom_graphfcn_module.stem, custom_graphfcn_module
-        )
-        custom_graphfcn_module = importlib.util.module_from_spec(spec)  # type: ignore[attr-defined]
-        spec.loader.exec_module(custom_graphfcn_module)
+    import_modules(config.get("custom_graphfcn_modules", []))
 
     # Validate the import
     validate_graphfcn_list(config.get("graphfcn_list", []))
@@ -355,28 +373,24 @@ def check_and_register_custom_metrics(config: dict):
     Raises:
         ValueError: If the custom metrics module does not exist.
     """
-    for custom_metric_module in config.get("custom_metric_modules", []):
-        custom_metric_module = Path(custom_metric_module)
-
-        # Check that the custom graphfcn exists
-        if not custom_metric_module.exists():
-            raise FileNotFoundError(
-                f"Custom graphfcn not found at {custom_metric_module}"
-            )
-
-        # Import the custom graphfcn module
-        spec = importlib.util.spec_from_file_location(  # type: ignore[attr-defined]
-            custom_metric_module.stem, custom_metric_module
-        )
-        custom_metric_module = importlib.util.module_from_spec(spec)  # type: ignore[attr-defined]
-        spec.loader.exec_module(custom_metric_module)
+    import_modules(config.get("custom_metric_modules", []))
 
     # Validate metric list
     validate_metric_list(config.get("metric_list", []))
 
     return config
 
 
+def register_custom_parameters(config: dict):
+    """Register custom parameter modules.
+
+    Args:
+        config (dict): The configuration.
+    """
+    import_modules(config.get("custom_parameter_modules", []))
+    return config
+
+
 def save_config(config: dict, config_path: Path):
     """Save the configuration to a file.
 
@@ -441,9 +455,12 @@ def load_config(
     # Check and register custom metrics
     config = check_and_register_custom_metrics(config)
 
-    # Check custom graphfcns
+    # Check and register custom graphfcns
     config = check_and_register_custom_graphfcns(config)
 
+    # Register custom parameters
+    config = register_custom_parameters(config)
+
     return config
 
 

tests/test_data/custom_parameters.py

@@ -0,0 +1,16 @@
+from __future__ import annotations
+
+from swmmanywhere import parameters
+
+
+@parameters.register_parameter_group(name="new_params")
+class new_params(parameters.BaseModel):
+    """New parameters."""
+
+    new_param: int = parameters.Field(
+        default=1,
+        ge=0,
+        le=10,
+        unit="-",
+        description="A new parameter.",
+    )