Hla 1142 module creation docs

README.md

@@ -24,9 +24,9 @@ The demo_configuration folder in this repository can be passed as the config_fol
 the deploy-tools commands. The deployment_root needs to be a writeable location for all
 files to get deployed under.
 
-In normal use these commands are not run by hand: they act on a shared deployment area and
-belong in a CI pipeline, gated by change review. Running them manually against the demo
-configuration, as below, is just the quickest way to see what each does — the
+These commands act on a shared deployment area, so we recommend running them from a CI
+pipeline gated by change review rather than by hand. Running them manually against the demo
+configuration, as below, is the quickest way to see what each does — the
 [documentation](https://diamondlightsource.github.io/deploy-tools) has a hands-on tutorial
 and a guide to driving them from CI.
 
@@ -46,7 +46,7 @@ deploy-tools validate $deployment_root $config_folder
 deploy-tools sync $deployment_root $config_folder
 
 # Compare the current deployment snapshot against what is actually deployed in the
-# deployment area. CI/CD should run this before a deploy to confirm a healthy state.
+# deployment area. Run this before a deploy to confirm a healthy state.
 deploy-tools compare $deployment_root
 
 ```

docs/conf.py

@@ -60,6 +60,9 @@
 # So we can use the ::: syntax
 myst_enable_extensions = ["colon_fence"]
 
+# Generate anchors for headings (levels 1-3) so pages can link to sections.
+myst_heading_anchors = 3
+
 # If true, Sphinx will warn about all references where the target cannot
 # be found.
 nitpicky = True

docs/explanations/deployment-area.md

@@ -37,7 +37,8 @@ corresponding `modulefile`. Users put only the *modulefiles* directories on thei
 
 Deprecation is therefore cheap and reversible: the built files never move, only the symlink
 moves between `modulefiles/` and `deprecated/modulefiles/`. See
-[the release lifecycle](deprecation-lifecycle.md) for the full set of transitions.
+[the release lifecycle](deprecation-lifecycle.md#configuration-is-declarative) for
+the full set of transitions.
 
 ## The build area
 

docs/explanations/deployment-steps.md

@@ -7,7 +7,7 @@ a command of their own. For the commands themselves see the [CLI reference](../c
 | Step | Description | Run by |
 |------|-------------|--------|
 | Compare | Compare the current deployment snapshot against the modulefiles and built modules that actually exist, confirming the [deployment area](deployment-area.md) is healthy. | `compare` |
-| Validate | Diff the new configuration against the current snapshot to determine the set of actions to take, and check those actions are permitted by the [release lifecycle](deprecation-lifecycle.md). | `validate`, `sync` |
+| Validate | Diff the new configuration against the current snapshot to determine the set of actions to take, and check those actions are permitted by the [lifecycle guard rails](deprecation-lifecycle.md#the-guard-rails). | `validate`, `sync` |
 | Build | Generate entrypoint scripts, configuration files and environment variables for each changed Module, writing them to the build area. | `sync` (`validate --test-build`) |
 | Deploy | Move the built Modules from the build area into the Modules Area, link each modulefile into the live or deprecated tree according to its status, and update default versions. | `sync` |
 
@@ -23,4 +23,4 @@ place, or nothing changes.
 read-only checks run before it. See
 [snapshots and the compare safety net](snapshots-and-compare.md) for how the snapshot
 ties them together, and [drive deploy-tools from CI](../how-to/ci-pipeline.md) for the
-order a pipeline runs them in.
+order to run them in.

docs/explanations/deprecation-lifecycle.md

@@ -9,8 +9,8 @@ deleted — is expressed by adding or removing a Release in configuration and to
 
 You never tell `deploy-tools` to "deprecate" or "remove" something directly. You describe
 the set of Releases you want, and the tool compares that against the
-[snapshot](snapshots-and-compare.md) of the last `sync` to derive the actions needed. Each
-Release falls into one of these cases:
+[snapshot](snapshots-and-compare.md#what-the-snapshot-is) of the last `sync` to derive
+the actions needed. Each Release falls into one of these cases:
 
 | Transition | Detected when… | Effect on the deployment area |
 |------------|----------------|-------------------------------|

docs/explanations/snapshots-and-compare.md

@@ -37,7 +37,7 @@ will not be detected. The risk of corruption is avoided at build time instead: a
 is built on the same filesystem as the deployment area and published by a single atomic
 rename, so a partial or failed build (including `.sif` files) is never moved into place.
 
-This is why CI should run `compare` *before* every `sync`: it confirms the area is in the
+This is why `compare` should be run *before* every `sync`: it confirms the area is in the
 healthy state the last `sync` claimed to leave it in.
 
 ## Recovery is manual
@@ -55,7 +55,7 @@ Two facilities help here:
   often easier to rollback the configuration to a previous state rather than fix the
   latest deployment.
 - `compare --from-scratch` asserts only that the deployment root exists and is *empty* —
-  this is the check to run in CI before the very first `sync`, when no snapshot exists yet.
+  this is the check to run before the very first `sync`, when no snapshot exists yet.
 
 The git repository in the deployment area exists only to give `compare --use-ref` this
 reference point. It deliberately excludes the build area and Apptainer images, and is

docs/glossary.md

@@ -9,8 +9,9 @@ for the on-disk layout these terms refer to.
 | Environment Modules | A [standard package for Linux](https://modules.readthedocs.io/en/latest/) that provides the commands for loading and unloading 'Environment Modules' (each defined by a Modulefile). Note that while we are using this system, our definition of Module is separate. If we are referring to an Environment Module, we will use the full name. |
 | Modulefile | Used by the Environment Modules package to specify all details of an Environment Module. This can include executables to add to the path, environment variables to set, etc. |
 | Module | A set of files that can be used to provide applications on your path, provide configuration, and set environment variables. We do this using the Environment Modules system by providing a Modulefile with the relevant configuration. |
-| Application | Each Module can be configured with multiple Applications, each one providing one or more executables. There are 3 types of Application: `Apptainer` (an executable container image), `Shell` (a Bash script) and `Binary` (an executable downloaded from a URL and verified against a hash). |
+| Application | Each Module can be configured with multiple Applications, each one providing one or more executables. There are 3 types of Application: `Apptainer` (an executable container image), `Shell` (a Bash script) and `Binary` (an executable downloaded from a URL and verified against a hash). See [the configuration reference](reference/configuration.md). |
 | Release (noun) | A Module, including version, alongside its [lifecycle](explanations/deprecation-lifecycle.md) (i.e. deprecation) status. |
+| Release file | The `<name>/<version>.yaml` file you author to define one Release. See [the configuration reference](reference/configuration.md). |
 | Deployment | The declared configuration for a Deployment Area: all Releases (deprecated or not) to be maintained there, plus global settings. Written to the `deployment.yaml` snapshot by `sync`. The act of deploying is written lowercase. |
 | Deployment Step | Refers to one of the primary steps that make up the deployment process. See [the deployment process](explanations/deployment-steps.md) for a breakdown. |
 | End User | Refers to anybody who is intending to make use of a deployed Module. This can include the people modifying configuration themselves. |

docs/how-to.md

@@ -5,6 +5,7 @@ Practical step-by-step guides for the more experienced user.
 ```{toctree}
 :maxdepth: 1
 
+how-to/write-module-configuration
 how-to/ci-pipeline
 how-to/run-container
 how-to/vscode-tasks

docs/how-to/ci-pipeline.md

@@ -1,12 +1,15 @@
 # Drive deploy-tools from CI
 
-In normal use, nobody should run `sync`, `validate` or `compare` by hand. Those commands
-touch a shared deployment area, so they belong to a CI pipeline in the configuration
-repository, gated by change review. End users only edit configuration and open a change;
-merging it deploys.
+`sync`, `validate` and `compare` touch a shared deployment area, so we recommend driving
+them from a CI pipeline in the configuration repository, gated by change review: end users
+only edit configuration and open a change, and merging it deploys. `deploy-tools` does not
+require this — an administrator can run the commands by hand — but a pipeline gives you
+review, repeatability and one place to serialise runs.
 
-This guide describes what that pipeline must do. It is deliberately independent of any
-particular CI system — translate the responsibilities below into your own.
+This guide describes the responsibilities such a pipeline has. It is deliberately
+independent of any particular CI system — translate them into your own. If you run the
+commands by hand, the same ordering and the [one at a time](#run-one-at-a-time) rule still
+apply.
 
 ## On a proposed change
 
@@ -15,9 +18,9 @@ When someone opens a change (before it is merged), run, without altering the are
 - `deploy-tools compare <area>` — confirm the area still matches its last snapshot, so
   the change is being checked against a known-healthy baseline.
 - `deploy-tools validate <area> <config>` — confirm the new configuration is valid and its
-  [lifecycle transitions](../explanations/deprecation-lifecycle.md) are permitted. Add
-  `--test-build` to build every changed Module in a temporary directory, catching build
-  failures before merge.
+  [lifecycle transitions](../explanations/deprecation-lifecycle.md#the-guard-rails) are
+  permitted. Add `--test-build` to build every changed Module in a temporary directory,
+  catching build failures before merge.
 
 This gives reviewers a green light that the change is deployable without changing anything
 on the filesystem.
@@ -32,15 +35,15 @@ When the change is merged to the main branch, deploy it:
 
 ## Run one at a time
 
-`deploy-tools` has no locking of its own. The pipeline must ensure only one run
+`deploy-tools` has no locking of its own. Whatever drives it must ensure only one run
 touches the area at a time — two concurrent `sync`s, or a `sync` racing a `compare`,
 can corrupt the area or report false drift. Serialise the relevant jobs (and, if
 possible, restrict them to a single runner).
 
 ## Manual operations
 
-Some tasks fall outside the automatic flows and are best run from a manually-triggered
-pipeline, exposing the relevant options as parameters:
+Some tasks fall outside the automatic flows and are run by hand, or from a
+manually-triggered pipeline exposing the relevant options as parameters:
 
 - **The first deployment.** A brand-new area has no snapshot to compare against, so it is
   run manually rather than triggered by a merge. Use `--from-scratch`, which assumes the

docs/how-to/regenerate-schemas.md

@@ -12,7 +12,7 @@ schemas and commit again.
 
 If you bypass the hooks (for example with `git commit --no-verify`), regenerate the
 schemas manually. The simplest way is the **Generate Schema** VSCode task (see
-[the VSCode tasks guide](vscode-tasks.md)), which writes to the correct location.
+[the VSCode tasks guide](vscode-tasks.md#running-a-task)), which writes to the correct location.
 Equivalently, run the CLI, pointing it at that folder:
 
 ```bash

docs/how-to/run-container.md

@@ -12,3 +12,6 @@ $ docker run ghcr.io/diamondlightsource/deploy-tools:latest --version
 ```
 
 To get a released version, use a numbered release instead of `latest`.
+
+This image is a convenient way to drive `deploy-tools` against a shared deployment area,
+typically from a CI pipeline — see [drive deploy-tools from CI](ci-pipeline.md).

docs/how-to/write-module-configuration.md

@@ -0,0 +1,88 @@
+# Write a Module configuration
+
+To add or change a Module, you edit YAML in the configuration folder. This guide covers
+the mechanics. For the shape of the files and every field they take, see
+[the configuration reference](../reference/configuration.md); to point your editor at
+the matching schema, the [schema reference](../schemas.md).
+
+## Add a new Module version
+
+1. Create the Release file at `<config folder>/<name>/<version>.yaml`. The folder name
+   is the Module `name` and the filename is the `version`, so `phoebus/0.1.yaml` defines
+   version `0.1` of `phoebus`. The `name` and `version` inside the file must match the
+   path.
+
+2. Add the schema line as the first line so your editor validates as you type:
+
+   ```yaml
+   # yaml-language-server: $schema=https://raw.githubusercontent.com/DiamondLightSource/deploy-tools/main/src/deploy_tools/models/schemas/release.json
+   ```
+
+   The line is read by [`yaml-language-server`](https://github.com/redhat-developer/yaml-language-server),
+   so it takes effect in VS Code with the Red Hat YAML extension or any other editor
+   running that language server; elsewhere it is an inert comment. See the
+   [schema reference](../schemas.md) for details.
+
+3. Define the Module. Most Modules provide one or more applications; the smallest useful
+   one is a single shell script:
+
+   ```yaml
+   module:
+     name: my-module
+     version: "1.0"
+     description: What this Module provides
+     applications:
+       - app_type: shell
+         name: hello
+         script:
+           - echo "hello from my-module"
+   ```
+
+   Swap the application for an `apptainer` or `binary` one as needed — see
+   [the three application types](../reference/configuration.md#the-three-application-types).
+
+   A Module doesn't have to provide an application: it can instead just set environment
+   variables or pull in other Modules as
+   [dependencies](../reference/configuration.md#a-module). Give such a Module an
+   empty `applications: []`.
+
+## Set the default version
+
+`module load <name>` with no version loads the default. If you don't choose one the
+highest version is picked automatically; to pin a specific version, add it to
+`settings.yaml`:
+
+```yaml
+default_versions:
+  my-module: "1.0"
+```
+
+`settings.yaml` can take a schema line of its own, as
+[above](#add-a-new-module-version), pointing at `deployment-settings.json`.
+
+To keep a version out of automatic selection — an alpha or release candidate, say — while
+still allowing an explicit `module load <name>/<version>`, set
+`exclude_from_defaults: true` on that Module.
+
+See [default version resolution](../explanations/default-versions.md) for how the
+automatic choice is made.
+
+## Get your change deployed
+
+How your change reaches the deployment area depends on how your site runs `deploy-tools`.
+The recommended setup is a CI pipeline in the configuration repository: you open a merge
+request, CI validates the change, and merging it deploys (see
+[drive deploy-tools from CI](ci-pipeline.md)). CI is not a requirement — an administrator
+can run the same `validate` and `sync` commands by hand instead. Either way, the
+`yaml-language-server` schema line catches structural mistakes in your editor as you type,
+before anyone else looks at the change.
+
+## Change or retire a version
+
+- **Update an existing version in place.** Rejected by default so published versions stay
+  stable; prefer publishing a new version. If you must, set `allow_updates: true` on the
+  Module — see [the guard rails](../explanations/deprecation-lifecycle.md#the-guard-rails).
+- **Retire a version.** Set `deprecated: true` in the Release file to steer users away
+  from it. Deleting it outright has to wait until after it is deprecated (unless the
+  Module has `allow_updates: true`). See
+  [the guard rails](../explanations/deprecation-lifecycle.md#the-guard-rails).

docs/reference.md

@@ -6,6 +6,7 @@ Technical reference material including APIs and release notes.
 :maxdepth: 1
 :glob:
 
+reference/configuration
 CLI <cli>
 API <_api/deploy_tools>
 Schemas <schemas>