Add how to add a recipe documentation

esmvaltool/utils/recipe_test_workflow/doc/source/add_a_recipe.rst

@@ -0,0 +1,122 @@
+How to add a recipe to the |RTW|
+================================
+
+.. include:: common.txt
+
+.. note::
+   Before you follow these steps to add your recipe, you must be able to
+   successfully run the recipe with the latest version of ESMValTool on the
+   compute server you use at your site, as detailed by the ``platform`` option
+   in the ``[[COMPUTE]]`` section in the site-specific ``.cylc`` file in the
+   ``esmvaltool/utils/recipe_test_workflow/recipe_test_workflow/site/``
+   directory.
+
+#. Open a `new ESMValTool issue`_ on GitHub, assign yourself to the issue, and
+   add the ``Recipe Test Workflow (RTW)`` label to the issue, see
+   `ESMValTool issue #3663`_ for an example.
+
+#. Create a branch.
+
+#. Obtain the duration and memory usage of the recipe from the messages printed
+   to screen, or at the end of the ``run/main_log.txt`` file in the recipe
+   output directory after running your recipe on the compute cluster you use at
+   your site; these messages will look something like::
+
+     YYYY-MM-DD HH:MM:SS:sss UTC [12345] INFO    Time for running the recipe was: 0:02:13.334742
+     YYYY-MM-DD HH:MM:SS:sss UTC [12345] INFO    Maximum memory used (estimate): 2.4 GB
+     [...]
+     YYYY-MM-DD HH:MM:SS:sss UTC [12345] INFO    Run was successful
+
+#. Add the recipe to the ``[task parameters]`` section in the
+   ``esmvaltool/utils/recipe_test_workflow/recipe_test_workflow/flow.cylc``
+   file.
+
+   .. hint::
+      If the recipe takes less than 10 minutes to run then it should be added
+      to the ``fast`` option. Recipes that take longer than ten minutes should
+      be added to the ``medium`` option.
+
+   .. hint::
+      The line added should follow the format of ``recipe_new_recipe, \``,
+      unless the line is the last one in the list, in which case the line added
+      should follow the format of ``recipe_new_recipe``.
+
+#. If the duration of the recipe is larger than the value specified by the
+   ``execution time limit`` option in the ``[[COMPUTE]]`` section in the
+   aforementioned site-specific ``.cylc`` file, and / or the memory usage of
+   the recipe is larger than the value specified by the ``--mem`` option in the
+   ``[[[directives]]]`` section in the ``[[COMPUTE]]`` section, add a section
+   (in alphabetical order) to this file as shown below (round the duration to
+   the nearest second)::
+
+     [[process<fast=recipe_albedolandcover>]]
+     # Actual: 0m31s, 2.5 GB on 2024-04-08.
+     execution time limit = PT2M
+     [[[directives]]]
+         --mem = 3G
+
+   .. hint::
+      The ``fast`` key in the example task definition above
+      (``[[process<fast=recipe_albedolandcover>]]``) should match name of the
+      option the recipe was added to in the ``[task parameters]`` section in
+      the
+      ``esmvaltool/utils/recipe_test_workflow/recipe_test_workflow/flow.cylc``
+      file
+
+   .. hint::
+      Set the ``execution time limit`` to 10-20% more than the actual duration.
+      For actual durations of up to ``1m45s``, set the ``execution time limit``
+      to ``PT2M`` (2 minutes).
+
+   .. hint::
+      Try not to regularly waste more than 500 MiB in memory usage. Typically,
+      rounding the actual memory usage up to the nearest integer is acceptable.
+
+#. Stop any running ``recipe_test_workflow`` workflows::
+
+    cylc stop recipe_test_workflow/*
+
+#. Run the |RTW|, as detailed in the
+   :doc:`Quick Start Guide <user_guide/quick_start>`; it is expected that the
+   ``compare`` task will fail.
+
+#. Update the Known Good Outputs (|KGOs|):
+
+   * Recursively copy the recipe output directory (i.e.
+     ``recipe_<recipe>_<date>_<time>``) from the
+     ``${HOME}/cylc-run/recipe_test_workflow/run1/share/cycle/<cycle>``
+     directory to your site-specific KGO directory, as detailed by the
+     ``KGO_ROOT_PATH`` option in the site-specific ``.conf`` file in the
+     ``esmvaltool/utils/recipe_test_workflow/recipe_test_workflow/opt/``
+     directory::
+
+       cp -r ${HOME}/cylc-run/recipe_test_workflow/run1/share/cycle/<cycle>/recipe_<recipe>_<date>_<time> <KGO_ROOT_PATH>
+
+   * Enable write permissions for all users on the recipe output directory in
+     your site-specific KGO directory::
+
+       chmod -R a+w <KGO_ROOT_PATH>/recipe_<recipe>_<date>_<time>
+
+#. Stop any running ``recipe_test_workflow`` workflows::
+
+     cylc stop recipe_test_workflow/*
+
+#. Run the |RTW| again, as detailed in the
+   :doc:`Quick Start Guide <user_guide/quick_start>`; the ``compare`` task
+   should now succeed.
+
+#. Add the recipe to the documentation; add a link to the recipe to the list of
+   "Currently tested recipes" in ``doc/source/tested_recipes.rst`` in
+   alphabetical order:
+
+   * Add a link to the recipe in ``doc/source/common.txt`` under the
+     ``.. Links`` section in alphabetical order (follow the format
+     ``.. _<name>: <URL>``).
+
+   * Add the link to the list of "Currently tested recipes" in
+     ``doc/source/tested_recipes.rst`` in alphabetical order (follow the
+     format ``* `<name>`_``).
+
+#. Commit and push your changes, create a PR, assign yourself to the PR, and
+   add the Recipe Test Workflow (RTW) label to the PR, see
+   `ESMValTool PR #3664`_ for an example.

esmvaltool/utils/recipe_test_workflow/doc/source/common.txt

@@ -8,6 +8,7 @@
 .. |ESMs| replace:: :term:`ESMs <ESM>`
 .. |ESMValCore| replace:: :term:`ESMValCore`
 .. |ESMValTool| replace:: :term:`ESMValTool`
+.. |KGOs| replace:: :term:`KGOs`
 .. |Rose| replace:: :term:`Rose`
 
 .. Links
@@ -19,9 +20,12 @@
 .. _Cylc Review: https://cylc.github.io/cylc-doc/7.9.3/html/tutorial.html#viewing-suite-logs-in-a-web-browser-cylc-review
 .. _Ensclus: https://docs.esmvaltool.org/en/latest/recipes/recipe_ensclus.html
 .. _ESMValTool Documentation: https://esmvaltool.org/
+.. _ESMValTool issue #3663: https://github.com/ESMValGroup/ESMValTool/issues/3663
+.. _ESMValTool PR #3664: https://github.com/ESMValGroup/ESMValTool/pull/3664
 .. _Examples python: https://docs.esmvaltool.org/en/latest/recipes/recipe_examples.html
 .. _Heatwaves coldwaves: https://docs.esmvaltool.org/en/latest/recipes/recipe_heatwaves_coldwaves.html
 .. _Landcover - Albedo: https://docs.esmvaltool.org/en/latest/recipes/recipe_albedolandcover.html#landcover-albedo
+.. _new ESMValTool issue: https://github.com/ESMValGroup/ESMValTool/issues/new
 .. _Ocean multimap: https://docs.esmvaltool.org/en/latest/recipes/recipe_oceans.html#recipe-ocean-multimap-yml
 .. _Radiation budget: https://docs.esmvaltool.org/en/latest/recipes/recipe_radiation_budget.html
 .. _Rose Documentation: https://metomi.github.io/rose/doc/html/index.html

esmvaltool/utils/recipe_test_workflow/doc/source/conf.py

@@ -16,7 +16,7 @@
 
 # -- Project information -----------------------------------------------------
 
-project = 'Recipe Test Workflow (RTW)'
+project = 'RTW'
 version = "0.1.0"
 
 # -- General configuration ---------------------------------------------------

esmvaltool/utils/recipe_test_workflow/doc/source/glossary.rst

@@ -24,6 +24,13 @@ Glossary
      routine evaluation of |ESMs| in |CMIP|, see the
      `ESMValTool Documentation`_.
 
+   KGOs
+    Known Good Outputs (KGOs) are outputs from an ESMValTool recipe that have
+    been confirmed (typically by a scientist) to be as expected. KGOs are used
+    in the compare task in the |RTW| to check whether any changes have been
+    introduced to the outputs from an ESMValTool recipe after changes are made
+    to either ESMValCore or ESMValTool.
+
    Rose
      Rose is a toolkit for writing, editing and running application
      configurations, see the `Rose Documentation`_.

esmvaltool/utils/recipe_test_workflow/doc/source/index.rst

@@ -13,4 +13,6 @@ Welcome to the Recipe Test Workflow's documentation!
    about
    tested_recipes
    user_guide/index.rst
+   add_a_recipe
+
    glossary

esmvaltool/utils/recipe_test_workflow/recipe_test_workflow/flow.cylc

@@ -3,8 +3,7 @@
     UTC mode = True
 
 [task parameters]
-    # `fast` recipes are those taking less than 10 mins.
-    # Other recipes should be included in `medium`.
+    # See "How to add a recipe to the RTW" documentation for more information.
     # Recipe paths are specified relative to esmvaltool/recipes. For recipes in
     # subdirectories, `--` stands for `/` since the latter is an illegal char.
     fast = recipe_radiation_budget, \