Merge branch 'main' into cmorize_ersst

CITATION.cff

@@ -388,11 +388,11 @@ authors:
     orcid: "https://orcid.org/0000-0003-3780-0784"
 
 cff-version: 1.2.0
-date-released: 2023-07-06
+date-released: 2023-12-20
 doi: "10.5281/zenodo.3401363"
 license: "Apache-2.0"
 message: "If you use this software, please cite it using these metadata."
 repository-code: "https://github.com/ESMValGroup/ESMValTool/"
 title: ESMValTool
-version: "v2.9.0"
+version: "v2.10.0"
 ...

doc/sphinx/source/community/maintainer.rst

@@ -8,7 +8,7 @@ the interface of library functions may change when updating to new versions. Thi
 withdrawal of datasets used by a recipe can result in an existing recipe to stop working. Such "broken"
 recipes might require some work to fix such problems and make the recipe fully functional again.
 
-A first **contact point** for the technical lead development team (`@ESMValGroup/technical-lead-development-team`_) in such cases is the recipe "maintainer". The recipe
+A first **contact point** for the technical lead development team (:team:`technical-lead-development-team`) in such cases is the recipe "maintainer". The recipe
 maintainer is then asked to check the affected recipe and if possible, fix the problems or work with the technical
 lead development team to find a solution. Ideally, a recipe maintainer is able to tell whether the results of a fixed
 recipe are scientifically valid and look as expected. Being a recipe maintainer consists of the following tasks:
@@ -22,10 +22,7 @@ recipe are scientifically valid and look as expected. Being a recipe maintainer
 * informing the core development team when no longer available as maintainer
 
 Ideally, a recipe maintainer is named when contributing a new recipe to the ESMValTool. Recipe maintainers are asked to inform
-the core development team (`@ESMValGroup/esmvaltool-coreteam`_) when they are no longer able to act as maintainer or when they would like to step down from this duty
+the core development team (:team:`esmvaltool-coreteam`) when they are no longer able to act as maintainer or when they would like to step down from this duty
 for any reason. The core development team will then try to find a successor. If no recipe maintainer can be found, the
 :ref:`policy on unmaintained broken (non-working) recipes<broken-recipe-policy>` might apply eventually leading to
 retirement of the affected recipe.
-
-.. _`@ESMValGroup/technical-lead-development-team`: https://github.com/orgs/ESMValGroup/teams/technical-lead-development-team
-.. _`@ESMValGroup/esmvaltool-coreteam`: https://github.com/orgs/ESMValGroup/teams/esmvaltool-coreteam

doc/sphinx/source/community/release_strategy/detailed_release_procedure.rst

@@ -136,8 +136,8 @@ Recipe output can be copied by doing from the VM:
 
 .. code-block:: bash
 
-  nohup rsync -rlt /path_to_testing/esmvaltool_output/* /shared/esmvaltool/v2.x.x/
-  
+  nohup rsync --exclude preproc/ -rlt /path_to_testing/esmvaltool_output/* /shared/esmvaltool/v2.x.x/
+
 By copying the debug.html and index.html files into /shared/esmvaltool/v2.x.x/, the output
 becomes available online, see for `example
 <https://esmvaltool.dkrz.de/shared/esmvaltool/v2.7.0>`_.
@@ -151,13 +151,13 @@ Link the overview webpage to the release issue.
 This makes it much easier to ask for feedback from recipe developers and analyse failures.
 
 Results produced with the final ESMValCore release candidate should be put in a VM directory
-named after the version number, e.g. ``v2.x.x``. 
+named after the version number, e.g. ``v2.x.x``.
 Once the release process is over, test results produced with previous release candidates can be deleted to save space on the VM.
 
 .. note::
 
   If you wrote recipe runs output to Levante's `/scratch` partition, be aware that
-  the data will be removed after two weeks, so you will have to quickly move the 
+  the data will be removed after two weeks, so you will have to quickly move the
   output data to the VM, using the ``nohup`` command above.
 
 Running the comparison
@@ -189,15 +189,15 @@ The steps to running the compare tool on the VM are the following:
 - prerequisite - install `imagehash`: `pip install imagehash`
 - reference run (v2.7.0; previous stable release): `export reference_dir=/work/bd0854/b382109/v270` (contains `preproc/` dirs too, 122 recipes)
 - current run (v2.8.0): `export current_dir=path_to_current_run`
-- run the :ref:`comparison script<compare_recipe_runs>` with: 
+- run the :ref:`comparison script<compare_recipe_runs>` with:
 
 .. code-block:: bash
 
   nohup python ESMValTool/esmvaltool/utils/testing/regression/compare.py --reference $reference_dir --current $current_dir > compare_v280_output.txt
 
 Copy the comparison txt file to the release issue.
-Some of the recipes will appear as having identical output to the one from previous release. 
-However, others will need human inspection. 
+Some of the recipes will appear as having identical output to the one from previous release.
+However, others will need human inspection.
 Ask the recipe maintainers (`@ESMValGroup/esmvaltool-recipe-maintainers`_) and ESMValTool Development Team (`@ESMValGroup/esmvaltool-developmentteam`_) to provide assistance in checking the results.
 Here are some guidelines on how to perform the human inspection:
 

doc/sphinx/source/community/release_strategy/release_strategy.rst

@@ -54,34 +54,40 @@ With the following release schedule, we strive to have three releases per year a
 Upcoming releases
 ^^^^^^^^^^^^^^^^^
 
-- 2.10.0 (Release Manager: `Klaus Zimmermann`_)
+- 2.11.0 (Release Manager: TBD)
 
-+------------+--------------------------+
-| 2023-10-02 |ESMValCore feature freeze |
-+------------+--------------------------+
-| 2023-10-09 |ESMValCore release        |
-+------------+--------------------------+
-| 2023-10-16 |ESMValTool feature freeze |
-+------------+--------------------------+
-| 2023-10-23 |ESMValTool release        |
-+------------+--------------------------+
+Planned for February or March 2024
 
 Past releases
 ^^^^^^^^^^^^^
 
+- 2.10.0 (Release Manager: `Klaus Zimmermann`_)
+
++------------+------------+----------------------------------------+-------------------------------------+
+|  Planned   |    Done    |            Event                       |             Changelog               |
++============+============+========================================+=====================================+
+| 2023-10-02 |            | ESMValCore `Feature Freeze`_           |                                     |
++------------+------------+----------------------------------------+-------------------------------------+
+| 2023-10-09 | 2023-12-19 | :esmvalcore-release:`v2.10.0` released | :ref:`esmvalcore:changelog-v2-10-0` |
++------------+------------+----------------------------------------+-------------------------------------+
+| 2023-10-16 |            | ESMValTool `Feature Freeze`_           |                                     |
++------------+------------+----------------------------------------+-------------------------------------+
+| 2023-10-16 | 2023-12-20 | :release:`v2.10.0` released            | :ref:`changelog-v2-10-0`            |
++------------+------------+----------------------------------------+-------------------------------------+
+
 - 2.9.0 (Release Manager: `Bouwe Andela`_)
 
-+------------+------------+---------------------------------------------------------------------------------------------+------------------------------------+
-|  Planned   |    Done    |                                            Event                                            |             Changelog              |
-+============+============+=============================================================================================+====================================+
-| 2023-06-05 |            |                                  ESMValCore Feature Freeze                                  |                                    |
-+------------+------------+---------------------------------------------------------------------------------------------+------------------------------------+
-| 2023-06-12 | 2023-07-04 | `ESMValCore Release 2.9.0 <https://github.com/ESMValGroup/ESMValCore/releases/tag/v2.9.0>`_ | :ref:`esmvalcore:changelog-v2-9-0` |
-+------------+------------+---------------------------------------------------------------------------------------------+------------------------------------+
-| 2023-06-19 |            |                                  ESMValTool Feature Freeze                                  |                                    |
-+------------+------------+---------------------------------------------------------------------------------------------+------------------------------------+
-| 2023-06-26 | 2023-07-06 | `ESMValTool Release 2.9.0 <https://github.com/ESMValGroup/ESMValTool/releases/tag/v2.9.0>`_ |      :ref:`changelog-v2-9-0`       |
-+------------+------------+---------------------------------------------------------------------------------------------+------------------------------------+
++------------+------------+---------------------------------------+-------------------------------------+
+|  Planned   |    Done    |            Event                      |             Changelog               |
++============+============+=======================================+=====================================+
+| 2023-06-05 |            | ESMValCore `Feature Freeze`_          |                                     |
++------------+------------+---------------------------------------+-------------------------------------+
+| 2023-06-12 | 2023-07-04 | :esmvalcore-release:`v2.9.0` released | :ref:`esmvalcore:changelog-v2-9-0`  |
++------------+------------+---------------------------------------+-------------------------------------+
+| 2023-06-19 |            | ESMValTool `Feature Freeze`_          |                                     |
++------------+------------+---------------------------------------+-------------------------------------+
+| 2023-06-26 | 2023-07-06 | :release:`v2.9.0` released            | :ref:`changelog-v2-9-0`             |
++------------+------------+---------------------------------------+-------------------------------------+
 
 - 2.8.1 (Bugfix, Release Manager: `Valeriu Predoi`_)
 
@@ -287,6 +293,8 @@ These are the detailed steps to take to make a release.
    - If a bug is discovered that needs to be fixed before the release, a pull request can be made to the main branch to fix the bug. The person making the pull request can then ask the release manager to cherry-pick that commit into the release branch.
    - Update the :ref:`list of broken recipes <broken-recipe-list>` with new recipes that could not be run successfully during the testing.
      Open a separate GitHub issue for each failing recipe and assign the next milestone.
+     Open an overview issue, see :issue:`3484` for an example, and review past overview issues.
+     Take action to ensure that the broken recipe policy is followed.
 
 
 #. ESMValCore release
@@ -339,7 +347,7 @@ Glossary
 
 Feature freeze
 ~~~~~~~~~~~~~~
-The date on which no new features may be submitted for the upcoming release. 
+The date on which no new features may be submitted for the upcoming release.
 After this date, only critical bug fixes can still be included to the :ref:`release_branch`.
 Development work can continue in the main branch.
 If you are unsure whether new developments could interfere with the release, check with the :ref:`release_manager`.
@@ -411,7 +419,7 @@ All tests should pass before making a release (branch).
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
 The version number is automatically generated from the information provided by
-git using [setuptools-scm](https://pypi.org/project/setuptools-scm/), but a
+git using `setuptools-scm <https://pypi.org/project/setuptools-scm/>`__, but a
 static version number is stored in ``CITATION.cff``.
 Make sure to update the version number and release date in ``CITATION.cff``.
 See https://semver.org for more information on choosing a version number.
@@ -464,8 +472,8 @@ and create the new release from the release branch (i.e. not from ``main``).
 The release tag always starts with the letter ``v`` followed by the version
 number, e.g. ``v2.1.0``.
 
-6. Mark the release in the main branch
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+6. Merge the release branch back into the main branch
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
 When the (pre-)release is tagged, it is time to merge the release branch back into `main`.
 We do this for two reasons, namely, one, to mark the point up to which commits in `main`

doc/sphinx/source/conf.py

@@ -63,6 +63,7 @@
 extensions = [
     'sphinx.ext.autodoc',
     'sphinx.ext.doctest',
+    'sphinx.ext.extlinks',
     'sphinx.ext.intersphinx',
     'sphinx.ext.todo',
     'sphinx.ext.coverage',
@@ -441,6 +442,40 @@
     'sklearn': ('https://scikit-learn.org/stable', None),
 }
 
+# -- Extlinks extension -------------------------------------------------------
+# See https://www.sphinx-doc.org/en/master/usage/extensions/extlinks.html
+
+extlinks = {
+    "discussion": (
+        "https://github.com/ESMValGroup/ESMValTool/discussions/%s",
+        "Discussion #%s",
+    ),
+    "issue": (
+        "https://github.com/ESMValGroup/ESMValTool/issues/%s",
+        "Issue #%s",
+    ),
+    "pull": (
+        "https://github.com/ESMValGroup/ESMValTool/pull/%s",
+        "Pull request #%s",
+    ),
+    "release": (
+        "https://github.com/ESMValGroup/ESMValTool/releases/tag/%s",
+        "ESMValTool %s",
+    ),
+    "esmvalcore-release": (
+        "https://github.com/ESMValGroup/ESMValCore/releases/tag/%s",
+        "ESMValCore %s",
+    ),
+    "team": (
+        "https://github.com/orgs/ESMValGroup/teams/%s",
+        "@ESMValGroup/%s",
+    ),
+    "user": (
+        "https://github.com/%s",
+        "@%s",
+    ),
+}
+
 # -- Custom Document processing ----------------------------------------------
 
 import gensidebar

doc/sphinx/source/input.rst

@@ -259,9 +259,9 @@ A list of the datasets for which a CMORizers is available is provided in the fol
 | CERES-SYN1deg                | rlds, rldscs, rlus, rluscs, rlut, rlutcs, rsds, rsdscs, rsus, rsuscs, rsut, rsutcs (3hr)             |   3  | NCL             |
 |                              | rlds, rldscs, rlus, rlut, rlutcs, rsds, rsdt, rsus, rsut, rsutcs (Amon)                              |      |                 |
 +------------------------------+------------------------------------------------------------------------------------------------------+------+-----------------+
-| CLARA-AVHRR                  | clt, clivi, lwp (Amon)                                                                               |   3  | NCL             |
+| CLARA-AVHRR                  | clt, clivi, clwvi, lwp (Amon)                                                                        |   3  | NCL             |
 +------------------------------+------------------------------------------------------------------------------------------------------+------+-----------------+
-| CLOUDSAT-L2                  | clw, clivi, lwp (Amon)                                                                               |   3  | NCL             |
+| CLOUDSAT-L2                  | clw, clivi, clwvi, lwp (Amon)                                                                        |   3  | NCL             |
 +------------------------------+------------------------------------------------------------------------------------------------------+------+-----------------+
 | CowtanWay                    | tasa (Amon)                                                                                          |   2  | Python          |
 +------------------------------+------------------------------------------------------------------------------------------------------+------+-----------------+
@@ -288,7 +288,7 @@ A list of the datasets for which a CMORizers is available is provided in the fol
 +------------------------------+------------------------------------------------------------------------------------------------------+------+-----------------+
 | ESACCI-AEROSOL               | abs550aer, od550aer, od550aerStderr, od550lt1aer, od870aer, od870aerStderr (aero)                    |   2  | NCL             |
 +------------------------------+------------------------------------------------------------------------------------------------------+------+-----------------+
-| ESACCI-CLOUD                 | clivi, clt, cltStderr, lwp, rlut, rlutcs, rsut, rsutcs, rsdt, rlus, rsus, rsuscs (Amon)              |   2  | NCL             |
+| ESACCI-CLOUD                 | clivi, clt, cltStderr, clwvi, lwp, rlut, rlutcs, rsut, rsutcs, rsdt, rlus, rsus, rsuscs (Amon)       |   2  | NCL             |
 +------------------------------+------------------------------------------------------------------------------------------------------+------+-----------------+
 | ESACCI-FIRE                  | burntArea (Lmon)                                                                                     |   2  | NCL             |
 +------------------------------+------------------------------------------------------------------------------------------------------+------+-----------------+

doc/sphinx/source/recipes/broken_recipe_list.rst

@@ -20,7 +20,15 @@ More details can be found in the :ref:`broken recipe policy
      - `ERA5_native6`
      - Derivation of custom variables `rlus` and `rsus`
      - `#1388 <https://github.com/ESMValGroup/ESMValCore/issues/1388>`_
+   * - :ref:`recipe_julia.yml <recipe_examples>`
+     - `example`
+     - fill values are not interpreted, resulting in an unusable plot
+     - `#2595 <https://github.com/ESMValGroup/ESMValTool/issues/2595>`_
    * - :ref:`recipe_seaice_drift.yml <recipes_seaice_drift>`
      - `sea_ice_drift_SCICEX`
-     - ``shapely`` issue
+     - ``shapely 2`` issue
      - `#3243 <https://github.com/ESMValGroup/ESMValTool/issues/3243>`_
+   * - :ref:`recipe_pysplot.yml <recipes_psyplot_diag>`
+     - `plot_map`
+     - ``shapely 2`` issue
+     - `#3483 <https://github.com/ESMValGroup/ESMValTool/issues/3483>`_