Merge branch 'main' into ipccwg1ar6ch3

.circleci/config.yml

@@ -27,6 +27,7 @@ commands:
           name: Log versions
           command: |
             . /opt/conda/etc/profile.d/conda.sh
+            conda env export --name base > /logs/base_environment.yml
             conda activate esmvaltool
             esmvaltool version
             dpkg -l > /logs/versions.txt
@@ -40,6 +41,7 @@ commands:
           command: |
             . /opt/conda/etc/profile.d/conda.sh
             conda activate esmvaltool
+            mamba --version
             pytest -n 4 --junitxml=test-reports/report.xml
             esmvaltool version
             esmvaltool -- --help
@@ -232,7 +234,7 @@ jobs:
     # Test building documentation
     docker:
       - image: condaforge/mambaforge:latest
-    resource_class: small
+    resource_class: medium
     steps:
       - checkout
       - run:
@@ -265,7 +267,7 @@ jobs:
             # Install prerequisites
             mkdir /logs
             # Install ESMValTool in a new conda environment
-            mamba create -y --name esmvaltool -c conda-forge esmvaltool julia 'python=3.10' >> /logs/conda.txt 2>&1
+            mamba create -y --name esmvaltool -c conda-forge esmvaltool julia 'python=3.11' >> /logs/conda.txt 2>&1
             # Activate the environment
             set +x; conda activate esmvaltool; set -x
             # install the Julia dependencies

CITATION.cff

@@ -388,11 +388,11 @@ authors:
     orcid: "https://orcid.org/0000-0003-3780-0784"
 
 cff-version: 1.2.0
-date-released: 2023-07-06
+date-released: 2023-12-20
 doi: "10.5281/zenodo.3401363"
 license: "Apache-2.0"
 message: "If you use this software, please cite it using these metadata."
 repository-code: "https://github.com/ESMValGroup/ESMValTool/"
 title: ESMValTool
-version: "v2.9.0"
+version: "v2.10.0"
 ...

README.md

@@ -8,7 +8,7 @@
 [![Codacy Badge](https://app.codacy.com/project/badge/Coverage/79bf6932c2e844eea15d0fb1ed7e415c)](https://www.codacy.com/gh/ESMValGroup/ESMValTool?utm_source=github.com&utm_medium=referral&utm_content=ESMValGroup/ESMValTool&utm_campaign=Badge_Coverage)
 [![Codacy Badge](https://app.codacy.com/project/badge/Grade/79bf6932c2e844eea15d0fb1ed7e415c)](https://www.codacy.com/gh/ESMValGroup/ESMValTool?utm_source=github.com&utm_medium=referral&utm_content=ESMValGroup/ESMValTool&utm_campaign=Badge_Grade)
 [![Docker Build Status](https://img.shields.io/docker/cloud/build/esmvalgroup/esmvaltool.svg)](https://hub.docker.com/r/esmvalgroup/esmvaltool/)
-[![Anaconda-Server Badge](https://img.shields.io/badge/Anaconda.org-2.9.0-blue.svg)](https://anaconda.org/conda-forge/esmvaltool)
+[![Anaconda-Server Badge](https://img.shields.io/conda/vn/conda-forge/ESMValTool?color=blue&label=conda-forge&logo=conda-forge&logoColor=white)](https://anaconda.org/conda-forge/esmvaltool)
 ![stand with Ukraine](https://badgen.net/badge/stand%20with/UKRAINE/?color=0057B8&labelColor=FFD700)
 
 ![esmvaltoollogo](https://raw.githubusercontent.com/ESMValGroup/ESMValTool/main/doc/sphinx/source/figures/ESMValTool-logo-2.png)
@@ -53,11 +53,11 @@ ESMValTool can run with the following types of [data as input](https://docs.esmv
 
 # Getting started
 
-Please see [getting started](https://docs.esmvaltool.org/en/latest/quickstart/index.html) on readthedocs as well as [ESMValTool tutorial](https://tutorial.esmvaltool.org/index.html). The tutorial is a set of lessons that together teach skills needed to work with ESMValTool in climate-related domains.
+Please see [getting started](https://docs.esmvaltool.org/en/latest/quickstart/index.html) on our instance of Read the Docs as well as [ESMValTool tutorial](https://tutorial.esmvaltool.org/index.html). The tutorial is a set of lessons that together teach skills needed to work with ESMValTool in climate-related domains.
 
 ## Getting help
 
-The easiest way to get help if you cannot find the answer in the documentation on [readthedocs](https://docs.esmvaltool.org), is to open an [issue on GitHub](https://github.com/ESMValGroup/ESMValTool/issues).
+The easiest way to get help, if you cannot find the answer in the documentation in our [docs](https://docs.esmvaltool.org), is to open an [issue on GitHub](https://github.com/ESMValGroup/ESMValTool/issues).
 
 ## Contributing
 

doc/sphinx/source/api/esmvaltool.diag_scripts.monitor.rst

@@ -11,6 +11,7 @@ Examples
 --------
 
 * :ref:`recipe_monitor`
+* :ref:`recipe_model_evaluation`
 
 
 Diagnostic scripts

doc/sphinx/source/community/code_documentation.rst

@@ -442,7 +442,10 @@ name to the list of authors in ``CITATION.cff`` and generate the entry for the
 ::
 
    pip install cffconvert
-   cffconvert --format zenodo --outfile .zenodo.json
+   cffconvert --infile CITATION.cff --format zenodo --outfile .zenodo.json
+
+Presently, this method unfortunately discards entries `communities`
+and `grants` from that file; please restore them manually.
 
 Note that authors of recipes and/or diagnostics also need to be added to the file
 `esmvaltool/config-references.yml <https://github.com/ESMValGroup/ESMValTool/blob/main/esmvaltool/config-references.yml>`__,

doc/sphinx/source/community/maintainer.rst

@@ -8,7 +8,7 @@ the interface of library functions may change when updating to new versions. Thi
 withdrawal of datasets used by a recipe can result in an existing recipe to stop working. Such "broken"
 recipes might require some work to fix such problems and make the recipe fully functional again.
 
-A first **contact point** for the technical lead development team (`@ESMValGroup/technical-lead-development-team`_) in such cases is the recipe "maintainer". The recipe
+A first **contact point** for the technical lead development team (:team:`technical-lead-development-team`) in such cases is the recipe "maintainer". The recipe
 maintainer is then asked to check the affected recipe and if possible, fix the problems or work with the technical
 lead development team to find a solution. Ideally, a recipe maintainer is able to tell whether the results of a fixed
 recipe are scientifically valid and look as expected. Being a recipe maintainer consists of the following tasks:
@@ -22,10 +22,7 @@ recipe are scientifically valid and look as expected. Being a recipe maintainer
 * informing the core development team when no longer available as maintainer
 
 Ideally, a recipe maintainer is named when contributing a new recipe to the ESMValTool. Recipe maintainers are asked to inform
-the core development team (`@ESMValGroup/esmvaltool-coreteam`_) when they are no longer able to act as maintainer or when they would like to step down from this duty
+the core development team (:team:`esmvaltool-coreteam`) when they are no longer able to act as maintainer or when they would like to step down from this duty
 for any reason. The core development team will then try to find a successor. If no recipe maintainer can be found, the
 :ref:`policy on unmaintained broken (non-working) recipes<broken-recipe-policy>` might apply eventually leading to
 retirement of the affected recipe.
-
-.. _`@ESMValGroup/technical-lead-development-team`: https://github.com/orgs/ESMValGroup/teams/technical-lead-development-team
-.. _`@ESMValGroup/esmvaltool-coreteam`: https://github.com/orgs/ESMValGroup/teams/esmvaltool-coreteam

doc/sphinx/source/community/release_strategy/detailed_release_procedure.rst

@@ -136,8 +136,8 @@ Recipe output can be copied by doing from the VM:
 
 .. code-block:: bash
 
-  nohup rsync -rlt /path_to_testing/esmvaltool_output/* /shared/esmvaltool/v2.x.x/
-  
+  nohup rsync --exclude preproc/ -rlt /path_to_testing/esmvaltool_output/* /shared/esmvaltool/v2.x.x/
+
 By copying the debug.html and index.html files into /shared/esmvaltool/v2.x.x/, the output
 becomes available online, see for `example
 <https://esmvaltool.dkrz.de/shared/esmvaltool/v2.7.0>`_.
@@ -151,13 +151,13 @@ Link the overview webpage to the release issue.
 This makes it much easier to ask for feedback from recipe developers and analyse failures.
 
 Results produced with the final ESMValCore release candidate should be put in a VM directory
-named after the version number, e.g. ``v2.x.x``. 
+named after the version number, e.g. ``v2.x.x``.
 Once the release process is over, test results produced with previous release candidates can be deleted to save space on the VM.
 
 .. note::
 
   If you wrote recipe runs output to Levante's `/scratch` partition, be aware that
-  the data will be removed after two weeks, so you will have to quickly move the 
+  the data will be removed after two weeks, so you will have to quickly move the
   output data to the VM, using the ``nohup`` command above.
 
 Running the comparison
@@ -189,15 +189,15 @@ The steps to running the compare tool on the VM are the following:
 - prerequisite - install `imagehash`: `pip install imagehash`
 - reference run (v2.7.0; previous stable release): `export reference_dir=/work/bd0854/b382109/v270` (contains `preproc/` dirs too, 122 recipes)
 - current run (v2.8.0): `export current_dir=path_to_current_run`
-- run the :ref:`comparison script<compare_recipe_runs>` with: 
+- run the :ref:`comparison script<compare_recipe_runs>` with:
 
 .. code-block:: bash
 
   nohup python ESMValTool/esmvaltool/utils/testing/regression/compare.py --reference $reference_dir --current $current_dir > compare_v280_output.txt
 
 Copy the comparison txt file to the release issue.
-Some of the recipes will appear as having identical output to the one from previous release. 
-However, others will need human inspection. 
+Some of the recipes will appear as having identical output to the one from previous release.
+However, others will need human inspection.
 Ask the recipe maintainers (`@ESMValGroup/esmvaltool-recipe-maintainers`_) and ESMValTool Development Team (`@ESMValGroup/esmvaltool-developmentteam`_) to provide assistance in checking the results.
 Here are some guidelines on how to perform the human inspection:
 

doc/sphinx/source/community/release_strategy/release_strategy.rst

@@ -54,34 +54,40 @@ With the following release schedule, we strive to have three releases per year a
 Upcoming releases
 ^^^^^^^^^^^^^^^^^
 
-- 2.10.0 (Release Manager: `Klaus Zimmermann`_)
+- 2.11.0 (Release Manager: TBD)
 
-+------------+--------------------------+
-| 2023-10-02 |ESMValCore feature freeze |
-+------------+--------------------------+
-| 2023-10-09 |ESMValCore release        |
-+------------+--------------------------+
-| 2023-10-16 |ESMValTool feature freeze |
-+------------+--------------------------+
-| 2023-10-23 |ESMValTool release        |
-+------------+--------------------------+
+Planned for February or March 2024
 
 Past releases
 ^^^^^^^^^^^^^
 
+- 2.10.0 (Release Manager: `Klaus Zimmermann`_)
+
++------------+------------+----------------------------------------+-------------------------------------+
+|  Planned   |    Done    |            Event                       |             Changelog               |
++============+============+========================================+=====================================+
+| 2023-10-02 |            | ESMValCore `Feature Freeze`_           |                                     |
++------------+------------+----------------------------------------+-------------------------------------+
+| 2023-10-09 | 2023-12-19 | :esmvalcore-release:`v2.10.0` released | :ref:`esmvalcore:changelog-v2-10-0` |
++------------+------------+----------------------------------------+-------------------------------------+
+| 2023-10-16 |            | ESMValTool `Feature Freeze`_           |                                     |
++------------+------------+----------------------------------------+-------------------------------------+
+| 2023-10-16 | 2023-12-20 | :release:`v2.10.0` released            | :ref:`changelog-v2-10-0`            |
++------------+------------+----------------------------------------+-------------------------------------+
+
 - 2.9.0 (Release Manager: `Bouwe Andela`_)
 
-+------------+------------+---------------------------------------------------------------------------------------------+------------------------------------+
-|  Planned   |    Done    |                                            Event                                            |             Changelog              |
-+============+============+=============================================================================================+====================================+
-| 2023-06-05 |            |                                  ESMValCore Feature Freeze                                  |                                    |
-+------------+------------+---------------------------------------------------------------------------------------------+------------------------------------+
-| 2023-06-12 | 2023-07-04 | `ESMValCore Release 2.9.0 <https://github.com/ESMValGroup/ESMValCore/releases/tag/v2.9.0>`_ | :ref:`esmvalcore:changelog-v2-9-0` |
-+------------+------------+---------------------------------------------------------------------------------------------+------------------------------------+
-| 2023-06-19 |            |                                  ESMValTool Feature Freeze                                  |                                    |
-+------------+------------+---------------------------------------------------------------------------------------------+------------------------------------+
-| 2023-06-26 | 2023-07-06 | `ESMValTool Release 2.9.0 <https://github.com/ESMValGroup/ESMValTool/releases/tag/v2.9.0>`_ |      :ref:`changelog-v2-9-0`       |
-+------------+------------+---------------------------------------------------------------------------------------------+------------------------------------+
++------------+------------+---------------------------------------+-------------------------------------+
+|  Planned   |    Done    |            Event                      |             Changelog               |
++============+============+=======================================+=====================================+
+| 2023-06-05 |            | ESMValCore `Feature Freeze`_          |                                     |
++------------+------------+---------------------------------------+-------------------------------------+
+| 2023-06-12 | 2023-07-04 | :esmvalcore-release:`v2.9.0` released | :ref:`esmvalcore:changelog-v2-9-0`  |
++------------+------------+---------------------------------------+-------------------------------------+
+| 2023-06-19 |            | ESMValTool `Feature Freeze`_          |                                     |
++------------+------------+---------------------------------------+-------------------------------------+
+| 2023-06-26 | 2023-07-06 | :release:`v2.9.0` released            | :ref:`changelog-v2-9-0`             |
++------------+------------+---------------------------------------+-------------------------------------+
 
 - 2.8.1 (Bugfix, Release Manager: `Valeriu Predoi`_)
 
@@ -287,6 +293,8 @@ These are the detailed steps to take to make a release.
    - If a bug is discovered that needs to be fixed before the release, a pull request can be made to the main branch to fix the bug. The person making the pull request can then ask the release manager to cherry-pick that commit into the release branch.
    - Update the :ref:`list of broken recipes <broken-recipe-list>` with new recipes that could not be run successfully during the testing.
      Open a separate GitHub issue for each failing recipe and assign the next milestone.
+     Open an overview issue, see :issue:`3484` for an example, and review past overview issues.
+     Take action to ensure that the broken recipe policy is followed.
 
 
 #. ESMValCore release
@@ -339,7 +347,7 @@ Glossary
 
 Feature freeze
 ~~~~~~~~~~~~~~
-The date on which no new features may be submitted for the upcoming release. 
+The date on which no new features may be submitted for the upcoming release.
 After this date, only critical bug fixes can still be included to the :ref:`release_branch`.
 Development work can continue in the main branch.
 If you are unsure whether new developments could interfere with the release, check with the :ref:`release_manager`.
@@ -411,7 +419,7 @@ All tests should pass before making a release (branch).
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
 The version number is automatically generated from the information provided by
-git using [setuptools-scm](https://pypi.org/project/setuptools-scm/), but a
+git using `setuptools-scm <https://pypi.org/project/setuptools-scm/>`__, but a
 static version number is stored in ``CITATION.cff``.
 Make sure to update the version number and release date in ``CITATION.cff``.
 See https://semver.org for more information on choosing a version number.
@@ -464,8 +472,8 @@ and create the new release from the release branch (i.e. not from ``main``).
 The release tag always starts with the letter ``v`` followed by the version
 number, e.g. ``v2.1.0``.
 
-6. Mark the release in the main branch
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+6. Merge the release branch back into the main branch
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
 When the (pre-)release is tagged, it is time to merge the release branch back into `main`.
 We do this for two reasons, namely, one, to mark the point up to which commits in `main`

doc/sphinx/source/conf.py

@@ -63,6 +63,7 @@
 extensions = [
     'sphinx.ext.autodoc',
     'sphinx.ext.doctest',
+    'sphinx.ext.extlinks',
     'sphinx.ext.intersphinx',
     'sphinx.ext.todo',
     'sphinx.ext.coverage',
@@ -441,6 +442,40 @@
     'sklearn': ('https://scikit-learn.org/stable', None),
 }
 
+# -- Extlinks extension -------------------------------------------------------
+# See https://www.sphinx-doc.org/en/master/usage/extensions/extlinks.html
+
+extlinks = {
+    "discussion": (
+        "https://github.com/ESMValGroup/ESMValTool/discussions/%s",
+        "Discussion #%s",
+    ),
+    "issue": (
+        "https://github.com/ESMValGroup/ESMValTool/issues/%s",
+        "Issue #%s",
+    ),
+    "pull": (
+        "https://github.com/ESMValGroup/ESMValTool/pull/%s",
+        "Pull request #%s",
+    ),
+    "release": (
+        "https://github.com/ESMValGroup/ESMValTool/releases/tag/%s",
+        "ESMValTool %s",
+    ),
+    "esmvalcore-release": (
+        "https://github.com/ESMValGroup/ESMValCore/releases/tag/%s",
+        "ESMValCore %s",
+    ),
+    "team": (
+        "https://github.com/orgs/ESMValGroup/teams/%s",
+        "@ESMValGroup/%s",
+    ),
+    "user": (
+        "https://github.com/%s",
+        "@%s",
+    ),
+}
+
 # -- Custom Document processing ----------------------------------------------
 
 import gensidebar

doc/sphinx/source/faq.rst

@@ -113,9 +113,15 @@ a symbolic link to it so it gets picked up at every re-run iteration:
 Can ESMValTool plot arbitrary model output?
 ===========================================
 
-Recipe :ref:`recipe_monitor` allows for the plotting of any preprocessed model.
-The plotting parameters are set through a yaml configuration file, and the
-type of plots to be generated are determined in the recipe.
+:ref:`recipe_model_evaluation` provides a set of recipes that can be used for a
+basic climate model evaluation with observational data.
+This is especially useful to get an overview of the general performance of a
+simulation.
+
+Furthermore, recipe :ref:`recipe_monitor` allows for the plotting of any
+preprocessed model.
+The plotting parameters are set through a yaml configuration file, and the type
+of plots to be generated are determined in the recipe.
 
 Moreover, recipe :ref:`recipes_psyplot_diag` and the corresponding diagnostic
 :ref:`psyplot_diag.py <api.esmvaltool.diag_scripts.psyplot_diag>` provide a