Merge branch 'main' of github.com:esmvalgroup/esmvaltool into update-…

…release-tools
ESMValGroup · Dec 20, 2023 · 89bd9a8 · 89bd9a8
2 parents 06fe226 + 7e60941
commit 89bd9a8
Show file tree

Hide file tree

Showing 50 changed files with 413 additions and 348 deletions.
diff --git a/.circleci/config.yml b/.circleci/config.yml
@@ -234,7 +234,7 @@ jobs:
     # Test building documentation
     docker:
       - image: condaforge/mambaforge:latest
-    resource_class: small
+    resource_class: medium
     steps:
       - checkout
       - run:

diff --git a/CITATION.cff b/CITATION.cff
@@ -388,11 +388,11 @@ authors:
     orcid: "https://orcid.org/0000-0003-3780-0784"
 
 cff-version: 1.2.0
-date-released: 2023-07-06
+date-released: 2023-12-19
 doi: "10.5281/zenodo.3401363"
 license: "Apache-2.0"
 message: "If you use this software, please cite it using these metadata."
 repository-code: "https://github.com/ESMValGroup/ESMValTool/"
 title: ESMValTool
-version: "v2.9.0"
+version: "v2.10.0"
 ...
diff --git a/doc/sphinx/source/changelog.rst b/doc/sphinx/source/changelog.rst
@@ -3,6 +3,116 @@
 Changelog
 =========
 
+v2.10.0
+-------
+Highlights
+
+-  Add a realistic IPCC example recipe that reproduces figure 9.3 from AR6. It
+   computes the mean sea-surface temperature anomaly between 1850-2100 over all
+   available CMIP6 models. See the :ref:`recipe documentation <recipe_examples>`
+   or read the `blog post <https://blog.esciencecenter.nl/easy-ipcc-powered-by-esmvalcore-19a0b6366ea7>`__
+   for more information.
+
+-  Added more plot types to monitoring diagnostic: Hovmoeller Z vs. time,
+   Hovmoeller time vs latlon, variable vs. latitude are now available. See the
+   :ref:`recipe documentation <recipe_monitor>` for more information.
+
+-  Add support for 4 new datasets:
+
+   - NOAA-CIRES-20CR v3 reanalysis
+   - NASA MERRA reanalysis
+   - NOAA marine boundary layer data for CH4
+   - MOBO-DIC2004-2019
+
+   See :ref:`supported_datasets` and :ref:`inputdata_observations` for more
+   information.
+
+-  Many recipes now have up-to-date obs4MIPs dataset names so required data can
+   automatically be downloaded from ESGF.
+
+This release includes
+
+Bug fixes
+~~~~~~~~~
+
+-  Update recipe shapeselect to work with shapely v2 (`#3283 <https://github.com/ESMValGroup/ESMValTool/pull/3283>`__) `Lukas <https://github.com/lukruh>`__
+-  Correctly handle ``~`` when reading ``plot_folder`` option of monitoring diagnostic (`#3449 <https://github.com/ESMValGroup/ESMValTool/pull/3449>`__) `Manuel Schlund <https://github.com/schlunma>`__
+-  Fixed provenance tracking for NCL multipanel PNGs (`#3332 <https://github.com/ESMValGroup/ESMValTool/pull/3332>`__) `Manuel Schlund <https://github.com/schlunma>`__
+-  Fixed plot paths in NCL provenance tracking (`#3422 <https://github.com/ESMValGroup/ESMValTool/pull/3422>`__) `Manuel Schlund <https://github.com/schlunma>`__
+-  Fix erroneous file_type handling in certain NCL diagnostics (`#3474 <https://github.com/ESMValGroup/ESMValTool/pull/3474>`__) `Klaus Zimmermann <https://github.com/zklaus>`__
+-  Fix NCL provenance tracking (`#3477 <https://github.com/ESMValGroup/ESMValTool/pull/3477>`__) `Manuel Schlund <https://github.com/schlunma>`__
+-  Fix plots and provenance in Russell diagnostics (`#3479 <https://github.com/ESMValGroup/ESMValTool/pull/3479>`__) `Manuel Schlund <https://github.com/schlunma>`__
+
+Documentation
+~~~~~~~~~~~~~
+
+-  Add merge instructions to release instructions (`#3292 <https://github.com/ESMValGroup/ESMValTool/pull/3292>`__) `Rémi Kazeroni <https://github.com/remi-kazeroni>`__
+-  Update release schedule after release of v2.9.0 (`#3289 <https://github.com/ESMValGroup/ESMValTool/pull/3289>`__) `Rémi Kazeroni <https://github.com/remi-kazeroni>`__
+-  Add list of failing recipes for v2.9.0 release (`#3294 <https://github.com/ESMValGroup/ESMValTool/pull/3294>`__) `Rémi Kazeroni <https://github.com/remi-kazeroni>`__
+-  Update ``mamba`` version in readthedocs configuration docs builds (`#3310 <https://github.com/ESMValGroup/ESMValTool/pull/3310>`__) `Valeriu Predoi <https://github.com/valeriupredoi>`__
+-  Add Romain Beucher to citation file as contributor (`#3318 <https://github.com/ESMValGroup/ESMValTool/pull/3318>`__) `Valeriu Predoi <https://github.com/valeriupredoi>`__
+-  Removed recipe_carvalhais14nat from list of broken recipes (`#3319 <https://github.com/ESMValGroup/ESMValTool/pull/3319>`__) `Rémi Kazeroni <https://github.com/remi-kazeroni>`__
+-  Add `OBS-maintainers <https://github.com/orgs/ESMValGroup/teams/obs-maintainers>`__ team to documentation on OBS data maintenance and CMORizer reviews (`#3335 <https://github.com/ESMValGroup/ESMValTool/pull/3335>`__) `Rémi Kazeroni <https://github.com/remi-kazeroni>`__
+-  Add Pauline Bonnet to citation file (`#3347 <https://github.com/ESMValGroup/ESMValTool/pull/3347>`__) `Paulinebonnet111 <https://github.com/Paulinebonnet111>`__
+-  Ensure compatible zstandard and zstd in readthedocs builds (`#3362 <https://github.com/ESMValGroup/ESMValTool/pull/3362>`__) `Klaus Zimmermann <https://github.com/zklaus>`__
+-  Fix documentation build (`#3397 <https://github.com/ESMValGroup/ESMValTool/pull/3397>`__) `Bouwe Andela <https://github.com/bouweandela>`__
+-  Minor updates to release tools (`#3216 <https://github.com/ESMValGroup/ESMValTool/pull/3216>`__) `Bouwe Andela <https://github.com/bouweandela>`__
+-  Enhance provenance documentation (`#3305 <https://github.com/ESMValGroup/ESMValTool/pull/3305>`__) `Alistair Sellar <https://github.com/alistairsellar>`__
+-  Re-add communities and grants in zenodo file (`#3416 <https://github.com/ESMValGroup/ESMValTool/pull/3416>`__) `Valeriu Predoi <https://github.com/valeriupredoi>`__
+-  Update Anconda badge in README (`#3375 <https://github.com/ESMValGroup/ESMValTool/pull/3375>`__, `#3453 <https://github.com/ESMValGroup/ESMValTool/pull/3453>`__) `Valeriu Predoi <https://github.com/valeriupredoi>`__
+
+Diagnostics
+~~~~~~~~~~~
+
+-  Slight refactoring of diagnostic script ``galytska23/select_variables_for_tigramite.py`` for generality and portability (`#3298 <https://github.com/ESMValGroup/ESMValTool/pull/3298>`__) `Valeriu Predoi <https://github.com/valeriupredoi>`__ and `Evgenia Galytska <https://github.com/egalytska>`__
+-  Allow custom variable grouping in diagnostic script ``monitor/multi_datasets.py`` (`#3343 <https://github.com/ESMValGroup/ESMValTool/pull/3343>`__) `Manuel Schlund <https://github.com/schlunma>`__
+-  Extended monitor diagnostic with plot type variable vs. latitude (`#3340 <https://github.com/ESMValGroup/ESMValTool/pull/3340>`__) `ellensarauer <https://github.com/ellensarauer>`__
+-  Add Hovmoeller Z vs. time plot to monitoring diagnostic (`#3345 <https://github.com/ESMValGroup/ESMValTool/pull/3345>`__) `cubeme <https://github.com/cubeme>`__ and `helgehr <https://github.com/helgehr>`__
+-  Adding Hovmoeller time vs latlon plots to monitoring recipes (`#3341 <https://github.com/ESMValGroup/ESMValTool/pull/3341>`__) `Lukas <https://github.com/lukruh>`__ and `jeremykraftdlr <https://github.com/jeremykraftdlr>`__
+-  Implied heat transport new diagnostic (`#3177 <https://github.com/ESMValGroup/ESMValTool/pull/3177>`__) `Alejandro Bodas <https://github.com/mo-abodas>`__
+-  Recipe changes for new statistics preprocessors (percentiles) (`#3351 <https://github.com/ESMValGroup/ESMValTool/pull/3351>`__) `Manuel Schlund <https://github.com/schlunma>`__
+-  Add a realistic example recipe (`#3356 <https://github.com/ESMValGroup/ESMValTool/pull/3356>`__) `Peter9192 <https://github.com/Peter9191>`__ and `Bouwe Andela <https://github.com/bouweandela>`__
+-  Support ``CenteredNorm`` in diagnostic monitor/multidatasets.py (`#3415 <https://github.com/ESMValGroup/ESMValTool/pull/3415>`__) `Manuel Schlund <https://github.com/schlunma>`__
+-  Use new preprocessor statistics calling convention for recipe_easy_ipcc.yml (`#3418 <https://github.com/ESMValGroup/ESMValTool/pull/3418>`__) `Bouwe Andela <https://github.com/bouweandela>`__
+-  Adapt to changed style scheme name in matplotlib (`#3475 <https://github.com/ESMValGroup/ESMValTool/pull/3475>`__) `Klaus Zimmermann <https://github.com/zklaus>`__
+-  Add version to dataset in python example recipe to avoid "Unknown file format" issue on JASMIN (`#3322 <https://github.com/ESMValGroup/ESMValTool/pull/3322>`__) `Emma Hogan <https://github.com/ehogan>`__
+-  Add the dataset version in the heatwaves_coldwaves recipe to avoid the "Unknown file format" issue on JASMIN (`#3373 <https://github.com/ESMValGroup/ESMValTool/pull/3373>`__) `Emma Hogan <https://github.com/ehogan>`__
+
+Observational and re-analysis dataset support
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+-  Cmorizer for NOAA-CIRES-20CR v3 reanalysis (clt, clwvi, hus, prw, rlut, rlutcs, rsut, rsutcs) (`#3137 <https://github.com/ESMValGroup/ESMValTool/pull/3137>`__) `Lisa Bock <https://github.com/LisaBock>`__
+-  CMORizer for NASA MERRA reanalysis (`#3039 <https://github.com/ESMValGroup/ESMValTool/pull/3039>`__) `Axel Lauer <https://github.com/axel-lauer>`__
+-  Download and formatting of NOAA marine boundary layer data for CH4 (NOAA-MBL-CH4) (`#3301 <https://github.com/ESMValGroup/ESMValTool/pull/3301>`__) `FranziskaWinterstein <https://github.com/FranziskaWinterstein>`__
+-  Added CMORizer for MOBO-DIC2004-2019 (`#3297 <https://github.com/ESMValGroup/ESMValTool/pull/3297>`__) `Manuel Schlund <https://github.com/schlunma>`__
+-  Update obs4MIPs dataset names in quantilebias recipe (`#3330 <https://github.com/ESMValGroup/ESMValTool/pull/3330>`__) `Romain Beucher <https://github.com/rbeucher>`__
+-  Update obs4MIPs dataset names in Schlund20esd recipe (`#3329 <https://github.com/ESMValGroup/ESMValTool/pull/3329>`__) `Romain Beucher <https://github.com/rbeucher>`__
+-  Update obs4MIPs dataset names in flatoipcc recipes (`#3328 <https://github.com/ESMValGroup/ESMValTool/pull/3328>`__) `Romain Beucher <https://github.com/rbeucher>`__
+-  Update obs4mips dataset names in clouds recipes (`#3326 <https://github.com/ESMValGroup/ESMValTool/pull/3326>`__) `Romain Beucher <https://github.com/rbeucher>`__
+-  Update Obs4MIPs dataset names in ECS recipes (`#3327 <https://github.com/ESMValGroup/ESMValTool/pull/3327>`__) `Romain Beucher <https://github.com/rbeucher>`__
+-  Update obs4mips dataset names in Bock et al recipes (`#3324 <https://github.com/ESMValGroup/ESMValTool/pull/3324>`__, `#3389 <https://github.com/ESMValGroup/ESMValTool/pull/3389>`__ and `#3473 <https://github.com/ESMValGroup/ESMValTool/pull/3473>`__) `Romain Beucher <https://github.com/rbeucher>`__ and `Bouwe Andela <https://github.com/bouweandela>`__
+-  Update obs4mips dataset names in radiation budget recipe (`#3323 <https://github.com/ESMValGroup/ESMValTool/pull/3323>`__) `Romain Beucher <https://github.com/rbeucher>`__
+-  Update Obs4MIPs dataset names in perfmetrics CMIP5 recipe (`#3325 <https://github.com/ESMValGroup/ESMValTool/pull/3325>`__) `Romain Beucher <https://github.com/rbeucher>`__
+
+Automatic testing
+~~~~~~~~~~~~~~~~~
+
+-  Made sklearn test backwards-compatible with sklearn < 1.3 (`#3285 <https://github.com/ESMValGroup/ESMValTool/pull/3285>`__) `Manuel Schlund <https://github.com/schlunma>`__
+-  Update conda lock creation Github Action workflow and ship updated conda-lock file (`#3307 <https://github.com/ESMValGroup/ESMValTool/pull/3307>`__, `#3407 <https://github.com/ESMValGroup/ESMValTool/pull/3407>`__) `Valeriu Predoi <https://github.com/valeriupredoi>`__
+-  Compress all bash shell setters into one default option per GitHub Action workflow (`#3315 <https://github.com/ESMValGroup/ESMValTool/pull/3315>`__) `Valeriu Predoi <https://github.com/valeriupredoi>`__
+-  Remove deprecated option ``offline`` from CI configuration (`#3367 <https://github.com/ESMValGroup/ESMValTool/pull/3367>`__) `Manuel Schlund <https://github.com/schlunma>`__
+
+Installation
+~~~~~~~~~~~~
+
+-  Use ESMValCore v2.10 (`#3486 <https://github.com/ESMValGroup/ESMValTool/pull/3486>`__) `Bouwe Andela <https://github.com/bouweandela>`__
+
+Improvements
+~~~~~~~~~~~~
+
+-  Merge v2.9.x into main  (`#3286 <https://github.com/ESMValGroup/ESMValTool/pull/3286>`__) `Manuel Schlund <https://github.com/schlunma>`__
+-  Allow NCL unit conversion `kg s-1` -> `GtC y-1` (`#3300 <https://github.com/ESMValGroup/ESMValTool/pull/3300>`__) `Manuel Schlund <https://github.com/schlunma>`__
+
 .. _changelog-v2-9-0:
 
 v2.9.0

diff --git a/doc/sphinx/source/input.rst b/doc/sphinx/source/input.rst
@@ -112,6 +112,13 @@ ESMValTool currently supports two ways to perform this reformatting (aka
    checks and fixes').  Details on this second method are given at the
    :ref:`end of this chapter <inputdata_native_datasets>`.
 
+A collection of readily CMORized OBS and OBS6 datasets can be accessed directly on CEDA/JASMIN and DKRZ. At CEDA/JASMIN
+OBS and OBS6 data is stored in the `esmeval` Group Workspace (GWS), and to be granted read (and execute) permissions to the
+GWS, one must apply at https://accounts.jasmin.ac.uk/services/group_workspaces/esmeval/ ; after permission has been granted, the user
+is encouraged to use the data locally, and not move it elsewhere, to minimize both data transfers and
+stale disk usage; to note that Tier 3 data is subject to data protection restrictions; for further inquiries,
+the GWS is adminstered by [Valeriu Predoi](mailto:[email protected]).
+
 Using a CMORizer script
 -----------------------
 
@@ -252,9 +259,9 @@ A list of the datasets for which a CMORizers is available is provided in the fol
 | CERES-SYN1deg                | rlds, rldscs, rlus, rluscs, rlut, rlutcs, rsds, rsdscs, rsus, rsuscs, rsut, rsutcs (3hr)             |   3  | NCL             |
 |                              | rlds, rldscs, rlus, rlut, rlutcs, rsds, rsdt, rsus, rsut, rsutcs (Amon)                              |      |                 |
 +------------------------------+------------------------------------------------------------------------------------------------------+------+-----------------+
-| CLARA-AVHRR                  | clt, clivi, lwp (Amon)                                                                               |   3  | NCL             |
+| CLARA-AVHRR                  | clt, clivi, clwvi, lwp (Amon)                                                                        |   3  | NCL             |
 +------------------------------+------------------------------------------------------------------------------------------------------+------+-----------------+
-| CLOUDSAT-L2                  | clw, clivi, lwp (Amon)                                                                               |   3  | NCL             |
+| CLOUDSAT-L2                  | clw, clivi, clwvi, lwp (Amon)                                                                        |   3  | NCL             |
 +------------------------------+------------------------------------------------------------------------------------------------------+------+-----------------+
 | CowtanWay                    | tasa (Amon)                                                                                          |   2  | Python          |
 +------------------------------+------------------------------------------------------------------------------------------------------+------+-----------------+
@@ -281,7 +288,7 @@ A list of the datasets for which a CMORizers is available is provided in the fol
 +------------------------------+------------------------------------------------------------------------------------------------------+------+-----------------+
 | ESACCI-AEROSOL               | abs550aer, od550aer, od550aerStderr, od550lt1aer, od870aer, od870aerStderr (aero)                    |   2  | NCL             |
 +------------------------------+------------------------------------------------------------------------------------------------------+------+-----------------+
-| ESACCI-CLOUD                 | clivi, clt, cltStderr, lwp, rlut, rlutcs, rsut, rsutcs, rsdt, rlus, rsus, rsuscs (Amon)              |   2  | NCL             |
+| ESACCI-CLOUD                 | clivi, clt, cltStderr, clwvi, lwp, rlut, rlutcs, rsut, rsutcs, rsdt, rlus, rsus, rsuscs (Amon)       |   2  | NCL             |
 +------------------------------+------------------------------------------------------------------------------------------------------+------+-----------------+
 | ESACCI-FIRE                  | burntArea (Lmon)                                                                                     |   2  | NCL             |
 +------------------------------+------------------------------------------------------------------------------------------------------+------+-----------------+

diff --git a/doc/sphinx/source/recipes/broken_recipe_list.rst b/doc/sphinx/source/recipes/broken_recipe_list.rst
@@ -20,7 +20,15 @@ More details can be found in the :ref:`broken recipe policy
      - `ERA5_native6`
      - Derivation of custom variables `rlus` and `rsus`
      - `#1388 <https://github.com/ESMValGroup/ESMValCore/issues/1388>`_
+   * - :ref:`recipe_julia.yml <recipe_examples>`
+     - `example`
+     - fill values are not interpreted, resulting in an unusable plot
+     - `#2595 <https://github.com/ESMValGroup/ESMValTool/issues/2595>`_
    * - :ref:`recipe_seaice_drift.yml <recipes_seaice_drift>`
      - `sea_ice_drift_SCICEX`
-     - ``shapely`` issue
+     - ``shapely 2`` issue
      - `#3243 <https://github.com/ESMValGroup/ESMValTool/issues/3243>`_
+   * - :ref:`recipe_pysplot.yml <recipes_psyplot_diag>`
+     - `plot_map`
+     - ``shapely 2`` issue
+     - `#3483 <https://github.com/ESMValGroup/ESMValTool/issues/3483>`_
diff --git a/environment.yml b/environment.yml
@@ -4,13 +4,11 @@ channels:
   # The release candidate channel should only be activated
   # during the rc phase right before the next release of the
   # ESMValCore.
-  - conda-forge/label/esmvalcore_rc
+  # - conda-forge/label/esmvalcore_rc
   - conda-forge
   - nodefaults
 
 dependencies:
-  - pip !=21.3
-  - python >=3.9
   - aiohttp
   - cartopy
   - cdo >=1.9.7
@@ -24,11 +22,11 @@ dependencies:
   - ecmwf-api-client
   - eofs
   - esmpy
-  - esmvalcore =2.10.0rc1
+  - esmvalcore 2.10.*
   - fiona
   - fire
   - gdal
-  - iris >=3.6.0
+  - iris >=3.6.1
   - iris-esmf-regrid >=0.7.0
   - jinja2
   - joblib
@@ -40,16 +38,18 @@ dependencies:
   - netCDF4
   - numba
   - numpy !=1.24.3  # severe masking bug
-  - packaging
   - openpyxl
+  - packaging
   - pandas
+  - pip !=21.3
   - progressbar2
   - prov
   - psyplot
   - psy-maps
   - psy-reg
   - psy-simple
   - pyproj >=2.1
+  - python >=3.9
   - python-cdo
   - python-dateutil
   - pyyaml
@@ -61,7 +61,7 @@ dependencies:
   - scipy
   - seaborn
   - seawater
-  - shapely
+  - shapely >=2
   - xarray >=0.12.0
   - xesmf >=0.7.1
   - xgboost >1.6.1  # github.com/ESMValGroup/ESMValTool/issues/2779

diff --git a/environment_osx.yml b/environment_osx.yml
@@ -4,13 +4,11 @@ channels:
   # The release candidate channel should only be activated
   # during the rc phase right before the next release of the
   # ESMValCore.
-  - conda-forge/label/esmvalcore_rc
+  # - conda-forge/label/esmvalcore_rc
   - conda-forge
   - nodefaults
 
 dependencies:
-  - pip !=21.3
-  - python >=3.9
   - aiohttp
   - cartopy
   - cdo >=1.9.7
@@ -24,11 +22,11 @@ dependencies:
   - ecmwf-api-client
   - eofs
   - esmpy
-  - esmvalcore =2.10.0rc1
+  - esmvalcore 2.10.*
   - fiona
   - fire
   - gdal
-  - iris >=3.6.0
+  - iris >=3.6.1
   - iris-esmf-regrid >=0.7.0
   - jinja2
   - joblib
@@ -38,17 +36,20 @@ dependencies:
   - natsort
   - nc-time-axis
   - netCDF4
+  - numba
   - numpy !=1.24.3  # severe masking bug
-  - packaging
   - openpyxl
+  - packaging
   - pandas
+  - pip !=21.3
   - progressbar2
   - prov
   - psyplot
   - psy-maps
   - psy-reg
   - psy-simple
   - pyproj>=2.1
+  - python >=3.9
   - python-cdo
   - python-dateutil
   - pyyaml
@@ -60,7 +61,7 @@ dependencies:
   - scipy
   - seaborn
   - seawater
-  - shapely
+  - shapely >=2
   - xarray >=0.12.0
   - xesmf >=0.7.1
   - xgboost >1.6.1  # github.com/ESMValGroup/ESMValTool/issues/2779