Recipe testing for release 2.8.0 - Core release candidate rc1 #3076

remi-kazeroni · 2023-03-07T15:23:37Z

This issue documents the round of recipe testing performed using the Core release candidate v2.8.0rc1.

Release process

Tool: ESMValTool v2.8 release #3055
Core: ESMValCore v2.8 release ESMValCore#1924

System and settings

`conda`/`mamba`

(base) mamba --version
mamba 1.3.1
conda 23.1.0

Git branches and state

Tue 7 Mar 16:06:43 CET 2023

(base) :~/ESMValTool 
$ git status
On branch main
Your branch is up to date with 'origin/main'.

nothing to commit, working tree clean

(base) :~/ESMValCore 
$ git status
On branch v2.8.x
Your branch is up to date with 'origin/v2.8.x'.

nothing to commit, working tree clean

Installation and environment

$ cd ~/ESMValTool
$ mamba env create -n tool_280rc1 -f environment.yml
$ conda activate tool_280rc1
$ pip install --editable '.[develop]'
$ cd ~/ESMValCore
$ pip install --editable '.[develop]'

Config user file

Main options: all default except search_esgf: when_missing

output_dir: ./esmvaltool_output
max_parallel_tasks: 8
log_level: debug
exit_on_warning: false
output_file_type: png
remove_preproc_dir: true
compress_netcdf: false
save_intermediary_cubes: false
config_developer_file: null
profile_diagnostic: false

# Site-specific entries: DKRZ-Levante
search_esgf: when_missing
download_dir: /work/bd0854/DATA/ESMValTool2/download
auxiliary_data_dir: /work/bd0854/DATA/ESMValTool2/AUX
rootpath:
  CMIP6: /work/bd0854/DATA/ESMValTool2/CMIP6_DKRZ
  CMIP5: /work/bd0854/DATA/ESMValTool2/CMIP5_DKRZ
  CMIP3: /work/bd0854/DATA/ESMValTool2/CMIP3
  CORDEX: /work/ik1017/C3SCORDEX/data/c3s-cordex/output
  OBS: /work/bd0854/DATA/ESMValTool2/OBS
  OBS6: /work/bd0854/DATA/ESMValTool2/OBS
  obs4MIPs: /work/bd0854/DATA/ESMValTool2/OBS
  ana4mips: /work/bd0854/DATA/ESMValTool2/OBS
  native6: /work/bd0854/DATA/ESMValTool2/RAWOBS
  RAWOBS: /work/bd0854/DATA/ESMValTool2/RAWOBS
drs:
  CMIP6: DKRZ
  CMIP5: DKRZ
  CMIP3: DKRZ
  CORDEX: BADC
  obs4MIPs: default
  ana4mips: default
  OBS: default
  OBS6: default
  native6: default

ESMValTool version

$ esmvaltool version
ESMValCore: 2.8.0rc1
ESMValTool: 2.8.0.dev77+gd0c0c038e

Environment file

tool_280rc1.txt

Compute resources used

On DKRZ-Levante

When possible (small to medium jobs, in practice 4 jobs per node):

#SBATCH --partition=interactive
#SBATCH --mem=64G

Large jobs (nodes not shared)

#SBATCH --partition=compute

Note: no output comparison will be done for this round of testing. The main purpose is to identify issues with the Core and start fixing failing diagnostics. Results and overview webpage will be posted in a later post.

The text was updated successfully, but these errors were encountered:

remi-kazeroni · 2023-03-08T12:56:10Z

Overview of the results

Numbers of successes and failures

The first round of recipe testing produced:

115 successful runs (i.e. where a index.html file was generated)
28 failed runs:
- 7 for missing data/files
- 13 diagnostic issues, including:
  - 8 failures related to matplotlib recent unpinning and upgrade in our environment Unpin matplotlib #3068
  - 4 failures related to perfmetrics/main.ncl
  - 1 failure related to numpy deprecation
- 8 preprocessor issues:
  - several could be due to ESGF facet issues (recipes tested with search_esgf: when_missing)
  - recipe(s) requiring adaptation after Support wildcards in the recipe and improve support for ancillary variables and dataset versioning ESMValCore#1609
  - nothing critical at first sight!

For comparison, we released ESMValTool 2.7.0 with 4 non-working recipes (this could have been 8 if we used a stricter policy on missing data as done for this round of testing)

Overview webpage and path to data

Webpage: https://esmvaltool.dkrz.de/shared/esmvaltool/v2.8.0rc1/
Path to the runs on Levante (with preproc dirs for failed runs): /work/bd0854/b309192/recipe_testing/recipe_testing_v2p8/v08rc1/scripts/esmvaltool_output

Note: I will now write three more posts to this issue for each category of failures (missing data, diagnostic issues, preprocessor issues) and provide more details about failures and lists of affected recipes. Please do not use this issue to comment on single recipe failures but rather open a separate issue, linking to this one. This issue is intended to provide a general overview or discuss recurring problems. More info and tagging the community soon 👍

valeriupredoi · 2023-03-08T13:59:27Z

@remi-kazeroni great work on this, bud! I see some manifestations of #3061 - so that might not be just a mocking test error; maybe @bouweandela has some time to look at, at least, anav failure 🍺

remi-kazeroni · 2023-03-08T14:30:11Z

Recipe failures due to missing data or files or data issues

Recipe	Test run	Problem	Assigned to	Related issue PR
~~recipe_esacci_lst~~	link	new ancestor taks issue due to malformed CMORized obs	-	#3104
~~recipe_flato13ipcc~~	link	known missing data issue	@katjaweigel	#2156
~~recipe_tebaldi21esd~~	link	new missing data issue: `pr-tas, Amon, CMIP6, MPI-ESM1-2-HR, ScenarioMIP, ssp370, r*`	@remi-kazeroni	#3093
~~recipe_autoassess_landsurface_soilmoisture~~	link	known missing climatology files (non-public)	@alistairsellar @valeriupredoi	marked as broken in #3103
recipe_schlund20jgr_gpp*	link to one	known retracted data issue	@schlunma	successfully tested with an extended data pool

For recipe_collins13ipcc and recipe_tebaldi21esd, I would encourage recipe maintainers and friends to open dedicated issues to discuss the problem there and link here by editing the tables.
For recipe_autoassess_landsurface_soilmoisture and recipe_schlund20jgr_gpp*, I would strongly recommend to discuss in the issue Application of the broken recipe policy for releases #3065 and not here.

remi-kazeroni · 2023-03-08T14:46:08Z

Recipe failures due to diagnostic errors

Recipe	Test run	Problem	Assigned to	Related issue PR
~~recipe_arctic_ocean~~	link	matplotlib issue	-	#3112
~~recipe_autoassess_stratosphere~~	link	matplotlib issue	-	#3088
~~recipe_bock20jgr_fig_8-10~~	link	matplotlib issue	@schlunma	fixed by #3087
~~recipe_carvalhais14nat~~	link	matplotlib issue	-	-
~~recipe_hype~~	link	numpy issue	@Peter9192	#3101
~~recipe_monitor~~	link	matplotlib issue	@sloosvel	see #3085
recipe_perfmetrics_CMIP5*	link to one	issue with `perfmetrics/main.ncl`	@LisaBock	#3083 #3098
recipe_smpi*	link to one	issue with `perfmetrics/main.ncl`	-	#3098
~~recipe_sea_surface_salinity~~	link	matplotlib issue	-@sloosvel	see #3100

All these are new issues discovered in the release process of v2.8.0. I would encourage recipe maintainers and friends to open dedicated issues to discuss the problem there and link here by editing the tables.
There are different types of "matplotlib issues". These are issues still existing after Replace removed matplotlib.pyplot.savefig option additional_artists #3075 was merged and used for the testing.
If needed some problems could be moved to the preprocessor issues table (recipe_bock20jgr_fig_1-4 and recipe_esacci_lst)

remi-kazeroni · 2023-03-08T15:12:56Z

Recipe failures due to preprocessor errors

Recipe	Test run	Problem	Assigned to	Related issue PR
~~recipe_anav13jclim~~	link	issue with `weighting_landsea_fraction` for OBS data	@schlunma	run fine with #3064
recipe_autoassess_radiation_rms_cfMon_all	link	known CMOR check issue with `CMOR checks`	-	ESMValGroup/ESMValCore#1238 but won't fix for v2.8?
~~recipe_bock20jgr_fig_1-4~~	link	preproc issue reading `OBS_HadCRUT4`	-	ESMValGroup/ESMValCore#1962
recipe_check_obs	link	known derivation issue for ERA5	-	see ESMValGroup/ESMValCore#1388
~~recipe_climate_change_hotspot~~	link	new `from_datasets.py` issue	@sloosvel	#3086
~~recipe_collins13ipcc~~	link	new out of memory issue	-	#3106
~~recipe_meehl20sciadv~~	link	new `_io.py` concatenation issue	@schlunma	run fine with #3064
~~recipe_russell18jgr~~	link	new `preprocessor/_supplementary_vars.py` issue	@schlunma	run fine with #3064
~~recipe_schlund20esd~~	link	new `_io.py` concatenation issue	@schlunma	run fine with #3064
~~recipe_wenzel16jclim~~	link	new `preprocessor/_supplementary_vars.py` issue	@schlunma	#3084; run fine with #3064

I have the impression that some (all?) the 4-5 recipes could be run successfully when using search_esgf: never, i.e. without ESGF data. Said differently, some of the listed preprocessor issues could be ESGF facet issues...
The last 5 failures are new issues discovered in the release process of v2.8. I would encourage recipe maintainers and friends to open dedicated issues to discuss the problem there and link here by editing the tables.

remi-kazeroni · 2023-03-08T15:23:59Z

Hi @ESMValGroup/esmvaltool-developmentteam and @ESMValGroup/esmvaltool-recipe-maintainers, the results from the first round of recipe testing for the release of ESMValTool and ESMValCore v2.8 are now available. See summary and link to the overview webpage in this #3076 (comment) above.

It would be great if you could take a look at the results of your favourite recipes and help fixing the failing ones. In such cases, please assign yourselves in the tables above and if possible open separate issues to discuss the problems. Feel free to comment here if you have noticed other problems with the recipe output.

There are 3 main categories of failures:

Any help will be greatly appreciated 👍
Please tag me in the PRs for recipe fixing, I can take a look and merge them into the main and release branches. Thanks in advance for your support!

valeriupredoi · 2023-03-08T16:07:02Z

@remi-kazeroni one thing you should tell the users that will be running their recipes is to deactivate the esmvalcore 2.7.* pin in environment.yml and make it either 2.8.* or remove it completely, and to activate the rc channel 👍

valeriupredoi · 2023-03-08T16:18:48Z

@remi-kazeroni one thing you should tell the users that will be running their recipes is to deactivate the esmvalcore 2.7.* pin in environment.yml and make it either 2.8.* or remove it completely, and to activate the rc channel +1

hang on, we have an issue with the Tool env picking up Core 2.8.0rc1 off the rc channel/label - am looking into this atm 👍

valeriupredoi · 2023-03-13T12:55:14Z

@remi-kazeroni my apologies - for the autoassess soilmoisture recipe, I plopped the clima files at /home/b/b382109/autoassess_files - just did a run with the new rc1 and that recipe runs well! If you want me to, I can put those files elsewhere, if you don't have access to them from me HOME dir 👍

remi-kazeroni · 2023-03-13T13:13:58Z

@remi-kazeroni my apologies - for the autoassess soilmoisture recipe, I plopped the clima files at /home/b/b382109/autoassess_files - just did a run with the new rc1 and that recipe runs well! If you want me to, I can put those files elsewhere, if you don't have access to them from me HOME dir 👍

Thanks for checking, good to know it runs fine 👍

We have started to discuss how to handle this recipe in the issue on the broken recipe policy #3065 (see this #3065 (comment)). If the clima files cannot be publicly accessed and if that access cannot be documented, we will always have trouble testing such recipes for releases.

Remember that we have discussed a lot already to avoid these manual steps where the release manager would need to search for extra files to run particular recipes. This makes the release process too complicated otherwise. I think you have a PR that aims at simplifying the description of the release process: #3032 (I'm going to push it to the finish line soon) 👍

valeriupredoi · 2023-03-13T13:40:33Z

@remi-kazeroni my apologies - for the autoassess soilmoisture recipe, I plopped the clima files at /home/b/b382109/autoassess_files - just did a run with the new rc1 and that recipe runs well! If you want me to, I can put those files elsewhere, if you don't have access to them from me HOME dir +1

Thanks for checking, good to know it runs fine +1

We have started to discuss how to handle this recipe in the issue on the broken recipe policy #3065 (see this #3065 (comment)). If the clima files cannot be publicly accessed and if that access cannot be documented, we will always have trouble testing such recipes for releases.

Remember that we have discussed a lot already to avoid these manual steps where the release manager would need to search for extra files to run particular recipes. This makes the release process too complicated otherwise. I think you have a PR that aims at simplifying the description of the release process: #3032 (I'm going to push it to the finish line soon) +1

totally with you on that, bud! ah that #3032 - had forgotten about it 😁 BTW great work streamlining it, such a socialist/community PR ⚒️

A quick try to see if I can solve the hype issue detected in #3076

remi-kazeroni · 2023-03-15T16:41:48Z

Thanks everyone for your support with fixing recipes! Most failed recipes have received some attention and many of them got fixed already. That's great news for the upcoming release 👍

Up to now, it seems that 2 recipes have unfortunately not received attention:

recipe_arctic_ocean

  File "/home/b/b309192/ESMValTool/esmvaltool/diag_scripts/arctic_ocean/utils.py", line 223, in get_cmap
    cm.register_cmap(cmap=LinearSegmentedColormap(
  File "/work/bd0854/b309192/soft/mambaforge/envs/tool_280rc1/lib/python3.10/site-packages/matplotlib/_api/deprecation.py", line 200, in wrapper
    return func(*args, **kwargs)
  File "/work/bd0854/b309192/soft/mambaforge/envs/tool_280rc1/lib/python3.10/site-packages/matplotlib/cm.py", line 263, in register_cmap
    _colormaps.register(cmap, name=name, force=override_builtin)
  File "/work/bd0854/b309192/soft/mambaforge/envs/tool_280rc1/lib/python3.10/site-packages/matplotlib/cm.py", line 136, in register
    raise ValueError(
ValueError: A colormap named "cubehelix3" is already registered.

recipe_carvalhais14nat

Traceback (most recent call last):
  File "/home/b/b309192/ESMValTool/esmvaltool/diag_scripts/land_carbon_cycle/diag_global_turnover.py", line 817, in <module>
    main(config)
  File "/home/b/b309192/ESMValTool/esmvaltool/diag_scripts/land_carbon_cycle/diag_global_turnover.py", line 755, in main
    _plot_single_map(plot_path_mod, tau_ctotal,
  File "/home/b/b309192/ESMValTool/esmvaltool/diag_scripts/land_carbon_cycle/diag_global_turnover.py", line 660, in _plot_single_map
    _fix_map(_ax)
  File "/home/b/b309192/ESMValTool/esmvaltool/diag_scripts/land_carbon_cycle/diag_global_turnover.py", line 278, in _fix_map
    plt.gca().outline_patch.set_visible(False)
AttributeError: 'GeoAxes' object has no attribute 'outline_patch'

If you happen to know their maintainers or colleagues/friends or if you have encountered similar diagnostic issues recently, you are most welcome to jump in here and help us getting the recipes fixed for the upcoming release. Otherwise these would be marked as broken in ESMValTool v2.8.

valeriupredoi · 2023-03-15T16:43:48Z

@remi-kazeroni hold on tight buddy, Carvalhais will get a solution in the next few minutes, I know exactly what's up there

valeriupredoi · 2023-03-15T16:49:20Z

Carvalhais needs to get back to the other patches - @zklaus fixed that for when we pinned Matplotlib last release, see #2886 (comment) - now we need to revert that, I'll PR it now

valeriupredoi · 2023-03-15T16:58:06Z

Carvalhais14 fixed here #3111

TomasTorsvik · 2023-03-15T21:07:46Z

Hi, I looked at the recipe_arctic_ocean log, and apparently the cm.register_cmap function has been deprecated in matplotlib since version 3.7 (see https://matplotlib.org/stable/api/cm_api.html#matplotlib.cm.register_cmap )
and is suggested to be replaced by matplotlib.colormaps.register. Could it be a case of just replacing the deprecated function?

Unfortunately, I'm not able to run any test at the moment. The Norwegian national data storage is migrating all data to a new location, and have closed down access for some time.

remi-kazeroni · 2023-03-16T08:17:37Z

colormaps.register

Hi @TomasTorsvik, thanks very much for taking a look and your suggestion! It would be great if you or someone could make these modifications available in a branch and open a (draft) pull request with that. This would then ease the testing for developers like me with access to all necessary data at DKRZ. Note that you could also make use of our bot directly from that PR (see doc). This would allow you to request a fresh installation of ESMValTool and a test run at DKRZ directly via the PR.

remi-kazeroni · 2023-03-21T16:30:22Z

Thanks everyone for you help and support with fixing recipes! All cases have now been addressed. I'm closing this and will open tomorrow another issue for the second and last round of testing (currently ongoing) where the community will be asked to take a look at recipe output and status of the comparison with results from the previous release.

valeriupredoi · 2023-03-21T16:33:05Z

legend @remi-kazeroni 🎖️ Hopefully the next (and last) round will be a cake walk 🍰

remi-kazeroni added testing release labels Mar 7, 2023

remi-kazeroni added this to the v2.8.0 milestone Mar 7, 2023

This was referenced Mar 7, 2023

Replace removed matplotlib.pyplot.savefig option additional_artists #3075

Merged

recipe_perfmetrics_land_CMIP5 fails in multi_model_statistics step #2594

Closed

remi-kazeroni pinned this issue Mar 7, 2023

This was referenced Mar 9, 2023

recipe_monitor.yml failing due to matplotlib issues #3085

Closed

recipe_climate_change_hotspot failing due to ensemble list #3086

Closed

LisaBock mentioned this issue Mar 9, 2023

Fix call of output_type in aux_plotting.ncl #3083

Merged

11 tasks

remi-kazeroni mentioned this issue Mar 9, 2023

Missing preprocessed OBS data when running recipe_bock20jgr_fig_1-4 ESMValGroup/ESMValCore#1962

Closed

schlunma mentioned this issue Mar 9, 2023

Remove colorbar from bbox_extra_artists #3087

Merged

11 tasks

valeriupredoi mentioned this issue Mar 9, 2023

Recipe Autoassess Stratosphere recipe_autoassess_stratosphere.py fails with Matplotlib=3.7.1 #3088

Closed

axel-lauer mentioned this issue Mar 13, 2023

Fix perfmetrics v2.8.0 #3098

Merged

9 tasks

remi-kazeroni mentioned this issue Mar 13, 2023

Application of the broken recipe policy for releases #3065

Closed

sloosvel mentioned this issue Mar 13, 2023

recipe_sea_surface_salinity broken due to matplotlib issues #3100

Closed

Peter9192 added a commit that referenced this issue Mar 13, 2023

Fix numpy deprecation in hype diagnostic

413993b

A quick try to see if I can solve the hype issue detected in #3076

Peter9192 mentioned this issue Mar 13, 2023

Fix numpy deprecation in hype diagnostic #3101

Merged

12 tasks

remi-kazeroni mentioned this issue Mar 13, 2023

Fix the ESACCI-LST CMORizer #3104

Merged

13 tasks

valeriupredoi mentioned this issue Mar 15, 2023

Fix recipe_carvalhais14: replace outline patch with splines #3111

Merged

3 tasks

TomasTorsvik mentioned this issue Mar 16, 2023

Replace deprecated function "cm.register_cmap" with "mpl.colormaps.register" for recipe_ arctic_ocean #3112

Merged

11 tasks

Jon-Lillis unpinned this issue Mar 16, 2023

remi-kazeroni closed this as completed Mar 21, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Recipe testing for release 2.8.0 - Core release candidate rc1 #3076

Recipe testing for release 2.8.0 - Core release candidate rc1 #3076

remi-kazeroni commented Mar 7, 2023

remi-kazeroni commented Mar 8, 2023

valeriupredoi commented Mar 8, 2023

remi-kazeroni commented Mar 8, 2023 •

edited

Loading

remi-kazeroni commented Mar 8, 2023 •

edited

Loading

remi-kazeroni commented Mar 8, 2023 •

edited

Loading

remi-kazeroni commented Mar 8, 2023

valeriupredoi commented Mar 8, 2023

valeriupredoi commented Mar 8, 2023

valeriupredoi commented Mar 13, 2023

remi-kazeroni commented Mar 13, 2023

valeriupredoi commented Mar 13, 2023 •

edited

Loading

remi-kazeroni commented Mar 15, 2023

valeriupredoi commented Mar 15, 2023

valeriupredoi commented Mar 15, 2023

valeriupredoi commented Mar 15, 2023

TomasTorsvik commented Mar 15, 2023

remi-kazeroni commented Mar 16, 2023

remi-kazeroni commented Mar 21, 2023

valeriupredoi commented Mar 21, 2023

Recipe testing for release 2.8.0 - Core release candidate rc1 #3076

Recipe testing for release 2.8.0 - Core release candidate rc1 #3076

Comments

remi-kazeroni commented Mar 7, 2023

Release process

System and settings

conda/mamba

Git branches and state

Installation and environment

Config user file

ESMValTool version

Environment file

Compute resources used

remi-kazeroni commented Mar 8, 2023

Overview of the results

Numbers of successes and failures

Overview webpage and path to data

valeriupredoi commented Mar 8, 2023

remi-kazeroni commented Mar 8, 2023 • edited Loading

Recipe failures due to missing data or files or data issues

remi-kazeroni commented Mar 8, 2023 • edited Loading

Recipe failures due to diagnostic errors

remi-kazeroni commented Mar 8, 2023 • edited Loading

Recipe failures due to preprocessor errors

remi-kazeroni commented Mar 8, 2023

valeriupredoi commented Mar 8, 2023

valeriupredoi commented Mar 8, 2023

valeriupredoi commented Mar 13, 2023

remi-kazeroni commented Mar 13, 2023

valeriupredoi commented Mar 13, 2023 • edited Loading

remi-kazeroni commented Mar 15, 2023

valeriupredoi commented Mar 15, 2023

valeriupredoi commented Mar 15, 2023

valeriupredoi commented Mar 15, 2023

TomasTorsvik commented Mar 15, 2023

remi-kazeroni commented Mar 16, 2023

remi-kazeroni commented Mar 21, 2023

valeriupredoi commented Mar 21, 2023

`conda`/`mamba`

remi-kazeroni commented Mar 8, 2023 •

edited

Loading

remi-kazeroni commented Mar 8, 2023 •

edited

Loading

remi-kazeroni commented Mar 8, 2023 •

edited

Loading

valeriupredoi commented Mar 13, 2023 •

edited

Loading