Published recipes use deprecated `fx_variables` option in preprocessors #3056

schlunma · 2023-02-27T17:09:36Z

schlunma · 2023-02-27T17:10:13Z

@ESMValGroup/esmvaltool-coreteam Feel free to update this list if you encounter more recipes that need updates.

bouweandela · 2023-02-27T20:00:38Z

The dataset name and version number needs to be updated for recipes using obs4MIPs datasets: #2974

valeriupredoi · 2023-02-28T12:41:05Z

the issue with collins and sftgif should not be an issue - mask_landseaice doesn't explicitly need an fx/supplementary variable - it should automatically look for sftgif and the mocker recipe_load test that fails and @schlunma links should just spit a warning, not an outright error

schlunma · 2023-02-28T12:58:21Z

Yeah, I will remove them again from the list and only add recipes where we confirmed that they stopped working. I thought we can trust that automated test 🥲

valeriupredoi · 2023-02-28T15:14:36Z

the test can be trusted (if you talking about the GA test) - the reasons it fails are all upon us 😆 Do you want to try and run those collins et al recipes - like actual runs, as you suggested, Manu? Using a Core-dev env just like the test, but where data is available, if they pass it means it's some mocking issue like we've had before, if they fail then it's some Bouweing issue like we've had before 👿

schlunma · 2023-02-28T15:20:56Z

We will test the recipes soon anyway, so I'll wait until that (don't want to waste resources on these huge recipes 😄 )

valeriupredoi · 2023-02-28T15:23:23Z

We will test the recipes soon anyway, so I'll wait until that (don't want to waste resources on these huge recipes smile )

such an Ursula von der Leyen-like answer 🥇

remi-kazeroni · 2023-02-28T15:31:22Z

We will test the recipes soon anyway, so I'll wait until that (don't want to waste resources on these huge recipes 😄 )

Indeed, this is probably our most resource-intensive recipe and only runs on 512GB nodes at DKRZ... See https://esmvaltool.dkrz.de/shared/esmvaltool/v2.7.0/debug.html

schlunma · 2023-03-13T08:16:02Z

Partly addressed by #3064

valeriupredoi · 2023-03-17T12:04:20Z

well Collins13 is fixed now - I mean, it's still a memory guzzler, but at least Manu and Remi ran it - any more of those recipes need fixing?

remi-kazeroni · 2023-03-17T12:53:46Z

All the recipes not yet updated will continue to run fine with v2.8. fx_variables was set to deprecation in ESMValGroup/ESMValCore#1609 but we discussed there already that is still supported in v2.8. Nevertheless, it would be better, if possible, to remove it from our recipes so that we don't show "bad" examples. At the latest, this will have to be finished before v2.9. I'm sure the maintainer of examples/recipe_python.yml would take care of that one next Monday, right @schlunma? 😉

schlunma · 2023-03-20T09:38:00Z

I'm sure the maintainer of examples/recipe_python.yml would take care of that one next Monday, right @schlunma? wink

Done in #3116. I can't remember how/when I got maintainer of examples/recipe_python.yml 😅

remi-kazeroni · 2023-03-21T11:40:36Z

Thanks to everyone who contributed to this effort already! I bumped this issue to the next milestone because we need to move on with the final recipe testing in v2.8 release. The remaining few recipes would need to be updated before the v2.9 release.

bouweandela · 2023-06-06T14:15:49Z

Friendly ping @katjaweigel and @debe-kevin. Could you have a look at the recipes that you are maintaining and remove all mentions of fx_variables?

katjaweigel · 2023-06-08T07:16:44Z

@bouweandela thanks for the ping! I didn't notice it before, but I'm working on it now.

katjaweigel · 2023-06-09T10:28:43Z

Unfortunately I get an error (for ipccwg1ar5ch9/recipe_weigel21gmd_figures_13_16.yml and similar also for
ipccwg1ar5ch9/recipe_flato13ipcc_figures_938_941_cmip6.yml) from the preprocessor, which is possibly not even related (see below - seems to be caused by regridding).
It occurs under the current development version of the core, if I don't change anything (with fx_variables: null) but also without "fx_variables: null".
It does not occur with the release version of the core (branch v2.8x), tested without "fx_variables: null" for ipccwg1ar5ch9/recipe_weigel21gmd_figures_13_16.yml. I guess have to find the reason for this first before I can update the preprocessor options.

Error message (ipccwg1ar5ch9/recipe_flato13ipcc_figures_938_941_cmip6.ym):

Traceback (most recent call last):
  File "/work/bd1083/b380216/CORE202303/ESMValCore/esmvalcore/_main.py", line 524, in run
    fire.Fire(ESMValTool())
  File "/home/b/b380216/mambaforge/envs/esmvaltool202303/lib/python3.10/site-packages/fire/core.py", line 141, in Fire
    component_trace = _Fire(component, args, parsed_flag_args, context, name)
  File "/home/b/b380216/mambaforge/envs/esmvaltool202303/lib/python3.10/site-packages/fire/core.py", line 466, in _Fire
    component, remaining_args = _CallAndUpdateTrace(
  File "/home/b/b380216/mambaforge/envs/esmvaltool202303/lib/python3.10/site-packages/fire/core.py", line 681, in _CallAndUpdateTrace
    component = fn(*varargs, **kwargs)
  File "/work/bd1083/b380216/CORE202303/ESMValCore/esmvalcore/_main.py", line 411, in run
    self._run(recipe, session)
  File "/work/bd1083/b380216/CORE202303/ESMValCore/esmvalcore/_main.py", line 453, in _run
    process_recipe(recipe_file=recipe, session=session)
  File "/work/bd1083/b380216/CORE202303/ESMValCore/esmvalcore/_main.py", line 125, in process_recipe
    recipe.run()
  File "/work/bd1083/b380216/CORE202303/ESMValCore/esmvalcore/_recipe/recipe.py", line 1294, in run
    self.tasks.run(max_parallel_tasks=self.session['max_parallel_tasks'])
  File "/work/bd1083/b380216/CORE202303/ESMValCore/esmvalcore/_task.py", line 738, in run
    self._run_parallel(address, max_parallel_tasks)
  File "/work/bd1083/b380216/CORE202303/ESMValCore/esmvalcore/_task.py", line 782, in _run_parallel
    _copy_results(task, running[task])
  File "/work/bd1083/b380216/CORE202303/ESMValCore/esmvalcore/_task.py", line 805, in _copy_results
    task.output_files, task.products = future.get()
  File "/home/b/b380216/mambaforge/envs/esmvaltool202303/lib/python3.10/multiprocessing/pool.py", line 774, in get
    raise self._value
  File "/home/b/b380216/mambaforge/envs/esmvaltool202303/lib/python3.10/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
  File "/work/bd1083/b380216/CORE202303/ESMValCore/esmvalcore/_task.py", line 816, in _run_task
    output_files = task.run()
  File "/work/bd1083/b380216/CORE202303/ESMValCore/esmvalcore/_task.py", line 264, in run
    self.output_files = self._run(input_files)
  File "/work/bd1083/b380216/CORE202303/ESMValCore/esmvalcore/preprocessor/__init__.py", line 670, in _run
    product.apply(step, self.debug)
  File "/work/bd1083/b380216/CORE202303/ESMValCore/esmvalcore/preprocessor/__init__.py", line 470, in apply
    self.cubes = preprocess(self.cubes, step,
  File "/work/bd1083/b380216/CORE202303/ESMValCore/esmvalcore/preprocessor/__init__.py", line 379, in preprocess
    result.append(_run_preproc_function(function, item, settings,
  File "/work/bd1083/b380216/CORE202303/ESMValCore/esmvalcore/preprocessor/__init__.py", line 328, in _run_preproc_function
    return function(items, **kwargs)
  File "/work/bd1083/b380216/CORE202303/ESMValCore/esmvalcore/preprocessor/_regrid.py", line 678, in regrid
    cube = _rechunk(cube, target_grid)
  File "/work/bd1083/b380216/CORE202303/ESMValCore/esmvalcore/preprocessor/_regrid.py", line 705, in _rechunk
    cube.data = data.rechunk(chunks)
  File "/home/b/b380216/mambaforge/envs/esmvaltool202303/lib/python3.10/site-packages/dask/array/core.py", line 2767, in rechunk
    return rechunk(self, chunks, threshold, block_size_limit, balance, method)
  File "/home/b/b380216/mambaforge/envs/esmvaltool202303/lib/python3.10/site-packages/dask/array/rechunk.py", line 307, in rechunk
    chunks = normalize_chunks(
  File "/home/b/b380216/mambaforge/envs/esmvaltool202303/lib/python3.10/site-packages/dask/array/core.py", line 3127, in normalize_chunks
    raise ValueError(
ValueError: Chunks do not add up to shape. Got chunks=((26, 26, 26, 26, 26, 26, 26, 26, 26, 26, 26, 26), (120,), (180, 108)), shape=(312, 180, 288)

katjaweigel · 2023-06-09T11:52:29Z

Hm, seems there were some half finished things in there, yesterday. Getting the newest version now, it works again.

bouweandela · 2023-06-09T12:41:17Z

Yes, there was a mistake ESMValGroup/ESMValCore#2085 in the code that has now been fixed.

katjaweigel · 2023-06-09T13:27:08Z

Sorry, that I don't manage to finish the PR today, some tests will run to long so I continue on Monday.

remi-kazeroni · 2023-06-14T15:37:06Z

This issue can now be closed since we do not have recipes in main using fx_variables any more. Thanks to everyone who helped with recipe updates 👍

schlunma added the requires new ESMValCore release A new release of ESMValCore is needed to solve this issue/merge this pull request.. label Feb 27, 2023

schlunma added this to the v2.8.0 milestone Feb 27, 2023

schlunma mentioned this issue Feb 28, 2023

Tests in full development mode fail after PR 3040 #3061

Closed

schlunma changed the title ~~Recipe updates necessary for v2.8~~ Published recipes use deprecated fx_variables option in preprocessors Mar 13, 2023

schlunma mentioned this issue Mar 20, 2023

Replace fx_variables with supplementary_variables in recipe_python.yml #3116

Closed

12 tasks

axel-lauer mentioned this issue Mar 20, 2023

Removed fx_variables in recipe_mpqb_xch4 and recipe_lauer22jclim_fig8 #3117

Merged

5 tasks

bouweandela mentioned this issue Mar 20, 2023

Update Python example recipe #3119

Merged

10 tasks

remi-kazeroni modified the milestones: v2.8.0, v2.9.0 Mar 21, 2023

remi-kazeroni added diagnostic deprecated feature and removed requires new ESMValCore release A new release of ESMValCore is needed to solve this issue/merge this pull request.. labels Mar 21, 2023

This was referenced Jun 7, 2023

Remove "fx_variable" from recipe_tebaldi21esd.yml #3211

Merged

Remove "fx_variable" from recipe_wenzel14jgr.yml #3212

Merged

katjaweigel mentioned this issue Jun 9, 2023

Remove fx_variables from ipccwg1ar5ch9 recipes #3215

Merged

10 tasks

remi-kazeroni closed this as completed Jun 14, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Published recipes use deprecated `fx_variables` option in preprocessors #3056

Published recipes use deprecated `fx_variables` option in preprocessors #3056

schlunma commented Feb 27, 2023 •

edited by remi-kazeroni

Loading

schlunma commented Feb 27, 2023

bouweandela commented Feb 27, 2023 •

edited

Loading

valeriupredoi commented Feb 28, 2023

schlunma commented Feb 28, 2023

valeriupredoi commented Feb 28, 2023 •

edited

Loading

schlunma commented Feb 28, 2023

valeriupredoi commented Feb 28, 2023

remi-kazeroni commented Feb 28, 2023

schlunma commented Mar 13, 2023

valeriupredoi commented Mar 17, 2023

remi-kazeroni commented Mar 17, 2023

schlunma commented Mar 20, 2023 •

edited

Loading

remi-kazeroni commented Mar 21, 2023

bouweandela commented Jun 6, 2023

katjaweigel commented Jun 8, 2023

katjaweigel commented Jun 9, 2023 •

edited by bouweandela

Loading

katjaweigel commented Jun 9, 2023

bouweandela commented Jun 9, 2023

katjaweigel commented Jun 9, 2023

remi-kazeroni commented Jun 14, 2023

Published recipes use deprecated fx_variables option in preprocessors #3056

Published recipes use deprecated fx_variables option in preprocessors #3056

Comments

schlunma commented Feb 27, 2023 • edited by remi-kazeroni Loading

schlunma commented Feb 27, 2023

bouweandela commented Feb 27, 2023 • edited Loading

valeriupredoi commented Feb 28, 2023

schlunma commented Feb 28, 2023

valeriupredoi commented Feb 28, 2023 • edited Loading

schlunma commented Feb 28, 2023

valeriupredoi commented Feb 28, 2023

remi-kazeroni commented Feb 28, 2023

schlunma commented Mar 13, 2023

valeriupredoi commented Mar 17, 2023

remi-kazeroni commented Mar 17, 2023

schlunma commented Mar 20, 2023 • edited Loading

remi-kazeroni commented Mar 21, 2023

bouweandela commented Jun 6, 2023

katjaweigel commented Jun 8, 2023

katjaweigel commented Jun 9, 2023 • edited by bouweandela Loading

katjaweigel commented Jun 9, 2023

bouweandela commented Jun 9, 2023

katjaweigel commented Jun 9, 2023

remi-kazeroni commented Jun 14, 2023

Published recipes use deprecated `fx_variables` option in preprocessors #3056

Published recipes use deprecated `fx_variables` option in preprocessors #3056

schlunma commented Feb 27, 2023 •

edited by remi-kazeroni

Loading

bouweandela commented Feb 27, 2023 •

edited

Loading

valeriupredoi commented Feb 28, 2023 •

edited

Loading

schlunma commented Mar 20, 2023 •

edited

Loading

katjaweigel commented Jun 9, 2023 •

edited by bouweandela

Loading