Refactor regridding (#2231)

Co-authored-by: Valeriu Predoi <[email protected]>

doc/api/esmvalcore.regridding_schemes.rst

@@ -0,0 +1,23 @@
+.. _regridding_schemes:
+
+Regridding schemes
+==================
+
+Iris natively supports data regridding with its :meth:`iris.cube.Cube.regrid`
+method and a set of predefined regridding schemes provided in the
+:mod:`~iris.analysis` module (further details are given on `this
+<https://scitools-iris.readthedocs.io/en/latest/userguide/interpolation_and_regridding.html>`__
+page).
+Here, further regridding schemes are provided that are compatible with
+:meth:`iris.cube.Cube.regrid`.
+
+Example:
+
+.. code:: python
+
+   from esmvalcore.preprocessor.regrid_schemes import ESMPyAreaWeighted
+
+   regridded_cube = cube.regrid(target_grid, ESMPyAreaWeighted())
+
+.. automodule:: esmvalcore.preprocessor.regrid_schemes
+   :no-show-inheritance:

doc/api/esmvalcore.rst

@@ -17,5 +17,6 @@ library. This section documents the public API of ESMValCore.
    esmvalcore.iris_helpers
    esmvalcore.local
    esmvalcore.preprocessor
+   esmvalcore.regridding_schemes
    esmvalcore.typing
    esmvalcore.experimental

doc/quickstart/find_data.rst

@@ -121,13 +121,13 @@ ERA5
 - Supported variables: ``cl``, ``clt``, ``evspsbl``, ``evspsblpot``, ``mrro``, ``pr``, ``prsn``, ``ps``, ``psl``, ``ptype``, ``rls``, ``rlds``, ``rsds``, ``rsdt``, ``rss``, ``uas``, ``vas``, ``tas``, ``tasmax``, ``tasmin``, ``tdps``, ``ts``, ``tsn`` (``E1hr``/``Amon``), ``orog`` (``fx``)
 - Tier: 3
 
-.. note:: According to the description of Evapotranspiration and potential Evapotranspiration on the Copernicus page 
+.. note:: According to the description of Evapotranspiration and potential Evapotranspiration on the Copernicus page
   (https://cds.climate.copernicus.eu/cdsapp#!/dataset/reanalysis-era5-single-levels-monthly-means?tab=overview):
-  "The ECMWF Integrated Forecasting System (IFS) convention is that downward fluxes are positive. 
+  "The ECMWF Integrated Forecasting System (IFS) convention is that downward fluxes are positive.
   Therefore, negative values indicate evaporation and positive values indicate condensation."
-  
+
   In the CMOR table, these fluxes are defined as positive, if they go from the surface into the atmosphere:
-  "Evaporation at surface (also known as evapotranspiration): flux of water into the atmosphere due to conversion 
+  "Evaporation at surface (also known as evapotranspiration): flux of water into the atmosphere due to conversion
   of both liquid and solid phases to vapor (from underlying surface and vegetation)."
   Therefore, the ERA5 (and ERA5-Land) CMORizer switches the signs of ``evspsbl`` and ``evspsblpot`` to be compatible with the CMOR standard used e.g. by the CMIP models.
 
@@ -398,7 +398,7 @@ The UGRID conventions provide a standardized format to store data on
 unstructured grids, which is required by many software packages or tools to
 work correctly.
 An example is the horizontal regridding of native ICON data to a regular grid.
-While the built-in :ref:`unstructured_nearest scheme <built-in regridding
+While the built-in :ref:`nearest scheme <built-in regridding
 schemes>` can handle unstructured grids not in UGRID format, using more complex
 regridding algorithms (for example provided by the
 :doc:`iris-esmf-regrid:index` package through :ref:`generic regridding
@@ -420,7 +420,7 @@ This automatic UGRIDization is enabled by default, but can be switched off with
 the facet ``ugrid: false`` in the recipe or the extra facets (see below).
 This is useful for diagnostics that do not support input data in UGRID format
 (yet) like the :ref:`Psyplot diagnostic <esmvaltool:recipes_psyplot_diag>` or
-if you want to use the built-in :ref:`unstructured_nearest scheme <built-in
+if you want to use the built-in :ref:`nearest scheme <built-in
 regridding schemes>` regridding scheme.
 
 For 3D ICON variables, ESMValCore tries to add the pressure level information

doc/recipe/preprocessor.rst

@@ -898,43 +898,27 @@ third party regridding schemes designed for use with :doc:`Iris
 Built-in regridding schemes
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
-The schemes used for the interpolation and extrapolation operations needed by
-the horizontal regridding functionality directly map to their corresponding
-implementations in :mod:`iris`:
-
-* ``linear``: Linear interpolation without extrapolation, i.e., extrapolation
-  points will be masked even if the source data is not a masked array (uses
-  ``Linear(extrapolation_mode='mask')``, see :obj:`iris.analysis.Linear`).
-* ``linear_extrapolate``: Linear interpolation with extrapolation, i.e.,
-  extrapolation points will be calculated by extending the gradient of the
-  closest two points (uses ``Linear(extrapolation_mode='extrapolate')``, see
-  :obj:`iris.analysis.Linear`).
-* ``nearest``: Nearest-neighbour interpolation without extrapolation, i.e.,
-  extrapolation points will be masked even if the source data is not a masked
-  array (uses ``Nearest(extrapolation_mode='mask')``, see
-  :obj:`iris.analysis.Nearest`).
-* ``area_weighted``: Area-weighted regridding (uses ``AreaWeighted()``, see
-  :obj:`iris.analysis.AreaWeighted`).
-* ``unstructured_nearest``: Nearest-neighbour interpolation for unstructured
-  grids (uses ``UnstructuredNearest()``, see
-  :obj:`iris.analysis.UnstructuredNearest`).
+* ``linear``: Bilinear regridding.
+  For source data on a regular grid, uses :obj:`~iris.analysis.Linear` with
+  `extrapolation_mode='mask'`.
+  For source data on an irregular grid, uses
+  :class:`~esmvalcore.preprocessor.regrid_schemes.ESMPyLinear`.
+  Source data on an unstructured grid is not supported, yet.
+* ``nearest``: Nearest-neighbor regridding.
+  For source data on a regular grid, uses :obj:`~iris.analysis.Nearest` with
+  `extrapolation_mode='mask'`.
+  For source data on an irregular grid, uses
+  :class:`~esmvalcore.preprocessor.regrid_schemes.ESMPyNearest`.
+  For source data on an unstructured grid, uses
+  :class:`~esmvalcore.preprocessor.regrid_schemes.UnstructuredNearest`.
+* ``area_weighted``: First-order conservative (area-weighted) regridding.
+  For source data on a regular grid, uses :obj:`~iris.analysis.AreaWeighted`.
+  For source data on an irregular grid, uses
+  :class:`~esmvalcore.preprocessor.regrid_schemes.ESMPyAreaWeighted`.
+  Source data on an unstructured grid is not supported, yet.
 
 See also :func:`esmvalcore.preprocessor.regrid`
 
-.. note::
-
-   Controlling the extrapolation mode allows us to avoid situations where
-   extrapolating values makes little physical sense (e.g. extrapolating beyond
-   the last data point).
-
-.. note::
-
-   The regridding mechanism is (at the moment) done with fully realized data in
-   memory, so depending on how fine the target grid is, it may use a rather
-   large amount of memory. Empirically target grids of up to ``0.5x0.5``
-   degrees should not produce any memory-related issues, but be advised that
-   for resolutions of ``< 0.5`` degrees the regridding becomes very slow and
-   will use a lot of memory.
 
 .. _generic regridding schemes:
 
@@ -971,6 +955,28 @@ tolerance.
             reference: iris.analysis:AreaWeighted
             mdtol: 0.7
 
+Another example is bilinear regridding with extrapolation.
+This can be achieved with the :class:`iris.analysis.Linear` scheme and the
+``extrapolation_mode`` keyword.
+Extrapolation points will be calculated by extending the gradient of the
+closest two points.
+
+.. code-block:: yaml
+
+    preprocessors:
+      regrid_preprocessor:
+        regrid:
+          target_grid: 2.5x2.5
+          scheme:
+            reference: iris.analysis:Linear
+            extrapolation_mode: extrapolate
+
+.. note::
+
+   Controlling the extrapolation mode allows us to avoid situations where
+   extrapolating values makes little physical sense (e.g. extrapolating beyond
+   the last data point).
+
 The value of the ``reference`` key has two parts that are separated by a
 ``:`` with no surrounding spaces. The first part is an importable Python
 module, the second refers to the scheme, i.e. some callable that will be called

esmvalcore/_recipe/check.py

@@ -17,6 +17,12 @@
 from esmvalcore.local import _get_start_end_year, _parse_period
 from esmvalcore.preprocessor import TIME_PREPROCESSORS, PreprocessingTask
 from esmvalcore.preprocessor._multimodel import _get_operator_and_kwargs
+from esmvalcore.preprocessor._regrid import (
+    HORIZONTAL_SCHEMES_IRREGULAR,
+    HORIZONTAL_SCHEMES_REGULAR,
+    HORIZONTAL_SCHEMES_UNSTRUCTURED,
+    _load_generic_scheme,
+)
 from esmvalcore.preprocessor._shared import get_iris_aggregator
 from esmvalcore.preprocessor._supplementary_vars import (
     PREPROCESSOR_SUPPLEMENTARIES,
@@ -486,3 +492,43 @@ def _check_mm_stat(step, step_settings):
             raise RecipeError(
                 f"Invalid options for {step}: {exc}"
             )
+
+
+def regridding_schemes(settings: dict):
+    """Check :obj:`str` regridding schemes."""
+    if 'regrid' not in settings:
+        return
+
+    # Note: If 'scheme' is missing, this will be detected in
+    # PreprocessorFile.check() later
+    scheme = settings['regrid'].get('scheme')
+
+    # Check built-in regridding schemes (given as str)
+    if isinstance(scheme, str):
+        scheme = settings['regrid']['scheme']
+        allowed_regridding_schemes = list(
+            set(
+                list(HORIZONTAL_SCHEMES_IRREGULAR) +
+                list(HORIZONTAL_SCHEMES_REGULAR) +
+                list(HORIZONTAL_SCHEMES_UNSTRUCTURED)
+            )
+        )
+        if scheme not in allowed_regridding_schemes:
+            raise RecipeError(
+                f"Got invalid built-in regridding scheme '{scheme}', expected "
+                f"one of {allowed_regridding_schemes} or a generic scheme "
+                f"(see https://docs.esmvaltool.org/projects/ESMValCore/en/"
+                f"latest/recipe/preprocessor.html#generic-regridding-schemes)."
+            )
+
+    # Check generic regridding schemes (given as dict)
+    if isinstance(scheme, dict):
+        try:
+            _load_generic_scheme(scheme)
+        except ValueError as exc:
+            raise RecipeError(
+                f"Failed to load generic regridding scheme: {str(exc)} See "
+                f"https://docs.esmvaltool.org/projects/ESMValCore/en/latest"
+                f"/recipe/preprocessor.html#generic-regridding-schemes for "
+                f"details."
+            )

esmvalcore/_recipe/recipe.py

@@ -656,6 +656,7 @@ def _update_preproc_functions(settings, dataset, datasets, missing_vars):
     if dataset.facets.get('frequency') == 'fx':
         check.check_for_temporal_preprocs(settings)
     check.statistics_preprocessors(settings)
+    check.regridding_schemes(settings)
     check.bias_type(settings)
 
 

esmvalcore/cmor/_fixes/fix.py

@@ -23,10 +23,10 @@
     _get_new_generic_level_coord,
     _get_simplified_calendar,
     _get_single_cube,
-    _is_unstructured_grid,
 )
 from esmvalcore.cmor.fixes import get_time_bounds
 from esmvalcore.cmor.table import get_var_info
+from esmvalcore.iris_helpers import has_unstructured_grid
 
 if TYPE_CHECKING:
     from esmvalcore.cmor.table import CoordinateInfo, VariableInfo
@@ -727,7 +727,7 @@ def _fix_coord_bounds(
             return
 
         # Skip guessing bounds for unstructured grids
-        if _is_unstructured_grid(cube) and cube_coord.standard_name in (
+        if has_unstructured_grid(cube) and cube_coord.standard_name in (
                 'latitude', 'longitude'):
             self._debug_msg(
                 cube,
@@ -763,7 +763,7 @@ def _fix_coord_direction(
             return (cube, cube_coord)
         if cube_coord.dtype.kind == 'U':
             return (cube, cube_coord)
-        if _is_unstructured_grid(cube) and cube_coord.standard_name in (
+        if has_unstructured_grid(cube) and cube_coord.standard_name in (
                 'latitude', 'longitude'
         ):
             return (cube, cube_coord)

esmvalcore/cmor/_utils.py

@@ -7,7 +7,6 @@
 
 from iris.coords import Coord
 from iris.cube import Cube
-from iris.exceptions import CoordinateNotFoundError
 
 from esmvalcore.cmor.table import CMOR_TABLES, CoordinateInfo, VariableInfo
 
@@ -173,19 +172,6 @@ def _get_simplified_calendar(calendar: str) -> str:
     return calendar_aliases.get(calendar, calendar)
 
 
-def _is_unstructured_grid(cube: Cube) -> bool:
-    """Check if cube uses unstructured grid."""
-    try:
-        lat = cube.coord('latitude')
-        lon = cube.coord('longitude')
-    except CoordinateNotFoundError:
-        pass
-    else:
-        if lat.ndim == 1 and (cube.coord_dims(lat) == cube.coord_dims(lon)):
-            return True
-    return False
-
-
 def _get_single_cube(
     cube_list: Sequence[Cube],
     short_name: str,

esmvalcore/cmor/check.py

@@ -25,10 +25,10 @@
     _get_generic_lev_coord_names,
     _get_new_generic_level_coord,
     _get_simplified_calendar,
-    _is_unstructured_grid,
 )
 from esmvalcore.cmor.table import CoordinateInfo, get_var_info
 from esmvalcore.exceptions import ESMValCoreDeprecationWarning
+from esmvalcore.iris_helpers import has_unstructured_grid
 
 
 class CheckLevels(IntEnum):
@@ -140,7 +140,7 @@ def __init__(self,
     @cached_property
     def _unstructured_grid(self) -> bool:
         """Cube uses unstructured grid."""
-        return _is_unstructured_grid(self._cube)
+        return has_unstructured_grid(self._cube)
 
     def check_metadata(self, logger: Optional[logging.Logger] = None) -> Cube:
         """Check the cube metadata.

esmvalcore/iris_helpers.py

@@ -10,7 +10,7 @@
 import numpy as np
 from iris.coords import Coord
 from iris.cube import Cube
-from iris.exceptions import CoordinateMultiDimError
+from iris.exceptions import CoordinateMultiDimError, CoordinateNotFoundError
 
 from esmvalcore.typing import NetCDFAttr
 
@@ -268,3 +268,53 @@ def rechunk_cube(
     _rechunk_dim_metadata(cube, complete_dims, remaining_dims=remaining_dims)
 
     return cube
+
+
+def has_irregular_grid(cube: Cube) -> bool:
+    """Check if a cube has an irregular grid.
+
+    Parameters
+    ----------
+    cube:
+        Cube to be checked.
+
+    Returns
+    -------
+    bool
+        ``True`` if input cube has an irregular grid, else ``False``.
+
+    """
+    try:
+        lat = cube.coord('latitude')
+        lon = cube.coord('longitude')
+    except CoordinateNotFoundError:
+        return False
+    if lat.ndim == 2 and lon.ndim == 2:
+        return True
+    return False
+
+
+def has_unstructured_grid(cube: Cube) -> bool:
+    """Check if a cube has an unstructured grid.
+
+    Parameters
+    ----------
+    cube:
+        Cube to be checked.
+
+    Returns
+    -------
+    bool
+        ``True`` if input cube has an unstructured grid, else ``False``.
+
+    """
+    try:
+        lat = cube.coord('latitude')
+        lon = cube.coord('longitude')
+    except CoordinateNotFoundError:
+        return False
+    if lat.ndim != 1 or lon.ndim != 1:
+        return False
+    if cube.coord_dims(lat) != cube.coord_dims(lon):
+        return False
+    return True

esmvalcore/preprocessor/_io.py

@@ -6,7 +6,7 @@
 import os
 from itertools import groupby
 from pathlib import Path
-from typing import Optional, NamedTuple
+from typing import NamedTuple, Optional
 from warnings import catch_warnings, filterwarnings
 
 import cftime