Recipe to plot generic output from the preprocessor (#2184)

* First try

* Generalize a bit

* Add mapgenerator to requirements

* Fix tests

* Make it work with main

* Fix flake8

* Add provenance

* Update mapgenerator version

* Update recipe to use CMIP6 data

* Fix doc and add more examples

* Some improvements

* Apply yapf

* Fix codacy issues

* Fix some graphical glitches

* Add doc for recipe

* Apply yapf

* Fix doc format

* Improve doc

* Improve doc

* Update monitor

* Add provenance and new default plot path

* Add final dependency on mapgenerator

* Fix doc

* Fix flake8

* Update doc

* Update doc

* Update mapgenerator

* Fix docstrings

* Apply suggestions from code review

Co-authored-by: Valeriu Predoi <[email protected]>

* Comment eofs code

* Improve doc

* Fix style issues

* Update doc/sphinx/source/recipes/recipe_monitor.rst

Co-authored-by: Valeriu Predoi <[email protected]>

* Apply suggestions from code review

Co-authored-by: Manuel Schlund <[email protected]>

* Add entry in FAQ

* Reference list of plots

* Update reference

* Update reference

* Improve doc

Co-authored-by: Javier Vegas-Regidor <[email protected]>
Co-authored-by: Saskia Loosveldt Tomas <[email protected]>
Co-authored-by: sloosvel <[email protected]>
Co-authored-by: Valeriu Predoi <[email protected]>
Co-authored-by: Manuel Schlund <[email protected]>

.gitignore

@@ -85,6 +85,7 @@ test-reports/
 
 # Build folder
 doc/sphinx/build
+doc/sphinx/_build
 doc/sphinx/source/_sidebar.rst.inc
 doc/sphinx/source/gallery.rst
 

doc/sphinx/source/faq.rst

@@ -108,3 +108,11 @@ a symbolic link to it so it gets picked up at every re-run iteration:
 .. way to have this mode run. The information provided will help you obtain any data
 .. that is missing and/or create fixes for the datasets and variables that failed the
 .. CMOR checks and could not be fixed on the fly.
+
+
+Can ESMValTool plot arbitrary model output?
+===========================================
+
+Recipe :ref:`recipe_monitor` allows for the plotting of any preprocessed model.
+The plotting parameters are set through a yaml configuration file, and the
+type of plots to be generated are determined in the recipe. 

doc/sphinx/source/recipes/index.rst

@@ -114,6 +114,7 @@ Other
    recipe_cmorizers
    recipe_ensclus
    recipe_esacci_lst
+   recipe_monitor
    recipe_multimodel_products
    recipe_rainfarm
    recipe_pv_capacity_factor

doc/sphinx/source/recipes/recipe_monitor.rst

@@ -0,0 +1,250 @@
+.. _recipe_monitor:
+
+Monitor
+#######
+
+Overview
+========
+
+Available recipes and diagnostics
+=================================
+
+Recipes are stored in `recipes/`
+
+  - recipe_monitor.yml
+
+Diagnostics are stored in `diag_scripts/monitor/`
+
+  - monitor.py:
+    Plots preprocessor output directly from the preprocessor.
+  - compute_eofs.py:
+    An example on how to use the monitor structure to show other metrics.
+    Computes and plots the map of the first EOF and the associated PC timeseries.
+
+
+List of plot types available in monitor.py
+------------------------------------------
+
+- Climatology (plot_type `clim`): Plots climatology. Supported coordinates:
+  (`latitude`, `longitude`, `month_number`).
+
+- Seasonal climatologies (plot_type `seasonclim`): It produces a multi panel (2x2) plot
+  with the seasonal climatologies. Supported coordinates:
+  (`latitude`, `longitude`, `month_number`).
+
+- Monthly climatologies (plot_type `monclim`): It produces a multi panel (3x4) plot with
+  the monthly climatologies. Can be customized to show only certain months
+  and to rearrange the number of columns and rows. Supported coordinates:
+  (`latitude`, `longitude`, `month_number`).
+
+- Time series (plot_type `timeseries`): Generate time series plots. It will always
+  generate the full period time series, but if the period is longer than 75
+  years, it will also generate two extra time series for the first and last 50
+  years. It will produce multi panel plots for data with `shape_id` or `region`
+  coordinates of length > 1. Supported coordinates: `time`, `shape_id`
+  (optional) and `region` (optional).
+
+- Annual cycle (plot_type `annual_cycle`): Generate an annual cycle plot (timeseries
+  like climatological from January to December). It will produce multi panel
+  plots for data with `shape_id` or `region` coordinates of length > 1.
+  Supported coordinates: `time`, `shape_id` (optional) and `region` (optional).
+
+User settings
+=============
+
+User setting files are stored in recipes and in a dedicated yaml config file.
+
+In the variable definitions, users can set the attribute `plot_name` to fix
+the variable name that will be used for the plot's title. If it is not set,
+mapgenerator will try to choose a sensible one from the standard name
+attributes (`long_name`, `standard_name` and `var_name`).
+
+monitor.py
+----------
+
+  * plots:
+    a dictionary containing the plots to make, with its own options.
+    Available types of plot are listed in section `List of plot types available in monitor.py`_.
+  * cartopy_data_dir:
+    Path to cartopy data dir. Defaults to None.
+    See https://scitools.org.uk/cartopy/docs/latest/cartopy.html.
+  * plot_folder:
+    Path to the folder to store the figures. It is defined as the
+    input paths in ``config-developer.yml``. See
+    https://docs.esmvaltool.org/projects/ESMValCore/en/latest/quickstart/configure.html#input-file-paths
+    for more details. Defaults to ``~/plots/{dataset}/{exp}/{modeling_realm}/{real_name}``.
+  * plot_filename:
+    Filename pattern for the plots. It is defined as the input
+    files in in ``config-developer.yml``. See
+    https://docs.esmvaltool.org/projects/ESMValCore/en/latest/quickstart/configure.html#input-file-paths
+    for more details. Defaults to ``{plot_type}_{real_name}_{dataset}_{mip}_{exp}_{ensemble}``.
+  * config_file:
+    Path to the monitor config file. Defaults to
+    ``monitor_config.yml`` in the same folder as the diagnostic script.
+
+Plot specific options:
+^^^^^^^^^^^^^^^^^^^^^^
+
+- monclim:
+   + maps:
+     List of maps to plot, as defined in the config file. Defaults to ``[global]``.
+   + months:
+     Select only specific months. Defaults to ``None`` (i.e. it does not select any month).
+   + plot_size:
+     Size of each individual figure. Default's to ``(5, 4)``.
+   + columns:
+     Number of columns in the plot. Defaults to ``3``.
+   + rows:
+     Number of rows in the plot. Defaults to ``4``.
+- seasonclim:
+   + maps:
+     List of maps to plot, as defined in the config file. Defaults to ``[global]``.
+- clim:
+   + maps:
+     List of maps to plot, as defined in the config file. Defaults to ``[global]``.
+- annual_cycle: No options.
+- timeseries: No options.
+
+compute_eofs.py
+---------------
+
+  * cartopy_data_dir:
+    Path to cartopy data dir. Defaults to None.
+    See https://scitools.org.uk/cartopy/docs/latest/cartopy.html.
+  * plot_folder:
+    Path to the folder to store the figures. It is defined as the
+    input paths in ``config-developer.yml``. See
+    https://docs.esmvaltool.org/projects/ESMValCore/en/latest/quickstart/configure.html#input-file-paths
+    for more details. Defaults to ``~/plots/{dataset}/{exp}/{modeling_realm}/{real_name}``.
+  * plot_filename:
+    Filename pattern for the plots. It is defined as the input
+    files in in ``config-developer.yml``. See
+    https://docs.esmvaltool.org/projects/ESMValCore/en/latest/quickstart/configure.html#input-file-paths
+    for more details. Defaults to ``{plot_type}_{real_name}_{dataset}_{mip}_{exp}_{ensemble}``.
+  * config_file:
+    Path to the monitor config file. Defaults to
+    ``monitor_config.yml`` in the same folder as the diagnostic script.
+
+.. hint::
+
+   Extra arguments are ignored, so it is safe to use yaml anchors to share the
+   configuration of common arguments with the `monitor.py` diagnostic script.
+
+monitor_config.yml
+------------------
+
+A yaml file containing map and variable specific options.
+
+Contains two dictionaries, ``maps`` and ``variables``.
+
+Each entry in ``maps`` corresponds to a map definition. See below for a sample with
+comments to define each option
+
+.. code-block:: yaml
+
+   maps:
+      global: # Map name, choose a meaningful one
+         projection: PlateCarree # Cartopy projection to use
+         projection_kwargs: # Dictionary with Cartopy's projection keyword arguments.
+            central_longitude: 285
+         smooth: true # If true, interpolate values to get smoother maps. If not, all points in a cells will get the exact same color
+         lon: [-120, -60, 0, 60, 120, 180] # Set longitude ticks
+         lat: [-90, -60, -30, 0, 30, 60, 90] # Set latitude ticks
+         colorbar_location: bottom
+         extent: null # If defined, restrict the projection to a region. Format [lon1, lon2, lat1, lat2]
+         suptitle_pos: 0.87 # Title position in the figure.
+
+Each entry in ``variable`` corresponds to a variable definition.
+Use the default entry to apply generic options to all variables.
+See below a sample with comments to define each option
+
+.. code-block:: yaml
+
+   variables:
+      # Define default. Variable definitions completely override the default
+      # not just the values defined. If you want to override only the defined
+      # values, use yaml anchors as shown
+      default: &default
+         colors: RdYlBu_r # Matplotlib colormap to use for the colorbar
+         N: 20 # Number of map intervals to plot
+         bad: [0.9, 0.9, 0.9] # Color to use when no data
+      pr:
+         <<: *default
+         colors: gist_earth_r
+         # Define bounds of the colorbar, as a list of
+         bounds: 0-10.5,0.5 # Set colorbar bounds, as a list or in the format min-max,interval
+         extend: max # Set extend parameter of mpl colorbar. See https://matplotlib.org/stable/api/_as_gen/matplotlib.pyplot.colorbar.html
+      sos:
+         # If default is defined, entries are treated as map specific option.
+         # Missing values in map definitionas are taken from variable's default
+         # definition
+         default:
+            <<: *default
+            bounds: 25-41,1
+            extend: both
+         arctic:
+            bounds: 25-40,1
+         antarctic:
+            bounds: 30-40,0.5
+      nao: &nao
+         <<: *default
+         extend: both
+         # Variable definitions can override map parameters. Use with caution.
+         bounds: [-0.03, -0.025, -0.02, -0.015, -0.01, -0.005, 0., 0.005, 0.01, 0.015, 0.02, 0.025, 0.03]
+         projection: PlateCarree
+         smooth: true
+         lon: [-90, -60, -30, 0, 30]
+         lat: [20, 40, 60, 80]
+         colorbar_location: bottom
+         suptitle_pos: 0.87
+      sam:
+         <<: *nao
+         lat: [-90, -80, -70, -60, -50]
+         projection: SouthPolarStereo
+         projection_kwargs:
+            central_longitude: 270
+         smooth: true
+         lon: [-120, -60, 0, 60, 120, 180]
+
+Variables
+=========
+
+* Any, but the variables' number of dimensions should match the ones expected by each plot.
+
+Example plots
+=============
+
+.. _fig_climglobal:
+.. figure::  /recipes/figures/monitor/clim.png
+   :align:   center
+   :width:   14cm
+
+Global climatology of tas.
+
+.. _fig_seasonclimglobal:
+.. figure::  /recipes/figures/monitor/seasonclim.png
+   :align:   center
+   :width:   14cm
+
+Seasonal climatology of pr, with a custom colorbar.
+
+.. _fig_monthlyclimglobal:
+.. figure::  /recipes/figures/monitor/monclim.png
+   :align:   center
+   :width:   14cm
+
+Monthly climatology of sivol, only for March and September.
+
+.. _fig_timeseries:
+.. figure::  /recipes/figures/monitor/timeseries.png
+   :align:   center
+   :width:   14cm
+
+Timeseries of Niño 3.4 index, computed directly with the preprocessor.
+
+.. _fig_annual_cycle:
+.. figure::  /recipes/figures/monitor/annualcycle.png
+   :align:   center
+   :width:   14cm
+
+Annual cycle of tas.

esmvaltool/diag_scripts/monitor/__init__.py

@@ -0,0 +1 @@
+"""Model monitoring recipe for ESMValTool."""

esmvaltool/diag_scripts/monitor/compute_eofs.py

@@ -0,0 +1,90 @@
+"""Diagnostic to compute and plot the first EOF of an arbitrary input.
+
+It is part of the monitoring recipe. Imports class MonitorBase and
+class PlotMap from mapgenerator in order to create plots, build
+paths and record provenance.
+"""
+import logging
+
+import iris
+import matplotlib.pyplot as plt
+from eofs.iris import Eof
+from mapgenerator.plotting.plotmap import PlotMap
+
+import esmvaltool.diag_scripts.shared
+from esmvaltool.diag_scripts.monitor.monitor_base import MonitorBase
+from esmvaltool.diag_scripts.shared import group_metadata
+
+logger = logging.getLogger(__name__)
+
+
+class Eofs(MonitorBase):
+    """Diagnostic to compute EOFs and plot them.
+
+    It is also an example on how to derive the monitor class to use its
+    plotting capabilities in diagnostics that can not be done using only
+    the preprocessor.
+    """
+
+    def compute(self):
+        """Compute the diagnostic."""
+        for module in ['matplotlib', 'fiona']:
+            module_logger = logging.getLogger(module)
+            module_logger.setLevel(logging.WARNING)
+
+        data = group_metadata(self.cfg['input_data'].values(), 'alias')
+        # Loop over datasets
+        for alias in data:
+            # Loop over variables
+            variables = group_metadata(data[alias], 'variable_group')
+            for var_name, var_info in variables.items():
+                logger.info('Plotting variable %s', var_name)
+                var_info = var_info[0]
+                # Load variable
+                cube = iris.load_cube(var_info['filename'])
+                # Initialise solver
+                solver = Eof(cube, weights='coslat')
+                # Get variable options as defined in monitor_config.yml
+                variable_options = self._get_variable_options(
+                    var_info['variable_group'], '')
+                # Initialise PlotMap class
+                plot_map = PlotMap(loglevel='INFO')
+                # Compute EOF
+                eof = solver.eofs(neofs=1)[0, ...]
+                # Set metadata
+                eof.long_name = var_info.get('eof_name', eof.long_name)
+                eof.standard_name = None
+                # Plot EOF map using plot_cube from PlotMap
+                plot_map.plot_cube(eof, save=False, **variable_options)
+                # Get filename for the EOF plot
+                filename = self.get_plot_path('eof', var_info, 'png')
+                # Save figure
+                plt.savefig(filename,
+                            bbox_inches='tight',
+                            pad_inches=.2,
+                            dpi=plot_map.dpi)
+                plt.close(plt.gcf())
+                # Record provenance for EOF plot
+                self.record_plot_provenance(filename, var_info, 'eof')
+
+                # Compute PC
+                pcomp = solver.pcs(npcs=1, pcscaling=1)[:, 0]
+                # Set metadata
+                pcomp.long_name = var_info.get('pc_name', pcomp.long_name)
+                pcomp.standard_name = None
+                # Get filename for the PC plot
+                filename = self.get_plot_path('pc', var_info)
+                # Plot PC timeseries using plot_cube from MonitorBase
+                self.plot_cube(pcomp, filename)
+                # Record provenance for the PC plot
+                self.record_plot_provenance(filename, var_info, 'pc')
+
+
+def main():
+    """Run EOFs diagnostic."""
+    with esmvaltool.diag_scripts.shared.run_diagnostic() as config:
+        Eofs(config).compute()
+
+
+if __name__ == "__main__":
+    main()