Failing to load cubes due to cell_method issues

[remi-kazeroni]

🐛 Bug Report

Hi! Until recently, ESMValTool was pinned in a way that we used iris 3.2.1. This has recently been changed (see ESMValGroup/ESMValTool#2945) so that we could use iris 3.4.0 in our environment for a development installation of ESMValTool. After running some recipes using iris 3.4.0, I noticed that some of our CMORized OBS data files (CMIP5-like standards) could not be loaded with iris any more due to cell_method issues. It looks like our files are missing some information, e.g. ts:cell_methods = "time" ; but such files could be loaded with iris 3.2.1.

How To Reproduce

Steps to reproduce the behaviour:

Assuming access to Jasmin:

1. Load the cube

import iris
import.iris("/gws/nopw/j04/esmeval/obsdata-v2/Tier2/HadISST/OBS_HadISST_reanaly_1_Amon_ts_187001-201712.nc")

Expected behaviour

Cube successfully loaded

Screenshots

The following traceback is shown:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/work/bd0854/b381141/mambaforge/envs/esmvaltool-v2.8.0-dev/lib/python3.10/site-packages/iris/__init__.py", line 318, in load
    return _load_collection(uris, constraints, callback).merged().cubes()
  File "/work/bd0854/b381141/mambaforge/envs/esmvaltool-v2.8.0-dev/lib/python3.10/site-packages/iris/__init__.py", line 284, in _load_collection
    result = _CubeFilterCollection.from_cubes(cubes, constraints)
  File "/work/bd0854/b381141/mambaforge/envs/esmvaltool-v2.8.0-dev/lib/python3.10/site-packages/iris/cube.py", line 104, in from_cubes
    for cube in cubes:
  File "/work/bd0854/b381141/mambaforge/envs/esmvaltool-v2.8.0-dev/lib/python3.10/site-packages/iris/__init__.py", line 269, in _generate_cubes
    for cube in iris.io.load_files(part_names, callback, constraints):
  File "/work/bd0854/b381141/mambaforge/envs/esmvaltool-v2.8.0-dev/lib/python3.10/site-packages/iris/io/__init__.py", line 223, in load_files
    for cube in handling_format_spec.handler(
  File "/work/bd0854/b381141/mambaforge/envs/esmvaltool-v2.8.0-dev/lib/python3.10/site-packages/iris/fileformats/netcdf/loader.py", line 574, in load_cubes
    cube = _load_cube(engine, cf, cf_var, filename)
  File "/work/bd0854/b381141/mambaforge/envs/esmvaltool-v2.8.0-dev/lib/python3.10/site-packages/iris/fileformats/netcdf/loader.py", line 287, in _load_cube
    engine.activate()
  File "/work/bd0854/b381141/mambaforge/envs/esmvaltool-v2.8.0-dev/lib/python3.10/site-packages/iris/fileformats/_nc_load_rules/engine.py", line 97, in activate
    run_actions(self)
  File "/work/bd0854/b381141/mambaforge/envs/esmvaltool-v2.8.0-dev/lib/python3.10/site-packages/iris/fileformats/_nc_load_rules/actions.py", line 521, in run_actions
    action_default(engine)  # This should run the default rules.
  File "/work/bd0854/b381141/mambaforge/envs/esmvaltool-v2.8.0-dev/lib/python3.10/site-packages/iris/fileformats/_nc_load_rules/actions.py", line 74, in inner
    rule_name = func(engine, *args, **kwargs)
  File "/work/bd0854/b381141/mambaforge/envs/esmvaltool-v2.8.0-dev/lib/python3.10/site-packages/iris/fileformats/_nc_load_rules/actions.py", line 90, in action_default
    hh.build_cube_metadata(engine)
  File "/work/bd0854/b381141/mambaforge/envs/esmvaltool-v2.8.0-dev/lib/python3.10/site-packages/iris/fileformats/_nc_load_rules/helpers.py", line 216, in build_cube_metadata
    cube.cell_methods = parse_cell_methods(nc_att_cell_methods)
  File "/work/bd0854/b381141/mambaforge/envs/esmvaltool-v2.8.0-dev/lib/python3.10/site-packages/iris/fileformats/netcdf/saver.py", line 283, in parse_cell_methods
    for m in _split_cell_methods(nc_cell_methods):
  File "/work/bd0854/b381141/mambaforge/envs/esmvaltool-v2.8.0-dev/lib/python3.10/site-packages/iris/fileformats/netcdf/saver.py", line 242, in _split_cell_methods
    method_indices.append((name_start_inds[-1], len(nc_cell_methods)))
IndexError: list index out of range

Environment

• OS & Version: Red Hat Enterprise Linux 8.5 (Ootpa)
• Iris Version: '3.4.0': bug, '3.2.1': no issues

Additional context

More info on the cube that can't be loaded with iris 3.4.0

Click to expand this section...

ncdump -h /gws/nopw/j04/esmeval/obsdata-v2/Tier2/HadISST/OBS_HadISST_reanaly_1_Amon_ts_187001-201712.nc
netcdf OBS_HadISST_reanaly_1_Amon_ts_187001-201712 {
dimensions:
	time = UNLIMITED ; // (1776 currently)
	lat = 180 ;
	lon = 360 ;
	bnds = 2 ;
variables:
	double time(time) ;
		time:bounds = "time_bnds" ;
		time:calendar = "gregorian" ;
		time:long_name = "time" ;
		time:axis = "T" ;
		time:units = "days since 1950-01-01 00:00:00" ;
		time:standard_name = "time" ;
	double time_bnds(time, bnds) ;
	double lat(lat) ;
		lat:bounds = "lat_bnds" ;
		lat:long_name = "latitude" ;
		lat:axis = "Y" ;
		lat:units = "degrees_north" ;
		lat:standard_name = "latitude" ;
	double lat_bnds(lat, bnds) ;
	double lon(lon) ;
		lon:standard_name = "longitude" ;
		lon:units = "degrees_east" ;
		lon:axis = "X" ;
		lon:long_name = "longitude" ;
		lon:bounds = "lon_bnds" ;
	double lon_bnds(lon, bnds) ;
	float ts(time, lat, lon) ;
		ts:standard_name = "surface_temperature" ;
		ts:units = "K" ;
		ts:cell_methods = "time" ;
		ts:cell_measures = "area" ;
		ts:long_name = "Surface Temperature" ;
		ts:comment = "\"\"skin\"\" temperature (i.e., SST for open ocean)" ;
		ts:_FillValue = 1.e+20f ;

// global attributes:
		:title = "HadISST data reformatted for the ESMValTool v2.0" ;
		:tier = 2 ;
		:source = "http://www.metoffice.gov.uk/hadobs/hadisst/data/download.html" ;
		:reference = "Rayner et al., J. Geophys. Res., doi:10.1029/2002JD002670, 2013" ;
		:user = "b309057" ;
		:host = "m11517.hpc.dkrz.de" ;
		:history = "Created on Thu Feb 21 16:26:28 CET 2019" ;
		:conventions = "CF/CMOR" ;
}

[pp-mo]

Related to #5067, but I believe not the same

• In that case, Iris couldn't cope with var:cell_methods = "";
• In this case, we have cell_methods of var:cell_methods = "time";,
  but it should be var:cell_methods = "time: <method>"; (for some 'method')

I think our best response to this is the same : we could do with relaxing the Iris load rules to workaround failing cases (while still raising the issue as a warning).
But the file itself is not good CF. To check, I generated a file based on the CDL above, and indeed the cf-checker does complain that the cell-method syntax is invalid.

[pp-mo]

Hi @trexfeathers would you be interested in looking at #5126 ?

[remi-kazeroni]

Update on the ESMValTool side: we have found and fixed a bug that led to the generation of this malformed datasets with incomplete cell_methods. This occurred when CMORizing some obs datasets. The newly CMORized data have a complete var:cell_methods = "time: <method>"; and can be successfully loaded with iris 3.4.0.

It would still be appreciated if Iris would load successfully (with a warning?) such files with incomplete/non-fully compliant cell_methods since this is not always well-defined in obs datasets.

[trexfeathers]

Update on the ESMValTool side: we have found and fixed a bug that led to the generation of this malformed datasets with incomplete cell_methods. This occurred when CMORizing some obs datasets. The newly CMORized data have a complete var:cell_methods = "time: <method>"; and can be successfully loaded with iris 3.4.0.

It would still be appreciated if Iris would load successfully (with a warning?) such files with incomplete/non-fully compliant cell_methods since this is not always well-defined in obs datasets.

Out of interest: if Iris had successfully loaded the malformed dataset, do you think you would still have made this improvement in CF compliance?

This is one of the many questions involved in how we plan Iris' intended loading behaviour.

[remi-kazeroni]

Out of interest: if Iris had successfully loaded the malformed dataset, do you think you would still have made this improvement in CF compliance?

This is one of the many questions involved in how we plan Iris' intended loading behaviour.

That's a good question. I guess it depends on the use cases but here, we would probably not have fixed it since the definition of cell_methods for some obs data can be unclear. With the bug fixed on the ESMValTool side, loading malformed files with incomplete cell_methods is no longer needed for us at the moment. I understand if this issue is closed.

[trexfeathers]

With the bug fixed on the ESMValTool side, loading malformed files with incomplete cell_methods is no longer needed for us at the moment. I understand if this issue is closed.

It's part of a much larger issue about what Iris should and should not load, which we're only just starting to explore. That's why I was curious.