Add option to load selected or all years available in an experiment

[sloosvel]

Is your feature request related to a problem? Please describe.
In #771 we tried to add the functionality to load all DCPP data without having to specify the start_year and the end_year, as the current time range handling is not ideal (#345). To do so, a new tag ` to load all the data available was introduced. @zklaus pointed out that it could work for all experiments.

Would you be able to help out?
Yes

[bouweandela]

The philosophy behind recipes is that they are as explicit as possible, so e.g. all datasets should be fully written out so it is clear what the recipe actually computes (and with which datasets it actually works).

However, this is not very user-friendly for people writing recipes and can become difficult to read too. See e.g. the requests for the use of wildcards '*' for specifying datasets in ESMValGroup/ESMValTool#671, #589, and #1082. A 'recipe filler' tool was also developed to deal with this.

Maybe we could compromise by allowing this kind of feature, but only accepting recipes that do not contain wildcards in the ESMValTool repository? For easy reproducibility, we could write out a copy of the recipe with all wildcards replaced by actual datasets in the output directory? Any opinions @ESMValGroup/esmvaltool-coreteam?

Maybe this is something for discussion at the next monthly meeting ESMValGroup/ESMValTool#2173 or one of the upcoming @ESMValGroup/technical-lead-development-team or @ESMValGroup/scientific-lead-development-team meetings?

[Peter9192]

Perhaps an another compromise could be to allow wildcards in the original recipe, but then to 'compile' the recipe as soon as it is run and to store that explicit (compiled) version of the recipe with the rest of the output? This way you retain full reproducibility, but you also provide the user-friendliness.

[axel-lauer]

I like @Peter9192 's idea. The recipe is stored already in the output directory anyways and in case of wildcards this could simply be the "compiled" recipe containing all datasets that are acutally used. But I think wildcards should probably be a new run-time option only, i.e. all pull requests are still required to provide a recipe with an explicit list of datasets because otherwise testing might become really difficult if not impossible.

[schlunma]

I also like @Peter9192's idea and fully agree with @axel-lauer here. Using wildcards for quick testing and to get started with a new recipe would be a really nice feature, but the repository should only contain datasets that are fully written out.

[zklaus]

This issue is not about a general wildcard mechanism, though. @sloosvel, perhaps you can elaborate on the use case a little bit? Why is it impossible or very difficult to use the normal start_year, end_year mechanism in this case?

[sloosvel]

Thank you @zklaus. As you mention, it's not so much about the readability of recipes with wildcards. It's just that the current way of specifying start and end years, while clipping the dates from January 1st to December 31st, is not very convenient for certain datasets

Below you can see how files for a DCPP experiment can look like. These are files from the same experiment (dcppA-hindcast). They have an extra sub-experiment tag to indicate the initialization of the run (s1960, s1961, ..., s2018) and each one of these sub-experiments ranges a different time period (November 1960 - October 1971 for s1960, November 1961 - October 1972 for s1961, and so on for all sub-experiments until s2018)

tas_Amon_EC-Earth3_dcppA-hindcast_s1960-r1i1p1f1_gr_196011-196110.nc 
tas_Amon_EC-Earth3_dcppA-hindcast_s1960-r1i1p1f1_gr_196111-196210.nc
tas_Amon_EC-Earth3_dcppA-hindcast_s1960-r1i1p1f1_gr_196211-196310.nc
tas_Amon_EC-Earth3_dcppA-hindcast_s1960-r1i1p1f1_gr_196311-196410.nc
tas_Amon_EC-Earth3_dcppA-hindcast_s1960-r1i1p1f1_gr_196411-196510.nc
tas_Amon_EC-Earth3_dcppA-hindcast_s1960-r1i1p1f1_gr_196511-196610.nc
tas_Amon_EC-Earth3_dcppA-hindcast_s1960-r1i1p1f1_gr_196611-196710.nc 
tas_Amon_EC-Earth3_dcppA-hindcast_s1960-r1i1p1f1_gr_196711-196810.nc
tas_Amon_EC-Earth3_dcppA-hindcast_s1960-r1i1p1f1_gr_196811-196910.nc
tas_Amon_EC-Earth3_dcppA-hindcast_s1960-r1i1p1f1_gr_196911-197010.nc
tas_Amon_EC-Earth3_dcppA-hindcast_s1960-r1i1p1f1_gr_197011-197110.nc
tas_Amon_EC-Earth3_dcppA-hindcast_s1961-r1i1p1f1_gr_196111-196210.nc 
tas_Amon_EC-Earth3_dcppA-hindcast_s1961-r1i1p1f1_gr_196211-196310.nc 
tas_Amon_EC-Earth3_dcppA-hindcast_s1961-r1i1p1f1_gr_196311-196410.nc
tas_Amon_EC-Earth3_dcppA-hindcast_s1961-r1i1p1f1_gr_196411-196510.nc
tas_Amon_EC-Earth3_dcppA-hindcast_s1961-r1i1p1f1_gr_196511-196610.nc
tas_Amon_EC-Earth3_dcppA-hindcast_s1961-r1i1p1f1_gr_196611-196710.nc 
tas_Amon_EC-Earth3_dcppA-hindcast_s1961-r1i1p1f1_gr_196711-196810.nc
tas_Amon_EC-Earth3_dcppA-hindcast_s1961-r1i1p1f1_gr_196811-196910.nc
tas_Amon_EC-Earth3_dcppA-hindcast_s1961-r1i1p1f1_gr_196911-197010.nc 
tas_Amon_EC-Earth3_dcppA-hindcast_s1961-r1i1p1f1_gr_197011-197110.nc  
tas_Amon_EC-Earth3_dcppA-hindcast_s1961-r1i1p1f1_gr_197111-197210.nc  

So what would happen when someone tries to work with the full dataset with ESMValTool? First they would have to define 58 entries in the recipe to call all the sub-experiments:

- {sub_experiment: 's1960', start_year: 1960, end_year: 1971}
- {sub_experiment: 's1961', start_year: 1961, end_year: 1972}

... 56 lines later we are done loading one dataset (in a tool that is meant to compare multiple datasets).

I think it would be much more user friendly to have an interface like that:

-{sub_experiment: 's(1960:2018)', select_years: all}

And what happens if we only want to take into account, let's say, a subset of years from each sub_experiment? (I don't know if this a real-life application)

- {sub_experiment: 's1960', start_year: 1960, end_year: 1963}
- {sub_experiment: 's1961', start_year: 1961, end_year: 1964}

Wouldn't it be more user friendly to do so?

- {sub_experiment: 's(1960:2018)', select_years:  first 3}

But the problem here is another one. The clip_start_end_year function makes the cube lose points because it automatically sets the range to a period that does not align with the period of the dataset:

cube after loading: <iris 'Cube' of air_temperature / (K) (time: 48; latitude: 256; longitude: 512)>
cube after `clip_start_end_year`:  <iris 'Cube' of air_temperature / (K) (time: 38; latitude: 256; longitude: 512)>

So now it's not only a problem of user-friendliness, but rather that a preprocessing function is returning wrong values.

In summary, what I was trying to address here is trying to find a compact way of loading these types of datasets, as well as fixing the clipping of the time ranges. Whether this should be done with wildcards, compiling recipes or not accepting this kind of general recipes in the repository because of readability, I'm all for doing whatever is most convenient to everyone. But the main issue is not this one.

[bouweandela]

Thanks for explaining the feature in more detail @sloosvel, I did not get this from the issue text at the top. I created a new issue about the use of wildcards in #1138 and hid the discussion on the that topic here.

[sloosvel]

I did not get this from the issue text at the top

My bad! I was not explaining things very well.

[bouweandela]

Wouldn't it be more user friendly to do so?

- {sub_experiment: 's(1960:2018)', select_years:  first 3}

I'm not sure if it's a good idea to move preprocessor functionality to the dataset section. Wouldn't it make more sense to create a preprocessor that selects the first or last data from all datasets, if that's something that's required?

But the problem here is another one. The clip_start_end_year function makes the cube lose points because it automatically sets the range to a period that does not align with the period of the dataset:

Would it be possible to set the range to the correct values when expanding the sub_experiment: 's(1960:2018)'? Maybe you could just omit specifying the start and end year?

[sloosvel]

I'm not sure if it's a good idea to move preprocessor functionality to the dataset section. Wouldn't it make more sense to create a preprocessor that selects the first or last data from all datasets, if that's something that's required?

The start_year and end_year tags work in the same way as this tag would work. They load the data looking at the filenames, but the clipping does not happen until the corresponding preprocessing step . So in a way the start_year and end_year tags are also a preprocessor functionality specified in the dataset section.

Would it be possible to set the range to the correct values when expanding the sub_experiment: 's(1960:2018)'? Maybe you could just omit specifying the start and end year?

@zklaus did not let me do that in #771, because we should allow people to work with a subset of years for each sub-experiment. And that means that the clipping should be generalised for these type of cases.

[zklaus]

My point was that there is nothing special about the sub-experiment. I think of every sub-experiment as its own experiment. So if we want to say that leaving out the start year means "take everything from the beginning of the experiment", and leaving out the end year means "take everything to the end of the experiment", that could work, but there is no reason to tie this to the sub-experiment. Indeed, this functionality would be useful also to compare, for example, spin-ups of varying lengths, etc.

So I like the feature very much. I am a bit concerned about using just the absence of the tag as a marker since that seems to invite accidentally voluminous analysis. Hence, my suggestion to have a different syntax in the recipe.

tl;dr
I like the feature. Three questions:

• Do you agree that it isn't really tied to sub-experiments?
• Do we want the feature in general?
• Do we need more expressive syntax?