Monthly ESMValTool meeting November 2021

[bouweandela]

The next monthly ESMValTool meeting will be in November. The result of the poll is that it will be on

Friday, November 5 at 13:00 CET (12:00 GMT).

If none of the proposed times work for you, please let us know by commenting below.

The duration will be one hour. Join the zoom meeting.

Calendar invitations to the meeting are sent to the mailing list, join it here.

@ESMValGroup/esmvaltool-developmentteam

Format and agenda

• Anyone with an interest in ESMValTool is welcome to join and ask questions
• We start the meeting by doing a round where everyone gets one minute to introduce themselves and tell what they will be working on in the next month or what they would like help with.
• Using the input from the introduction round, we can decide to talk about specific topics with the entire group or make smaller breakout groups to continue the discussion there.
• If you already know that you would like to bring up a particular topic in the meeting, you can mention the GitHub issue on the topic in a comment below. That will make it easy to find for other people in the meeting and we can already start discussing there before the meeting. Just create a new GitHub issue about your topic if there is no open issue yet.
• GitHub issues are our preferred way to communicate outside the meeting and keep track of work, so any feature requests/action items/etc discussed in the meeting should ideally result in a GitHub issue.

Notes from the October meeting can be found here: ESMValGroup/ESMValTool#2316

[bouweandela]

In case you're wondering why there are so few options in the doodle: I'm only working Thursdays and Fridays at the moment. If there is a preference for another day, I would need someone else to take over chairing the meeting. Please comment here if you would like to have the meeting on another day of the week.

[senesis]

I won't be able to attend, but would like to know if proposal in Issue 1377 is relevant. It addresses handling extra_facets when testing datafile access

[bouweandela]

Thanks for being there today everyone!

Here are the personal updates:

• @rogerkuou from the Netherlands eScience Center joined. He may be helping us next year improving the dask performance and is also working on a project with ESMValTool together with @SarahAlidoost
• @bettina-gier Plans to port anav recipes from NCL to Python next year
• @valeriupredoi Working on supporting the release
• @rswamina Cross team meeting took place, discussed: communicating and planning backward incompatible changes, keeping the tutorial up to date, gave a talk for ESMS at the Met Office, met with atmospheric chemistry UKESM modeling group that would like to do a tutorial
• @zklaus Working on release, attended cross-team talk, planning work on station data and similar data in new EU horizon project
• @bsolino No update, just here for the fun 🥳
• @schlunma Testing and fixes for the release
• @remi-kazeroni Testing recipes for the release, worked on @esmvalbot, drafted an agenda for the workshop ESMValTool workshop Nov 2021 ESMValTool#2311 together with @hb326 and @axel-lauer.
• @sloosvel Fixed things for the release and working on IS-ENES3 deliverables
• @bouweandela Helped out with running the recipes for the release and is only working Thursdays and Fridays at the moment

Release

Most recipes were tested successfully (ESMValGroup/ESMValTool#2354 (comment)), but missing data is a problem. There appear to be various reasons why data can be missing, it would be good to investigate this and open an issue per problem. The people in the meeting gave @zklaus the green light to go ahead with the release, even if some recipes are not currently working due to various missing input data problems. @bouweandela Recommended creating an issue per broken recipe, labelling it with the 'bug' label, and assigning the recipe maintainers and/or authors, to communicate to the outside world and the recipe maintainers that some work is required here.

Really great to see that so many people in our community are putting in effort to get as many recipes working for the upcoming release as possible! 🎉

IS-ENES3 deliverables

• @slooslvel will continue timerange support in Allow to load all files, first X years or last X years in an experiment ESMValCore#1133 (and maybe merge Generalise clip_timerange to custom time periods. ESMValCore#1214 into it because these pull requests are very intertwined). @bouweandela has limited availability at the moment, so recommends not waiting with this until the end of the year.
• @remi-kazeroni Mentions that it is mostly report writing now

Workshop topics

• @bettina-gier Would like to discuss what to do with CMIP data that has been retracted or even worse, disappeared from ESGF. Should it be kept in recipes for historical correctness or should it be removed so recipes actually work?
• Several people would like to discuss how to improve support for ERA5/big observational datasets at workshop: these are typically too big to download multiple copies. Should we add support for reading the grib files provided centrally on Mistral? Any other thoughts/issues? Some previous discussion too place in Strategy to integrate large observational data sets ESMValTool#1109 and Why do we want to CMORize all observational datasets? ESMValTool#1120. We have a working and documented mechanism to read in such datasets without copying, but it does need to be coded per dataset just like CMORizers: https://docs.esmvaltool.org/projects/esmvalcore/en/latest/develop/fixing_data.html#natively-supported-non-cmip-datasets
• @remi-kazeroni @zklaus @valeriupredoi Would like to discuss what to do with auxiliary data - how should it be provided? @bouweandela Says that the current policy is that the recipe documentation needs to provide instructions for how to obtain this data (example). However, if you want to run many recipes, this can be a time-consuming task. Therefore @remi-kazeroni is maintaining a central copy on Mistral and Jasmin that can be found in the default config-user.yml file.