Context and Goals

A first attempt at organising the separation of concerns between science and IT governance of model evaluation (see Fig. 1) has been initiated by the CLIVAR ENSO Research Focus group of experts. Initial discussion at the Dubrovnik CMIP5 conference showed that several model evaluation infrastructures wanted to include ENSO metrics in their software (PMP, ESMValTool, CLIMAF). From the CLIVAR ENSO experts perspective, it soon became clear that programming the same metrics three times should be avoided, both for the sake of efficiency and maintenance, hence the need to define a common science/IT interface.

The process started by defining 3 test science questions that users may want to ask themselves (What is the ENSO performance in historical simulations ? How are ENSO teleconnections represented in historical simulations ? How are ENSO process represented in models ?). These questions may be asked by different types of users: model developers, climate impact experts or ENSO experts. For each question a collection of metrics was defined by the ENSO experts, together with collection-specific requirements (such as length of simulation, period to be used from observations, etc.). Note that a metric may be used by several collections.

Each metrics was then precisely defined with:

• Documentation: whys and how’s of the metric including a published reference to demonstrate the robustness of metric
• Algorithmic definition of the metric (RMSE, averaging area, composite,…)
• Frequency of data needed (daily, monthly, seasonal,…)
• Reference observations/reanalysis (as many as possible), including period to use
• Minimum requirements for metric to make sense (number of members in a ensemble, length of simulation needed,…)
• A list of « dive down » diagnostics to explore in more detail the metric (map of RMS, hovmoellers…)
• Normalisation to perform to share a single colour bar in a multimetric/multimodel view of the results

This pilot implementation of an independent ENSO metrics package which science contents and choices are governed by the CLIVAR ENSO experts, and currently written in python and used by an external PMP driver, offers a first proof of concept of a clear science / IT interface. The current step is to include this package in the ESMValTool software infrastructure, further improving the science/IT interface. This initial attempt will help us understand the concepts for the science governance and associated technical challenges to be respectively addressed by the science community (within WCRP) and by software and data engineers (as initially planned in the European IS-ENES3 effort).

Figure 1: Illustration of the proposed framework for community consensus climate model evaluation. The separation of concerns is organised around three type of actors: climate information users, climate experts and software and data engineers. Recognising that these three types of actors have different scopes, expectations, community organisations and workflow (vertical arrows), the governance of each type and the interfaces between them (grey arrows) need to be articulated.