ESMValTool user configuration files are not ignored by the REF

[schlunma]

The problem

Currently, ESMValTool diagnostics are invoked with the --config-dir option. This will NOT prevent ESMValTool from parsing user configuration files at ~/.config/esmvaltool/ (see here).

This might lead to errors during REF runs or unit tests if the user specified invalid or non-suitable (e.g., additional rootpaths) options in their own configuration file.

Definition of "done"

ESMValTool should be "encapsulated" within the REF, i.e., tests and runs should be independent from potential ESMValTool user configuration files and not fail because of bad ESMValTool user configuration files.

Additional context

This can be solved fairly easily by using the environment variable ESMVALTOOL_CONFIG_DIR instead of the command line option --config_dir. In addition, it would be nice to ensure that potential old configuration files are ignored (requires ESMValGroup/ESMValCore#2736).

[bouweandela]

In some cases, it may be desirable that user configuration is picked up by the esmvaltool executable run by the REF, for example, for configuring Dask. Or would you have better ideas on how we could solve that?

[bouweandela]

In the current set-up, users can use the ESMVALTOOL_CONFIG_DIR environmental variable to point the esmvaltool run by the REF to a different directory than what they would want to use for their plain esmvaltool work.

[bouweandela]

Good point about picking up configuration in the unit tests, I'll see if I can do something about that.

[schlunma]

Maybe that's subjective, but to me it feels weird that the REF uses the default ESMValTool configuration directory by default. The way it works (it gets installed automatically, input and output paths get set up automatically, it is run automatically, etc.), it's very surprising to me that it uses the configuration of my own ESMValTool installation. It would feel more natural if there was a REF-specific ESMValTool configuration that can be adapted (maybe located in the REF config directory). If desired, this can be changed to our default user configuration dir.

If that's not desired, then you need to at least make sure that some specific settings (like rootpath or drs) are ignored from the default ESMValTool config dir. I have the case where I specify a rootpath entry for CMIP6, which then messes up the CMIP6 data finding in the REF. It would be really awkward if one needs to modify configuration files back and forth to use ESMValTool and the REF in parallel.