From ece90573dbf5b8c557c1cbc69eb07a55e2b45032 Mon Sep 17 00:00:00 2001 From: rswamina Date: Mon, 1 May 2023 17:36:40 +0100 Subject: [PATCH 01/15] changed version --- _episodes/03-configuration.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_episodes/03-configuration.md b/_episodes/03-configuration.md index a4a3d676..d4c4e9c3 100644 --- a/_episodes/03-configuration.md +++ b/_episodes/03-configuration.md @@ -2,7 +2,7 @@ title: "Configuration" teaching: 10 exercises: 10 -compatibility: ESMValTool v2.6.0 +compatibility: ESMValTool v2.8.0 questions: - What is the user configuration file and how should I use it? From 64c6b0be61be0765db65d07e547b585affbbfaf8 Mon Sep 17 00:00:00 2001 From: rswamina Date: Mon, 1 May 2023 18:19:17 +0100 Subject: [PATCH 02/15] changes to reflect auto download options --- _episodes/03-configuration.md | 5 +++-- data/dataset.urls | 4 +++- setup.md | 5 ++++- 3 files changed, 10 insertions(+), 4 deletions(-) diff --git a/_episodes/03-configuration.md b/_episodes/03-configuration.md index d4c4e9c3..e19f60ee 100644 --- a/_episodes/03-configuration.md +++ b/_episodes/03-configuration.md @@ -153,10 +153,11 @@ download directory and by setting the option to download data as shown below. ```yaml # Directory for storing downloaded climate data download_dir: ~/climate_data -offline: false +search_esgf: always ``` If you are working offline or do not want to download the data then set the -option above to `true`. +option above to `never`. If you want to download data only when the necessary files +are missing at the usual location, you can set the option to `when_missing`. The ``rootpath`` specifies the directories where ESMValTool will look for input data. For each category, you can define either one path or several paths as a list. For example: diff --git a/data/dataset.urls b/data/dataset.urls index 11fe67e8..df9a3ced 100644 --- a/data/dataset.urls +++ b/data/dataset.urls @@ -1,4 +1,6 @@ # Dataset urls required fo tutorial, described at https://esmvalgroup.github.io/ESMValTool_Tutorial/setup.html#using-your-own-machine # For episodes 4 and 5 +http://esgf2.dkrz.de/thredds/fileServer/lta_dataroot/cmip5/output1/BCC/bcc-csm1-1/historical/mon/atmos/Amon/r1i1p1/v1/tas/tas_Amon_bcc-csm1-1_historical_r1i1p1_185001-201212.nc +http://esgf2.dkrz.de/thredds/fileServer/lta_dataroot/cmip5/output1/BCC/bcc-csm1-1/historical/fx/atmos/fx/r0i0p0/v1/areacella/areacella_fx_bcc-csm1-1_historical_r0i0p0.nc http://esgf3.dkrz.de/thredds/fileServer/cmip6/CMIP/BCC/BCC-ESM1/historical/r1i1p1f1/Amon/tas/gn/v20181214/tas_Amon_BCC-ESM1_historical_r1i1p1f1_gn_185001-201412.nc -http://esgf2.dkrz.de/thredds/fileServer/lta_dataroot/cmip5/output1/CCCma/CanESM2/historical/mon/atmos/Amon/r1i1p1/v20120718/tas/tas_Amon_CanESM2_historical_r1i1p1_185001-200512.nc +http://esgf3.dkrz.de/thredds/fileServer/cmip6/CMIP/BCC/BCC-ESM1/1pctCO2/r1i1p1f1/fx/areacella/gn/v20190613/areacella_fx_BCC-ESM1_1pctCO2_r1i1p1f1_gn.nc diff --git a/setup.md b/setup.md index 3600bd76..a581aab0 100644 --- a/setup.md +++ b/setup.md @@ -235,7 +235,7 @@ local machine and go [here](#github-account-advanced). If you are planning on running ESMValTool on your own machine, please make sure that you are able to download CMIP data and that you have several GB of space available to install conda and ESMValTool, but also enough to make a copy of some -data (~2Gb). +data (~125MB). To download the data required by this tutorial, run the following command using [wget](https://en.wikipedia.org/wiki/Wget): @@ -245,6 +245,9 @@ wget --no-clobber --input-file \ https://github.com/ESMValGroup/ESMValTool_Tutorial/raw/main/data/dataset.urls \ --directory-prefix $HOME/esmvaltool_tutorial/data/ ~~~ +You can also use ESMValTool to automatically download data needed for test recipes. +Please see the [Configuration][lesson-configuration] episode or the [(documentation +pages)](https://docs.esmvaltool.org/projects/ESMValCore/en/latest/quickstart/configure.html#user-configuration-file) for more information. ## GitHub account (Advanced) From 099b7d964f53671f7575f2d81f57e3558583710e Mon Sep 17 00:00:00 2001 From: rswamina Date: Mon, 1 May 2023 23:16:23 +0100 Subject: [PATCH 03/15] changes to recipe example --- _episodes/04-recipe.md | 310 ++++++++++++++++++----------------- _includes/example_output.txt | 208 +++++++++++++---------- 2 files changed, 281 insertions(+), 237 deletions(-) diff --git a/_episodes/04-recipe.md b/_episodes/04-recipe.md index 18df8f5d..fd081f78 100644 --- a/_episodes/04-recipe.md +++ b/_episodes/04-recipe.md @@ -2,7 +2,7 @@ title: "Running your first recipe" teaching: 15 exercises: 15 -compatibility: ESMValTool v2.6.0 +compatibility: ESMValTool v2.8.0 questions: - "How to run a recipe?" @@ -36,7 +36,7 @@ We will start by running [examples/recipe_python.yml](https://docs.esmvaltool. org/en/latest/recipes/recipe_examples.html) ``` -esmvaltool run examples/recipe_python.yml +esmvaltool run recipe_python.yml ``` @@ -85,8 +85,7 @@ Let's dissect what's happening here. > > 1. ESMValTool found the recipe in its installation directory, something like > > `/home/users/username/mambaforge/envs/esmvaltool/bin/esmvaltool/recipes/examples/` >> or if you are using a pre-installed module on a server, something like - `/apps/jasmin/community/esmvaltool/ESMValTool_/esmvaltool/recipes -/examples/recipe_python.yml`, where `` is the latest release. + `/apps/jasmin/community/esmvaltool/ESMValTool_/esmvaltool/recipes/examples/recipe_python.yml`, where `` is the latest release. > > 1. ESMValTool creates a time-stamped output directory for every run. In this > > case, it should be something like `recipe_python_YYYYMMDD_HHMMSS`. This > > folder is made inside the output directory specified in the previous @@ -126,12 +125,11 @@ distinguished in the log messages: > > > ## Answer > > -> > Just after 'creating tasks' and before 'executing tasks', we find the +> > Just after all the 'creating tasks' and before 'executing tasks', we find the > > following line in the output: > > > > ``` -> >[190720] INFO These tasks will be executed: map/script1, map/tas, timeseries/tas_amsterdam, - timeseries/script1, timeseries/tas_global +> >[18776] INFO These tasks will be executed: map/tas, timeseries/tas_global, timeseries/script1, map/script1, timeseries/tas_amsterdam > > ``` > > > > So there are three tasks related to timeseries: global temperature, @@ -163,106 +161,115 @@ For reference, you can also view the recipe by unfolding the box below. > ## recipe_python.yml > > ```yaml -> # ESMValTool -> # recipe_python.yml -> --- -> documentation: -> description: | -> Example recipe that plots a map and timeseries of temperature. -> -> title: Recipe that runs an example diagnostic written in Python. -> -> authors: -> - andela_bouwe -> - righi_mattia -> -> maintainer: -> - schlund_manuel -> -> references: -> - acknow_project -> -> projects: -> - esmval -> - c3s-magic -> -> datasets: -> - {dataset: BCC-ESM1, project: CMIP6, exp: historical, ensemble: r1i1p1f1, grid: gn} -> - {dataset: CanESM2, project: CMIP5, exp: historical, ensemble: r1i1p1} -> -> preprocessors: -> -> select_january: -> extract_month: -> month: 1 -> -> annual_mean_amsterdam: -> extract_point: -> latitude: 52.379189 -> longitude: 4.899431 -> scheme: linear -> annual_statistics: -> operator: mean -> multi_model_statistics: -> statistics: -> - mean -> span: overlap -> -> annual_mean_global: -> area_statistics: -> operator: mean -> fx_variables: -> areacella: -> annual_statistics: -> operator: mean -> -> diagnostics: -> -> map: -> description: Global map of temperature in January 2000. -> themes: -> - phys -> realms: -> - atmos -> variables: -> tas: -> mip: Amon -> preprocessor: select_january -> start_year: 2000 -> end_year: 2000 -> scripts: -> script1: -> script: examples/diagnostic.py -> write_netcdf: true -> output_file_type: pdf -> quickplot: -> plot_type: pcolormesh -> cmap: Reds -> -> timeseries: -> description: Annual mean temperature in Amsterdam and global mean since 1850. -> themes: -> - phys -> realms: -> - atmos -> variables: -> tas_amsterdam: -> short_name: tas -> mip: Amon -> preprocessor: annual_mean_amsterdam -> start_year: 1850 -> end_year: 2000 -> tas_global: -> short_name: tas -> mip: Amon -> preprocessor: annual_mean_global -> start_year: 1850 -> end_year: 2000 -> scripts: -> script1: -> script: examples/diagnostic.py -> quickplot: -> plot_type: plot +># ESMValTool +># recipe_python.yml +># +># See https://docs.esmvaltool.org/en/latest/recipes/recipe_examples.html +># for a description of this recipe. +># +># See https://docs.esmvaltool.org/projects/esmvalcore/en/latest/recipe/overview.html +># for a description of the recipe format. +>--- +>documentation: +> description: | +> Example recipe that plots a map and timeseries of temperature. +> +> title: Recipe that runs an example diagnostic written in Python. +> +> authors: +> - andela_bouwe +> - righi_mattia +> +> maintainer: +> - schlund_manuel +> +> references: +> - acknow_project +> +> projects: +> - esmval +> - c3s-magic +> +>datasets: +> - {dataset: BCC-ESM1, project: CMIP6, exp: historical, ensemble: r1i1p1f1, grid: gn} +> - {dataset: bcc-csm1-1, project: CMIP5, exp: historical, ensemble: r1i1p1} +> +>preprocessors: +> # See https://docs.esmvaltool.org/projects/esmvalcore/en/latest/recipe/preprocessor.html +> # for a description of the preprocessor functions. +> +> to_degrees_c: +> convert_units: +> units: degrees_C +> +> annual_mean_amsterdam: +> extract_location: +> location: Amsterdam +> scheme: linear +> annual_statistics: +> operator: mean +> multi_model_statistics: +> statistics: +> - mean +> span: overlap +> convert_units: +> units: degrees_C +> +> annual_mean_global: +> area_statistics: +> operator: mean +> annual_statistics: +> operator: mean +> convert_units: +> units: degrees_C +> +>diagnostics: +> +> map: +> description: Global map of temperature in January 2000. +> themes: +> - phys +> realms: +> - atmos +> variables: +> tas: +> mip: Amon +> preprocessor: to_degrees_c +> timerange: 2000/P1M +> caption: | +> Global map of {long_name} in January 2000 according to {dataset}. +> scripts: +> script1: +> script: examples/diagnostic.py +> quickplot: +> plot_type: pcolormesh +> cmap: Reds +> +> timeseries: +> description: Annual mean temperature in Amsterdam and global mean since 1850. +> themes: +> - phys +> realms: +> - atmos +> variables: +> tas_amsterdam: +> short_name: tas +> mip: Amon +> preprocessor: annual_mean_amsterdam +> timerange: 1850/2000 +> caption: Annual mean {long_name} in Amsterdam according to {dataset}. +> tas_global: +> short_name: tas +> mip: Amon +> preprocessor: annual_mean_global +> timerange: 1850/2000 +> caption: Annual global mean {long_name} according to {dataset}. +> scripts: +> script1: +> script: examples/diagnostic.py +> quickplot: +> plot_type: plot +> > ``` > {: .solution} @@ -288,10 +295,10 @@ Do you recognize the basic recipe structure that was introduced in episode 1? > > ## Answers > > > > 1. The example recipe is written by Bouwe Andela and Mattia Righi. -> > 1. Manual Schlund is listed as the maintainer of this recipe. +> > 1. Manuel Schlund is listed as the maintainer of this recipe. > > 1. Two datasets are analysed: > > - CMIP6 data from the model BCC-ESM1 -> > - CMIP5 data from the model CANESM2 +> > - CMIP5 data from the model bcc-csm1-1 > > 1. The preprocessor `annual_mean_global` computes an area mean as well as > > annual means > > 1. The diagnostic called `map` executes a script referred to as `script1`. @@ -299,7 +306,7 @@ Do you recognize the basic recipe structure that was introduced in episode 1? > > 1. There are two diagnostics: `map` and `timeseries`. Under the diagnostic > > `map` we find two tasks: > > - a preprocessor task called `tas`, applying the preprocessor called -> > `select_january` to the variable `tas`. +> > `to_degrees_c` to the variable `tas`. > > - a diagnostic task called `script1`, applying the script > > `examples/diagnostic.py` to the preprocessed data (`map/tas`). > > @@ -335,14 +342,16 @@ Do you recognize the basic recipe structure that was introduced in episode 1? > > ## Answer > > > > - **map/tas**: creates `/preproc/map/tas`, which contains preprocessed data -> > for each of the input datasets, and a file called `metadata.yml` -> > describing the contents of these datasets. +> > for each of the input datasets, a file called `metadata.yml` +> > describing the contents of these datasets and provenance information in the +>> form of `.xml` files. > > - **timeseries/tas_global**: creates `/preproc/timeseries/tas_global`, which -> > contains preprocessed data for each of the input datasets, and -> > `metadata.yml`. +> > contains preprocessed data for each of the input datasets, a +> > `metadata.yml` file and provenance information in the +>> form of `.xml` files. > > - **timeseries/tas_amsterdam**: creates `/preproc/timeseries/tas_amsterdam`, > > which contains preprocessed data for each of the input datasets, plus a -> > combined `MultiModelMean`, and `metadata.yml`. +> > combined `MultiModelMean`, a`metadata.yml` file and provenance files. > > - **map/script1**: creates `/run/map/script1` with general information and a > > log of the diagnostic script run. It also creates `/plots/map/script1` and > > `/work/map/script1`, which contain output figures and output datasets, @@ -399,42 +408,41 @@ ESMValTool. > > [here](https://en.wikipedia.org/wiki/Diff#Unified_format). > > > > ```diff -> > --- recipe_python.yml -> > +++ recipe_python_london.yml -> > @@ -31,10 +31,10 @@ -> > extract_month: -> > month: 1 -> > -> > - annual_mean_amsterdam: -> > + annual_mean_london: -> > extract_point: -> > - latitude: 52.379189 -> > - longitude: 4.899431 -> > + latitude: 51.5074 -> > + longitude: 0.1278 -> > scheme: linear -> > annual_statistics: -> > operator: mean -> > @@ -73,16 +73,16 @@ -> > cmap: Reds -> > -> > timeseries: -> > - description: Annual mean temperature in Amsterdam and global mean since 1850. -> > + description: Annual mean temperature in London and global mean since 1850. -> > themes: -> > - phys -> > realms: -> > - atmos -> > variables: -> > - tas_amsterdam: -> > + tas_london: -> > short_name: tas -> > mip: Amon -> > - preprocessor: annual_mean_amsterdam -> > + preprocessor: annual_mean_london -> > start_year: 1850 -> > end_year: 2000 -> > tas_global: +>>--- recipe_python.yml +>>+++ recipe_python_london.yml +>>@@ -39,9 +39,9 @@ +>> convert_units: +>> units: degrees_C +>> +>>- annual_mean_amsterdam: +>>+ annual_mean_london: +>> extract_location: +>>- location: Amsterdam +>>+ location: London +>> scheme: linear +>> annual_statistics: +>> operator: mean +>>@@ -83,7 +83,7 @@ +>> cmap: Reds +>> +>> timeseries: +>>- description: Annual mean temperature in Amsterdam and global mean since 1850. +>>+ description: Annual mean temperature in London and global mean since 1850. +>> themes: +>> - phys +>> realms: +>>@@ -92,9 +92,9 @@ +>> tas_amsterdam: +>> short_name: tas +>> mip: Amon +>>- preprocessor: annual_mean_amsterdam +>>+ preprocessor: annual_mean_london +>> timerange: 1850/2000 +>>- caption: Annual mean {long_name} in Amsterdam according to {dataset}. +>>+ caption: Annual mean {long_name} in London according to {dataset}. +>> tas_global: +>> short_name: tas +>> mip: Amon > > ``` > > > {: .solution} diff --git a/_includes/example_output.txt b/_includes/example_output.txt index 3bedaa04..71f5b7c8 100644 --- a/_includes/example_output.txt +++ b/_includes/example_output.txt @@ -1,7 +1,8 @@ > ## Example output > > ``` ->2022-07-29 17:31:48,745 UTC [190720] INFO +> +>2023-05-01 20:59:33,345 UTC [18776] INFO >______________________________________________________________________ > _____ ____ __ ____ __ _ _____ _ > | ____/ ___|| \/ \ \ / /_ _| |_ _|__ ___ | | @@ -10,11 +11,12 @@ > |_____|____/|_| |_| \_/ \__,_|_| |_|\___/ \___/|_| >______________________________________________________________________ > +> >ESMValTool - Earth System Model Evaluation Tool. > >http://www.esmvaltool.org > -> CORE DEVELOPMENT TEAM AND CONTACTS: +>CORE DEVELOPMENT TEAM AND CONTACTS: > Birgit Hassler (Co-PI; DLR, Germany - birgit.hassler@dlr.de) > Alistair Sellar (Co-PI; Met Office, UK - alistair.sellar@metoffice.gov.uk) > Bouwe Andela (Netherlands eScience Center, The Netherlands - b.andela@esciencecenter.nl) @@ -38,90 +40,124 @@ >For further help, please read the documentation at >http://docs.esmvaltool.org. Have fun! > ->2022-07-29 17:31:48,745 UTC [190720] INFO Package versions ->2022-07-29 17:31:48,746 UTC [190720] INFO ---------------- ->2022-07-29 17:31:48,746 UTC [190720] INFO ESMValCore: 2.6.0 ->2022-07-29 17:31:48,746 UTC [190720] INFO ESMValTool: 2.6.0 ->2022-07-29 17:31:48,746 UTC [190720] INFO ---------------- ->2022-07-29 17:31:48,746 UTC [190720] INFO Using config file /home/users/username/esmvaltool-tutorial/config-user.yml ->2022-07-29 17:31:48,746 UTC [190720] INFO Writing program log files to: ->/home/users/username/esmvaltool-tutorial/esmvaltool_output/recipe_python_20220729_173145/run/main_log.txt ->/home/users/username/esmvaltool-tutorial/esmvaltool_output/recipe_python_20220729_173145/run/main_log_debug.txt ->2022-07-29 17:31:48,747 UTC [190720] INFO Starting the Earth System Model Evaluation Tool at time: 2022-07-29 17:31:48 UTC ->2022-07-29 17:31:48,747 UTC [190720] INFO ---------------------------------------------------------------------- ->2022-07-29 17:31:48,747 UTC [190720] INFO RECIPE = /apps/jasmin/community/esmvaltool/ESMValTool_2.6.0/esmvaltool/recipes/examples/recipe_python.yml ->2022-07-29 17:31:48,747 UTC [190720] INFO RUNDIR = /home/users/username/esmvaltool-tutorial/esmvaltool_output/recipe_python_20220729_173145/run ->2022-07-29 17:31:48,747 UTC [190720] INFO WORKDIR = /home/users/username/esmvaltool-tutorial/esmvaltool_output/recipe_python_20220729_173145/work ->2022-07-29 17:31:48,747 UTC [190720] INFO PREPROCDIR = /home/users/username/esmvaltool-tutorial/esmvaltool_output/recipe_python_20220729_173145/preproc ->2022-07-29 17:31:48,747 UTC [190720] INFO PLOTDIR = /home/users/username/esmvaltool-tutorial/esmvaltool_output/recipe_python_20220729_173145/plots ->2022-07-29 17:31:48,747 UTC [190720] INFO ---------------------------------------------------------------------- ->2022-07-29 17:31:48,747 UTC [190720] INFO Running tasks using at most 24 processes ->2022-07-29 17:31:48,747 UTC [190720] INFO If your system hangs during execution, it may not have enough memory for keeping this number of tasks in memory. ->2022-07-29 17:31:48,747 UTC [190720] INFO If you experience memory problems, try reducing 'max_parallel_tasks' in your user configuration file. ->2022-07-29 17:31:48,805 UTC [190720] INFO Creating tasks from recipe ->2022-07-29 17:31:48,805 UTC [190720] INFO Creating tasks for diagnostic map ->2022-07-29 17:31:48,805 UTC [190720] INFO Creating preprocessor task map/tas ->2022-07-29 17:31:48,805 UTC [190720] INFO Creating preprocessor 'select_january' task for variable 'tas' ->2022-07-29 17:31:48,835 UTC [190720] INFO Found input files for CMIP6 ->2022-07-29 17:31:48,900 UTC [190720] INFO Found input files for CMIP5 ->2022-07-29 17:31:48,961 UTC [190720] INFO PreprocessingTask map/tas created. ->2022-07-29 17:31:48,961 UTC [190720] INFO Creating diagnostic task map/script1 ->2022-07-29 17:31:48,962 UTC [190720] INFO Creating tasks for diagnostic timeseries ->2022-07-29 17:31:48,962 UTC [190720] INFO Creating preprocessor task timeseries/tas_amsterdam ->2022-07-29 17:31:48,963 UTC [190720] INFO Creating preprocessor 'annual_mean_amsterdam' task for variable 'tas' ->2022-07-29 17:31:48,969 UTC [190720] INFO Found input files for CMIP6 ->2022-07-29 17:31:49,019 UTC [190720] INFO Found input files for CMIP5 ->2022-07-29 17:31:49,063 UTC [190720] INFO PreprocessingTask timeseries/tas_amsterdam created. ->2022-07-29 17:31:49,064 UTC [190720] INFO Creating preprocessor task timeseries/tas_global ->2022-07-29 17:31:49,064 UTC [190720] INFO Creating preprocessor 'annual_mean_global' task for variable 'tas' ->2022-07-29 17:31:49,065 UTC [190720] WARNING Missing data for fx variable 'areacella' of dataset CMIP6 ->2022-07-29 17:31:49,071 UTC [190720] INFO Found input files for CMIP6 ->2022-07-29 17:31:49,138 UTC [190720] INFO Found input files for CMIP5 ->2022-07-29 17:31:49,183 UTC [190720] INFO PreprocessingTask timeseries/tas_global created. ->2022-07-29 17:31:49,183 UTC [190720] INFO Creating diagnostic task timeseries/script1 ->2022-07-29 17:31:49,184 UTC [190720] INFO These tasks will be executed: map/script1, map/tas, timeseries/tas_amsterdam, timeseries/script1, timeseries/tas_global ->2022-07-29 17:31:49,195 UTC [190720] INFO Running 5 tasks using 5 processes ->2022-07-29 17:31:49,239 UTC [190776] INFO Starting task map/tas in process [190776] ->2022-07-29 17:31:49,240 UTC [190777] INFO Starting task timeseries/tas_amsterdam in process [190777] ->2022-07-29 17:31:49,240 UTC [190778] INFO Starting task timeseries/tas_global in process [190778] ->2022-07-29 17:31:49,335 UTC [190720] INFO Progress: 3 tasks running, 2 tasks waiting for ancestors, 0/5 done ->2022-07-29 17:31:55,735 UTC [190776] INFO Successfully completed task map/tas (priority 0) in 0:00:06.494906 ->2022-07-29 17:31:55,847 UTC [190720] INFO Progress: 2 tasks running, 2 tasks waiting for ancestors, 1/5 done ->2022-07-29 17:31:55,852 UTC [190779] INFO Starting task map/script1 in process [190779] ->2022-07-29 17:31:55,861 UTC [190779] INFO Running command ['/apps/jasmin/community/esmvaltool/miniconda3/envs/esmvaltool/bin/python', '/apps/jasmin/community/esmvaltool/ESMValTool_2.6.0/esmvaltool/diag_scripts/examples/diagnostic.py', '/home/users/username/esmvaltool-tutorial/esmvaltool_output/recipe_python_20220729_173145/run/map/script1/settings.yml'] ->2022-07-29 17:31:55,862 UTC [190779] INFO Writing output to /home/users/username/esmvaltool-tutorial/esmvaltool_output/recipe_python_20220729_173145/work/map/script1 ->2022-07-29 17:31:55,862 UTC [190779] INFO Writing plots to /home/users/username/esmvaltool-tutorial/esmvaltool_output/recipe_python_20220729_173145/plots/map/script1 ->2022-07-29 17:31:55,862 UTC [190779] INFO Writing log to /home/users/username/esmvaltool-tutorial/esmvaltool_output/recipe_python_20220729_173145/run/map/script1/log.txt ->2022-07-29 17:31:55,862 UTC [190779] INFO To re-run this diagnostic script, run: ->cd /home/users/username/esmvaltool-tutorial/esmvaltool_output/recipe_python_20220729_173145/run/map/script1; MPLBACKEND="Agg" /apps/jasmin/community/esmvaltool/miniconda3/envs/esmvaltool/bin/python /apps/jasmin/community/esmvaltool/ESMValTool_2.6.0/esmvaltool/diag_scripts/examples/diagnostic.py /home/users/username/esmvaltool-tutorial/esmvaltool_output/recipe_python_20220729_173145/run/map/script1/settings.yml ->2022-07-29 17:31:55,947 UTC [190720] INFO Progress: 3 tasks running, 1 tasks waiting for ancestors, 1/5 done ->2022-07-29 17:31:58,538 UTC [190777] INFO Generated PreprocessorFile: /home/users/username/esmvaltool-tutorial/esmvaltool_output/recipe_python_20220729_173145/preproc/timeseries/tas_amsterdam/MultiModelMean_Amon_tas_1850-2000.nc ->2022-07-29 17:31:58,762 UTC [190777] INFO Successfully completed task timeseries/tas_amsterdam (priority 2) in 0:00:09.521837 ->2022-07-29 17:31:58,953 UTC [190720] INFO Progress: 2 tasks running, 1 tasks waiting for ancestors, 2/5 done ->2022-07-29 17:31:59,700 UTC [190778] INFO Successfully completed task timeseries/tas_global (priority 3) in 0:00:10.459256 ->2022-07-29 17:31:59,855 UTC [190720] INFO Progress: 1 tasks running, 1 tasks waiting for ancestors, 3/5 done ->2022-07-29 17:31:59,863 UTC [190780] INFO Starting task timeseries/script1 in process [190780] ->2022-07-29 17:31:59,871 UTC [190780] INFO Running command ['/apps/jasmin/community/esmvaltool/miniconda3/envs/esmvaltool/bin/python', '/apps/jasmin/community/esmvaltool/ESMValTool_2.6.0/esmvaltool/diag_scripts/examples/diagnostic.py', '/home/users/username/esmvaltool-tutorial/esmvaltool_output/recipe_python_20220729_173145/run/timeseries/script1/settings.yml'] ->2022-07-29 17:31:59,872 UTC [190780] INFO Writing output to /home/users/username/esmvaltool-tutorial/esmvaltool_output/recipe_python_20220729_173145/work/timeseries/script1 ->2022-07-29 17:31:59,872 UTC [190780] INFO Writing plots to /home/users/username/esmvaltool-tutorial/esmvaltool_output/recipe_python_20220729_173145/plots/timeseries/script1 ->2022-07-29 17:31:59,872 UTC [190780] INFO Writing log to /home/users/username/esmvaltool-tutorial/esmvaltool_output/recipe_python_20220729_173145/run/timeseries/script1/log.txt ->2022-07-29 17:31:59,872 UTC [190780] INFO To re-run this diagnostic script, run: ->cd /home/users/username/esmvaltool-tutorial/esmvaltool_output/recipe_python_20220729_173145/run/timeseries/script1; MPLBACKEND="Agg" /apps/jasmin/community/esmvaltool/miniconda3/envs/esmvaltool/bin/python /apps/jasmin/community/esmvaltool/ESMValTool_2.6.0/esmvaltool/diag_scripts/examples/diagnostic.py /home/users/username/esmvaltool-tutorial/esmvaltool_output/recipe_python_20220729_173145/run/timeseries/script1/settings.yml ->2022-07-29 17:31:59,956 UTC [190720] INFO Progress: 2 tasks running, 0 tasks waiting for ancestors, 3/5 done ->2022-07-29 17:32:01,586 UTC [190779] INFO Successfully completed task map/script1 (priority 1) in 0:00:05.733018 ->2022-07-29 17:32:01,760 UTC [190720] INFO Progress: 1 tasks running, 0 tasks waiting for ancestors, 4/5 done ->2022-07-29 17:32:06,079 UTC [190780] INFO Maximum memory used (estimate): 0.2 GB ->2022-07-29 17:32:06,081 UTC [190780] INFO Sampled every second. It may be inaccurate if short but high spikes in memory consumption occur. ->2022-07-29 17:32:06,760 UTC [190780] INFO Successfully completed task timeseries/script1 (priority 4) in 0:00:06.896972 ->2022-07-29 17:32:06,771 UTC [190720] INFO Progress: 0 tasks running, 0 tasks waiting for ancestors, 5/5 done ->2022-07-29 17:32:06,771 UTC [190720] INFO Successfully completed all tasks. ->2022-07-29 17:32:07,764 UTC [190720] INFO Wrote recipe output to: ->file:///home/users/username/esmvaltool-tutorial/esmvaltool_output/recipe_python_20220729_173145/index.html ->2022-07-29 17:32:07,764 UTC [190720] INFO Ending the Earth System Model Evaluation Tool at time: 2022-07-29 17:32:07 UTC ->2022-07-29 17:32:07,764 UTC [190720] INFO Time for running the recipe was: 0:00:19.017514 ->2022-07-29 17:32:08,702 UTC [190720] INFO Maximum memory used (estimate): 1.3 GB ->2022-07-29 17:32:08,703 UTC [190720] INFO Sampled every second. It may be inaccurate if short but high spikes in memory consumption occur. ->2022-07-29 17:32:08,704 UTC [190720] INFO Run was successful +>2023-05-01 20:59:33,345 UTC [18776] INFO Package versions +>2023-05-01 20:59:33,345 UTC [18776] INFO ---------------- +>2023-05-01 20:59:33,345 UTC [18776] INFO ESMValCore: 2.8.0 +>2023-05-01 20:59:33,346 UTC [18776] INFO ESMValTool: 2.9.0.dev4+g6948d5512 +>2023-05-01 20:59:33,346 UTC [18776] INFO ---------------- +>2023-05-01 20:59:33,346 UTC [18776] INFO Using config file /home/users/username/esmvaltool_tutorial/config-user.yml +>2023-05-01 20:59:33,346 UTC [18776] INFO Writing program log files to: +>/home/users/username/esmvaltool_tutorial/esmvaltool_output/recipe_python_20230501_205933/run/main_log.txt +>/home/users/username/esmvaltool_tutorial/esmvaltool_output/recipe_python_20230501_205933/run/main_log_debug.txt +>2023-05-01 20:59:35,209 UTC [18776] INFO Starting the Earth System Model Evaluation Tool at time: 2023-05-01 20:59:35 UTC +>2023-05-01 20:59:35,210 UTC [18776] INFO ---------------------------------------------------------------------- +>2023-05-01 20:59:35,210 UTC [18776] INFO RECIPE = /apps/jasmin/community/esmvaltool/ESMValTool_2.8.0/esmvaltool/recipes/examples/recipe_python.yml +>2023-05-01 20:59:35,210 UTC [18776] INFO RUNDIR = /home/users/username/esmvaltool_tutorial/esmvaltool_output/recipe_python_20230501_205933/run +>2023-05-01 20:59:35,211 UTC [18776] INFO WORKDIR = /home/users/username/esmvaltool_tutorial/esmvaltool_output/recipe_python_20230501_205933/work +>2023-05-01 20:59:35,211 UTC [18776] INFO PREPROCDIR = /home/users/username/esmvaltool_tutorial/esmvaltool_output/recipe_python_20230501_205933/preproc +>2023-05-01 20:59:35,211 UTC [18776] INFO PLOTDIR = /home/users/username/esmvaltool_tutorial/esmvaltool_output/recipe_python_20230501_205933/plots +>2023-05-01 20:59:35,211 UTC [18776] INFO ---------------------------------------------------------------------- +>2023-05-01 20:59:35,211 UTC [18776] INFO Running tasks using at most 8 processes +>2023-05-01 20:59:35,211 UTC [18776] INFO If your system hangs during execution, it may not have enough memory for keeping this number of tasks in memory. +>2023-05-01 20:59:35,211 UTC [18776] INFO If you experience memory problems, try reducing 'max_parallel_tasks' in your user configuration file. +>2023-05-01 20:59:35,485 UTC [18776] INFO For Dataset: tas, Amon, CMIP6, BCC-ESM1, historical, r1i1p1f1, gn, supplementaries: areacella, *, *, *, *: ignoring supplementary variable 'areacella', unable to expand wildcards 'mip', 'exp', 'ensemble', 'activity', 'institute'. +>2023-05-01 20:59:35,542 UTC [18776] INFO Creating tasks from recipe +>2023-05-01 20:59:35,542 UTC [18776] INFO Creating tasks for diagnostic map +>2023-05-01 20:59:35,542 UTC [18776] INFO Creating diagnostic task map/script1 +>2023-05-01 20:59:35,544 UTC [18776] INFO Creating preprocessor task map/tas +>2023-05-01 20:59:35,544 UTC [18776] INFO Creating preprocessor 'to_degrees_c' task for variable 'tas' +>2023-05-01 20:59:35,565 UTC [18776] INFO Found input files for Dataset: tas, Amon, CMIP6, BCC-ESM1, CMIP, historical, r1i1p1f1, gn, v20181214 +>2023-05-01 20:59:35,571 UTC [18776] INFO Found input files for Dataset: tas, Amon, CMIP5, bcc-csm1-1, historical, r1i1p1, v1 +>2023-05-01 20:59:35,573 UTC [18776] INFO PreprocessingTask map/tas created. +>2023-05-01 20:59:35,574 UTC [18776] INFO Creating tasks for diagnostic timeseries +>2023-05-01 20:59:35,574 UTC [18776] INFO Creating diagnostic task timeseries/script1 +>2023-05-01 20:59:35,574 UTC [18776] INFO Creating preprocessor task timeseries/tas_amsterdam +>2023-05-01 20:59:35,574 UTC [18776] INFO Creating preprocessor 'annual_mean_amsterdam' task for variable 'tas_amsterdam' +>2023-05-01 20:59:35,583 UTC [18776] INFO Found input files for Dataset: tas, Amon, CMIP6, BCC-ESM1, CMIP, historical, r1i1p1f1, gn, v20181214 +>2023-05-01 20:59:35,586 UTC [18776] INFO Found input files for Dataset: tas, Amon, CMIP5, bcc-csm1-1, historical, r1i1p1, v1 +>2023-05-01 20:59:35,588 UTC [18776] INFO PreprocessingTask timeseries/tas_amsterdam created. +>2023-05-01 20:59:35,589 UTC [18776] INFO Creating preprocessor task timeseries/tas_global +>2023-05-01 20:59:35,589 UTC [18776] INFO Creating preprocessor 'annual_mean_global' task for variable 'tas_global' +>2023-05-01 20:59:35,589 UTC [18776] WARNING Preprocessor function area_statistics works best when at least one supplementary variable of ['areacella', 'areacello'] is defined in the recipe for Dataset: +>{'diagnostic': 'timeseries', +> 'variable_group': 'tas_global', +> 'dataset': 'BCC-ESM1', +> 'project': 'CMIP6', +> 'mip': 'Amon', +> 'short_name': 'tas', +> 'activity': 'CMIP', +> 'alias': 'CMIP6', +> 'caption': 'Annual global mean {long_name} according to {dataset}.', +> 'ensemble': 'r1i1p1f1', +> 'exp': 'historical', +> 'frequency': 'mon', +> 'grid': 'gn', +> 'institute': ['BCC'], +> 'long_name': 'Near-Surface Air Temperature', +> 'modeling_realm': ['atmos'], +> 'original_short_name': 'tas', +> 'preprocessor': 'annual_mean_global', +> 'recipe_dataset_index': 0, +> 'standard_name': 'air_temperature', +> 'timerange': '1850/2000', +> 'units': 'K'} +>session: 'recipe_python_20230501_205933'. +>2023-05-01 20:59:35,598 UTC [18776] INFO Found input files for Dataset: tas, Amon, CMIP6, BCC-ESM1, CMIP, historical, r1i1p1f1, gn, v20181214 +>2023-05-01 20:59:35,603 UTC [18776] INFO Found input files for Dataset: tas, Amon, CMIP5, bcc-csm1-1, historical, r1i1p1, v1, supplementaries: areacella, fx, 1pctCO2, r0i0p0 +>2023-05-01 20:59:35,604 UTC [18776] INFO PreprocessingTask timeseries/tas_global created. +>2023-05-01 20:59:35,605 UTC [18776] INFO These tasks will be executed: map/tas, timeseries/tas_global, timeseries/script1, map/script1, timeseries/tas_amsterdam +>2023-05-01 20:59:35,623 UTC [18776] INFO Wrote recipe with version numbers and wildcards to: +>file:///home/users/username/esmvaltool_tutorial/esmvaltool_output/recipe_python_20230501_205933/run/recipe_python_filled.yml +>2023-05-01 20:59:35,623 UTC [18776] INFO Running 5 tasks using 5 processes +>2023-05-01 20:59:35,682 UTC [18794] INFO Starting task map/tas in process [18794] +>2023-05-01 20:59:35,682 UTC [18795] INFO Starting task timeseries/tas_amsterdam in process [18795] +>2023-05-01 20:59:35,684 UTC [18796] INFO Starting task timeseries/tas_global in process [18796] +>2023-05-01 20:59:35,772 UTC [18776] INFO Progress: 3 tasks running, 2 tasks waiting for ancestors, 0/5 done +>2023-05-01 20:59:36,762 UTC [18795] INFO Extracting data for Amsterdam, Noord-Holland, Nederland (52.3730796 °N, 4.8924534 °E) +>2023-05-01 20:59:37,685 UTC [18794] INFO Successfully completed task map/tas (priority 1) in 0:00:02.002836 +>2023-05-01 20:59:37,880 UTC [18776] INFO Progress: 2 tasks running, 2 tasks waiting for ancestors, 1/5 done +>2023-05-01 20:59:37,892 UTC [18797] INFO Starting task map/script1 in process [18797] +>2023-05-01 20:59:37,903 UTC [18797] INFO Running command ['/apps/jasmin/community/esmvaltool/miniconda3_py310_23.1.0/envs/esmvaltool/bin/python', '/apps/jasmin/community/esmvaltool/ESMValTool_2.8.0/esmvaltool/diag_scripts/examples/diagnostic.py', '/home/users/username/esmvaltool_tutorial/esmvaltool_output/recipe_python_20230501_205933/run/map/script1/settings.yml'] +>2023-05-01 20:59:37,904 UTC [18797] INFO Writing output to /home/users/username/esmvaltool_tutorial/esmvaltool_output/recipe_python_20230501_205933/work/map/script1 +>2023-05-01 20:59:37,904 UTC [18797] INFO Writing plots to /home/users/username/esmvaltool_tutorial/esmvaltool_output/recipe_python_20230501_205933/plots/map/script1 +>2023-05-01 20:59:37,904 UTC [18797] INFO Writing log to /home/users/username/esmvaltool_tutorial/esmvaltool_output/recipe_python_20230501_205933/run/map/script1/log.txt +>2023-05-01 20:59:37,904 UTC [18797] INFO To re-run this diagnostic script, run: +>cd /home/users/username/esmvaltool_tutorial/esmvaltool_output/recipe_python_20230501_205933/run/map/script1; MPLBACKEND="Agg" /apps/jasmin/community/esmvaltool/miniconda3_py310_23.1.0/envs/esmvaltool/bin/python /apps/jasmin/community/esmvaltool/ESMValTool_2.8.0/esmvaltool/diag_scripts/examples/diagnostic.py /home/users/username/esmvaltool_tutorial/esmvaltool_output/recipe_python_20230501_205933/run/map/script1/settings.yml +>2023-05-01 20:59:37,980 UTC [18776] INFO Progress: 3 tasks running, 1 tasks waiting for ancestors, 1/5 done +>2023-05-01 20:59:38,992 UTC [18795] INFO Extracting data for Amsterdam, Noord-Holland, Nederland (52.3730796 °N, 4.8924534 °E) +>2023-05-01 20:59:40,791 UTC [18795] INFO Generated PreprocessorFile: /home/users/username/esmvaltool_tutorial/esmvaltool_output/recipe_python_20230501_205933/preproc/timeseries/tas_amsterdam/MultiModelMean_historical_Amon_tas_1850-2000.nc +>2023-05-01 20:59:41,059 UTC [18795] INFO Successfully completed task timeseries/tas_amsterdam (priority 3) in 0:00:05.375697 +>2023-05-01 20:59:41,189 UTC [18776] INFO Progress: 2 tasks running, 1 tasks waiting for ancestors, 2/5 done +>2023-05-01 20:59:41,417 UTC [18796] INFO Successfully completed task timeseries/tas_global (priority 4) in 0:00:05.732484 +>2023-05-01 20:59:41,590 UTC [18776] INFO Progress: 1 tasks running, 1 tasks waiting for ancestors, 3/5 done +>2023-05-01 20:59:41,602 UTC [18798] INFO Starting task timeseries/script1 in process [18798] +>2023-05-01 20:59:41,611 UTC [18798] INFO Running command ['/apps/jasmin/community/esmvaltool/miniconda3_py310_23.1.0/envs/esmvaltool/bin/python', '/apps/jasmin/community/esmvaltool/ESMValTool_2.8.0/esmvaltool/diag_scripts/examples/diagnostic.py', '/home/users/username/esmvaltool_tutorial/esmvaltool_output/recipe_python_20230501_205933/run/timeseries/script1/settings.yml'] +>2023-05-01 20:59:41,612 UTC [18798] INFO Writing output to /home/users/username/esmvaltool_tutorial/esmvaltool_output/recipe_python_20230501_205933/work/timeseries/script1 +>2023-05-01 20:59:41,612 UTC [18798] INFO Writing plots to /home/users/username/esmvaltool_tutorial/esmvaltool_output/recipe_python_20230501_205933/plots/timeseries/script1 +>2023-05-01 20:59:41,612 UTC [18798] INFO Writing log to /home/users/username/esmvaltool_tutorial/esmvaltool_output/recipe_python_20230501_205933/run/timeseries/script1/log.txt +>2023-05-01 20:59:41,612 UTC [18798] INFO To re-run this diagnostic script, run: +>cd /home/users/username/esmvaltool_tutorial/esmvaltool_output/recipe_python_20230501_205933/run/timeseries/script1; MPLBACKEND="Agg" /apps/jasmin/community/esmvaltool/miniconda3_py310_23.1.0/envs/esmvaltool/bin/python /apps/jasmin/community/esmvaltool/ESMValTool_2.8.0/esmvaltool/diag_scripts/examples/diagnostic.py /home/users/username/esmvaltool_tutorial/esmvaltool_output/recipe_python_20230501_205933/run/timeseries/script1/settings.yml +>2023-05-01 20:59:41,691 UTC [18776] INFO Progress: 2 tasks running, 0 tasks waiting for ancestors, 3/5 done +>2023-05-01 20:59:46,069 UTC [18797] INFO Maximum memory used (estimate): 0.3 GB +>2023-05-01 20:59:46,072 UTC [18797] INFO Sampled every second. It may be inaccurate if short but high spikes in memory consumption occur. +>2023-05-01 20:59:47,192 UTC [18797] INFO Successfully completed task map/script1 (priority 0) in 0:00:09.298761 +>2023-05-01 20:59:47,306 UTC [18776] INFO Progress: 1 tasks running, 0 tasks waiting for ancestors, 4/5 done +>2023-05-01 20:59:49,777 UTC [18798] INFO Maximum memory used (estimate): 0.3 GB +>2023-05-01 20:59:49,780 UTC [18798] INFO Sampled every second. It may be inaccurate if short but high spikes in memory consumption occur. +>2023-05-01 20:59:50,866 UTC [18798] INFO Successfully completed task timeseries/script1 (priority 2) in 0:00:09.263219 +>2023-05-01 20:59:50,914 UTC [18776] INFO Progress: 0 tasks running, 0 tasks waiting for ancestors, 5/5 done +>2023-05-01 20:59:50,914 UTC [18776] INFO Successfully completed all tasks. +>2023-05-01 20:59:50,959 UTC [18776] INFO Wrote recipe with version numbers and wildcards to: +>file:///home/users/username/esmvaltool_tutorial/esmvaltool_output/recipe_python_20230501_205933/run/recipe_python_filled.yml +>2023-05-01 20:59:51,233 UTC [18776] INFO Wrote recipe output to: +>file:///home/users/username/esmvaltool_tutorial/esmvaltool_output/recipe_python_20230501_205933/index.html +>2023-05-01 20:59:51,233 UTC [18776] INFO Ending the Earth System Model Evaluation Tool at time: 2023-05-01 20:59:51 UTC +>2023-05-01 20:59:51,233 UTC [18776] INFO Time for running the recipe was: 0:00:16.023887 +>2023-05-01 20:59:51,787 UTC [18776] INFO Maximum memory used (estimate): 1.7 GB +>2023-05-01 20:59:51,788 UTC [18776] INFO Sampled every second. It may be inaccurate if short but high spikes in memory consumption occur. +>2023-05-01 20:59:51,790 UTC [18776] INFO Removing `preproc` directory containing preprocessed data +>2023-05-01 20:59:51,790 UTC [18776] INFO If this data is further needed, then set `remove_preproc_dir` to `false` in your user configuration file +>2023-05-01 20:59:51,827 UTC [18776] INFO Run was successful > ``` > {: .solution} From 09eb25f9f7e0c04e6c58c5a14a4fb5dffb7028f3 Mon Sep 17 00:00:00 2001 From: rswamina Date: Mon, 1 May 2023 23:37:01 +0100 Subject: [PATCH 04/15] added info on location and timerange --- _episodes/04-recipe.md | 15 +++++++++++++-- _includes/links.md | 2 ++ 2 files changed, 15 insertions(+), 2 deletions(-) diff --git a/_episodes/04-recipe.md b/_episodes/04-recipe.md index fd081f78..704a0613 100644 --- a/_episodes/04-recipe.md +++ b/_episodes/04-recipe.md @@ -291,6 +291,8 @@ Do you recognize the basic recipe structure that was introduced in episode 1? > 1. What does the preprocessor called `annual_mean_global` do? > 1. Which script is applied for the diagnostic called `map`? > 1. Can you link specific lines in the recipe to the tasks that we saw before? +> 1. How is the location of the city specified? +> 1. How is the temporal range of the data specified? > > > ## Answers > > @@ -319,6 +321,15 @@ Do you recognize the basic recipe structure that was introduced in episode 1? > > `examples/diagnostic.py` to the preprocessed data > > (`timeseries/tas_global` and `timeseries/tas_amsterdam`). > > +>> +>> 1. The `extract_location` preprocessor is used to get data for a specific location +>> here. ESMValTool interpolates to the location based on the chosen scheme. +>> Can you tell the scheme used here? For more ways to extract areas, see the +>> [Area operations][preproc-area-manipulation] page. +>> 1. The `timerange` tag is used to extract data from a specific time period here. +>> The start time is `01/01/2000` and the span of time to calculate means is +>> `1 Month` given by `P1M`. For more options on how to specify time ranges, +>> see the [timerange documentation][timeranges]. > {: .solution} {: .challenge} @@ -395,14 +406,14 @@ ESMValTool. > > > ## Solution > > -> > In principle, you only have to modify the latitude and longitude coordinates +> > In principle, you only have to modify the location > > in the preprocessor called `annual_mean_amsterdam`. However, it is good > > practice to also replace all instances of `amsterdam` with the correct name > > of your location. Otherwise the log messages and output will be confusing. > > You are free to modify the names of preprocessors or diagnostics. > > > > In the `diff` file below you will see the changes we have made to the file. -> > The top 2 lines are the filenames and the lines like `@@ -31,10 +31,10 @@` +> > The top 2 lines are the filenames and the lines like `@@ -39,9 +39,9 @@` > > represent the line numbers in the original and modified file, respectively. > > For more info on this format, see > > [here](https://en.wikipedia.org/wiki/Diff#Unified_format). diff --git a/_includes/links.md b/_includes/links.md index 2af951eb..f73eab21 100644 --- a/_includes/links.md +++ b/_includes/links.md @@ -46,6 +46,7 @@ [osi]: https://opensource.org [pandoc]: https://pandoc.org/ [paper-now]: https://github.com/PeerJ/paper-now +[preproc-area-manipulation]: https://docs.esmvaltool.org/projects/ESMValCore/en/latest/recipe/preprocessor.html#area-operations [provenance]: https://docs.esmvaltool.org/en/latest/community/diagnostic.html?highlight=provenance#recording-provenance [python-gapminder]: https://swcarpentry.github.io/python-novice-gapminder/ [pyyaml]: https://pypi.python.org/pypi/PyYAML @@ -62,5 +63,6 @@ [theatoga.nc]: http://esgf-data1.ceda.ac.uk/thredds/fileServer/esg_dataroot/cmip5/output1/MOHC/HadGEM2-ES/historical/mon/ocean/Omon/r1i1p1/v20110916/thetaoga/thetaoga_Omon_HadGEM2-ES_historical_r1i1p1_185912-200512.nc [token-authentication-requirements]: https://github.blog/2020-12-15-token-authentication-requirements-for-git-operations/#what-you-need-to-do-today [training]: https://carpentries.github.io/instructor-training/ +[timeranges]: https://docs.esmvaltool.org/projects/ESMValCore/en/latest/recipe/overview.html#time-ranges [workshop-repo]: {{ site.workshop_repo }} [yaml]: http://yaml.org/ From e6cdf4cc362fcff8a5945b73a4950881a49819d7 Mon Sep 17 00:00:00 2001 From: rswamina Date: Mon, 1 May 2023 23:41:36 +0100 Subject: [PATCH 05/15] fixed esmvaltool version and removed python version --- _episodes/02-installation.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/_episodes/02-installation.md b/_episodes/02-installation.md index b4d234dc..7bdfd9f9 100644 --- a/_episodes/02-installation.md +++ b/_episodes/02-installation.md @@ -2,7 +2,7 @@ title: "Installation" teaching: 10 exercises: 10 -compatibility: ESMValTool v2.6.0 +compatibility: ESMValTool v2.8.0 questions: - "What are the prerequisites for installing ESMValTool?" - "How do I confirm that the installation was successful?" @@ -123,7 +123,7 @@ For the tutorial, we will install the complete package. Thus, to install the ESMValTool package, run ```bash -mamba create --name esmvaltool esmvaltool 'python=3.10' +mamba create --name esmvaltool esmvaltool ``` On MacOSX ESMValTool functionalities in Julia, NCL, and R are not supported. To install From 1b0abd258292d713b822a80db1fd18592e1c7a28 Mon Sep 17 00:00:00 2001 From: rswamina Date: Mon, 1 May 2023 23:46:52 +0100 Subject: [PATCH 06/15] fixed esmvaltool version and some broken links --- _episodes/05-conclusions.md | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/_episodes/05-conclusions.md b/_episodes/05-conclusions.md index a4b84857..0af107ca 100644 --- a/_episodes/05-conclusions.md +++ b/_episodes/05-conclusions.md @@ -2,7 +2,7 @@ title: "Conclusion of the basic tutorial" teaching: 10 exercises: 0 -compatibility: ESMValTool v2.6.0 +compatibility: ESMValTool v2.8.0 questions: - "What do I do now?" @@ -61,7 +61,7 @@ to be suitable for use by ESMValTool. > > - [Documenation](https://docs.esmvaltool.org) > - [ESMValTool home page](https://www.esmvaltool.org/) -> - [Papers](https://www.esmvaltool.org/references.html) +> - [Papers](https://esmvaltool.org/references/) > - [Source code (ESMValTool)](https://github.com/ESMValGroup/ESMValTool) > - [Source code (ESMValCore )](https://github.com/ESMValGroup/ESMValCore) {: .callout} @@ -77,7 +77,8 @@ is a good place to learn about general issues, or to see if your question has If you have a GitHub account, you can also post your questions on the page. If you get stuck, a great starting point is to create a -[new issue](https://github.com/ESMValGroup/ESMValTool/issues/new/choose). +[new issue](https://github.com/ESMValGroup/ESMValTool/issues/new/choose). Note +that you will need a GitHub account for this. There is also an ESMValTool email list. Please see From 881e4a9f9cc259dacea2d385bc6358cd221e71d3 Mon Sep 17 00:00:00 2001 From: rswamina Date: Tue, 2 May 2023 00:03:02 +0100 Subject: [PATCH 07/15] version change and fixed a link in the keypoint section but could do that better --- _episodes/01-quickstart.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/_episodes/01-quickstart.md b/_episodes/01-quickstart.md index 209a42f2..d753d2ad 100644 --- a/_episodes/01-quickstart.md +++ b/_episodes/01-quickstart.md @@ -2,7 +2,7 @@ title: "Quickstart guide" teaching: 2 exercises: 8 -compatibility: ESMValTool v2.6.0 +compatibility: ESMValTool v2.8.0 questions: @@ -24,7 +24,7 @@ keypoints: to run ESMValTool as quickly as possible without having to go through the whole tutorial" - "Use the `module load` command to load the ESMValTool environment, - see the [installation]({{ page.root }}{% link _episodes/02-installation.md %}) + see the [Installation](https://esmvalgroup.github.io/ESMValTool_Tutorial/02-installation/index.html) episode for more details and use `esmvaltool --help` to check the ESMValTool environment" - "Use `esmvaltool config get_config_user` to create the ESMValTool user From 254ac11ad4c1a50b6232e1f032b52c370c13044a Mon Sep 17 00:00:00 2001 From: rswamina Date: Tue, 2 May 2023 00:17:45 +0100 Subject: [PATCH 08/15] changed line lengths --- _episodes/04-recipe.md | 9 ++++++--- setup.md | 4 ++-- 2 files changed, 8 insertions(+), 5 deletions(-) diff --git a/_episodes/04-recipe.md b/_episodes/04-recipe.md index 704a0613..533b7fcd 100644 --- a/_episodes/04-recipe.md +++ b/_episodes/04-recipe.md @@ -82,10 +82,12 @@ Let's dissect what's happening here. > > 1. The config file should be the one we edited in the previous episode, > > something like `/home//.esmvaltool/config-user.yml` or `~/esmvaltool_tutorial/config-user.yml`. -> > 1. ESMValTool found the recipe in its installation directory, something like +> > 1. ESMValTool found the recipe in its installation directory, +>> something like > > `/home/users/username/mambaforge/envs/esmvaltool/bin/esmvaltool/recipes/examples/` >> or if you are using a pre-installed module on a server, something like - `/apps/jasmin/community/esmvaltool/ESMValTool_/esmvaltool/recipes/examples/recipe_python.yml`, where `` is the latest release. +>> `/apps/jasmin/community/esmvaltool/ESMValTool_/esmvaltool/recipes/examples/recipe_python.yml`, +>> where `` is the latest release. > > 1. ESMValTool creates a time-stamped output directory for every run. In this > > case, it should be something like `recipe_python_YYYYMMDD_HHMMSS`. This > > folder is made inside the output directory specified in the previous @@ -129,7 +131,8 @@ distinguished in the log messages: > > following line in the output: > > > > ``` -> >[18776] INFO These tasks will be executed: map/tas, timeseries/tas_global, timeseries/script1, map/script1, timeseries/tas_amsterdam +> >[18776] INFO These tasks will be executed: map/tas, timeseries/tas_global, +>>timeseries/script1, map/script1, timeseries/tas_amsterdam > > ``` > > > > So there are three tasks related to timeseries: global temperature, diff --git a/setup.md b/setup.md index a581aab0..8d3650d2 100644 --- a/setup.md +++ b/setup.md @@ -246,8 +246,8 @@ wget --no-clobber --input-file \ --directory-prefix $HOME/esmvaltool_tutorial/data/ ~~~ You can also use ESMValTool to automatically download data needed for test recipes. -Please see the [Configuration][lesson-configuration] episode or the [(documentation -pages)](https://docs.esmvaltool.org/projects/ESMValCore/en/latest/quickstart/configure.html#user-configuration-file) for more information. +Please see the [Configuration][lesson-configuration] episode or the [documentation +pages](https://docs.esmvaltool.org/projects/ESMValCore/en/latest/quickstart/configure.html#user-configuration-file) for more information. ## GitHub account (Advanced) From cf5c2b609881b79a581dd2036ce288624185f295 Mon Sep 17 00:00:00 2001 From: rswamina Date: Tue, 2 May 2023 00:26:49 +0100 Subject: [PATCH 09/15] changed line lengths --- _episodes/04-recipe.md | 3 ++- _includes/links.md | 1 + setup.md | 4 ++-- 3 files changed, 5 insertions(+), 3 deletions(-) diff --git a/_episodes/04-recipe.md b/_episodes/04-recipe.md index 533b7fcd..c471e597 100644 --- a/_episodes/04-recipe.md +++ b/_episodes/04-recipe.md @@ -86,7 +86,8 @@ Let's dissect what's happening here. >> something like > > `/home/users/username/mambaforge/envs/esmvaltool/bin/esmvaltool/recipes/examples/` >> or if you are using a pre-installed module on a server, something like ->> `/apps/jasmin/community/esmvaltool/ESMValTool_/esmvaltool/recipes/examples/recipe_python.yml`, +>> `/apps/jasmin/community/esmvaltool/ESMValTool_ +>>/esmvaltool/recipes/examples/recipe_python.yml`, >> where `` is the latest release. > > 1. ESMValTool creates a time-stamped output directory for every run. In this > > case, it should be something like `recipe_python_YYYYMMDD_HHMMSS`. This diff --git a/_includes/links.md b/_includes/links.md index f73eab21..fc512f82 100644 --- a/_includes/links.md +++ b/_includes/links.md @@ -7,6 +7,7 @@ [coc-reporting]: https://docs.carpentries.org/topic_folders/policies/incident-reporting.html [coc]: https://docs.carpentries.org/topic_folders/policies/code-of-conduct.html [concept-maps]: https://carpentries.github.io/instructor-training/05-memory/ +[config-file]: https://docs.esmvaltool.org/projects/ESMValCore/en/latest/quickstart/configure.html [contrib-covenant]: https://contributor-covenant.org/ [contributing]: {{ repo_url }}/blob/{{ source_branch }}/CONTRIBUTING.md [cran-checkpoint]: https://cran.r-project.org/package=checkpoint diff --git a/setup.md b/setup.md index 8d3650d2..13f32ca7 100644 --- a/setup.md +++ b/setup.md @@ -246,8 +246,8 @@ wget --no-clobber --input-file \ --directory-prefix $HOME/esmvaltool_tutorial/data/ ~~~ You can also use ESMValTool to automatically download data needed for test recipes. -Please see the [Configuration][lesson-configuration] episode or the [documentation -pages](https://docs.esmvaltool.org/projects/ESMValCore/en/latest/quickstart/configure.html#user-configuration-file) for more information. +Please see the [Configuration][lesson-configuration] episode +or the [configuration file documentation][config-file] for more information. ## GitHub account (Advanced) From e66125db1c6a434fbbca2b477f1078438d26267b Mon Sep 17 00:00:00 2001 From: rswamina Date: Tue, 2 May 2023 10:25:18 +0100 Subject: [PATCH 10/15] links checked and blank callout box removed --- _episodes/01-quickstart.md | 2 -- index.md | 2 +- 2 files changed, 1 insertion(+), 3 deletions(-) diff --git a/_episodes/01-quickstart.md b/_episodes/01-quickstart.md index d753d2ad..a14b083d 100644 --- a/_episodes/01-quickstart.md +++ b/_episodes/01-quickstart.md @@ -33,8 +33,6 @@ keypoints: --- -- {: .callout} - > ## What is the purpose of the quickstart guide? > > - The purpose of the quickstart guide is to enable a user of ESMValTool to diff --git a/index.md b/index.md index 8f01c7ee..5b2224a1 100644 --- a/index.md +++ b/index.md @@ -68,7 +68,7 @@ independently. > - [Documentation](https://docs.esmvaltool.org) > - [ESMValTool home page](https://www.esmvaltool.org/) > - [Discussions page](https://github.com/ESMValGroup/ESMValTool/discussions) -> - [Papers](https://www.esmvaltool.org/references.html) +> - [Papers](https://esmvaltool.org/references/) > - [ESMValTool Source code](https://github.com/ESMValGroup/ESMValTool) > - [ESMValCore Source code](https://github.com/ESMValGroup/ESMValCore) > - [ESMValTool Citation info](https://esmvalgroup.github.io/ESMValTool_Tutorial/about/index.html) From 9fdecca7a8d08688c30e8ebc1d49cc575c925c6d Mon Sep 17 00:00:00 2001 From: rswamina Date: Tue, 2 May 2023 10:39:36 +0100 Subject: [PATCH 11/15] fixed run recipe command --- _episodes/04-recipe.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_episodes/04-recipe.md b/_episodes/04-recipe.md index c471e597..d7849569 100644 --- a/_episodes/04-recipe.md +++ b/_episodes/04-recipe.md @@ -36,7 +36,7 @@ We will start by running [examples/recipe_python.yml](https://docs.esmvaltool. org/en/latest/recipes/recipe_examples.html) ``` -esmvaltool run recipe_python.yml +esmvaltool run examples/recipe_python.yml ``` From 7406db2e0ea4b4a239178bdb09eeed60f929c671 Mon Sep 17 00:00:00 2001 From: rswamina Date: Tue, 2 May 2023 10:51:35 +0100 Subject: [PATCH 12/15] fixed spaces and version in installation episode --- _episodes/02-installation.md | 7 +++++-- _episodes/04-recipe.md | 4 ++-- 2 files changed, 7 insertions(+), 4 deletions(-) diff --git a/_episodes/02-installation.md b/_episodes/02-installation.md index 7bdfd9f9..84d4201a 100644 --- a/_episodes/02-installation.md +++ b/_episodes/02-installation.md @@ -229,10 +229,13 @@ to display the command line help. > > {: .bash} > > The version of ESMValTool installed should be displayed on the screen as: > > ~~~ -> > ESMValCore: 2.6.0 -> > ESMValTool: 2.6.0 +> > ESMValCore: 2.8.0 +> > ESMValTool: 2.8.0 > > ~~~ > > {: .output} +> Note that on HPC servers such as JASMIN, sometimes a more recent development +> version may be displayed for ESMValTool, for e.g. +> `ESMValTool: 2.9.0.dev4+g6948d5512` > {: .solution} {: .challenge} diff --git a/_episodes/04-recipe.md b/_episodes/04-recipe.md index d7849569..9f402eb9 100644 --- a/_episodes/04-recipe.md +++ b/_episodes/04-recipe.md @@ -362,11 +362,11 @@ Do you recognize the basic recipe structure that was introduced in episode 1? >> form of `.xml` files. > > - **timeseries/tas_global**: creates `/preproc/timeseries/tas_global`, which > > contains preprocessed data for each of the input datasets, a -> > `metadata.yml` file and provenance information in the +> > `metadata.yml` file and provenance information in the >> form of `.xml` files. > > - **timeseries/tas_amsterdam**: creates `/preproc/timeseries/tas_amsterdam`, > > which contains preprocessed data for each of the input datasets, plus a -> > combined `MultiModelMean`, a`metadata.yml` file and provenance files. +> > combined `MultiModelMean`, a `metadata.yml` file and provenance files. > > - **map/script1**: creates `/run/map/script1` with general information and a > > log of the diagnostic script run. It also creates `/plots/map/script1` and > > `/work/map/script1`, which contain output figures and output datasets, From 303d1f537165cc1a961493712acab83053f4c6b1 Mon Sep 17 00:00:00 2001 From: rswamina Date: Tue, 2 May 2023 10:59:15 +0100 Subject: [PATCH 13/15] updates to auto download suggestion --- setup.md | 16 ++++++++++------ 1 file changed, 10 insertions(+), 6 deletions(-) diff --git a/setup.md b/setup.md index 13f32ca7..0faddab0 100644 --- a/setup.md +++ b/setup.md @@ -233,11 +233,18 @@ Please skip this section if you are not going to use ESMValTool on your local machine and go [here](#github-account-advanced). If you are planning on running ESMValTool on your own machine, please make sure -that you are able to download CMIP data and that you have several GB of space +that you are able to download CMIP data and that you have a few GB of space available to install conda and ESMValTool, but also enough to make a copy of some -data (~125MB). +data (~125MB) needed for this tutorial. -To download the data required by this tutorial, run the following command using +You can use ESMValTool to automatically download data needed for test recipes. +Please see the [Configuration][lesson-configuration] episode +or the [configuration file documentation][config-file] for more information. +This the recommended option as it has the advantage that data is +stored in subdirectories, and features such as wildcards and recording +the version of the data will work automatically. + +Alternatively, you can run the following command using [wget](https://en.wikipedia.org/wiki/Wget): ~~~shell @@ -245,9 +252,6 @@ wget --no-clobber --input-file \ https://github.com/ESMValGroup/ESMValTool_Tutorial/raw/main/data/dataset.urls \ --directory-prefix $HOME/esmvaltool_tutorial/data/ ~~~ -You can also use ESMValTool to automatically download data needed for test recipes. -Please see the [Configuration][lesson-configuration] episode -or the [configuration file documentation][config-file] for more information. ## GitHub account (Advanced) From 7ed151adb8cc9403b588072182724cf06b0b310c Mon Sep 17 00:00:00 2001 From: rswamina Date: Tue, 2 May 2023 10:59:53 +0100 Subject: [PATCH 14/15] fixed broken link for papers --- _episodes/00-introduction.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_episodes/00-introduction.md b/_episodes/00-introduction.md index 6051f898..c2c420d7 100644 --- a/_episodes/00-introduction.md +++ b/_episodes/00-introduction.md @@ -2,7 +2,7 @@ title: "Introduction" teaching: 5 exercises: 10 -compatibility: ESMValTool v2.6.0 +compatibility: ESMValTool v2.8.0 questions: - What is ESMValTool? From c8a5ff7df927b023119fa1338c36a9a61f4ae277 Mon Sep 17 00:00:00 2001 From: rswamina Date: Tue, 2 May 2023 12:46:43 +0100 Subject: [PATCH 15/15] fixed white spaces --- _episodes/04-recipe.md | 2 +- setup.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/_episodes/04-recipe.md b/_episodes/04-recipe.md index 9f402eb9..a5a9f512 100644 --- a/_episodes/04-recipe.md +++ b/_episodes/04-recipe.md @@ -362,7 +362,7 @@ Do you recognize the basic recipe structure that was introduced in episode 1? >> form of `.xml` files. > > - **timeseries/tas_global**: creates `/preproc/timeseries/tas_global`, which > > contains preprocessed data for each of the input datasets, a -> > `metadata.yml` file and provenance information in the +> > `metadata.yml` file and provenance information in the >> form of `.xml` files. > > - **timeseries/tas_amsterdam**: creates `/preproc/timeseries/tas_amsterdam`, > > which contains preprocessed data for each of the input datasets, plus a diff --git a/setup.md b/setup.md index 0faddab0..4b27bffe 100644 --- a/setup.md +++ b/setup.md @@ -241,7 +241,7 @@ You can use ESMValTool to automatically download data needed for test recipes. Please see the [Configuration][lesson-configuration] episode or the [configuration file documentation][config-file] for more information. This the recommended option as it has the advantage that data is -stored in subdirectories, and features such as wildcards and recording +stored in subdirectories, and features such as wildcards and recording the version of the data will work automatically. Alternatively, you can run the following command using