From 6496c78fb85ca8f7607107d2426a6e369e2f19e9 Mon Sep 17 00:00:00 2001 From: Soren Rasmussen Date: Mon, 21 Aug 2023 17:14:16 -0700 Subject: [PATCH 1/5] New documentation for building and running on Derecho. Mkdocs deprecation fixes and reorganization of running documentation files --- docs/compiling.md | 36 ++++++++++++++++++-- docs/{ => running}/running.md | 25 ++++++++++++++ docs/{ => running}/settings_documentation.md | 0 mkdocs.yml | 5 +-- 4 files changed, 62 insertions(+), 4 deletions(-) rename docs/{ => running}/running.md (87%) rename docs/{ => running}/settings_documentation.md (100%) diff --git a/docs/compiling.md b/docs/compiling.md index 6b0cbc8b..724887a5 100644 --- a/docs/compiling.md +++ b/docs/compiling.md @@ -13,7 +13,7 @@ Also to set the compiler for your machine if necessary (defaults to gfortran) make install # compile if necessary, then install in the install directory [~/bin] -### Options: +### Options MODE=fast # more optimization, slower compile, WARNING:not safe optimizations MODE=profile # set profiling options for gnu or intel compilers MODE=debug # debug compile with optimizations @@ -29,5 +29,37 @@ Also to set the compiler for your machine if necessary (defaults to gfortran) add -jn to parallelize the compile over n processors -### Example: +### Example make install MODE=debug -j4 # uses 4 processes to compile in debug mode + + +### Derecho System Specifics +Instructions for building ICAR on [Derecho](https://arc.ucar.edu/knowledge_base/74317833), a supercomputer at NCAR. +See [instructions](running/running.md#derecho-system-specifics) for running ICAR on Derecho. + +#### Cray Compiler +``` bash +module load cce ncarcompilers cray-mpich netcdf fftw +make -j 4 +``` + +#### GNU Compiler +``` bash +module load gcc ncarcompilers cray-mpich netcdf fftw caf/derecho-2.10.1 +COMPILER=gnu make -j 4 +``` + +#### Intel Compilers + - __Note__: currently the classic Intel compiler is recommended for production runs but testing `ifx` and reporting issues would be useful. + - __Note__: test `debugslow` mode is required for successful running and compilation, issue is being worked on. + +##### Classic +``` bash +module load intel-classic ncarcompilers intel-mpi netcdf-mpi fftw-mpi +COMPILER=intel MODE=debugslow make -j 4 +``` +##### OneAPI +``` bash +module load intel-oneapi ncarcompilers intel-mpi netcdf-mpi fftw-mpi +COMPILER=intel MODE=debugslow F90=ifx FC=ifx make -j 4 +``` diff --git a/docs/running.md b/docs/running/running.md similarity index 87% rename from docs/running.md rename to docs/running/running.md index 4990802d..c2eda655 100644 --- a/docs/running.md +++ b/docs/running/running.md @@ -69,3 +69,28 @@ LONG = Longitude on ICAR mass grid (degrees) [SOIL] = Soil type classification for land surface model ``` Must be in a (nearly) constant dx,dy projection (e.g. Lambert Conformal Conic) + + +### Derecho System Specifics +In the following examples one Derecho node is used with 128 processes. +See [instructions](../compiling.md#derecho-system-specifics) for compiling ICAR on Derecho and the list of modules to use. +#### Cray Compiler +``` bash +mpiexec -np 128 ./icar icar_options.nml +``` + +#### GNU Compiler +``` bash +cafrun -n 128 ./icar icar_options.nml +``` + +#### Intel Compilers +##### Classic +``` bash +FOR_COARRAY_NUM_IMAGES=128 ./icar icar_options.nml +``` + +##### OneAPI +``` bash +FOR_COARRAY_NUM_IMAGES=128 ./icar icar_options.nml +``` diff --git a/docs/settings_documentation.md b/docs/running/settings_documentation.md similarity index 100% rename from docs/settings_documentation.md rename to docs/running/settings_documentation.md diff --git a/mkdocs.yml b/mkdocs.yml index 48f253cd..67d1f094 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -1,8 +1,9 @@ site_name: ICAR -pages: +nav: - Home: index.md - Compiling: compiling.md -- Running: running.md +- Running: running/running.md +- Settings Documentation: running/settings_documentation.md - Developing: developing.md - Errors: errors.md theme: readthedocs From 9ba0b36da88eab9db2a875f3a1bf31f1221f9321 Mon Sep 17 00:00:00 2001 From: Soren Rasmussen Date: Mon, 28 Aug 2023 12:06:39 -0600 Subject: [PATCH 2/5] Fixes two deprecation warning messages: - meets readthedocs' upcoming requirement for a version 2 .readthedocs.yaml file. - Use build.os instead of build.image in .readthedocs.yaml --- .readthedocs.yaml | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) create mode 100644 .readthedocs.yaml diff --git a/.readthedocs.yaml b/.readthedocs.yaml new file mode 100644 index 00000000..0b4e63bc --- /dev/null +++ b/.readthedocs.yaml @@ -0,0 +1,19 @@ +# .readthedocs.yaml +# Read the Docs configuration file +# See https://docs.readthedocs.io/en/stable/config-file/v2.html for details +# on both this MkDocs setup and available Sphinx setup +version: 2 + +# Set the version of Python and other tools you might need +build: + os: ubuntu-22.04 + tools: + python: "3.11" + +mkdocs: + configuration: mkdocs.yml + +# Optionally declare the Python requirements required to build your docs +# python: +# install: +# - requirements: docs/requirements.txt From 2fb06cba8a869d4daddd63163235a6ae3bd450b6 Mon Sep 17 00:00:00 2001 From: Soren Rasmussen Date: Mon, 28 Aug 2023 13:11:17 -0600 Subject: [PATCH 3/5] Enhancements: better organization of mkdocs.yml and removal of that deprecated method for doing google analytics. Small formatting changes if markdown files --- docs/developing.md | 16 ++++++++-------- docs/running/settings_documentation.md | 2 +- mkdocs.yml | 20 +++++++++++++------- 3 files changed, 22 insertions(+), 16 deletions(-) diff --git a/docs/developing.md b/docs/developing.md index 33400a3f..95b87506 100644 --- a/docs/developing.md +++ b/docs/developing.md @@ -9,14 +9,14 @@ For an outline of the basic code structure see the [ICAR code overview](icar_cod For a more complete documentation of ICAR, see the [detailed ICAR code description](http://ncar.github.io/icar/). This full description is being updated as the doxygen markup is added to the code. It also has detailed interactive diagrams of the inter-relationship between all functions. For the overview, start by looking at the [documentation for the main program](http://ncar.github.io/icar/driver_8f90.html). -ICAR main is now a significant change to the internal workings of ICAR, and has made some components vastly more dynamic and easier to implement / modify. However, to support that, some elements became more complicated. For example, adding a new variable to the output file is incredibly easy if that variable already exists. Just add it to a call (almost anywhere) to output%add_variables()... however, if that variables has not been added before there are several steps that need to occur. Below is a quick outline of the process: - -1) The variable has to be a a “variable_t” type, not just an array (and preferably part of the domain object though not technically required, but other steps below would be different if it is not.) -2) That variable needs to have an entry in the kVARS structure in the `constants/icar_constants.f90` file. -3) That variable needs to have an entry in the default metadata array `io/default_output_metadata.f90`. -4) That variable needs to be initialized somewhere appropriate (for example in `objects/domain_obj.f90` `create_variables`.) -5) Somewhere you need to tell it to write that variable too. The easiest place to do this is in the domain_obj routine `var_request`, add it to the list of variables to be allocated, **and** to the list of “restart” variables (i.e. output variables for now). -6) The one change that needs to be made in `io/output_obj.f90` is to add a single line for that variable to the `add_variables` routine. +ICAR main is now a significant change to the internal workings of ICAR, and has made some components vastly more dynamic and easier to implement/modify. However, to support that, some elements became more complicated. For example, adding a new variable to the output file is incredibly easy if that variable already exists. Just add it to a call (almost anywhere) to `output%add_variables()` however, if that variables has not been added before there are several steps that need to occur. Below is a quick outline of the process: + +1. The variable has to be a a “variable_t” type, not just an array (and preferably part of the domain object though not technically required, but other steps below would be different if it is not.) +2. That variable needs to have an entry in the kVARS structure in the `constants/icar_constants.f90` file. +3. That variable needs to have an entry in the default metadata array `io/default_output_metadata.f90`. +4. That variable needs to be initialized somewhere appropriate (for example in `objects/domain_obj.f90` `create_variables`.) +5. Somewhere you need to tell it to write that variable too. The easiest place to do this is in the domain_obj routine `var_request`, add it to the list of variables to be allocated, **and** to the list of “restart” variables (i.e. output variables for now). +6. The one change that needs to be made in `io/output_obj.f90` is to add a single line for that variable to the `add_variables` routine. I know that ended up being more complicated than it could be. Once a variable is in that format, it is really easy to add and remove variables (and for that matter create multiple output files with different variable lists in each, it is almost magical), but the initial setup is non-trivial to get there. diff --git a/docs/running/settings_documentation.md b/docs/running/settings_documentation.md index 35986f48..78b51f74 100644 --- a/docs/running/settings_documentation.md +++ b/docs/running/settings_documentation.md @@ -1,6 +1,6 @@ ## ICAR settings file documentation -Reading the comments in the run/short_icar_options.nml or run/complete_icar_options.nml files may be some of the best documentation available. +Reading the comments in the [run/short_icar_options.nml](https://github.com/NCAR/icar/blob/main/run/short_icar_options.nml) or [run/complete_icar_options.nml](https://github.com/NCAR/icar/blob/main/run/complete_icar_options.nml) files may be some of the best documentation available. ### Main Namelists __REQUIRED__: diff --git a/mkdocs.yml b/mkdocs.yml index 67d1f094..a2852710 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -1,10 +1,16 @@ site_name: ICAR nav: -- Home: index.md -- Compiling: compiling.md -- Running: running/running.md -- Settings Documentation: running/settings_documentation.md -- Developing: developing.md -- Errors: errors.md + - Home: index.md + - ICAR Code Overview: icar_code_overview.md + - Compiling: compiling.md + - Running: running/running.md + - Settings Documentation: running/settings_documentation.md + - Developing: developing.md + - Errors: errors.md + - FAQ: FAQ.md + - How To: + - Build MkDocs: howto/build_mkdocs.md + - ICAR and Git: howto/icar_and_git_howto.md + - ICAR Git Workflow: howto/icar_git_workflow.md theme: readthedocs -google_analytics: ['UA-108421993-4', 'icar.readthedocs.org'] +# google_analytics: ['UA-108421993-4', 'icar.readthedocs.org'] From 3843a56814e8c645b0f25be78c5698366e4c097d Mon Sep 17 00:00:00 2001 From: Soren Rasmussen Date: Tue, 29 Aug 2023 15:32:24 -0600 Subject: [PATCH 4/5] Since markdown files and readthedocs settings on the main branch are being edited, changing the readthedocs link to the 'latest' for right now. The 'latest' url is built from the main branch --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index fd734d15..a1474c06 100644 --- a/README.md +++ b/README.md @@ -8,7 +8,7 @@ ICAR is a simplified atmospheric model designed primarily for climate downscalin In ICAR 2.0 (currently early alpha), ICAR supports parallelization across hundreds of computing nodes (the basic physics have been shown to scale up to nearly 100,000 processors) using coarray fortran. This version of the code has a significant overhaul of the original code base, and as a result not all functionality has been restored yet. -Documentation is (slowly) being built on [readthedocs](http://icar.readthedocs.org/en/develop/) and doxygen based documentation can be built now by running "make doc", and is available through [github-pages](http://NCAR.github.io/icar). +Documentation is (slowly) being built on [readthedocs](http://icar.readthedocs.org/en/latest/) and doxygen based documentation can be built now by running "make doc", and is available through [github-pages](http://NCAR.github.io/icar). #### Requirements To run the model 3D time-varying atmospheric data are required, though an ideal test case can be generated for simple simulations as well. See "Running the Model" below. There are some sample python scripts to help make input forcing files, but the WRF pre-processing system can also be used. Low-resolution WRF output files can be used directly, various reanalysis and GCM output files can be used with minimal pre-processing (just get all the variables in the same netcdf file.) In addition, a high-resolution netCDF topography file is required. This will define the grid that ICAR will run on. Finally and ICAR options file is used to specify various parameters for the model. A sample options file is provided in the run/ directory. From 2a0501647b2661321434306a9c785b3d27ad1b4a Mon Sep 17 00:00:00 2001 From: Soren Rasmussen Date: Tue, 23 Jan 2024 10:35:54 -0700 Subject: [PATCH 5/5] documentation: adding general and Derecho specific debugging information --- docs/compiling.md | 51 +++++++++++++++++++++++++++++++++++------------ 1 file changed, 38 insertions(+), 13 deletions(-) diff --git a/docs/compiling.md b/docs/compiling.md index 724887a5..8a662f1c 100644 --- a/docs/compiling.md +++ b/docs/compiling.md @@ -39,27 +39,52 @@ See [instructions](running/running.md#derecho-system-specifics) for running ICAR #### Cray Compiler ``` bash -module load cce ncarcompilers cray-mpich netcdf fftw +module load ncarenv cce cray-mpich netcdf fftw make -j 4 ``` #### GNU Compiler ``` bash -module load gcc ncarcompilers cray-mpich netcdf fftw caf/derecho-2.10.1 +module load gcc ncarenv/23.09 cray-mpich netcdf fftw opencoarrays COMPILER=gnu make -j 4 ``` -#### Intel Compilers - - __Note__: currently the classic Intel compiler is recommended for production runs but testing `ifx` and reporting issues would be useful. - - __Note__: test `debugslow` mode is required for successful running and compilation, issue is being worked on. -##### Classic -``` bash -module load intel-classic ncarcompilers intel-mpi netcdf-mpi fftw-mpi -COMPILER=intel MODE=debugslow make -j 4 -``` -##### OneAPI +### Derecho Debugging +When running ICAR on larger domains there is a chance the program will run out of memory available for coarrays. +This may be difficult to diagnose because the program will stop without outputting any information but the following steps might help. +*NOTE*: The following debugging information is only for Cray compilers, GNU's OpenCoarrays is not setup to use the SHMEM library. + +#### Set Helpful Debug Variables +Set the following variables, further detail and more variables can be found in the [Cray OpenSHMEMX](https://cray-openshmemx.readthedocs.io/en/latest/intro_shmem.html#cray-openshmemx-setup-and-running-specific-environment-variables) documentation. +* `SHMEM_MEMINFO_DISPLAY=1` to display information about the job's memory allocation during initialization +* `SHMEM_ABORT_ON_ERROR=1` and `MPICH_ABORT_ON_ERROR=1`, these are set when loading the ATP module + +#### Increase Memory Available for Coarrays +The default symmetric heap size on Derecho is 64MB per process. +To increase the size set the variable `XT_SYMMETRIC_HEAP_SIZE` to an integer value with the suffix `M` for megabyte or `G` for gigabyte. ``` bash -module load intel-oneapi ncarcompilers intel-mpi netcdf-mpi fftw-mpi -COMPILER=intel MODE=debugslow F90=ifx FC=ifx make -j 4 +export XT_SYMMETRIC_HEAP_SIZE=128M ``` + +#### Further Tools +* [Derecho debugging and profiling documentation](https://arc.ucar.edu/knowledge_base/149323810) +* Use [gdb](https://sourceware.org/gdb/current/onlinedocs/gdb) (with [cheatsheet](https://sourceware.org/gdb/current/onlinedocs/gdb)) or [gdb4hpc](https://cpe.ext.hpe.com/docs/debugging-tools/gdb4hpc.1.html) +* [Linaro Forge Tools](https://docs.linaroforge.com/23.0/html/forge/index.html) such as DTT or MAP. +* If the program returned `died from signal XYZ`, check the signal error against list shown by `$ kill -L` or `$ kill -l` + + + + + + + + + + + + + + + +