Merge remote-tracking branch 'upstream/main' into develop

.readthedocs.yaml

@@ -0,0 +1,19 @@
+# .readthedocs.yaml
+# Read the Docs configuration file
+# See https://docs.readthedocs.io/en/stable/config-file/v2.html for details
+# on both this MkDocs setup and available Sphinx setup
+version: 2
+
+# Set the version of Python and other tools you might need
+build:
+  os: ubuntu-22.04
+  tools:
+    python: "3.11"
+
+mkdocs:
+  configuration: mkdocs.yml
+
+# Optionally declare the Python requirements required to build your docs
+# python:
+#   install:
+#   - requirements: docs/requirements.txt

README.md

@@ -8,7 +8,7 @@ ICAR is a simplified atmospheric model designed primarily for climate downscalin
 
 In ICAR 2.0 (currently early alpha), ICAR supports parallelization across hundreds of computing nodes (the basic physics have been shown to scale up to nearly 100,000 processors) using coarray fortran. This version of the code has a significant overhaul of the original code base, and as a result not all functionality has been restored yet.
 
-Documentation is (slowly) being built on [readthedocs](http://icar.readthedocs.org/en/develop/) and doxygen based documentation can be built now by running "make doc", and is available through [github-pages](http://NCAR.github.io/icar).
+Documentation is (slowly) being built on [readthedocs](http://icar.readthedocs.org/en/latest/) and doxygen based documentation can be built now by running "make doc", and is available through [github-pages](http://NCAR.github.io/icar).
 
 #### Requirements
 To run the model 3D time-varying atmospheric data are required, though an ideal test case can be generated for simple simulations as well.  See "Running the Model" below. There are some sample python scripts to help make input forcing files, but the WRF pre-processing system can also be used.  Low-resolution WRF output files can be used directly, various reanalysis and GCM output files can be used with minimal pre-processing (just get all the variables in the same netcdf file.)  In addition, a high-resolution netCDF topography file is required.  This will define the grid that ICAR will run on.  Finally and ICAR options file is used to specify various parameters for the model.  A sample options file is provided in the run/ directory.

docs/compiling.md

@@ -13,7 +13,7 @@ Also to set the compiler for your machine if necessary (defaults to gfortran)
     make install
         # compile if necessary, then install in the install directory [~/bin]
 
-### Options:
+### Options
     MODE=fast           # more optimization, slower compile, WARNING:not safe optimizations
     MODE=profile        # set profiling options for gnu or intel compilers
     MODE=debug          # debug compile with optimizations
@@ -29,5 +29,62 @@ Also to set the compiler for your machine if necessary (defaults to gfortran)
 
     add -jn to parallelize the compile over n processors
 
-### Example:
+### Example
     make install MODE=debug -j4  # uses 4 processes to compile in debug mode
+
+
+### Derecho System Specifics
+Instructions for building ICAR on [Derecho](https://arc.ucar.edu/knowledge_base/74317833), a supercomputer at NCAR.
+See [instructions](running/running.md#derecho-system-specifics) for running ICAR on Derecho.
+
+#### Cray Compiler
+``` bash
+module load ncarenv cce cray-mpich netcdf fftw
+make -j 4
+```
+
+#### GNU Compiler
+``` bash
+module load gcc ncarenv/23.09 cray-mpich netcdf fftw opencoarrays
+COMPILER=gnu make -j 4
+```
+
+
+### Derecho Debugging
+When running ICAR on larger domains there is a chance the program will run out of memory available for coarrays.
+This may be difficult to diagnose because the program will stop without outputting any information but the following steps might help.
+*NOTE*: The following debugging information is only for Cray compilers, GNU's OpenCoarrays is not setup to use the SHMEM library.
+
+#### Set Helpful Debug Variables
+Set the following variables, further detail and more variables can be found in the [Cray OpenSHMEMX](https://cray-openshmemx.readthedocs.io/en/latest/intro_shmem.html#cray-openshmemx-setup-and-running-specific-environment-variables) documentation.
+* `SHMEM_MEMINFO_DISPLAY=1` to display information about the job's memory allocation during initialization
+* `SHMEM_ABORT_ON_ERROR=1` and `MPICH_ABORT_ON_ERROR=1`, these are set when loading the ATP module
+
+#### Increase Memory Available for Coarrays
+The default symmetric heap size on Derecho is 64MB per process.
+To increase the size set the variable `XT_SYMMETRIC_HEAP_SIZE` to an integer value with the suffix `M` for megabyte or `G` for gigabyte.
+``` bash
+export XT_SYMMETRIC_HEAP_SIZE=128M
+```
+
+#### Further Tools
+* [Derecho debugging and profiling documentation](https://arc.ucar.edu/knowledge_base/149323810)
+* Use [gdb](https://sourceware.org/gdb/current/onlinedocs/gdb) (with [cheatsheet](https://sourceware.org/gdb/current/onlinedocs/gdb)) or [gdb4hpc](https://cpe.ext.hpe.com/docs/debugging-tools/gdb4hpc.1.html)
+* [Linaro Forge Tools](https://docs.linaroforge.com/23.0/html/forge/index.html) such as DTT or MAP.
+* If the program returned `died from signal XYZ`, check the signal error against list shown by `$ kill -L` or `$ kill -l`
+
+<!-- NOTE: removing Intel compiler information until tested more -->
+<!-- #### Intel Compilers -->
+<!--   - __Note__: currently the classic Intel compiler is recommended for production runs but testing `ifx` and reporting issues would be useful. -->
+<!--   - __Note__: test `debugslow` mode is required for successful running and compilation, issue is being worked on. -->
+
+<!-- ##### Classic -->
+<!-- ``` bash -->
+<!-- module load intel-classic ncarcompilers intel-mpi netcdf-mpi fftw-mpi -->
+<!-- COMPILER=intel MODE=debugslow make -j 4 -->
+<!-- ``` -->
+<!-- ##### OneAPI -->
+<!-- ``` bash -->
+<!-- module load intel-oneapi ncarcompilers intel-mpi netcdf-mpi fftw-mpi -->
+<!-- COMPILER=intel MODE=debugslow F90=ifx FC=ifx make -j 4 -->
+<!-- ``` -->

docs/developing.md

@@ -9,14 +9,14 @@ For an outline of the basic code structure see the [ICAR code overview](icar_cod
 
 For a more complete documentation of ICAR, see the [detailed ICAR code description](http://ncar.github.io/icar/).  This full description is being updated as the doxygen markup is added to the code.  It also has detailed interactive diagrams of the inter-relationship between all functions.  For the overview, start by looking at the [documentation for the main program](http://ncar.github.io/icar/driver_8f90.html).
 
-ICAR main is now a significant change to the internal workings of ICAR, and has made some components vastly more dynamic and easier to implement / modify.  However, to support that, some elements became more complicated.  For example, adding a new variable to the output file is incredibly easy if that variable already exists.  Just add it to a call (almost anywhere) to output%add_variables()... however, if that variables has not been added before there are several steps that need to occur.  Below is a quick outline of the process:
-
-1) The variable has to be a a “variable_t” type, not just an array (and preferably part of the domain object though not technically required, but other steps below would be different if it is not.)
-2) That variable needs to have an entry in the kVARS structure in the `constants/icar_constants.f90` file.
-3) That variable needs to have an entry in the default metadata array `io/default_output_metadata.f90`.
-4) That variable needs to be initialized somewhere appropriate (for example in `objects/domain_obj.f90` `create_variables`.)
-5) Somewhere you need to tell it to write that variable too.  The easiest place to do this is in the domain_obj routine `var_request`, add it to the list of variables to be allocated, **and** to the list of “restart” variables (i.e. output variables for now).
-6) The one change that needs to be made in `io/output_obj.f90` is to add a single line for that variable to the `add_variables` routine.
+ICAR main is now a significant change to the internal workings of ICAR, and has made some components vastly more dynamic and easier to implement/modify.  However, to support that, some elements became more complicated.  For example, adding a new variable to the output file is incredibly easy if that variable already exists.  Just add it to a call (almost anywhere) to `output%add_variables()` however, if that variables has not been added before there are several steps that need to occur.  Below is a quick outline of the process:
+
+1. The variable has to be a a “variable_t” type, not just an array (and preferably part of the domain object though not technically required, but other steps below would be different if it is not.)
+2. That variable needs to have an entry in the kVARS structure in the `constants/icar_constants.f90` file.
+3. That variable needs to have an entry in the default metadata array `io/default_output_metadata.f90`.
+4. That variable needs to be initialized somewhere appropriate (for example in `objects/domain_obj.f90` `create_variables`.)
+5. Somewhere you need to tell it to write that variable too.  The easiest place to do this is in the domain_obj routine `var_request`, add it to the list of variables to be allocated, **and** to the list of “restart” variables (i.e. output variables for now).
+6. The one change that needs to be made in `io/output_obj.f90` is to add a single line for that variable to the `add_variables` routine.
 
 I know that ended up being more complicated than it could be.  Once a variable is in that format, it is really easy to add and remove variables (and for that matter create multiple output files with different variable lists in each, it is almost magical), but the initial setup is non-trivial to get there.
 

docs/running.md → docs/running/running.md

@@ -69,3 +69,28 @@ LONG    = Longitude on ICAR mass grid               (degrees)
 [SOIL]  = Soil type classification for land surface model
 ```
 Must be in a (nearly) constant dx,dy projection (e.g. Lambert Conformal Conic)
+
+
+### Derecho System Specifics
+In the following examples one Derecho node is used with 128 processes.
+See [instructions](../compiling.md#derecho-system-specifics) for compiling ICAR on Derecho and the list of modules to use.
+#### Cray Compiler
+``` bash
+mpiexec -np 128 ./icar icar_options.nml
+```
+
+#### GNU Compiler
+``` bash
+cafrun -n 128 ./icar icar_options.nml
+```
+
+#### Intel Compilers
+##### Classic
+``` bash
+FOR_COARRAY_NUM_IMAGES=128 ./icar icar_options.nml
+```
+
+##### OneAPI
+``` bash
+FOR_COARRAY_NUM_IMAGES=128 ./icar icar_options.nml
+```

docs/settings_documentation.md → docs/running/settings_documentation.md

@@ -1,6 +1,6 @@
 ## ICAR settings file documentation
 
-Reading the comments in the run/short_icar_options.nml or run/complete_icar_options.nml files may be some of the best documentation available.
+Reading the comments in the [run/short_icar_options.nml](https://github.com/NCAR/icar/blob/main/run/short_icar_options.nml) or [run/complete_icar_options.nml](https://github.com/NCAR/icar/blob/main/run/complete_icar_options.nml) files may be some of the best documentation available.
 
 ### Main Namelists
 __REQUIRED__:

mkdocs.yml

@@ -1,9 +1,16 @@
 site_name: ICAR
-pages:
-- Home: index.md
-- Compiling: compiling.md
-- Running: running.md
-- Developing: developing.md
-- Errors: errors.md
+nav:
+  - Home: index.md
+  - ICAR Code Overview: icar_code_overview.md
+  - Compiling: compiling.md
+  - Running: running/running.md
+  - Settings Documentation: running/settings_documentation.md
+  - Developing: developing.md
+  - Errors: errors.md
+  - FAQ: FAQ.md
+  - How To:
+    - Build MkDocs: howto/build_mkdocs.md
+    - ICAR and Git: howto/icar_and_git_howto.md
+    - ICAR Git Workflow: howto/icar_git_workflow.md
 theme: readthedocs
-google_analytics: ['UA-108421993-4', 'icar.readthedocs.org']
+# google_analytics: ['UA-108421993-4', 'icar.readthedocs.org']