Debug 1M microphysics GPU bug #1371
Description
When running 1M microphysics in coupled AMIP simulations, we get this error in step! after 2-3 days (e.g. in this build):
ERROR: a exception was thrown during kernel execution on thread (161, 1, 1) in block (7, 1, 1).
Stacktrace not available, run Julia on debug level 2 for more details (by passing -g2 to the executable).
ERROR: LoadError: KernelException: exception thrown during kernel execution on device NVIDIA A100-SXM4-80GB
It looks like this comes from an error in generated code, so the specific error itself isn't printed (see this discussion). To debug it, we'll need to run Julia on debug level 2, using julia -g2 run_amip.jl --config_file config/longrun_configs/amip_edonly_1M.yml or just julia -g2 to get a REPL