Skip to content

Work around CCE 19.0.0 compiler bugs for Cray+OpenACC builds #1286

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
sbryngelson wants to merge 19 commits into
base: master
Choose a base branch
from
+216 −116
Open

Work around CCE 19.0.0 compiler bugs for Cray+OpenACC builds #1286

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
19 commits
Select commit Hold shift + click to select a range
b9da01e
Work around CCE 19.0.0 compiler bugs for Cray+OpenACC builds
Mar 5, 2026
1aa4cf5
Temporarily disable Phoenix + Frontier AMD CI (pre-existing failures …
Mar 5, 2026
835a2b8
Address code review findings: bounds warning, assert patch applied, C…
Mar 5, 2026
ddcaa4a
Temporarily disable Phoenix (NVHPC) benchmark jobs (QOS job limit iss…
Mar 5, 2026
6314820
Address CodeRabbit review: CCE PROHIBIT guards + pyrometheus forward-…
Mar 5, 2026
c274109
Add comment noting pyrometheus upstream issue for thermochem GPU_ROUT…
Mar 5, 2026
05d1dc0
Merge branch 'master' into fix/cce-cray-inline-routine
sbryngelson Mar 5, 2026
1dadcc3
Fix Frontier benchmark SLURM: use batch+1:59+normal QOS
Mar 6, 2026
e208275
Fix bench.yml: restore timeout-minutes to 480 (revert accidental 240)
Mar 6, 2026
2d1b359
Address review: CCE_MAX_SPECIES constant, GPU error for n_species ove…
Mar 6, 2026
810056d
Fix ##-> #! Fypp comment in m_chemistry.fpp top-level scope
Mar 6, 2026
8a6398c
Fix ##-> #! Fypp comment in parallel_macros.fpp cray_noinline block
Mar 6, 2026
23309f6
Extend -Oipa0 workaround to all Cray builds, not just Cray+OpenACC
Mar 6, 2026
5d177b7
Fix -Oipa0 guard: exclude Cray+OpenMP, cover Cray+OpenACC and Cray CPU
Mar 6, 2026
9fc072a
Remove persistent build cache for self-hosted test runners
sbryngelson Mar 6, 2026
2cdade9
Remove build cache from benchmark jobs on Phoenix and Frontier
sbryngelson Mar 6, 2026
6e97695
Fix submit.sh to survive monitor SIGKILL by re-checking SLURM state
sbryngelson Mar 6, 2026
61924d8
Extract monitor SIGKILL recovery into shared run_monitored_slurm_job.sh
sbryngelson Mar 6, 2026
ac28127
bench: update Phoenix tmpbuild path to project storage
sbryngelson Mar 7, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
37 changes: 37 additions & 0 deletions .github/scripts/run_monitored_slurm_job.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
#!/bin/bash
# Run monitor_slurm_job.sh and recover if the monitor is killed (e.g. SIGKILL
# from the runner OS) before the SLURM job completes. When the monitor exits
# non-zero, sacct is used to verify the job's actual final state; if the SLURM
# job succeeded we exit 0 so the CI step is not falsely marked as failed.
#
# Usage: run_monitored_slurm_job.sh <job_id> <output_file>

set -euo pipefail

if [ $# -ne 2 ]; then
echo "Usage: $0 <job_id> <output_file>"
exit 1
fi

job_id="$1"
output_file="$2"

SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"

monitor_exit=0
bash "$SCRIPT_DIR/monitor_slurm_job.sh" "$job_id" "$output_file" || monitor_exit=$?

if [ "$monitor_exit" -ne 0 ]; then
echo "Monitor exited with code $monitor_exit; re-checking SLURM job $job_id final state..."
# Give the SLURM epilog time to finalize if the job just finished
sleep 30
final_state=$(sacct -j "$job_id" -n -X -P -o State 2>/dev/null | head -n1 | cut -d'|' -f1 | tr -d ' ' || echo "UNKNOWN")
final_exit=$(sacct -j "$job_id" --format=ExitCode --noheader --parsable2 2>/dev/null | head -n1 | tr -d ' ' || echo "")
echo "Final SLURM state=$final_state exit=$final_exit"
if [ "$final_state" = "COMPLETED" ] && [ "$final_exit" = "0:0" ]; then
echo "SLURM job $job_id completed successfully despite monitor failure — continuing."
else
echo "ERROR: SLURM job $job_id did not complete successfully (state=$final_state exit=$final_exit)"
exit 1
fi
fi
25 changes: 1 addition & 24 deletions .github/workflows/bench.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,30 +37,7 @@ jobs:
fail-fast: false
matrix:
include:
- cluster: phoenix
name: Georgia Tech | Phoenix (NVHPC)
group: phoenix
labels: gt
flag: p
device: cpu
interface: none
build_script: ""
- cluster: phoenix
name: Georgia Tech | Phoenix (NVHPC)
group: phoenix
labels: gt
flag: p
device: gpu
interface: acc
build_script: ""
- cluster: phoenix
name: Georgia Tech | Phoenix (NVHPC)
group: phoenix
labels: gt
flag: p
device: gpu
interface: omp
build_script: ""
# Phoenix (NVHPC) — TEMPORARILY DISABLED (QOS job limit / pre-existing SLURM failures)
- cluster: frontier
name: Oak Ridge | Frontier (CCE)
group: phoenix
Expand Down
5 changes: 1 addition & 4 deletions .github/workflows/frontier/build.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,10 +20,7 @@ build_opts="$gpu_opts"

. ./mfc.sh load -c $compiler_flag -m $([ "$job_device" = "gpu" ] && echo "g" || echo "c")

# Only set up build cache for test suite, not benchmarks
if [ "$run_bench" != "bench" ]; then
source .github/scripts/setup-build-cache.sh "$cluster_name" "$job_device" "$job_interface"
fi
rm -rf build

source .github/scripts/retry-build.sh
if [ "$run_bench" == "bench" ]; then
Expand Down
11 changes: 5 additions & 6 deletions .github/workflows/frontier/submit.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -45,10 +45,10 @@ fi

# Select SBATCH params based on job type
if [ "$job_type" = "bench" ]; then
sbatch_account="#SBATCH -A ENG160"
sbatch_time="#SBATCH -t 05:59:00"
sbatch_partition="#SBATCH -p extended"
sbatch_extra=""
sbatch_account="#SBATCH -A CFD154"
sbatch_time="#SBATCH -t 01:59:00"
sbatch_partition="#SBATCH -p batch"
sbatch_extra="#SBATCH --qos=normal"
else
sbatch_account="#SBATCH -A CFD154"
sbatch_time="#SBATCH -t 01:59:00"
Expand Down Expand Up @@ -102,5 +102,4 @@ fi

echo "Submitted batch job $job_id"

# Use resilient monitoring instead of sbatch -W
bash "$SCRIPT_DIR/../../scripts/monitor_slurm_job.sh" "$job_id" "$output_file"
bash "$SCRIPT_DIR/../../scripts/run_monitored_slurm_job.sh" "$job_id" "$output_file"
4 changes: 3 additions & 1 deletion .github/workflows/phoenix/bench.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

source .github/scripts/bench-preamble.sh

tmpbuild=/storage/scratch1/6/sbryngelson3/mytmp_build
tmpbuild=/storage/project/r-sbryngelson3-0/sbryngelson3/mytmp_build
currentdir=$tmpbuild/run-$(( RANDOM % 900 ))
mkdir -p $tmpbuild
mkdir -p $currentdir
Expand All @@ -15,6 +15,8 @@ else
bench_opts="--mem 1"
fi

rm -rf build

source .github/scripts/retry-build.sh
RETRY_CLEAN_CMD="./mfc.sh clean" retry_build ./mfc.sh build -j $(nproc) $build_opts || exit 1

Expand Down
3 changes: 1 addition & 2 deletions .github/workflows/phoenix/submit.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -94,6 +94,5 @@ fi

echo "Submitted batch job $job_id"

# Use resilient monitoring instead of sbatch -W
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
bash "$SCRIPT_DIR/../../scripts/monitor_slurm_job.sh" "$job_id" "$output_file"
bash "$SCRIPT_DIR/../../scripts/run_monitored_slurm_job.sh" "$job_id" "$output_file"
3 changes: 1 addition & 2 deletions .github/workflows/phoenix/test.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,8 +3,7 @@
source .github/scripts/gpu-opts.sh
build_opts="$gpu_opts"

# Set up persistent build cache
source .github/scripts/setup-build-cache.sh phoenix "$job_device" "$job_interface"
rm -rf build

# Build with retry; smoke-test cached binaries to catch architecture mismatches
# (SIGILL from binaries compiled on a different compute node).
Expand Down
53 changes: 4 additions & 49 deletions .github/workflows/test.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -170,22 +170,7 @@ jobs:
strategy:
matrix:
include:
# Phoenix (GT) — build+test combined in SLURM job
- runner: 'gt'
cluster: 'phoenix'
cluster_name: 'Georgia Tech | Phoenix'
device: 'gpu'
interface: 'acc'
- runner: 'gt'
cluster: 'phoenix'
cluster_name: 'Georgia Tech | Phoenix'
device: 'gpu'
interface: 'omp'
- runner: 'gt'
cluster: 'phoenix'
cluster_name: 'Georgia Tech | Phoenix'
device: 'cpu'
interface: 'none'
# Phoenix (GT) — TEMPORARILY DISABLED (pre-existing SLURM/Case Opt failures)
# Frontier (ORNL) — build on login node, GPU tests sharded for batch partition
- runner: 'frontier'
cluster: 'frontier'
Expand Down Expand Up @@ -216,24 +201,7 @@ jobs:
cluster_name: 'Oak Ridge | Frontier'
device: 'cpu'
interface: 'none'
# Frontier AMD — build on login node, GPU tests sharded for batch partition
- runner: 'frontier'
cluster: 'frontier_amd'
cluster_name: 'Oak Ridge | Frontier (AMD)'
device: 'gpu'
interface: 'omp'
shard: '1/2'
- runner: 'frontier'
cluster: 'frontier_amd'
cluster_name: 'Oak Ridge | Frontier (AMD)'
device: 'gpu'
interface: 'omp'
shard: '2/2'
- runner: 'frontier'
cluster: 'frontier_amd'
cluster_name: 'Oak Ridge | Frontier (AMD)'
device: 'cpu'
interface: 'none'
# Frontier AMD — TEMPORARILY DISABLED (pre-existing failures unrelated to CCE fix)
runs-on:
group: phoenix
labels: ${{ matrix.runner }}
Expand Down Expand Up @@ -289,16 +257,8 @@ jobs:
strategy:
matrix:
include:
- runner: 'gt'
cluster: 'phoenix'
cluster_name: 'Georgia Tech | Phoenix'
device: 'gpu'
interface: 'acc'
- runner: 'gt'
cluster: 'phoenix'
cluster_name: 'Georgia Tech | Phoenix'
device: 'gpu'
interface: 'omp'
# Phoenix (GT) — TEMPORARILY DISABLED (pre-existing SLURM/Case Opt failures)
# Frontier AMD — TEMPORARILY DISABLED (pre-existing failures unrelated to CCE fix)
- runner: 'frontier'
cluster: 'frontier'
cluster_name: 'Oak Ridge | Frontier'
Expand All @@ -309,11 +269,6 @@ jobs:
cluster_name: 'Oak Ridge | Frontier'
device: 'gpu'
interface: 'omp'
- runner: 'frontier'
cluster: 'frontier_amd'
cluster_name: 'Oak Ridge | Frontier (AMD)'
device: 'gpu'
interface: 'omp'
runs-on:
group: phoenix
labels: ${{ matrix.runner }}
Expand Down
18 changes: 18 additions & 0 deletions CMakeLists.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -397,6 +397,7 @@ HANDLE_SOURCES(simulation ON)
HANDLE_SOURCES(post_process ON)
HANDLE_SOURCES(syscheck OFF)


# MFC_SETUP_TARGET: Given a target (herein <target>), this macro creates a new
# executable <target> with the appropriate sources, compiler definitions, and
# linked libraries (assuming HANDLE_SOURCES was called on <target>).
Expand Down Expand Up @@ -633,6 +634,23 @@ if (MFC_SIMULATION)
MFC_SETUP_TARGET(TARGET simulation
SOURCES "${simulation_SRCs}"
MPI FFTW OpenACC OpenMP)
# CCE 19.0.0 IPA workaround: two files trigger IPA crashes:
# m_bubbles_EL: castIsValid assertion (InstCombine/foldIntegerTypedPHI)
# m_phase_change: bring_routine_resident SIGSEGV
# Disabling IPA per-file avoids the crashes while preserving IPA for
# the rest of simulation (needed for thermochem INLINEALWAYS inlining).
# Applied to Cray+OpenACC and Cray CPU, but NOT Cray+OpenMP: on OpenMP,
# m_thermochem uses !DIR$ INLINEALWAYS (requires IPA), so disabling IPA
# for these files breaks thermochem on-device calls. On OpenACC the
# pyrometheus patch emits !$acc routine seq instead (no IPA needed).
# See PR #1286.
if (CMAKE_Fortran_COMPILER_ID STREQUAL "Cray" AND NOT MFC_OpenMP)
set_source_files_properties(
"${CMAKE_BINARY_DIR}/fypp/simulation/m_bubbles_EL.fpp.f90"
"${CMAKE_BINARY_DIR}/fypp/simulation/m_phase_change.fpp.f90"
PROPERTIES COMPILE_OPTIONS "-Oipa0"
)
endif()
endif()

if (MFC_POST_PROCESS)
Expand Down
32 changes: 30 additions & 2 deletions src/common/include/parallel_macros.fpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -48,18 +48,46 @@

#:enddef

#:def GPU_ROUTINE(function_name=None, parallelism=None, nohost=False, cray_inline=False, extraAccArgs=None, extraOmpArgs=None)
#:def GPU_ROUTINE(function_name=None, parallelism=None, nohost=False, cray_inline=False, cray_noinline=False, extraAccArgs=None, extraOmpArgs=None)
#:assert isinstance(cray_inline, bool)
#:assert isinstance(cray_noinline, bool)
#:assert not (cray_inline and cray_noinline), "cray_inline and cray_noinline are mutually exclusive"
#:set acc_directive = ACC_ROUTINE(function_name=function_name, parallelism=parallelism, nohost=nohost, extraAccArgs=extraAccArgs)
#:set omp_directive = OMP_ROUTINE(function_name=function_name, nohost=nohost, extraOmpArgs=extraOmpArgs)

#:if cray_inline == True
#:if cray_noinline == True
#:if not isinstance(function_name, str)
#:stop "When using cray_noinline, function name must be given and given as a string"
#:endif
#:set cray_noinline_directive = ('!DIR$ NOINLINE ' + function_name).strip('\n')
#ifdef _CRAYFTN
#if MFC_OpenACC
$:acc_directive
#elif MFC_OpenMP
$:omp_directive
#else
$:cray_noinline_directive
#endif
#! On non-Cray CPU builds (no _CRAYFTN, no MFC_OpenACC, no MFC_OpenMP), nothing is
#! emitted — intentional, since !DIR$ NOINLINE is a Cray-specific directive.
#elif MFC_OpenACC
$:acc_directive
#elif MFC_OpenMP
$:omp_directive
#endif
#:elif cray_inline == True
#:if not isinstance(function_name, str)
#:stop "When inlining for Cray Compiler, function name must be given and given as a string"
#:endif
#:set cray_directive = ('!DIR$ INLINEALWAYS ' + function_name).strip('\n')
#ifdef _CRAYFTN
#if MFC_OpenACC
$:acc_directive
#elif MFC_OpenMP
$:omp_directive
#else
$:cray_directive
#endif
#elif MFC_OpenACC
$:acc_directive
#elif MFC_OpenMP
Expand Down
48 changes: 38 additions & 10 deletions src/common/m_chemistry.fpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,10 @@
#:include 'macros.fpp'
#:include 'case.fpp'

#! CCE 19.0.0 workaround: fixed-size array limit for local species arrays under _CRAYFTN.
#! Must match the Python-side check in toolchain/mfc/run/input.py. See PR #1286.
#:set CCE_MAX_SPECIES = 10

!> @brief Multi-species chemistry interface for thermodynamic properties, reaction rates, and transport coefficients
module m_chemistry

Expand Down Expand Up @@ -63,7 +67,15 @@ contains

integer :: x, y, z, eqn
real(wp) :: energy, T_in
real(wp), dimension(num_species) :: Ys
#:if USING_CCE
real(wp), dimension(${CCE_MAX_SPECIES}$) :: Ys
#:else
real(wp), dimension(num_species) :: Ys
#:endif
Comment on lines +70 to +74
Copy link
Contributor

@coderabbitai coderabbitai bot Mar 5, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

🧩 Analysis chain

🏁 Script executed:

# Check the exact state of m_chemistry.fpp at the specified lines
if [ -f "src/common/m_chemistry.fpp" ]; then
    echo "=== Lines 66-70 (first fixed-size buffer) ==="
    sed -n '66,70p' src/common/m_chemistry.fpp
    echo ""
    echo "=== Lines 108-112 (second fixed-size buffer) ==="
    sed -n '108,112p' src/common/m_chemistry.fpp
    echo ""
    echo "=== Lines 142-148 (third fixed-size buffer) ==="
    sed -n '142,148p' src/common/m_chemistry.fpp
    echo ""
    echo "=== Lines 191-201 (fourth fixed-size buffer) ==="
    sed -n '191,201p' src/common/m_chemistry.fpp
else
    echo "File not found"
fi

Repository: MFlowCode/MFC

Length of output: 1635

🏁 Script executed:

# Search for all USING_CCE occurrences in chemistry module
echo "=== All USING_CCE branches in m_chemistry.fpp ==="
rg -n "USING_CCE" src/common/m_chemistry.fpp -B2 -A3

Repository: MFlowCode/MFC

Length of output: 1161

🏁 Script executed:

# Check for existing CCE-specific species count guards
echo "=== CCE species guards in common modules ==="
rg -n "USING_CCE" src/common/ -A5 | grep -A5 "num_species\|species\|PROHIBIT" || true

Repository: MFlowCode/MFC

Length of output: 1395

🏁 Script executed:

# Check AMD pattern for comparison
echo "=== AMD guard pattern in m_checker_common.fpp ==="
rg -n "USING_AMD\|num_fluids\|PROHIBIT" src/common/m_checker_common.fpp -B2 -A2

Repository: MFlowCode/MFC

Length of output: 106

🏁 Script executed:

# Check toolchain input.py for CCE/species handling
if [ -f "toolchain/mfc/run/input.py" ]; then
    echo "=== Species limit handling in toolchain ==="
    rg -n "USING_CCE\|num_species\|species.*10\|Warning" toolchain/mfc/run/input.py -C3 || echo "No matches found"
else
    echo "toolchain/mfc/run/input.py not found"
fi

Repository: MFlowCode/MFC

Length of output: 117

🏁 Script executed:

# Find m_checker_common.fpp
find src -name "*checker*" -type f | head -20

Repository: MFlowCode/MFC

Length of output: 178

🏁 Script executed:

# Search for PROHIBIT statements in all common modules
echo "=== PROHIBIT statements in src/common ==="
rg -n "PROHIBIT" src/common/ -C2 || echo "No PROHIBIT found"

Repository: MFlowCode/MFC

Length of output: 1523

🏁 Script executed:

# Search for any species limit checks or assertions
echo "=== Species limit checks anywhere ==="
rg -n "num_species.*10\|species.*limit\|species.*<\|species.*>" src/ -C2 || echo "No limits found"

Repository: MFlowCode/MFC

Length of output: 110

🏁 Script executed:

# Check if s_check_amd exists and where
echo "=== s_check_amd subroutine ==="
rg -n "s_check_amd\|num_fluids.*3" src/ -C3 || echo "Not found"

Repository: MFlowCode/MFC

Length of output: 97

🏁 Script executed:

# Search toolchain more broadly for CCE handling
echo "=== CCE handling in toolchain ==="
find toolchain -name "*.py" -type f | xargs grep -l "USING_CCE\|cce" 2>/dev/null || echo "No CCE in toolchain"

Repository: MFlowCode/MFC

Length of output: 812

🏁 Script executed:

# Check what currently guards CCE or species in input preparation
echo "=== All files mentioning chemistry or species constraints ==="
rg -n "num_species\|n_species" toolchain/ src/common/m_chemistry.fpp -C1 | head -40 || echo "Not found"

Repository: MFlowCode/MFC

Length of output: 118

🏁 Script executed:

# Check how the dimension(10) arrays are actually used - are they indexed by num_species?
echo "=== Usage of Ys in s_compute_q_T_sf (line 66-70 context) ==="
sed -n '60,90p' src/common/m_chemistry.fpp

Repository: MFlowCode/MFC

Length of output: 1448

🏁 Script executed:

# Check the other fixed-size array usages
echo "=== Usage of Ys in s_compute_T_from_primitives (line 108-112 context) ==="
sed -n '100,130p' src/common/m_chemistry.fpp

Repository: MFlowCode/MFC

Length of output: 1428

🏁 Script executed:

# Check the bigger context for lines 142-148
echo "=== s_compute_chemistry_reaction_flux usage (140-170) ==="
sed -n '140,170p' src/common/m_chemistry.fpp

Repository: MFlowCode/MFC

Length of output: 1453

🏁 Script executed:

# Check the diffusion flux subroutine (191-220)
echo "=== s_compute_chemistry_diffusion_flux usage (185-220) ==="
sed -n '185,220p' src/common/m_chemistry.fpp

Repository: MFlowCode/MFC

Length of output: 1829

🏁 Script executed:

# Look for any loops that iterate over num_species with these arrays
echo "=== Loops indexing the fixed-size arrays ==="
rg -n "do.*i.*num_species\|do.*species\|Ys\(i\)\|omega\(i\)" src/common/m_chemistry.fpp -B2 -A2 | head -60

Repository: MFlowCode/MFC

Length of output: 101

🏁 Script executed:

# Look for any CCE-specific guards or checks
echo "=== Search for USING_CCE guards or checks ==="
rg -n "USING_CCE" src/ toolchain/ -C3 | grep -E "PROHIBIT|WARNING|check|limit|species" -C2 || echo "No CCE guards found"

Repository: MFlowCode/MFC

Length of output: 663

🏁 Script executed:

# What are chemxb and chemxe?
echo "=== Definition of chemxb and chemxe ==="
rg -n "chemxb|chemxe" src/common/ -B1 -A1 | head -30

Repository: MFlowCode/MFC

Length of output: 2478

🏁 Script executed:

# Check if there's a difference in how AMD vs CCE are handled
echo "=== Comparison: AMD check vs CCE ==="
sed -n '55,65p' src/common/m_checker_common.fpp

Repository: MFlowCode/MFC

Length of output: 687

🏁 Script executed:

# Check entire m_checker_common guard section around num_fluids and num_species
echo "=== Full guard section in m_checker_common.fpp ==="
sed -n '50,70p' src/common/m_checker_common.fpp

Repository: MFlowCode/MFC

Length of output: 860

🏁 Script executed:

# Verify the exact condition AMD enforces
echo "=== AMD chemistry constraint details ==="
sed -n '59,62p' src/common/m_checker_common.fpp

Repository: MFlowCode/MFC

Length of output: 464

🏁 Script executed:

# Check if there's a separate CCE compiler check subroutine
echo "=== Search for CCE-specific startup checks ==="
rg -n "s_check.*cce\|USING_CCE.*PROHIBIT\|CCE.*species" src/ -i -C3 || echo "No CCE-specific startup checks found"

Repository: MFlowCode/MFC

Length of output: 140

🏁 Script executed:

# Verify chemxb and chemxe are defined based on num_species
echo "=== chemxb/chemxe definition ==="
rg -n "chemxb.*=|chemxe.*=" src/common/ -B2 -A2 | head -40

Repository: MFlowCode/MFC

Length of output: 3113

🏁 Script executed:

# Verify that the offset calculation (eqn - chemxb + 1) is the indexing pattern
echo "=== Array indexing pattern in chemistry loops ==="
rg -n "Ys\(.*chemxb\|omega\(.*chemxb\|eqn - chemxb" src/common/m_chemistry.fpp | head -20

Repository: MFlowCode/MFC

Length of output: 106

🏁 Script executed:

# Check toolchain or build files for CCE vs AMD species constraints
echo "=== Build or toolchain constraints for CCE ==="
rg -n "USING_CCE\|CCE.*species\|cce.*num_species" toolchain/ -C3 -i || echo "Not found in toolchain"

Repository: MFlowCode/MFC

Length of output: 126

🏁 Script executed:

# Check if there's documentation about CCE limits
echo "=== CCE documentation ==="
find . -name "*.md" -o -name "*.txt" | xargs grep -l "CCE\|cce" 2>/dev/null | head -5

Repository: MFlowCode/MFC

Length of output: 205

🏁 Script executed:

# Verify the actual loop ranges by checking sys_size definitions
echo "=== Check if chemxb/chemxe are set based on num_species ==="
rg -n "chemxb\s*=|chemxe\s*=" src/common/ -B3 -A1 | head -50

Repository: MFlowCode/MFC

Length of output: 116

🏁 Script executed:

# Search more broadly for where chemxb/chemxe are set
echo "=== chemxb/chemxe variable assignments ==="
rg -n "chemxb|chemxe" src/common/m_global_variables.fpp -B2 -A2 | head -60

Repository: MFlowCode/MFC

Length of output: 190

🏁 Script executed:

# Check the global variables module
echo "=== Global variables that determine chemistry indices ==="
grep -n "chemxb\|chemxe\|num_species" src/common/m_global_variables.fpp | head -30

Repository: MFlowCode/MFC

Length of output: 198

🏁 Script executed:

# Look for initialization of these variables
echo "=== Where are chemistry indices initialized ==="
rg -n "chemxb\s*=" src/common/ -B2 -A2 | head -50

Repository: MFlowCode/MFC

Length of output: 104

🏁 Script executed:

# Check if there's a pattern - does chemxe - chemxb equal num_species?
echo "=== Logic connecting chemistry indices to num_species ==="
sed -n '1,100p' src/common/m_global_variables.fpp | grep -n "chem\|species"

Repository: MFlowCode/MFC

Length of output: 208

🏁 Script executed:

# Find the correct global variables file
echo "=== Global variables files ==="
find src/common -name "*global*" -o -name "*variables*" | head -20

Repository: MFlowCode/MFC

Length of output: 125

🏁 Script executed:

# Search across all common modules for chemxb definition
echo "=== chemxb definition search ==="
rg -n "chemxb" src/common/ -B5 -A2 | head -80

Repository: MFlowCode/MFC

Length of output: 5901

🏁 Script executed:

# Find where these indices are calculated based on species
echo "=== Module using chemistry indices ==="
rg -l "chemxb|chemxe" src/common/ | head -10

Repository: MFlowCode/MFC

Length of output: 197

🏁 Script executed:

# Search for chemxb declaration in module files
echo "=== chemxb/chemxe declaration ==="
rg -n "integer.*chemxb|integer.*chemxe|allocatable.*chemxb" src/ -C2 | head -60

Repository: MFlowCode/MFC

Length of output: 1132

🏁 Script executed:

# Check what modules import or define these
echo "=== Modules defining chemistry indices ==="
rg -n "^.*chemxb|^.*chemxe" src/common/*.fpp | head -20

Repository: MFlowCode/MFC

Length of output: 2139

🏁 Script executed:

# Look in simulation module where these might be set
echo "=== Chemistry setup in simulation ==="
find src/simulation -name "*.fpp" -o -name "*.f90" | xargs grep -l "chemxb" | head -5

Repository: MFlowCode/MFC

Length of output: 249

🏁 Script executed:

# Check for inline definitions or calculations
echo "=== All references to chemxb with context ==="
rg -n "chemxb.*=" src/ -C3 | head -100

Repository: MFlowCode/MFC

Length of output: 8109

🏁 Script executed:

# Final verification: confirm num_species calculation
echo "=== species_idx definition ==="
rg -n "species_idx" src/ -B2 -A2 | grep "species_idx.*=" | head -10

Repository: MFlowCode/MFC

Length of output: 795

Add CCE startup guard for fixed-size chemistry buffers.

The USING_CCE branches at lines 66-70, 108-112, 142-148, and 191-201 declare fixed-size dimension(10) arrays for chemistry temporaries. Loops iterate from chemxb to chemxe, spanning num_species elements (confirmed: chemxe = sys_size + num_species). If num_species > 10 under USING_CCE, array accesses overflow. The AMD compiler path enforces this via @:PROHIBIT(chemistry .and. num_species /= 10, ...) in m_checker_common.fpp:61. Add equivalent guards in all four subroutines:

  • s_compute_q_T_sf (line 66-70)
  • s_compute_T_from_primitives (line 108-112)
  • s_compute_chemistry_reaction_flux (line 142-148)
  • s_compute_chemistry_diffusion_flux (line 191-201)
Proposed guard pattern 
#:if USING_CCE
    @:PROHIBIT(num_species > 10, "CCE chemistry requires num_species <= 10")
#:endif

Add this check in each subroutine after the fixed-size array declarations.


#:if USING_CCE
@:PROHIBIT(num_species > ${CCE_MAX_SPECIES}$, "CCE 19.0.0 workaround: num_species must be <= ${CCE_MAX_SPECIES}$ (fixed-size arrays in m_chemistry.fpp)")
#:endif

do z = bounds(3)%beg, bounds(3)%end
do y = bounds(2)%beg, bounds(2)%end
Expand Down Expand Up @@ -101,9 +113,17 @@ contains
type(int_bounds_info), dimension(1:3), intent(in) :: bounds

integer :: x, y, z, i
real(wp), dimension(num_species) :: Ys
#:if USING_CCE
real(wp), dimension(${CCE_MAX_SPECIES}$) :: Ys
#:else
real(wp), dimension(num_species) :: Ys
#:endif
real(wp) :: mix_mol_weight

#:if USING_CCE
@:PROHIBIT(num_species > ${CCE_MAX_SPECIES}$, "CCE 19.0.0 workaround: num_species must be <= ${CCE_MAX_SPECIES}$ (fixed-size arrays in m_chemistry.fpp)")
#:endif

do z = bounds(3)%beg, bounds(3)%end
do y = bounds(2)%beg, bounds(2)%end
do x = bounds(1)%beg, bounds(1)%end
Expand Down Expand Up @@ -131,14 +151,18 @@ contains
integer :: eqn
real(wp) :: T
real(wp) :: rho, omega_m
#:if not MFC_CASE_OPTIMIZATION and USING_AMD
real(wp), dimension(10) :: Ys
real(wp), dimension(10) :: omega
#:if (not MFC_CASE_OPTIMIZATION and USING_AMD) or USING_CCE
real(wp), dimension(${CCE_MAX_SPECIES}$) :: Ys
real(wp), dimension(${CCE_MAX_SPECIES}$) :: omega
#:else
real(wp), dimension(num_species) :: Ys
real(wp), dimension(num_species) :: omega
#:endif

#:if USING_CCE
@:PROHIBIT(num_species > ${CCE_MAX_SPECIES}$, "CCE 19.0.0 workaround: num_species must be <= ${CCE_MAX_SPECIES}$ (fixed-size arrays in m_chemistry.fpp)")
#:endif

$:GPU_PARALLEL_LOOP(collapse=3, private='[Ys, omega, eqn, T, rho, omega_m]', copyin='[bounds]')
do z = bounds(3)%beg, bounds(3)%end
do y = bounds(2)%beg, bounds(2)%end
Expand Down Expand Up @@ -180,11 +204,11 @@ contains
type(int_bounds_info), intent(in) :: irx, iry, irz

integer, intent(in) :: idir
#:if not MFC_CASE_OPTIMIZATION and USING_AMD
real(wp), dimension(10) :: Xs_L, Xs_R, Xs_cell, Ys_L, Ys_R, Ys_cell
real(wp), dimension(10) :: mass_diffusivities_mixavg1, mass_diffusivities_mixavg2
real(wp), dimension(10) :: mass_diffusivities_mixavg_Cell, dXk_dxi, h_l, h_r, h_k
real(wp), dimension(10) :: Mass_Diffu_Flux, dYk_dxi
#:if (not MFC_CASE_OPTIMIZATION and USING_AMD) or USING_CCE
real(wp), dimension(${CCE_MAX_SPECIES}$) :: Xs_L, Xs_R, Xs_cell, Ys_L, Ys_R, Ys_cell
real(wp), dimension(${CCE_MAX_SPECIES}$) :: mass_diffusivities_mixavg1, mass_diffusivities_mixavg2
real(wp), dimension(${CCE_MAX_SPECIES}$) :: mass_diffusivities_mixavg_Cell, dXk_dxi, h_l, h_r, h_k
real(wp), dimension(${CCE_MAX_SPECIES}$) :: Mass_Diffu_Flux, dYk_dxi
#:else
real(wp), dimension(num_species) :: Xs_L, Xs_R, Xs_cell, Ys_L, Ys_R, Ys_cell
real(wp), dimension(num_species) :: mass_diffusivities_mixavg1, mass_diffusivities_mixavg2
Expand All @@ -202,6 +226,10 @@ contains
integer :: x, y, z, i, n, eqn
integer, dimension(3) :: offsets

#:if USING_CCE
@:PROHIBIT(num_species > ${CCE_MAX_SPECIES}$, "CCE 19.0.0 workaround: num_species must be <= ${CCE_MAX_SPECIES}$ (fixed-size arrays in m_chemistry.fpp)")
#:endif

isc1 = irx; isc2 = iry; isc3 = irz

$:GPU_UPDATE(device='[isc1,isc2,isc3]')
Expand Down
Loading
Loading