Fix self-hosted CI robustness: build cache, SLURM QOS, and submit resilience#1295

Open

sbryngelson wants to merge 15 commits intoMFlowCode:masterfrom

sbryngelson:fix/ci-robustness

Commits on Mar 6, 2026

Fix Frontier benchmark SLURM: use batch+1:59+normal QOS

Spencer Bryngelson
and
claude
committed
Fix bench.yml: restore timeout-minutes to 480 (revert accidental 240)

Spencer Bryngelson
and
claude
committed
Remove persistent build cache for self-hosted test runners

authored and
Spencer Bryngelson
committed
Remove build cache from benchmark jobs on Phoenix and Frontier

sbryngelson
authored and
Spencer Bryngelson
committed
Fix submit.sh to survive monitor SIGKILL by re-checking SLURM state

authored and
Spencer Bryngelson
committed
Extract monitor SIGKILL recovery into shared run_monitored_slurm_job.sh

sbryngelson
and
claude
committed
Reduce benchmark steps and switch Frontier bench to batch/normal QOS

sbryngelson
and
claude
committed
Cap bench script parallelism at 64 to fix GNR node failures

sbryngelson
and
claude
committed
Disable AVX-512 FP16 to fix build on Granite Rapids nodes

sbryngelson
and
claude
committed
Fix Rich MarkupError crash when build output contains bracket paths

sbryngelson
and
claude
committed
Merge branch 'master' into fix/ci-robustness
sbryngelson
authored
Address bot review comments: sacct -X flag, dead job_type var, stale comment
Spencer Bryngelson
committed
Fix bench: use PR's submit.sh for master job to get SIGKILL recovery

sbryngelson
and
claude
committed
Fix submit_and_monitor_bench.sh: define SCRIPT_DIR before use

sbryngelson
and
claude
committed

Commits on Mar 7, 2026

bench: update Phoenix tmpbuild path to project storage
sbryngelson
committed