PyAutoLabs · Jammy2211 · Aug 24, 2026 · Aug 24, 2026 · Aug 24, 2026 · Aug 24, 2026
diff --git a/hpc/batch_gpu/submit_slogdet_nan_attribution_a100 b/hpc/batch_gpu/submit_slogdet_nan_attribution_a100
@@ -0,0 +1,65 @@
+#!/bin/bash -l
+#
+# W7 — CP-4 follow-up: per-draw NaN attribution on the AdaptSplit NaN wall
+# (autolens_profiling#164). Copy of ../batch_gpu/submit_slogdet_ab_adaptsplit_a100
+# pointed at scripts/misc/searches/slogdet_nan_attribution.py's tier-attack mode.
+#
+# NOT SUBMITTED as part of W7 (2026-08-24): the laptop/CPU replay of both
+# tier files (a100, ral_cpu) already answered the tier-attack question — see
+# results/notes/inference/phase_08_regularization/slogdet_ab/attribution/ and
+# RESULTS.md's "W7 addendum" section. This script exists so the SAME
+# comparison can be re-run natively on an A100 (rather than replayed on CPU
+# under JAX_PLATFORMS=cpu) if a future phase needs GPU-native
+# cholesky/slogdet numbers for the tier_flip / marginal_band draws specifically
+# — e.g. to check whether cuSOLVER's slogdet/cholesky agree with the CPU
+# LAPACK reference the same way OpenBLAS does. Submit manually if that
+# question comes up; do not submit as a matter of course.
+#
+# --matrix-only skips the fit_from() gradient closures (the expensive part —
+# see the module docstring's cost note) and only forms
+# curvature_reg_matrix_reduced, comparing LAPACK cholesky, np.linalg.slogdet,
+# an eigvalsh-reference logdet, and jnp cholesky/slogdet on it. --classes
+# tier_flip,marginal_band targets exactly the draws whose classification
+# depends on hardware (see PROGRAMME.md #164 / RESULTS.md).
+
+#SBATCH -J slogdet_nan_attribution_a100
+#SBATCH --partition=gpu
+#SBATCH --gres=gpu:1
+#SBATCH --ntasks=1
+#SBATCH --cpus-per-task=8
+#SBATCH --mem=64gb
+#SBATCH --time=2:00:00
+#SBATCH -o output/output.%A.out
+#SBATCH -e error/error.%A.err
+#SBATCH --mail-type=END,FAIL
+#SBATCH --mail-user=james.w.nightingale@durham.ac.uk
+
+export AP_ROOT=/mnt/ral/jnightin/autolens_profiling
+source $AP_ROOT/activate.sh
+
+export JAX_PLATFORM_NAME=cuda
+export JAX_PLATFORMS=cuda,cpu
+export XLA_PYTHON_CLIENT_PREALLOCATE=false
+export JAX_ENABLE_X64=True
+export NUMBA_CACHE_DIR=/tmp/numba_cache
+export MPLCONFIGDIR=/tmp/matplotlib
+
+nvidia-smi
+
+echo "=========================================="
+date
+echo "Experiment: W7 slogdet NaN attribution — GPU-native tier attack"
+echo "Cell:       imaging/delaunay_adapt_split/hst"
+
+cd $AP_ROOT
+
+python3 scripts/misc/searches/slogdet_nan_attribution.py \
+    --dataset-class imaging \
+    --model-type delaunay_adapt_split \
+    --instrument hst \
+    --tier-file a100 \
+    --classes tier_flip,marginal_band \
+    --matrix-only
+
+echo "Finished."
+date
diff --git a/results/notes/inference/DECISIONS.md b/results/notes/inference/DECISIONS.md
@@ -393,3 +393,78 @@ not run.
 **Records:** `scripts/misc/searches/_targets.py`,
 `results/baselines/InferenceRefs_v1/`, PROGRAMME.md Phase 1 section + W4
 table row (this commit).
+
+---
+
+---
+
+## 2026-08-24 — W7 (CP-4 follow-up, #164): per-draw attribution + CP-4 re-scored on the clean subset — verdict unchanged
+
+**Record:** `scripts/misc/searches/slogdet_nan_attribution.py` replayed
+individual stored `delaunay_adapt_split` draws (both tiers) non-jitted,
+drilling into `inversion.*` matrices and small `jax.grad` closures to
+assign each sampled draw one mechanism label (170 draws classified across
+both tiers: 8 full-gradient, 87 A100 + 83 RAL CPU matrix-only). Two driver
+artefacts identified and fixed at harvest (`slogdet_ab.py`): (1) six
+`descent`-source draws carried a NEGATIVE physical regularization
+coefficient and ten more were dead (entirely-NaN) checkpointed lanes — the
+prior "Ops notes" misread this as a log axis; both are now dropped at
+harvest. (2) all 128 `lambda_transect` draws sat at the anchor where
+`ell_comps`/shear are exactly (0, 0), an undefined-gradient point in
+`autogalaxy/convert.py`'s sqrt-magnitude conversion unrelated to the
+regularization wall; the anchor is now jittered 1e-3 in unit-cube space off
+any exact zero. A tier-attack on draw 96 (coefficients [3.5e5, 536]: finite
+under both arms on A100 with a 9,619-nat slogdet/cholesky delta, NaN under
+both arms on RAL CPU for the IDENTICAL input vector) traced the
+tier-dependence to `cond(curvature_reg_matrix_reduced) = 4.5e18` — far past
+float64's ~1e16 precision floor — where the log-det terms remain
+individually computable (and agree with the original A100 value to 5
+significant figures) while the reconstruction linear solve on the SAME
+matrix is itself NaN; different BLAS/LAPACK backends (cuSOLVER vs OpenBLAS)
+diverge on whether that solve returns a number. Cross-tier value agreement
+on mutually-finite shared draws: slogdet max|Δ| 6.72 nats, cholesky 1.30
+nats — both far above the ~1e-4 clean-PD floor, confirming disagreement
+concentrates in the marginal band.
+
+**CP-4 re-scored on the clean subset (excluding dead-lane,
+invalid-coefficient and anchor-singularity draws):** A100 n=416 → 272 clean
+(144 excluded: 10 dead-lane, 6 invalid-coefficient, 128
+anchor-singularity); RAL CPU n=384 → 256 clean (128 excluded, all
+anchor-singularity — this tier's harvest never ran the descent arm, so it
+has no dead-lane/invalid-coefficient rows). On the clean subset, all three
+per-draw criteria STILL fail on both tiers — criterion 1 (zero slogdet
+NaNs): 22/272 A100, 20/256 CPU still NaN; criterion 2 (value equality):
+48/218 A100 and 40/207 CPU mutually-finite comparisons exceed tolerance,
+max|Δ| unchanged (9,619 nats A100, 1.62 nats CPU); criterion 3 (finite
+gradients): 22/272 A100, 23/256 CPU still non-finite. Criterion 4 (runtime)
+is population-level and unchanged (1.03× A100, 3.74× CPU). **Verdict: FAIL
+on both tiers, unchanged from the original Phase 8A run.** Matrix-only
+population sampling (170 draws) confirms the residual failures are not
+driver artefacts: once the two fixed bugs are excluded, the sampled
+`nan_both` draws are 53% (A100) / 80% (RAL CPU) `genuinely_singular`
+(cond ≥ 1e16 or LAPACK-Cholesky failure) — the genuinely-singular λ⁴
+population (`prior`-source draws, regularization coefficient ~4e5-9e5)
+that CP-4 was measuring in the first place. A `marginal_tier_flippable`
+band (`cond` in `[1e12, 1e16)`) is also a real, sizeable population
+(18/87 A100, 18/83 RAL CPU sampled draws, every class except `truth_bar`)
+— the conditioning range where a hardware/BLAS change can flip a draw's
+NaN verdict even though the matrix is nominally still invertible.
+
+**Human call (2026-08-24), confirming the W8 adoption stands unchanged:**
+the re-score does not overturn anything — slogdet remains the GPU
+gradient-work default in this repo (W8, already shipped), the library
+default stays cholesky (opt-in), and the reminder to revisit the
+PyAutoArray default is still owed once W9 lands. W9 (#166) is unblocked
+with three concrete inputs from this investigation: (i) quantify whether
+slogdet ever returns a wrong-but-finite number where cholesky legitimately
+NaNs, using the `cond >= 1e16` threshold and the max|Δ| vs an
+eigvalsh-reference logdet (established here: up to 9,619 nats,
+tier-dependent, concentrated in the `[1e12, 1e16)` marginal band); (ii) the
+library-default recommendation is slogdet on GPU / cholesky on CPU, with
+the 3.7× CPU cost restated as the reason CPU keeps the historical default;
+(iii) two guards independent of log-det method, both now shipped in
+`slogdet_ab.py`: reject non-finite or out-of-prior-bound lane vectors at
+descent-harvest time, and never anchor a transect/probe at an exact-zero
+ell_comps/shear component. Record: `phase_08_regularization/RESULTS.md`
+"W7 addendum" + "CP-4 re-scored" sections; attribution artifacts under
+`slogdet_ab/attribution/`.
diff --git a/results/notes/inference/PROGRAMME.md b/results/notes/inference/PROGRAMME.md
@@ -32,7 +32,7 @@ expectation · **[CONTRADICTED]** existing evidence pushed back on the brief ·
 | Phase 0(e) searches README dashboard loop | **complete** | PR#139 (2026-08-18): nested-layout scanner, 34 rows render, truth-bar rows verified |
 | Phase 2 NSS mainline re-tune | **scan complete 2026-08-24 (CP-2 ✅, CP-5 ✅)** — H2.1 closed: logZ bias was inner-kernel under-mixing (inner=30: 5/5 seeds +1.0 ± 0.4 nats vs Nautilus); operating point n200/nd100/inner30 costs 5.0× the Nautilus sampler wall on MGE, 18.4× on Delaunay (same-night re-baselines: Nautilus mge 707 s, delaunay 1,891 s). Sample economy: Kish ESS 1,315 vs Nautilus 4,121 per run; ~940 vs ~15 evals/ESS. **GATE A CALLED 2026-08-24 — Nautilus stays baseline; af.NSS = tuned alternative** | `phase_02_nss_mainline/RESULTS.md` + DECISIONS.md 2026-08-24 |
 | Phase 3 Prodigy reliability | **wave 1 complete + adversarially reviewed (CP-3 ✅ 2026-08-23, positions-off)** — p̂_hit = 0.048 [0.037, 0.061] (lower bound; n256 tier, ~1,280 distinct draws — tiers share lane-index draws, n-dependence unmeasured); demonstrated ≥99% reliability at **n=256 only** (joint-95% worst case), 3.4–4.5× under the viz-stripped Nautilus wall; zero parameter-recovery impostors; ~half of lanes end pinned (H3.3, measured not closed); θ_E diagnostic uninformative for H3.1 (withdrawn). **Gate B pt 1 CALLED (human-ratified 2026-08-23** — DECISIONS.md): Prodigy(n=256, prior_box, autoconv, positions-off) ratified as global MAP searcher on MGE; pt 2 (PositionsLH) open. Fresh-seed tier 2026-08-24: n128 5/5, n256 15/15 cumulative (Wilson-95 lower 0.80); Nautilus re-baseline 707 s → 3.1–4.3× | `phase_03_prodigy_reliability/RESULTS.md` + `ADVERSARIAL_REVIEW.md` |
-| Phase 8A slogdet A/B (CP-4) | **complete 2026-08-24 — FAIL as pre-registered.** knn stressor never walls (VOID both tiers); on Delaunay+AdaptSplit slogdet rescues 64–73 % of NaNs, 0 regressions, but 20–32 NaN-under-both, λ-transect grads non-finite, marginal-band Δ up to 9,619 nats (A100), CPU 3.7×. Human call: adopt as this repo's GPU gradient-cell default (W8), library stays opt-in, residual NaNs → W7, 8B in parallel | `phase_08_regularization/RESULTS.md` + DECISIONS.md 2026-08-24 |
+| Phase 8A slogdet A/B (CP-4) | **complete 2026-08-24 — FAIL as pre-registered.** knn stressor never walls (VOID both tiers); on Delaunay+AdaptSplit slogdet rescues 64–73 % of NaNs, 0 regressions, but 20–32 NaN-under-both, λ-transect grads non-finite, marginal-band Δ up to 9,619 nats (A100), CPU 3.7×. Human call: adopt as this repo's GPU gradient-cell default (W8), library stays opt-in, residual NaNs → W7, 8B in parallel — re-scored 2026-08-24 (W7): per-draw attribution (170 draws classified) finds two harvest/driver bugs (negative-coefficient + dead-lane descent rows; zero-anchor λ-transect gradient) now fixed; CP-4 re-scored on the clean subset excluding both — **verdict unchanged, FAIL on both tiers** (residual failures are the genuinely-singular λ⁴ population, 53–80% of sampled `nan_both` draws); tier-dependence traced to `cond ~ 1e15-1e19` where the reconstruction solve NaNs despite finite log-det terms. W8 adoption stands; W9 unblocked | `phase_08_regularization/RESULTS.md` + DECISIONS.md 2026-08-24 |
 | Phases 1, 4–7, 8B–13 | not started | — |
 | Gates A–F | **A CALLED 2026-08-24** (Nautilus stays nested baseline; af.NSS tuned alternative); **B part 1 CALLED 2026-08-23** (Prodigy n=256 global MAP, MGE, positions-off; caveat (a) n=256-only stands 2026-08-24); B pt 2, C–F open — C criterion reworded 2026-08-24 (batched-pipeline value, not single-fit ESS/s vs Nautilus) | DECISIONS.md |