CP-4 / Phase 8A: slogdet vs cholesky A/B driver + RAL results — FAIL as pre-registered

hpc/batch_cpu/submit_slogdet_ab_adaptsplit_ral_cpu

@@ -0,0 +1,97 @@
+#!/bin/bash -l
+#
+# Phase 8A / CP-4 — slogdet vs cholesky A/B on the free-AdaptSplit NaN wall.
+# RAL CPU tier: the cheap, deterministic half of the experiment.
+#
+# PRE-REGISTERED PASS CRITERIA (all four must hold; stated before any run):
+#
+#   1. zero slogdet NaNs on the replay set
+#   2. value equality within float tolerance on PD points
+#   3. finite gradients
+#   4. runtime within 2x
+#
+# TWO CELLS, and the difference matters. The pre-registration names the
+# free-AdaptSplit stressor as `knn`, but this repo's #117 record says the KNN
+# mesh's high-coefficient region is a finite over-regularized floor, while the
+# NaN WALL is on the Delaunay mesh with the same free AdaptSplit reg
+# (_setup._delaunay_adapt_split_model; multi_start_prodigy/delaunay.py). Both
+# arms run here: `knn` is what was registered, `delaunay_adapt_split` is where
+# the failure slogdet exists to remove has actually been seen. A `knn` arm that
+# reports zero cholesky NaNs is a real finding about that cell, and the script
+# exits non-zero calling it VOID rather than reporting four vacuous passes — do
+# not read that exit code as a broken job without reading the verdict block.
+#
+# What this tier can and cannot answer. The conditioning wall is crossed
+# deterministically by the lambda transect at any hardware tier, which is what
+# makes this CPU-viable. What CPU cannot supply is a REAL search trajectory
+# through the wall: the 335003-5 arms measured zero value-NaN lane-steps here
+# (results/notes/inference/phase_00_unblocking/RESULTS.md:189), so the
+# trajectory-side observation remains GPU-scale. Run the A100 sibling
+# (../batch_gpu/submit_slogdet_ab_adaptsplit_a100, which adds --descent-steps).
+#
+# Runtime estimate: the knn cell measured ~59 s per 16-start batch_size=4
+# value_and_grad step on this tier (RESUME.md, jobs 335003-5) => ~3.7 s per
+# per-draw value_and_grad; a laptop-CPU validation of this script measured
+# ~10 s/draw with a ~30-50 s XLA compile per arm. 384 draws x 2 arms x 2 cells
+# ~= 1.5-4 h of warm wall. 6 h wall with headroom.
+#
+# NOTE on --partition: jobs 335003-5 ran on partition `ral`
+# (euclid-ral-compute-1). The older CPU exemplar in this directory uses `cpu`.
+# Check `sinfo` before submitting.
+
+#SBATCH -J slogdet_ab_adaptsplit_cpu
+#SBATCH --partition=ral
+#SBATCH --ntasks=1
+#SBATCH --cpus-per-task=32
+#SBATCH --mem=128gb
+#SBATCH --time=6:00:00
+#SBATCH -o output/output.%A.out
+#SBATCH -e error/error.%A.err
+#SBATCH --mail-type=END,FAIL
+#SBATCH --mail-user=james.w.nightingale@durham.ac.uk
+
+export AP_ROOT=/mnt/ral/jnightin/autolens_profiling
+source $AP_ROOT/activate.sh
+
+# fp64 is not optional here: an fp32 run manufactures conditioning failures of
+# its own and the cholesky-vs-slogdet delta on the PD set would be round-off
+# rather than the two algorithms. The script exits if this is unset.
+export JAX_ENABLE_X64=True
+export JAX_PLATFORM_NAME=cpu
+export JAX_PLATFORMS=cpu
+export NUMBA_CACHE_DIR=/tmp/numba_cache
+export MPLCONFIGDIR=/tmp/matplotlib
+
+echo "=========================================="
+date
+echo "Experiment: phase 8A slogdet A/B (CP-4)"
+echo "Cells:      imaging/knn/hst and imaging/delaunay_adapt_split/hst"
+echo "Arms:       cholesky (control) vs slogdet (treatment)"
+
+cd $AP_ROOT
+
+# One cell must not sink the other: a VOID verdict exits non-zero by design, and
+# without this the second cell would never run.
+run_cell () {
+    echo "=== cell: $1 ==="
+    date
+    python3 scripts/misc/searches/slogdet_ab.py \
+        --stage all \
+        --dataset-class imaging \
+        --model-type "$1" \
+        --instrument hst \
+        --seed 0 \
+        --n-prior 128 \
+        --n-truth-bar 128 \
+        --transect-points 128 \
+        --batch-size 4 \
+        --draws-tag ral_cpu \
+        --tag "CP-4 phase 8A, RAL CPU tier" \
+        || echo "!!! cell $1 exited non-zero — read its verdict block above (a VOID verdict exits 1)"
+}
+
+run_cell delaunay_adapt_split
+run_cell knn
+
+echo "Finished."
+date

hpc/batch_gpu/submit_slogdet_ab_adaptsplit_a100

@@ -0,0 +1,106 @@
+#!/bin/bash -l
+#
+# Phase 8A / CP-4 — slogdet vs cholesky A/B on the free-AdaptSplit NaN wall.
+# A100 tier: the half the CPU sibling cannot supply.
+#
+# PRE-REGISTERED PASS CRITERIA (all four must hold; stated before any run):
+#
+#   1. zero slogdet NaNs on the replay set
+#   2. value equality within float tolerance on PD points
+#   3. finite gradients
+#   4. runtime within 2x
+#
+# TWO CELLS, and the difference matters. The pre-registration names the
+# free-AdaptSplit stressor as `knn`, but this repo's #117 record says the KNN
+# mesh's high-coefficient region is a finite over-regularized floor, while the
+# NaN WALL is on the Delaunay mesh with the same free AdaptSplit reg
+# (_setup._delaunay_adapt_split_model; multi_start_prodigy/delaunay.py). Both
+# run here. A VOID verdict exits non-zero by design — read the verdict block
+# before treating a non-zero exit as a broken job.
+#
+# Why the A100 rather than only the CPU sibling
+# (../batch_cpu/submit_slogdet_ab_adaptsplit_ral_cpu): the deterministic lambda
+# transect crosses the conditioning wall on any tier, but the wall as a SEARCH
+# phenomenon has so far only been observed at GPU scale — the CPU arms of jobs
+# 335003-5 measured zero value-NaN lane-steps
+# (results/notes/inference/phase_00_unblocking/RESULTS.md:189), while the 62%
+# value-NaN MGE finding (#128) is a GPU result. This job therefore adds
+# --descent-steps: a short cholesky-arm MultiStartProdigy run whose checkpointed
+# lane vectors are harvested into the replay set, so the A/B is tested on draws
+# a real gradient search actually visited, not only on a synthetic transect.
+#
+# The harvested draws are written to their own npz (--draws-tag a100) so this
+# run cannot overwrite the CPU run's draw set.
+#
+# batch_size=4 is MANDATORY on these cells: the unbatched 16-start jvp fusion is
+# the ~58 GB allocation (see scripts/imaging/searches/multi_start_prodigy/knn.py).
+#
+# Runtime: the descent harvests dominate — 2 cells x 16 starts x 150 steps at
+# the A100's per-step cost, then 2 x 2 x ~600 draws of value_and_grad replay.
+# 8 h with headroom for the four XLA compiles.
+
+#SBATCH -J slogdet_ab_adaptsplit_a100
+#SBATCH --partition=gpu
+#SBATCH --gres=gpu:1
+#SBATCH --ntasks=1
+#SBATCH --cpus-per-task=8
+#SBATCH --mem=128gb
+#SBATCH --time=8:00:00
+#SBATCH -o output/output.%A.out
+#SBATCH -e error/error.%A.err
+#SBATCH --mail-type=END,FAIL
+#SBATCH --mail-user=james.w.nightingale@durham.ac.uk
+
+export AP_ROOT=/mnt/ral/jnightin/autolens_profiling
+source $AP_ROOT/activate.sh
+
+export JAX_PLATFORM_NAME=cuda
+export JAX_PLATFORMS=cuda,cpu
+export XLA_PYTHON_CLIENT_PREALLOCATE=false
+# fp64 is not optional here: an fp32 run manufactures conditioning failures of
+# its own and the cholesky-vs-slogdet delta on the PD set would be round-off
+# rather than the two algorithms. The script exits if this is unset. On this
+# cluster sbatch does NOT inherit it from the login shell — it must be exported
+# in the job script.
+export JAX_ENABLE_X64=True
+export NUMBA_CACHE_DIR=/tmp/numba_cache
+export MPLCONFIGDIR=/tmp/matplotlib
+
+nvidia-smi
+
+echo "=========================================="
+date
+echo "Experiment: phase 8A slogdet A/B (CP-4)"
+echo "Cells:      imaging/knn/hst and imaging/delaunay_adapt_split/hst"
+echo "Arms:       cholesky (control) vs slogdet (treatment)"
+echo "Draws:      prior + truth_bar + lambda_transect + descent trajectory"
+
+cd $AP_ROOT
+
+# One cell must not sink the other: a VOID verdict exits non-zero by design, and
+# without this the second cell would never run.
+run_cell () {
+    echo "=== cell: $1 ==="
+    date
+    python3 scripts/misc/searches/slogdet_ab.py \
+        --stage all \
+        --dataset-class imaging \
+        --model-type "$1" \
+        --instrument hst \
+        --seed 0 \
+        --n-prior 128 \
+        --n-truth-bar 128 \
+        --transect-points 128 \
+        --descent-starts 16 \
+        --descent-steps 150 \
+        --batch-size 4 \
+        --draws-tag a100 \
+        --tag "CP-4 phase 8A, A100 tier, with descent-trajectory draws" \
+        || echo "!!! cell $1 exited non-zero — read its verdict block above (a VOID verdict exits 1)"
+}
+
+run_cell delaunay_adapt_split
+run_cell knn
+
+echo "Finished."
+date

results/notes/inference/DECISIONS.md

@@ -246,3 +246,30 @@ tripling claim withdrawn on adversarial review — see
 PROGRAMME.md phase/gate table (this commit); overnight strengthening
 jobs 339065-339073 (n128 fresh seeds, n256 extra seeds, Nautilus
 re-baselines) submitted before the call and noted here for the record.
+
+---
+
+## 2026-08-24 — CP-4 / Phase 8A: slogdet FAILS the pre-registered A/B — human call: ADOPT as the profiling-repo testing default, keep library-optional, chase the residual NaNs
+
+**Record (Gates E/F remain open):** slogdet vs cholesky A/B run on RAL
+(A100 338808, CPU 338807). The pre-registered stressor (knn + free
+AdaptSplit) never walls — VOID on both tiers, as the driver's #117 note
+predicted. On the real wall (Delaunay + free AdaptSplit) slogdet rescues
+64–73 % of Cholesky NaNs with zero regressions but leaves 20–32 draws NaN
+under both arms, leaves all λ-transect gradients non-finite, disagrees
+with Cholesky by up to 9,619 nats (A100) in the marginal band, and costs
+3.7× on CPU. Three of four criteria fail.
+
+**Human call (2026-08-24), overriding the record's "not recommended":**
+zero regressions + 64–73 % rescue is worth having now. (1) `slogdet`
+becomes the **default `log_det_method` for gradient-work cells in this
+repo's searches framework** (`_setup.py`; GPU tiers — CPU keeps cholesky
+for the 3.7× cost) — W8. (2) The PyAutoArray default stays cholesky
+(opt-in), **to be revisited once (3) lands — reminder owed to the human at
+the end of the queue: "make slogdet standard".** (3) The NaN-under-both
+draws (32 A100 / 20 CPU) and the all-transect non-finite gradients are a
+named investigation, W7 — are they genuinely singular systems (λ⁴, #104)
+or a formula-independent overflow that a different scaling removes? Phase
+8B stays the live reparameterization candidate alongside. Record:
+`phase_08_regularization/RESULTS.md`.
+

results/notes/inference/PROGRAMME.md

@@ -32,7 +32,8 @@ expectation · **[CONTRADICTED]** existing evidence pushed back on the brief ·
 | Phase 0(e) searches README dashboard loop | **complete** | PR#139 (2026-08-18): nested-layout scanner, 34 rows render, truth-bar rows verified |
 | Phase 2 NSS mainline re-tune | **entry complete (CP-2 ✅ 2026-08-23)** — wiring + laptop smoke + A100 anchor row: mainline matches the fork answer at fork knobs (max logL 31786.62, logZ bias +8.4 persists at inner=5 per H2.1) but is ~1.35× slower sampler-wall; scan + Gate A pending | `phase_02_nss_mainline/RESULTS.md` |
 | Phase 3 Prodigy reliability | **wave 1 complete + adversarially reviewed (CP-3 ✅ 2026-08-23, positions-off)** — p̂_hit = 0.048 [0.037, 0.061] (lower bound; n256 tier, ~1,280 distinct draws — tiers share lane-index draws, n-dependence unmeasured); demonstrated ≥99% reliability at **n=256 only** (joint-95% worst case), 3.4–4.5× under the viz-stripped Nautilus wall; zero parameter-recovery impostors; ~half of lanes end pinned (H3.3, measured not closed); θ_E diagnostic uninformative for H3.1 (withdrawn). **Gate B pt 1 CALLED (human-ratified 2026-08-23** — DECISIONS.md): Prodigy(n=256, prior_box, autoconv, positions-off) ratified as global MAP searcher on MGE; pt 2 (PositionsLH) open. Fresh-seed tier 2026-08-24: n128 5/5, n256 15/15 cumulative (Wilson-95 lower 0.80); Nautilus re-baseline 707 s → 3.1–4.3× | `phase_03_prodigy_reliability/RESULTS.md` + `ADVERSARIAL_REVIEW.md` |
-| Phases 1, 4–13 | not started | — |
+| Phase 8A slogdet A/B (CP-4) | **complete 2026-08-24 — FAIL as pre-registered.** knn stressor never walls (VOID both tiers); on Delaunay+AdaptSplit slogdet rescues 64–73 % of NaNs, 0 regressions, but 20–32 NaN-under-both, λ-transect grads non-finite, marginal-band Δ up to 9,619 nats (A100), CPU 3.7×. Human call: adopt as this repo's GPU gradient-cell default (W8), library stays opt-in, residual NaNs → W7, 8B in parallel | `phase_08_regularization/RESULTS.md` + DECISIONS.md 2026-08-24 |
+| Phases 1, 4–7, 8B–13 | not started | — |
 | Gates A–F | **B part 1 CALLED 2026-08-23** (Prodigy n=256 global MAP, MGE, positions-off — DECISIONS.md); A, B pt 2, C–F open | DECISIONS.md |
 
 ---
@@ -921,9 +922,10 @@ Maximum information before large A100 commitment — strictly ordered:
    called below n=256 — `phase_03_prodigy_reliability/RESULTS.md` +
    `ADVERSARIAL_REVIEW.md`. Positions-on half awaits Phase-4 PositionsLH
    plumbing.**
-4. **CP-4 · slogdet A/B on the AdaptSplit NaN wall** (hours). Expected pass;
-   converts Phase 8A from plan to record and sets the gradient-work
-   likelihood profile early.
+4. **CP-4 · slogdet A/B on the AdaptSplit NaN wall** (hours). ~~Expected pass~~
+   **DONE 2026-08-24 — FAILED the pre-registration**, adopted anyway by
+   human call as the repo's GPU gradient-cell default (W8); residual
+   NaN-under-both → W7; Phase 8B in parallel (`phase_08_regularization/`).
 5. **CP-5 · NSS inner-steps/logZ-bias scan + one pixelized probe**
    (laptop/RAL + 1 short A100 slot). Decides Gate A and whether NSS's 7–11×
    pixelized deficit is configuration or structure.

results/notes/inference/phase_08_regularization/RESULTS.md

@@ -0,0 +1,94 @@
+# Phase 8A / CP-4 — slogdet vs cholesky A/B on the AdaptSplit NaN wall: results
+
+Driver: `scripts/misc/searches/slogdet_ab.py` (pre-registration, cell
+selection and verdict rules in its module docstring). Submits:
+`hpc/batch_gpu/submit_slogdet_ab_adaptsplit_a100` (RAL 338808, A100 fp64,
+29 min) and `hpc/batch_cpu/submit_slogdet_ab_adaptsplit_ral_cpu` (RAL
+338807, 5 h 37 m of a 6 h limit). Both COMPLETED 2026-08-23/24; artifacts
+under `slogdet_ab/` (verdict JSON + `per_draw.npz` per cell × tier, draw
+sets under `draws/`). Version stamp 2026.8.17.1, jax 0.10.2.
+
+Replay set per cell: 128 prior draws + 128 λ-transect draws + 128
+truth-bar draws (+32 descent-trajectory draws on the A100 tier); anchor =
+prior medians (no truth-basin vector recorded in-repo — see driver).
+
+## Verdict table (pre-registered criteria: all four must hold)
+
+| cell | tier | draws | chol NaN | slogdet NaN | rescued | regressed | NaN both | grad non-finite chol → slogdet | runtime ratio | criteria 1/2/3/4 | verdict |
+|---|---|---:|---:|---:|---:|---:|---:|---|---:|---|---|
+| knn | A100 | 416 | 0 | 0 | 0 | 0 | 0 | 128 → 128 | 1.03 | ✅ ✗ ✗ ✅ | **VOID** (control never walled) |
+| knn | RAL CPU | 384 | 0 | 0 | 0 | 0 | 0 | 129 → 129 | 2.52 | ✅ ✗ ✗ ✗ | **VOID** (control never walled) |
+| delaunay_adapt_split | A100 | 416 | 90 | 32 | **58** | **0** | 32 | 192 → 160 | 1.03 | ✗ ✗ ✗ ✅ | **FAIL** |
+| delaunay_adapt_split | RAL CPU | 384 | 73 | 20 | **53** | **0** | 20 | 180 → 151 | **3.74** | ✗ ✗ ✗ ✗ | **FAIL** |
+
+## Reading
+
+1. **knn is not a NaN wall — confirmed, on both tiers.** As the driver's
+   "WHICH CELL ACTUALLY WALLS" note predicted from #117, the pre-registered
+   stressor (KNN + free AdaptSplit) produced zero Cholesky NaNs over 416
+   draws spanning the prior, a λ transect to 8e5 and the truth bar. Its
+   criteria pass vacuously; the verdict is VOID as pre-registered, not a
+   pass. The 128/129 non-finite gradients are the λ-transect draws under
+   BOTH arms and are unrelated to `log_det_method` (attribution block:
+   `slogdet_introduces_no_new_grad_nans: true`).
+
+2. **On the real wall (Delaunay + free AdaptSplit) slogdet is a partial
+   mitigation, not a fix.** It rescues 64 % / 73 % of the Cholesky NaNs
+   (58/90 A100, 53/73 CPU) with **zero regressions** — every draw finite
+   under Cholesky stays finite under slogdet, and every draw whose gradient
+   is finite under Cholesky stays finite (attribution:
+   `grad_nonfinite_only_under_slogdet = 0`; 32 / 29 draws gain a finite
+   gradient). But 32 / 20 draws are NaN under both arms — those are
+   genuinely singular systems (the λ⁴ fragility of #104), where no log-det
+   formula returns a number — and the λ-transect gradients are non-finite
+   under both arms on all 128 draws. Criteria 1 and 3, scored on the
+   treatment alone as registered, fail.
+
+3. **Value equality holds on clean-PD points and breaks in the marginal
+   band, as the laptop probe warned.** `clean_pd` (251 / 243 draws):
+   max |Δ| 1.7e-4 / 1.0e-4 at |logL| ~ 1.6e5 (relative ~1e-9 — the target
+   has not moved where the Cholesky was healthy). `marginal_band` (75 / 68
+   draws, coefficients up to 8e5–9e5): max |Δ| **9,619 nats on the A100**
+   vs 2.27 on CPU. The two arms disagree by thousands of nats exactly
+   where the Cholesky is numerically marginal — and the disagreement is
+   tier-dependent, so neither arm can be called "the value" there. The 58
+   rescued draws all evaluate to logL ≤ −1.6e5 (median −2.2e5): slogdet
+   turns a NaN into a finite but catastrophically bad number, which is
+   what a gradient search needs (a direction) but not a category-2
+   equivalence claim.
+
+4. **Runtime: free on GPU, 3.7× on CPU.** A100 warm wall 67 → 69 s
+   (1.03×); RAL CPU 1,746 → 6,522 s (3.74×, fails the 2× ceiling; compile
+   also 45 → 100 s). LU-based slogdet has no cheap CPU path at this matrix
+   size; the GPU hides it.
+
+## Decision input for Gate E/F (superseded by the human call below)
+
+The pre-registration expected a clean pass; it did not get one. Recommended
+recording (DECISIONS.md 2026-08-24): **slogdet is NOT recommended as the
+gradient-work default profile.** It may be enabled per-fit on GPU tiers as a
+NaN-wall softener at zero cost and zero regression risk, but it does not
+remove the wall, it changes the value in the marginal band by a
+tier-dependent amount, and it is 3.7× slower on CPU. Phase 8B
+(log-coordinate stepping — a category-1 reparameterization that keeps the
+search away from the singular band altogether) is now the live candidate;
+8+ (analytic log-det on fixed-topology rectangular meshes) is the only
+lever that kills the Cholesky rather than dressing it.
+
+## Human call (2026-08-24)
+
+Adopt now, chase the rest: slogdet becomes the default for gradient-work
+cells in this repo on GPU tiers (W8, library default untouched); the
+NaN-under-both draws and transect gradients get their own investigation
+(W7); Phase 8B proceeds in parallel; "make slogdet standard in PyAutoArray"
+is a reminder owed once W7 reports. See DECISIONS.md 2026-08-24.
+
+## Ops notes
+
+- The CPU run's 5 h 37 m sits 23 min under its 6 h limit; any wider
+  transect on CPU needs a longer limit or a smaller replay set.
+- Descent-trajectory draws exist only on the A100 tier (32); on CPU the
+  driver skipped them (no per-source `descent` block).
+- `coefficient_min = −5.2` in the A100 marginal band is the driver
+  reporting the log-coefficient axis for transect draws; not a negative
+  regularization.