W7 · CP-4 follow-up: per-draw NaN attribution, harvest fixes, CP-4 re-scored on the clean subset (#164) - #171
Merged
Conversation
) Drop dead-lane and negative-coefficient descent checkpoint rows at harvest (harvest_descent), and jitter the lambda_transect anchor off any exact-zero ell_comps/shear component (anchor_vector / _jitter_anchor_off_zeros). Both were misread in RESULTS.md's prior "Ops notes" as a log-coefficient axis; per-draw attribution traced them to real harvest artefacts instead (negative/NaN Prodigy checkpoint vectors, and an undefined sqrt-magnitude gradient at the anchor's ell_comps=shear=0 point in autogalaxy/convert.py). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EDABYoH6giHXhFJUks8yd6
#164) Replays individual stored slogdet_ab draws non-jitted, drills into inversion.* matrices (mapping/data/curvature/reg matrices, cond via eigvalsh, LAPACK cholesky vs np.slogdet vs jnp cholesky/slogdet) and small jax.grad closures (compiled once per parameter subset, reused across every draw) to classify each sampled draw into one mechanism: dead_lane, invalid_coefficient, anchor_singularity, upstream_nan, genuinely_singular, marginal_tier_flippable, or clean. --matrix-only skips the gradient closures (the expensive path) for population-scale runs. Also adds the (not-submitted) A100 GPU-native submit script for the tier_flip/marginal_band conditioning scan, should a future phase need cuSOLVER-native numbers. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EDABYoH6giHXhFJUks8yd6
slogdet_nan_attribution.py output for delaunay_adapt_split, both tiers:
- imaging_delaunay_adapt_split_hst_a100.{json,npz} — 8-draw full-gradient
probe (nan_both, transect) that established anchor_singularity
- imaging_delaunay_adapt_split_hst_{a100,ral_cpu}_matrix_only.{json,npz} —
87 A100 / 83 RAL CPU draw matrix-only population classification (all
classes, max 15/class)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EDABYoH6giHXhFJUks8yd6
) RESULTS.md: correct the "coefficient_min=-5.2 is a log axis" ops note (it was a harvest bug); add "W7 addendum" with the per-draw classification table (170 draws across both tiers), the tier-attack finding (draw 96: cond=4.5e18 explains both the 9,619-nat A100 marginal-band outlier and its NaN-under-both status on RAL CPU for the identical vector), and "CP-4 re-scored on the clean subset" (excluding dead-lane/invalid-coefficient/ anchor-singularity draws — verdict unchanged, FAIL on both tiers; residual failures are the genuinely-singular lambda^4 population). DECISIONS.md: append-only entry recording the re-score and the human call that W8 adoption stands unchanged, with three concrete inputs for W9 (#166). PROGRAMME.md: append the re-score summary to the Phase 8A row. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EDABYoH6giHXhFJUks8yd6
…md: keep W4 and W7 entries in order)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Closes #164. Unblocks W9 (#166).
slogdet_nan_attribution.py— replays the stored CP-4 draws non-jitted and classifies each intodead_lane/invalid_coefficient/anchor_singularity/upstream_nan/genuinely_singular/marginal_tier_flippable/cleanfrom the inversion matrices (eigvalsh condition numbers, LAPACK cholesky vsnp.slogdetvsjnpon the same matrix) and per-termjax.gradclosures (compiled once per parameter subset). 170 draws classified across both tiers; artefacts underslogdet_ab/attribution/.slogdet_ab.py: 10 A100 "NaN-under-both" draws were dead-lane checkpoints with NaN input vectors and 6 more carried negative coefficients (harvest now drops them); all 128 λ-transect draws sat at an anchor withell_comps = shear = (0,0)exactly — an undefined-gradient point inautogalaxy/convert.py— so every transect gradient was NaN regardless of log-det (anchor now jittered 1e-3 off zero). Thecoefficient_min = −5.2ops note is corrected.cond(F + λ⁴H) = 4.5e18; the log-det terms agree with the stored A100 value to 5 s.f. but the reconstruction solve on that matrix is itself NaN — cuSOLVER vs OpenBLAS differ only on whether they return a number. Cross-tier max |Δ| on mutually finite draws: 6.7 nats (slogdet) / 1.3 (cholesky).Verification
ruff check/format --checkclean;build_readme --checkclean;pytest scripts/misc/test: 112 passed, 1 failed (test_hazards_prior_exit— worktree-layout artefact, passes on the canonical checkout).hpc/batch_gpu/submit_slogdet_nan_attribution_a100prepared, not submitted (matrix-only tier check is optional now that cond explains the flips).🤖 Generated with Claude Code
https://claude.ai/code/session_01EDABYoH6giHXhFJUks8yd6