Skip to content

Corner plots: fix the silent failure on repeated labels; per-component labels and plots; latent parameters - #1709

Merged
Jammy2211 merged 9 commits into
PyAutoLabs:mainfrom
samlange04:fix/corner-anesthetic-duplicate-labels
Oct 21, 2026
Merged

Jammy2211 merged 9 commits into
PyAutoLabs:mainfrom
samlange04:fix/corner-anesthetic-duplicate-labels

Conversation

@samlange04

@samlange04 samlange04 commented Oct 18, 2026 •

Copy link
Copy Markdown
Contributor

Discussion: https://github.com/orgs/PyAutoLabs/discussions/38.

corner_anesthetic produced no plot for any model with two components of one class (most lens models) and logged
"posterior estimate not yet sufficient" instead: anesthetic keys its columns by label, the LaTeX labels repeat
(every Gaussian of a basis is $\epsilon_1^{\rm g}$), plot_2d raised KeyError and @log_plot_exception
swallowed it. This PR fixes that and makes the plots of a many-component model readable.

Fix. Columns are keyed by position (p0, p1, ...) and the LaTeX labels passed as anesthetic display labels
(NestedSamples(labels=...), make_2d_axes(labels=...)); labels joins the reserved kwargs.

Labels. AbstractPriorModel.superscripts is the dot-joined path of the names the owning components were given
in the model (lens.bulge, source.bulge, lens.shear), via new component_paths: Collection containers,
tuple names and the list index of a basis sharing one prior are dropped, an index distinguishing independent priors
is kept, a prior shared between differently named components takes the path its samples column is named after
(unique_prior_paths). The notation config's class abbreviation applies only to a prior on a bare unnamed Model.
Paths three or more levels deep keep the first name and cut the rest to three characters
(superscript_from_component_path: source.pix.reg). Tests in test_model_mapper.py cover each rule.

Per-component plots. New parameter_indices_by_component (top-level component -> parameter indices; root
priors as "model") and a parameter_indices kwarg on corner_anesthetic; WeightedDefaults.plot_results writes
corner_anesthetic_<component>.png per component when there is more than one. Config
visualize/plots_search.yaml nest.corner_anesthetic_components: true; PosteriorDefaults.should_plot(name, default=) falls back for a key an older workspace config lacks.

Latent parameters. autofit/mapper/prior_model/latent_parameters.py: a component class declares
latent_parameters = {("gamma_1", "gamma_2"): fn}; fn takes every parameter of the component by name (free as
arrays over the samples, fixed as floats, tuples flattened name_0/name_1) and returns {name: array};
declarations merge down the MRO. AbstractPriorModel.latent_parameters_from(parameter_array, weights=None) returns a
LatentParameters (names, superscripts, component paths, full model paths, array, mask); has_latent_parameters.
Periodic(values, period).recentred(weights) wraps an angle into one period centred on its weighted circular mean,
so a posterior at the branch cut is not drawn as two peaks; exported as af.Periodic.

  • corner_anesthetic(latent=True) plots the latent set; WeightedDefaults.plot_latent_results writes
    corner_anesthetic_latent.png and corner_anesthetic_latent_<component>.png for the components with a latent
    parameter (none for a single-component model), under config nest.corner_anesthetic_latent (default false),
    independent of the other two switches.
  • latent_samples_from appends the model's latent parameters to the latent samples under their model path
    (galaxies.lens.shear.magnitude), with or without analysis latents, so they reach latent/samples.csv,
    latent.results and latent_summary.json.
  • The latent plot is latent output: a new plot_latent_results on NonLinearSearch / PosteriorDefaults (no-op
    for chain and point kinds), which SearchUpdater.visualize calls once the fit completes and during it only if
    output.yaml's latent_during_fit is on. plot_results and the iterations_per_full_update cadence are
    unchanged; searches overriding plot_results(samples) are unaffected.

Compatibility. Superscripts in model.results and plot labels change for named components (from the class
abbreviation to the component path). Two new config keys, both with in-code fallbacks. No search, samples or
output-format change otherwise.

Before / after (a SLACS lens's 19-parameter source-light stage; more in the discussion):

Before: old labels

Before: old labels
After: corner_anesthetic_lens.png

After: corner_anesthetic_lens.png
After: corner_anesthetic_latent_lens.png

After: corner_anesthetic_latent_lens.png

Tests. test_model_mapper.py (component paths, names only, shared priors, abbreviation),
test_latent_parameters.py (new), test_nest_plotters.py (repeated labels, subset, latent), test_family_bases.py
(per-component / latent plot files, after-fit hook, single-component), test_updater.py (latent plot schedule),
test_latent.py (model latents in the latent output). Full suite: 3720 passed locally (macOS, Python 3.13).

Commits: 9db2921aa fix, 941bd9d1b labels + per-component + latent plots, 43dcf0f65 latent plot off by default,
587a75b32 names only, 7db4b9cf8 deep-path abbreviation, 5d9ba5167 latent naming / latent.csv / after-fit,
13bd430cf shared priors.

🤖 Generated with Claude Code


Maintainer changes (f9377f3, tracked in #1710)

The review findings are resolved on this branch rather than in another review round. Maintainer decisions (2026-10-21): (a) superscripts are the component's dot-joined path everywhere, including model.results; (b) the latent corner plot stays a separate plot, behind plots_search.yaml nest.corner_anesthetic_latent (default off).

API Changes

Corner plots label each parameter with its component path, can be split per top-level component, and can show the model's declared latent parameters (e.g. a shear's magnitude and angle). Models can declare latents with a latent_parameters class attribute; af.Periodic marks angles. On top of the contributor's work, this commit adds two config keys and one keyword, and changes three behaviours: per-component plots follow corner_anesthetic when not configured, deep labels are guaranteed unique, and shared priors are converted by every holder. latent.csv now keeps an angle's fixed range.
See full details below.

Test Plan

  • pytest test_autofit: 3881 passed, 1 skipped, 1 xfailed (local, f9377f3 content)
  • New Witness tests: (i) test_weighted_plot_results_components_follow_corner_anesthetic_when_key_absent, (ii) test__visualize__raising_latent_conversion_is_logged_not_raised, (iii) test__latent_parameters__shared_prior_is_converted_by_every_holder, (iv) test__superscripts__deep_paths_stay_unique / ..._in_a_model_stay_unique. All 13 new or changed tests fail on bf43aa6.
  • Workspace smoke pyauto-heart smoke against this branch: 165/165 passed (autofit 10, autogalaxy 16, autolens 43, autolens_workspace_test 37, euclid 9, HowToLens 50). The autocti environments did not build locally (missing GSL headers); not run.
  • Independent review: no code defects. The docstring finding is fixed; workspace config keys follow in phase 3.
  • CI green on f9377f3
Full API Changes (for automation & release notes)

Added

  • AbstractPriorModel.latent_parameters_from(parameter_array, weights=None, recentre=False) — model latents (component latent_parameters declarations) over samples; recentre=True recentres Periodic latents on their circular mean (plots only)
  • af.Periodic(values, period) — marks a periodic latent (angle)
  • AbstractPriorModel.component_paths, parameter_indices_by_component
  • autofit.mapper.prior_model.abstract.superscripts_from_component_paths(paths), indices_by_top_level_component(paths)
  • Config output.yaml: latent_model_parameters: true — model latents join latent.csv / latent.results / latent_summary.json
  • Config visualize/plots_search.yaml nest: corner_anesthetic_components, corner_anesthetic_latent (default false), corner_anesthetic_latent_components
  • Plots corner_anesthetic_<component>.png, corner_anesthetic_latent[_<component>].png

Changed Behaviour

  • Superscripts (corner plots, model.results) are the dot-joined named-component path (lens.bulge, source.pix.reg), not the class's notation.yaml superscript: entry, which now applies only to priors directly on an unnamed Model
  • Deep-path names are shortened consistently and only as far as they stay unique among siblings (lens.bulge_i.she / lens.bulge_o.she)
  • When corner_anesthetic_components is absent from a config it follows corner_anesthetic; corner_anesthetic_latent_components follows corner_anesthetic_latent
  • A latent conversion that raises is logged; the fit's output continues without the latent plots and columns
  • Parameters with no named component group under "model", or "model_root" when a component is named model

Migration

  • Workspaces: add the new plots_search.yaml / output.yaml keys (library defaults apply meanwhile); update notation.yaml superscript: comments (phase 3, after release)

Heart RED development-only override

Live maintainer authorization, 2026-10-21: "do 2, Override and proceed" (start_dev for #1710). This covers commit, push and PR work only. Merge needs a separate /prm with every required check green. No release. This branch makes no claim to fix Heart. Heart readiness: STALE ("release validation incomplete: no rehearsal for current source"). Monitoring feed at ship: Heart: RED — Release RED; monitoring RED · 14/100 · 464 unresolved (updated 23h ago)

Exact RED reasons
  • PyAutoFit: CI failure — https://api.github.com/repos/PyAutoLabs/PyAutoFit/actions/runs/39111874604
  • PyAutoFit: PyAutoFit: CI failure
  • PyAutoFit/Tests: failure
  • PyAutoFit/workflows/Tests: failure
  • PyAutoHands/workflows/Python Version Matrix: failure
  • PyAutoHeart/workflows/Workspace Smoke: failure
  • PyAutoMind/workflows/Spawn Drift: failure
  • PyAutoArray/Tests: red
  • PyAutoArray/Tests: red
  • PyAutoFit/Tests: red
  • PyAutoFit/Tests: red
  • PyAutoFit/Tests: red
  • PyAutoFit/Tests: red
  • autofit_workspace_test/Smoke Tests: red
  • autogalaxy_workspace_test/Smoke Tests: red
  • autolens_workspace_test/Smoke Tests: red
  • CI wall-clock: 27 gates · slowest PyAutoBrain Nightly Release 49m · 9 hang events
  • PyAutoFit: PyAutoFit: CI failure
  • Workspace validation: NOT ready — 1414p / 1f / 133s @ cloud#39224774724
  • euclid: red
  • test_run: red
  • PyAutoArray: red
  • autolens_inference: red
  • autolens_profiling: red
  • Worktree drift: 5 orphan / 0 missing / 4 dirty
  • euclid_strong_lens_modeling_pipeline: red

Generated by the PyAutoLabs agent workflow.

🤖 Generated with Claude Code

samlange04 and others added 8 commits October 18, 2026 14:36
…ated labels

corner_anesthetic keyed anesthetic's columns by the LaTeX parameter labels, which
repeat whenever a model has several components of one type: every Gaussian of a
multi-Gaussian basis is labelled $\epsilon_{1}^{g}$, so a lens model with an MGE
light and an MGE source repeats y, x and both ellipticity components. anesthetic
then raises KeyError, @log_plot_exception reports it as "posterior estimate not
yet sufficient", and the corner plot never appears -- for a 19-parameter SLaM
source fit it was missing at every update and at completion.

Key the columns by position (always unique) and pass the LaTeX labels to
anesthetic for display, so the figure looks as before. `labels` joins the
reserved kwargs, since a caller passing it would now collide with them.

Tests: a model with a repeated label writes its figure and still displays the
repeated label (red without the fix), and `labels` is rejected as reserved.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…er plots

Looking at a 19-parameter SLaM source fit, the corner plot had three problems
beyond the repeated-label crash the previous commit fixed:

1. Labels did not tell components apart. The superscript was the immediate
   parent name or the notation config's class abbreviation, so every Gaussian
   of a multi-Gaussian basis was y^g whether it sat in the lens bulge, the lens
   disk or the source bulge. The superscript is now the dot-joined path of the
   named components owning the prior (new `component_paths`): Collection
   containers (`galaxies`, `profile_list`), the tuple name and a list index
   that only distinguishes copies of a shared prior are dropped, giving
   y^{lens.bulge}, y^{lens.disk}, y^{source.bulge}, theta_Ein^{lens.mass}. The
   config's class override replaces only the component it names (`lens.ext`),
   and a prior directly on a root model keeps the old behaviour, so existing
   labels of simple models are unchanged.

2. One 19x19 plot is too dense to read. `parameter_indices_by_component`
   groups parameters by top-level component and `WeightedDefaults.plot_results`
   writes `corner_anesthetic_<component>.png` per component beside the full
   plot (switch `nest.corner_anesthetic_components`, default on; none for a
   single-component model). `corner_anesthetic` gains `parameter_indices=`.

3. The sampled parameters are not the ones a model is read in. A component
   class may declare `derived_parameters = {("gamma_1", "gamma_2"): fn}`: the
   free parameters it replaces and a function of every parameter of the
   component (free ones as sample arrays, fixed ones as floats) returning the
   derived ones; declarations inherit down the MRO. `derived.py` applies them
   to a sample array in model order, keeping each component's superscript. A
   derived angle is returned as `Periodic(values, period)` and recentred on
   its weighted circular mean, so a posterior peaked at the conversion's branch
   cut is not drawn as two peaks. Switch `nest.corner_anesthetic_derived`
   (independent of the other two) writes `corner_anesthetic_derived.png` and,
   with the per-component switch, one per component; skipped when nothing in
   the model declares a conversion. `corner_anesthetic` gains `derived=`.

`should_plot` takes a `default` for a key an older workspace config does not
mention, so neither new switch breaks an existing config.

Tests: component paths, config override scope and shared-vs-independent index
(test_model_mapper), the derived set, fixed values, inheritance and periodic
recentring (test_derived), subset and derived plots (test_nest_plotters), the
per-component and derived plot calls and the missing-key fallback
(test_family_bases).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The conversions and a second set of anesthetic figures take a while on a large
sample set, so `nest.corner_anesthetic_derived` ships false and a user opts in.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…ly for a bare model

The notation config's per-class abbreviation (`Gaussian: g`) no longer replaces
a named component's name: every component is labelled by the path of names it
was given in the model, so a shear reads `lens.shear`, a basis of Gaussians
`lens.bulge`, and a collection of Gaussians `gaussian_0`, `gaussian_1` rather
than `g`, `g`. The abbreviation still applies to a prior directly on a bare
`Model`, which has no name of its own.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…e first

A parameter owned by a component three levels down, such as a source
galaxy's pixelization regularization, was labelled with the full path
`source.pixelization.regularization`, which runs into the neighbouring
axis labels on a corner plot. `superscript_from_component_path` now keeps
the first name whole (the galaxy or plane that distinguishes the
components) and cuts every later name to three characters, so the label
reads `source.pix.reg`. One- and two-level paths are unchanged.

Also removes two superscript tests that were duplicated verbatim into a
second test class.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
PyAutoFit already calls a quantity computed from the model but not sampled
a latent variable, so the component-declared conversions (a shear's magnitude
and angle for gamma_1 / gamma_2) take that name: `latent_parameters` on the
class, `autofit/mapper/prior_model/latent_parameters.py` (`LatentParameters`,
`latent_parameters_from`), `corner_anesthetic(latent=True)`, config
`corner_anesthetic_latent` and files `corner_anesthetic_latent[_<component>].png`.

They now join the analysis's latents in the latent output: `latent_samples_from`
appends them under their model path (`galaxies.lens.shear.magnitude`) to
`latent/samples.csv`, `latent.results` and `latent_summary.json`, with or
without analysis-defined latents (`LatentParameters.paths`, `latent_items`).

The latent corner plots follow the same rule as that output rather than the
`iterations_per_full_update` cadence: a `plot_latent_results` hook on the
search and `PosteriorDefaults`, which `SearchUpdater.visualize` calls once the
fit completes and during it only if `output.yaml`'s `latent_during_fit` is on.
Per-component latent plots are made only for components with a latent
parameter (`parameter_indices_by_latent_component`), and not at all for a
single-component model, so no plot repeats another.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…common prefix

A prior shared between differently named components (a power law's
ellipticity linked into its multipoles, `mass` / `multipole_3` /
`multipole_4`) was labelled by the components' common prefix alone,
`lens`, because every differing path element was dropped. Only a list
index that differs (the Gaussians of a basis) says nothing; a differing
name is kept, from the path the samples column is named after
(`unique_prior_paths`, the last component holding the prior), so label
and column agree: `q^{lens.mass}`.

`latent_parameters_from` indexed `model.paths`, which lists every path to
a shared prior, instead of `unique_prior_paths`, so the latent.csv keys of
such a model were misaligned.

Found by running the HST project's EPLS34 model (elliptical and circular
multipole classes) through labels, latent parameters, latent samples and
the corner plots.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
sphinx_autodoc_typehints could not resolve the string annotation on
latent_parameters_from, three new warnings over the baseline.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

@Jammy2211 Jammy2211 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @samlange04, thank you, this is a really careful piece of work. The silent KeyError hidden behind "posterior estimate not yet sufficient" was nasty, and the before/after plots make the case on their own.

On your two questions. (a) Yes, change the superscripts everywhere, not plot-only; \gamma_1^{\rm lens.shear} is clearer than \gamma^{\rm ext} twice. (b) We want the latent corner plot as a separate plot, switched by config and written next to corner_anesthetic.png as part of PyAutoFit, which is what you have built. Keeping corner_anesthetic_latent in visualize/plots_search.yaml next to corner_anesthetic, default off, is right.

Rather than ask for another round, I will push the remaining changes onto your branch myself, as we did for PyAutoGalaxy#635, and merge. For the record they are: (1) per-component plots fall back to on when an older config lacks corner_anesthetic_components, even with corner_anesthetic: false (posterior_kind.py:134-146); (2) latent_parameters_from in plot_latent_results runs outside the plot-exception guard, so a failing conversion would raise at the end of a fit; (3) model latents reach latent/samples.csv with no switch, so I will add one to the output.yaml latent block; (4) three-character path truncation can collide (lens.bulge_inner.shear and lens.bulge_outer.shear both become lens.bul.she); (5) a power law's ell_comps linked into a multipole can lose the mass's axis ratio and angle under the last-holder rule, which needs a test; (6) recentred angles will go into plots only, not latent.csv; (7) notation.yaml comments noting the superscript: block now applies only to a bare unnamed Model. Your workspace and PyAutoGalaxy branches will follow once this merges and releases. Thanks again!

- Per-component corner plots follow corner_anesthetic when an older config
  omits corner_anesthetic_components; latent per-component plots get their
  own corner_anesthetic_latent_components key.
- plot_latent_results logs a raising latent conversion instead of failing
  the fit's final visualize, and is skipped in test mode.
- output.yaml latent_model_parameters switches the model's declared latents
  out of latent.csv / latent.results / latent_summary.json.
- Deep-path superscripts are cut consistently and only while they stay unique
  (superscripts_from_component_paths); no more lens.bul.she twice.
- A prior shared between components is seen by every holder's declarations,
  so a power law's ell_comps linked into a multipole keeps its latents.
- Periodic latents are recentred for plots only; latent.csv keeps the
  conversion's range so it does not depend on the fit.
- Root group is "model_root" when a component is named model; component
  paths cached while the model is frozen.

Refs PyAutoLabs#1710

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Jammy2211 Jammy2211 added the pending-release PR queued for the next release build label Oct 21, 2026
@Jammy2211

Copy link
Copy Markdown
Collaborator

@samlange04 I've pushed the review changes onto your branch as f9377f3, so there's no extra round for you. Summary:

  • Per-component plots follow corner_anesthetic when a config doesn't mention corner_anesthetic_components. The latent per-component plots get their own key, corner_anesthetic_latent_components.
  • A raising latent conversion is now logged rather than failing the final visualize, and the latent plots are skipped in test mode.
  • output.yaml latent_model_parameters (default on) switches the declared latents in or out of latent.csv / latent.results / latent_summary.json.
  • Deep labels are shortened consistently and only as far as they stay unique: lens.bulge_i.she / lens.bulge_o.she, and a basis and its profiles read lens.bul / lens.bul.0.
  • Shared priors are seen by every holder's declarations, so a power law's ell_comps linked into its multipoles keeps axis_ratio/angle whichever component the column is named after.
  • Periodic latents are recentred for the corner plots only. latent.csv keeps the conversion's own range, so the column means the same thing in every fit.
  • Minor: the root group becomes model_root when a component is named model, and component paths are cached while the model is frozen.

The full suite passes locally (3881), and the workspace smoke tests pass against the branch (165/165). Once CI is green I'll merge. Your PyAutoGalaxy declarations and workspace config PRs can then follow the release, as you planned. Thanks for this!

@Jammy2211
Jammy2211 merged commit d3a7721 into PyAutoLabs:main Oct 21, 2026
5 checks passed
@Jammy2211 Jammy2211 removed the pending-release PR queued for the next release build label Oct 23, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants