Repository navigation
Corner plots: fix the silent failure on repeated labels; per-component labels and plots; latent parameters - #1709
Conversation
…ated labels
corner_anesthetic keyed anesthetic's columns by the LaTeX parameter labels, which
repeat whenever a model has several components of one type: every Gaussian of a
multi-Gaussian basis is labelled $\epsilon_{1}^{g}$, so a lens model with an MGE
light and an MGE source repeats y, x and both ellipticity components. anesthetic
then raises KeyError, @log_plot_exception reports it as "posterior estimate not
yet sufficient", and the corner plot never appears -- for a 19-parameter SLaM
source fit it was missing at every update and at completion.
Key the columns by position (always unique) and pass the LaTeX labels to
anesthetic for display, so the figure looks as before. `labels` joins the
reserved kwargs, since a caller passing it would now collide with them.
Tests: a model with a repeated label writes its figure and still displays the
repeated label (red without the fix), and `labels` is rejected as reserved.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…er plots
Looking at a 19-parameter SLaM source fit, the corner plot had three problems
beyond the repeated-label crash the previous commit fixed:
1. Labels did not tell components apart. The superscript was the immediate
parent name or the notation config's class abbreviation, so every Gaussian
of a multi-Gaussian basis was y^g whether it sat in the lens bulge, the lens
disk or the source bulge. The superscript is now the dot-joined path of the
named components owning the prior (new `component_paths`): Collection
containers (`galaxies`, `profile_list`), the tuple name and a list index
that only distinguishes copies of a shared prior are dropped, giving
y^{lens.bulge}, y^{lens.disk}, y^{source.bulge}, theta_Ein^{lens.mass}. The
config's class override replaces only the component it names (`lens.ext`),
and a prior directly on a root model keeps the old behaviour, so existing
labels of simple models are unchanged.
2. One 19x19 plot is too dense to read. `parameter_indices_by_component`
groups parameters by top-level component and `WeightedDefaults.plot_results`
writes `corner_anesthetic_<component>.png` per component beside the full
plot (switch `nest.corner_anesthetic_components`, default on; none for a
single-component model). `corner_anesthetic` gains `parameter_indices=`.
3. The sampled parameters are not the ones a model is read in. A component
class may declare `derived_parameters = {("gamma_1", "gamma_2"): fn}`: the
free parameters it replaces and a function of every parameter of the
component (free ones as sample arrays, fixed ones as floats) returning the
derived ones; declarations inherit down the MRO. `derived.py` applies them
to a sample array in model order, keeping each component's superscript. A
derived angle is returned as `Periodic(values, period)` and recentred on
its weighted circular mean, so a posterior peaked at the conversion's branch
cut is not drawn as two peaks. Switch `nest.corner_anesthetic_derived`
(independent of the other two) writes `corner_anesthetic_derived.png` and,
with the per-component switch, one per component; skipped when nothing in
the model declares a conversion. `corner_anesthetic` gains `derived=`.
`should_plot` takes a `default` for a key an older workspace config does not
mention, so neither new switch breaks an existing config.
Tests: component paths, config override scope and shared-vs-independent index
(test_model_mapper), the derived set, fixed values, inheritance and periodic
recentring (test_derived), subset and derived plots (test_nest_plotters), the
per-component and derived plot calls and the missing-key fallback
(test_family_bases).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The conversions and a second set of anesthetic figures take a while on a large sample set, so `nest.corner_anesthetic_derived` ships false and a user opts in. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…ly for a bare model The notation config's per-class abbreviation (`Gaussian: g`) no longer replaces a named component's name: every component is labelled by the path of names it was given in the model, so a shear reads `lens.shear`, a basis of Gaussians `lens.bulge`, and a collection of Gaussians `gaussian_0`, `gaussian_1` rather than `g`, `g`. The abbreviation still applies to a prior directly on a bare `Model`, which has no name of its own. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…e first A parameter owned by a component three levels down, such as a source galaxy's pixelization regularization, was labelled with the full path `source.pixelization.regularization`, which runs into the neighbouring axis labels on a corner plot. `superscript_from_component_path` now keeps the first name whole (the galaxy or plane that distinguishes the components) and cuts every later name to three characters, so the label reads `source.pix.reg`. One- and two-level paths are unchanged. Also removes two superscript tests that were duplicated verbatim into a second test class. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
PyAutoFit already calls a quantity computed from the model but not sampled a latent variable, so the component-declared conversions (a shear's magnitude and angle for gamma_1 / gamma_2) take that name: `latent_parameters` on the class, `autofit/mapper/prior_model/latent_parameters.py` (`LatentParameters`, `latent_parameters_from`), `corner_anesthetic(latent=True)`, config `corner_anesthetic_latent` and files `corner_anesthetic_latent[_<component>].png`. They now join the analysis's latents in the latent output: `latent_samples_from` appends them under their model path (`galaxies.lens.shear.magnitude`) to `latent/samples.csv`, `latent.results` and `latent_summary.json`, with or without analysis-defined latents (`LatentParameters.paths`, `latent_items`). The latent corner plots follow the same rule as that output rather than the `iterations_per_full_update` cadence: a `plot_latent_results` hook on the search and `PosteriorDefaults`, which `SearchUpdater.visualize` calls once the fit completes and during it only if `output.yaml`'s `latent_during_fit` is on. Per-component latent plots are made only for components with a latent parameter (`parameter_indices_by_latent_component`), and not at all for a single-component model, so no plot repeats another. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…common prefix
A prior shared between differently named components (a power law's
ellipticity linked into its multipoles, `mass` / `multipole_3` /
`multipole_4`) was labelled by the components' common prefix alone,
`lens`, because every differing path element was dropped. Only a list
index that differs (the Gaussians of a basis) says nothing; a differing
name is kept, from the path the samples column is named after
(`unique_prior_paths`, the last component holding the prior), so label
and column agree: `q^{lens.mass}`.
`latent_parameters_from` indexed `model.paths`, which lists every path to
a shared prior, instead of `unique_prior_paths`, so the latent.csv keys of
such a model were misaligned.
Found by running the HST project's EPLS34 model (elliptical and circular
multipole classes) through labels, latent parameters, latent samples and
the corner plots.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
sphinx_autodoc_typehints could not resolve the string annotation on latent_parameters_from, three new warnings over the baseline. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Jammy2211
left a comment
There was a problem hiding this comment.
Hi @samlange04, thank you, this is a really careful piece of work. The silent KeyError hidden behind "posterior estimate not yet sufficient" was nasty, and the before/after plots make the case on their own.
On your two questions. (a) Yes, change the superscripts everywhere, not plot-only; \gamma_1^{\rm lens.shear} is clearer than \gamma^{\rm ext} twice. (b) We want the latent corner plot as a separate plot, switched by config and written next to corner_anesthetic.png as part of PyAutoFit, which is what you have built. Keeping corner_anesthetic_latent in visualize/plots_search.yaml next to corner_anesthetic, default off, is right.
Rather than ask for another round, I will push the remaining changes onto your branch myself, as we did for PyAutoGalaxy#635, and merge. For the record they are: (1) per-component plots fall back to on when an older config lacks corner_anesthetic_components, even with corner_anesthetic: false (posterior_kind.py:134-146); (2) latent_parameters_from in plot_latent_results runs outside the plot-exception guard, so a failing conversion would raise at the end of a fit; (3) model latents reach latent/samples.csv with no switch, so I will add one to the output.yaml latent block; (4) three-character path truncation can collide (lens.bulge_inner.shear and lens.bulge_outer.shear both become lens.bul.she); (5) a power law's ell_comps linked into a multipole can lose the mass's axis ratio and angle under the last-holder rule, which needs a test; (6) recentred angles will go into plots only, not latent.csv; (7) notation.yaml comments noting the superscript: block now applies only to a bare unnamed Model. Your workspace and PyAutoGalaxy branches will follow once this merges and releases. Thanks again!
- Per-component corner plots follow corner_anesthetic when an older config omits corner_anesthetic_components; latent per-component plots get their own corner_anesthetic_latent_components key. - plot_latent_results logs a raising latent conversion instead of failing the fit's final visualize, and is skipped in test mode. - output.yaml latent_model_parameters switches the model's declared latents out of latent.csv / latent.results / latent_summary.json. - Deep-path superscripts are cut consistently and only while they stay unique (superscripts_from_component_paths); no more lens.bul.she twice. - A prior shared between components is seen by every holder's declarations, so a power law's ell_comps linked into a multipole keeps its latents. - Periodic latents are recentred for plots only; latent.csv keeps the conversion's range so it does not depend on the fit. - Root group is "model_root" when a component is named model; component paths cached while the model is frozen. Refs PyAutoLabs#1710 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
@samlange04 I've pushed the review changes onto your branch as f9377f3, so there's no extra round for you. Summary:
The full suite passes locally (3881), and the workspace smoke tests pass against the branch (165/165). Once CI is green I'll merge. Your PyAutoGalaxy declarations and workspace config PRs can then follow the release, as you planned. Thanks for this! |
Discussion: https://github.com/orgs/PyAutoLabs/discussions/38.
corner_anestheticproduced no plot for any model with two components of one class (most lens models) and logged"posterior estimate not yet sufficient" instead: anesthetic keys its columns by label, the LaTeX labels repeat
(every Gaussian of a basis is
$\epsilon_1^{\rm g}$),plot_2draisedKeyErrorand@log_plot_exceptionswallowed it. This PR fixes that and makes the plots of a many-component model readable.
Fix. Columns are keyed by position (
p0,p1, ...) and the LaTeX labels passed as anesthetic display labels(
NestedSamples(labels=...),make_2d_axes(labels=...));labelsjoins the reserved kwargs.Labels.
AbstractPriorModel.superscriptsis the dot-joined path of the names the owning components were givenin the model (
lens.bulge,source.bulge,lens.shear), via newcomponent_paths:Collectioncontainers,tuple names and the list index of a basis sharing one prior are dropped, an index distinguishing independent priors
is kept, a prior shared between differently named components takes the path its samples column is named after
(
unique_prior_paths). The notation config's class abbreviation applies only to a prior on a bare unnamedModel.Paths three or more levels deep keep the first name and cut the rest to three characters
(
superscript_from_component_path:source.pix.reg). Tests intest_model_mapper.pycover each rule.Per-component plots. New
parameter_indices_by_component(top-level component -> parameter indices; rootpriors as
"model") and aparameter_indiceskwarg oncorner_anesthetic;WeightedDefaults.plot_resultswritescorner_anesthetic_<component>.pngper component when there is more than one. Configvisualize/plots_search.yamlnest.corner_anesthetic_components: true;PosteriorDefaults.should_plot(name, default=)falls back for a key an older workspace config lacks.Latent parameters.
autofit/mapper/prior_model/latent_parameters.py: a component class declareslatent_parameters = {("gamma_1", "gamma_2"): fn};fntakes every parameter of the component by name (free asarrays over the samples, fixed as floats, tuples flattened
name_0/name_1) and returns{name: array};declarations merge down the MRO.
AbstractPriorModel.latent_parameters_from(parameter_array, weights=None)returns aLatentParameters(names, superscripts, component paths, full model paths, array, mask);has_latent_parameters.Periodic(values, period).recentred(weights)wraps an angle into one period centred on its weighted circular mean,so a posterior at the branch cut is not drawn as two peaks; exported as
af.Periodic.corner_anesthetic(latent=True)plots the latent set;WeightedDefaults.plot_latent_resultswritescorner_anesthetic_latent.pngandcorner_anesthetic_latent_<component>.pngfor the components with a latentparameter (none for a single-component model), under config
nest.corner_anesthetic_latent(default false),independent of the other two switches.
latent_samples_fromappends the model's latent parameters to the latent samples under their model path(
galaxies.lens.shear.magnitude), with or without analysis latents, so they reachlatent/samples.csv,latent.resultsandlatent_summary.json.plot_latent_resultsonNonLinearSearch/PosteriorDefaults(no-opfor chain and point kinds), which
SearchUpdater.visualizecalls once the fit completes and during it only ifoutput.yaml'slatent_during_fitis on.plot_resultsand theiterations_per_full_updatecadence areunchanged; searches overriding
plot_results(samples)are unaffected.Compatibility. Superscripts in
model.resultsand plot labels change for named components (from the classabbreviation to the component path). Two new config keys, both with in-code fallbacks. No search, samples or
output-format change otherwise.
Before / after (a SLACS lens's 19-parameter source-light stage; more in the discussion):
Before: old labels
After:
corner_anesthetic_lens.pngAfter:
corner_anesthetic_latent_lens.pngTests.
test_model_mapper.py(component paths, names only, shared priors, abbreviation),test_latent_parameters.py(new),test_nest_plotters.py(repeated labels, subset, latent),test_family_bases.py(per-component / latent plot files, after-fit hook, single-component),
test_updater.py(latent plot schedule),test_latent.py(model latents in the latent output). Full suite: 3720 passed locally (macOS, Python 3.13).Commits:
9db2921aafix,941bd9d1blabels + per-component + latent plots,43dcf0f65latent plot off by default,587a75b32names only,7db4b9cf8deep-path abbreviation,5d9ba5167latent naming / latent.csv / after-fit,13bd430cfshared priors.🤖 Generated with Claude Code
Maintainer changes (f9377f3, tracked in #1710)
The review findings are resolved on this branch rather than in another review round. Maintainer decisions (2026-10-21): (a) superscripts are the component's dot-joined path everywhere, including
model.results; (b) the latent corner plot stays a separate plot, behindplots_search.yamlnest.corner_anesthetic_latent(default off).API Changes
Corner plots label each parameter with its component path, can be split per top-level component, and can show the model's declared latent parameters (e.g. a shear's magnitude and angle). Models can declare latents with a
latent_parametersclass attribute;af.Periodicmarks angles. On top of the contributor's work, this commit adds two config keys and one keyword, and changes three behaviours: per-component plots followcorner_anestheticwhen not configured, deep labels are guaranteed unique, and shared priors are converted by every holder. latent.csv now keeps an angle's fixed range.See full details below.
Test Plan
pytest test_autofit: 3881 passed, 1 skipped, 1 xfailed (local, f9377f3 content)test_weighted_plot_results_components_follow_corner_anesthetic_when_key_absent, (ii)test__visualize__raising_latent_conversion_is_logged_not_raised, (iii)test__latent_parameters__shared_prior_is_converted_by_every_holder, (iv)test__superscripts__deep_paths_stay_unique/..._in_a_model_stay_unique. All 13 new or changed tests fail on bf43aa6.pyauto-heart smokeagainst this branch: 165/165 passed (autofit 10, autogalaxy 16, autolens 43, autolens_workspace_test 37, euclid 9, HowToLens 50). The autocti environments did not build locally (missing GSL headers); not run.Full API Changes (for automation & release notes)
Added
AbstractPriorModel.latent_parameters_from(parameter_array, weights=None, recentre=False)— model latents (componentlatent_parametersdeclarations) over samples;recentre=TruerecentresPeriodiclatents on their circular mean (plots only)af.Periodic(values, period)— marks a periodic latent (angle)AbstractPriorModel.component_paths,parameter_indices_by_componentautofit.mapper.prior_model.abstract.superscripts_from_component_paths(paths),indices_by_top_level_component(paths)output.yaml:latent_model_parameters: true— model latents join latent.csv / latent.results / latent_summary.jsonvisualize/plots_search.yamlnest:corner_anesthetic_components,corner_anesthetic_latent(default false),corner_anesthetic_latent_componentscorner_anesthetic_<component>.png,corner_anesthetic_latent[_<component>].pngChanged Behaviour
model.results) are the dot-joined named-component path (lens.bulge,source.pix.reg), not the class'snotation.yamlsuperscript:entry, which now applies only to priors directly on an unnamedModellens.bulge_i.she/lens.bulge_o.she)corner_anesthetic_componentsis absent from a config it followscorner_anesthetic;corner_anesthetic_latent_componentsfollowscorner_anesthetic_latent"model", or"model_root"when a component is namedmodelMigration
plots_search.yaml/output.yamlkeys (library defaults apply meanwhile); updatenotation.yamlsuperscript:comments (phase 3, after release)Heart RED development-only override
Live maintainer authorization, 2026-10-21: "do 2, Override and proceed" (start_dev for #1710). This covers commit, push and PR work only. Merge needs a separate
/prmwith every required check green. No release. This branch makes no claim to fix Heart. Heart readiness: STALE ("release validation incomplete: no rehearsal for current source"). Monitoring feed at ship:Heart: RED — Release RED; monitoring RED · 14/100 · 464 unresolved (updated 23h ago)Exact RED reasons
PyAutoFit: CI failure — https://api.github.com/repos/PyAutoLabs/PyAutoFit/actions/runs/39111874604PyAutoFit: PyAutoFit: CI failurePyAutoFit/Tests: failurePyAutoFit/workflows/Tests: failurePyAutoHands/workflows/Python Version Matrix: failurePyAutoHeart/workflows/Workspace Smoke: failurePyAutoMind/workflows/Spawn Drift: failurePyAutoArray/Tests: redPyAutoArray/Tests: redPyAutoFit/Tests: redPyAutoFit/Tests: redPyAutoFit/Tests: redPyAutoFit/Tests: redautofit_workspace_test/Smoke Tests: redautogalaxy_workspace_test/Smoke Tests: redautolens_workspace_test/Smoke Tests: redCI wall-clock: 27 gates · slowest PyAutoBrain Nightly Release 49m · 9 hang eventsPyAutoFit: PyAutoFit: CI failureWorkspace validation: NOT ready — 1414p / 1f / 133s @ cloud#39224774724euclid: redtest_run: redPyAutoArray: redautolens_inference: redautolens_profiling: redWorktree drift: 5 orphan / 0 missing / 4 dirtyeuclid_strong_lens_modeling_pipeline: redGenerated by the PyAutoLabs agent workflow.
🤖 Generated with Claude Code