|
| 1 | +--- |
| 2 | +title: Aggregator performance (result loading at catalogue scale) |
| 3 | +type: concept |
| 4 | +topics: [software, aggregator, performance, profiling, sqlite, database, io, pyautofit] |
| 5 | +sources: |
| 6 | + - PyAutoFit issues #1375, #1377, #1385; PyAutoConf #129; autolens_workspace_test #171 |
| 7 | + - Merged PRs: PyAutoFit #1376/#1380/#1384/#1386, PyAutoConf #130, autofit_workspace_test #48/#49/#50/#51, autolens_workspace_test #172 |
| 8 | + - Harnesses: autofit_workspace_test scripts/profiling/aggregator/ + autolens_workspace_test scripts/profiling/aggregator/ |
| 9 | + - Raw grids: <workspace>/output/profiling_aggregator/results/*.json (local, 2026-07-16/17) |
| 10 | +status: draft |
| 11 | +last_updated: 2026-07-17 |
| 12 | +--- |
| 13 | + |
| 14 | +# Aggregator performance (result loading at catalogue scale) |
| 15 | + |
| 16 | +## TL;DR |
| 17 | + |
| 18 | +A 2026-07-16/17 campaign (PyAutoFit #1375 arc, 10 PRs) profiled every way PyAutoFit |
| 19 | +loads modelling results at Euclid catalogue scale (3000+ lenses) using mock result |
| 20 | +trees fabricated in seconds via `PYAUTO_TEST_MODE_SAMPLES` bypass fits — no sampler |
| 21 | +ever runs. Central results: |
| 22 | + |
| 23 | +> **The scaling axis is samples-per-result, not number of results.** The directory |
| 24 | +> scan costs ~0.2 ms/result even at 3000 results; the full `samples.csv` parse |
| 25 | +> dominates everything. After the fixes, per-result costs: summaries ~6.5 ms, |
| 26 | +> model ~4.6 ms, full samples ~45 ms @1k rows (−45% at 10k rows). |
| 27 | +> **Every workflow stage is floor-bounded by JSON→object deserialization |
| 28 | +> (`from_dict`), not by lens math or file scanning.** |
| 29 | +
|
| 30 | +Three latent **crash-level bugs** were found by profiling realistic data, all fixed: |
| 31 | +`from_dict` silently dropped dict entries whose value is exactly `0.0`; |
| 32 | +`AggregateFITS` leaked one file handle per HDU per result (hard crash ~500 results); |
| 33 | +database aggregator slicing was inverted (`agg[:5]` returned `len−5` fits). |
| 34 | + |
| 35 | +## What was measured (per-result, mock results) |
| 36 | + |
| 37 | +| pathway | before | after | change | |
| 38 | +|---|---|---|---| |
| 39 | +| `values("samples")` @10k rows | 611 ms | 334 ms | −45% (csv headers parsed once, not per row) | |
| 40 | +| `values("samples")` @1k rows × 1000 results | 68 ms | 45 ms | −34% | |
| 41 | +| `values("model")` | 8.2 ms | 4.6 ms | −44% (prior-config lookup caching, Conf#130) | |
| 42 | +| `values("samples_summary")` | 8.7 ms | 6.6 ms | −25% | |
| 43 | +| `AggregateCSV` (csv_make) @15-param models | 110 ms | 83 ms | −25% (row caching + double-eval fix) | |
| 44 | +| `AggregateFITS` (fits_make) @2 HDUs | 5.6 ms | 4.5 ms | −19% + fd leak fixed | |
| 45 | +| `AggregateImages` (png_make) | ~1.5 ms | — | profiled clean, no change | |
| 46 | +| `Aggregator.from_directory` scan | ~0.2 ms | ~0.2 ms | zips no longer re-extracted on repeat opens | |
| 47 | + |
| 48 | +Lens-level (`TracerAgg`/`FitImagingAgg`, autolens_workspace_test#172): reconstruction |
| 49 | +adds only ~20–60% over the raw summary load at 7×7 fixture scale — deserialization |
| 50 | +dominates, not lens computation. |
| 51 | + |
| 52 | +## The sqlite verdict (revised at representative scale) |
| 53 | + |
| 54 | +- Small/simple mock data said sqlite reads were 2–10× slower than the directory |
| 55 | + Aggregator. **At representative scale** (10k samples × 18-param model, the |
| 56 | + ~9 MB SLaM parity target) **the full-samples read is comparable** (150 vs |
| 57 | + 165 ms/result). Small-data benchmarks mislead here — always measure at |
| 58 | + production shape. |
| 59 | +- What stands against sqlite: build cost ~0.35 s/result up front (~17.5 min + ~5 GB |
| 60 | + at 3000 lenses), dominated by loading every result's samples into the ORM. |
| 61 | +- What stands for it: single-file storage (HPC inode limits) and fast indexed |
| 62 | + queries (~1–6 ms) once built. |
| 63 | +- The direct-write (session) path is **not** broken as suspected — its real gap was |
| 64 | + `samples_summary` never being stored (`DatabasePaths` no-op, fixed in #1380); |
| 65 | + minimised samples (~1 row) are by design (`save_all_samples=False`). |
| 66 | + |
| 67 | +## Memory scaling limit (open) |
| 68 | + |
| 69 | +`values("samples")` across 3000 × 1k-sample results OOMs (~6.6 GB RSS): every |
| 70 | +`SearchOutput` caches its `Samples`, so even generator-style iteration accumulates. |
| 71 | +Workaround: process in slices. Candidate fix (not taken): opt-out/weak cache for |
| 72 | +bulk iteration. |
| 73 | + |
| 74 | +## Bugs found by profiling realistic data (all fixed) |
| 75 | + |
| 76 | +1. **`from_dict` dropped 0.0-valued dict entries** (Fit#1384): the `type=="dict"` |
| 77 | + branch filtered `if value`. Exposed by bypass output writing exact prior medians |
| 78 | + — lens centres/ell_comps are zero-median, so summaries lost 8/15 parameters and |
| 79 | + `TracerAgg` crashed. Pre-existing; affected any stored dict with legit zeros. |
| 80 | +2. **`AggregateFITS` fd leak** (Fit#1386): one `fits.open` per requested HDU per |
| 81 | + result, handles kept alive by returned memmaps — "Too many open files" at ~500 |
| 82 | + results (default ulimit 1024); a 3000-lens fits_make run would have died. |
| 83 | +3. **Database aggregator slicing inverted** (Fit#1380): `agg[:5]` on 26 fits |
| 84 | + returned 21 (`len − stop` instead of `stop − start`); the old unit test passed |
| 85 | + only by a 2-fit coincidence where `len − stop == stop`. |
| 86 | +4. **`JSONPriorConfig` re-sorted the whole flattened config on every lookup** |
| 87 | + (Conf#130) and linear-scanned identical repeated queries — 77% of the model |
| 88 | + deserialization floor was default-prior construction that `from_dict` then |
| 89 | + overwrote. |
| 90 | + |
| 91 | +## Durable methodology lessons |
| 92 | + |
| 93 | +- **Mock the outputs, never run the sampler**: one template result written through |
| 94 | + the real library write path (`PYAUTO_TEST_MODE=2/3` + `PYAUTO_TEST_MODE_SAMPLES=N` |
| 95 | + bypass fit), stamped N times with per-copy `unique_tag`/`dataset_name` — thousands |
| 96 | + of results in seconds, formats can never drift. The database keys fits by the |
| 97 | + (search, model, unique_tag) identifier hash: identical copies collapse to one row. |
| 98 | +- **Time each stage on a fresh Aggregator**: `SearchOutput` caches model/samples/ |
| 99 | + summaries per instance, so stage order otherwise contaminates the numbers. |
| 100 | +- **Benchmark grids need an idle machine**: a concurrent pytest run inverted a |
| 101 | + before/after signal once. Under unavoidable load, use interleaved in-process A/B |
| 102 | + of old-vs-new implementations instead of cross-run comparison. |
| 103 | +- **Background `cmd | tail` pipelines mask OOM kills** (pipeline exit code is |
| 104 | + tail's 0) — check `dmesg` for "Out of memory". |
| 105 | +- **`open_database` silently prepends `conf.instance.output_path`** (plus the |
| 106 | + test-mode segment) to relative sqlite paths — a decoy empty file appears at the |
| 107 | + literal path; use absolute paths in tooling. |
| 108 | + |
| 109 | +## Where the remaining time goes (deliberately not pursued) |
| 110 | + |
| 111 | +After all fixes the floor is ~4.6 ms/result for a model load: sample-json |
| 112 | +`from_dict` recursion and prior-object construction itself. No repeated-waste |
| 113 | +pattern remains — further gains would need a redesign (e.g. schema-aware |
| 114 | +deserialization or model memoization across identical `model.json` files), judged |
| 115 | +diminishing returns as of 2026-07-17. |
0 commit comments