Skip to content

feat(jax_compile): warm-compile baselines — cache_state, pins, dashboard #103

Description

@Jammy2211

Overview

Phase 2 of the compile-axis arc (phase 1 is PyAutoBrain#218 / PyAutoBrain#219).

Tracking warm compile per cell is the point of the arc: the persistent
compilation cache turned 117.0 s into 2.3 s (CPU MGE vag) and 5517.8 s into
937.1 s (A100 end-to-end), --xla_gpu_autotune_level=0 cut the pathological cold
probe 17×, and nothing watches for either reverting. Both are settings, so a
config drift or an XLA_FLAGS clobber — the PyAutoNerves#127 failure that went
undetected for two months — puts the 70-minute worst case straight back.

The blocker

Warm rows were not machine-identifiable. Warmness lived only in the free-text
tag, which carries ~40 ad-hoc spellings across the committed corpus
(census-warm, census-warm2, prodigy-census-warm-retry, cache-warm,
mb_homo_cold, idle-check, smoke, …). A dashboard whose whole job is tracking
warm compile could not tell which rows were warm.

The obvious shortcut is a trap: cache_dir is non-empty on cold rows too,
because the cold run is the one that populates the cache.

Legs

  • cache_state derived from probe behaviour — count cache entries either
    side of lowered.compile(). Wrote a new entry ⇒ cold (miss); wrote nothing
    into a populated cache ⇒ warm (hit); no --cache-dirnone;
    configured-but-empty-and-nothing-written ⇒ unknown, not silently warm.
    Derived per transform, since one invocation can legitimately miss on one
    transform and hit on another — something a per-run tag cannot express.
  • host_state (cpu_count, 1-minute load average). XLA compiles on the
    host cores; README.md records the first measurements being wrong by up to
    (851 s vs 117 s for the same compile) from host load alone, and the
    corpus already mixes a 32-core RAL allocation with laptop rows.
  • record_kind so the schema is self-describing — export_probe.py and
    trace_profile.py append their own shape into the same results tree.
  • Backfill of the 89 existing compile records — exact where cache_dir was
    empty (none), inferred only from an end-anchored cold/warm tag, left
    unknown otherwise. Result: 33 cold, 34 warm, 19 none, 3 unknown.
  • Warm-compile pins per cell, keyed on the comparability key
    (hardware, jax_version, mixed_precision, cache_state) — the compile
    equivalent of the runtime results' pinned_expected.
  • Dashboard rows through build_baseline.py / build_readme.py.
  • Brain legingest --axis compile, reporting unpinned and drifted rows
    strictly within one comparability key. Blocked on PyAutoBrain#219 merging
    (it needs that PR's load_compile_corpus / compile_tier_of).

The trap any pin comparison must not create

Never compare a cold row against a warm pin, an A100 row against a CPU pin, or
rows across a jax_version bump. Cache keys include the jax version, so a bump
recompiles once by design — expected behaviour, not drift. Phase 3 classifies
it as such; a tool that flags it is one people learn to ignore.

Note on CI

scripts/misc/test/ is not run by any workflowlint.yml runs ruff,
build_readme.py --check, lychee and a smoke script, but no pytest step. The new
test_jax_compile_cache_state.py (9 tests) therefore passes locally but is not a
CI gate, and test_vram_config.py::test_aggregate_matches_cell_prefixed_stems is
currently failing on main unnoticed. Worth a separate small PR adding the step;
flagging rather than folding it in here.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions