docs(1961): record the enableCompileCache() CLI startup measurement — not shipped

[thymikee]

Summary

Issue #1961 asked to measure module.enableCompileCache() on the CLI bin entry before shipping it — "measure first, ship only if it's a real win," bar = warm-run delta >= 50ms. Measured and not shipped. The code change was reverted; this PR retains the writeup, the harness, and the raw samples.

Part of #1961, not Closes — the issue asked for cold and warm OS-file-cache measurements and only the warm leg could be run here (see Limitation below).

Reproducibility

Harness at scripts/perf/compile-cache-ab.mjs, raw per-sample elapsed times at docs/cli-compile-cache-startup-samples.json. Every median and interval below recomputes without re-running anything:

node scripts/perf/compile-cache-ab.mjs stats --in docs/cli-compile-cache-startup-samples.json

The bootstrap uses a fixed seed, so a recompute reproduces the published intervals exactly.

Methodology

Elapsed wall-clock. Both arms are copies of the bin entry against the same unchanged dist/, sampled in one interleaved loop with arm order flipped each iteration, reported as median plus a bootstrap 95% CI on the median difference.

Interleaving removes order bias; it does not prove contention was absorbed. This host is shared and load was observable but not controllable, so load is reported per route rather than as one global claim.

Numbers

Warm compile cache — the issue's bar (n=120/arm)

Route	load	baseline	compile-cache	delta	95% CI	verdict
--version	6.4→7.1	30.4 ms	30.3 ms	-0.1 ms	[-0.5, +0.5]	no sig. difference
--help	7.1→7.7	49.3 ms	45.7 ms	-3.6 ms	[-4.3, -2.9]	faster
open CLI-side path	7.7→7.4	59.9 ms	54.8 ms	-5.1 ms	[-5.6, -4.5]	faster

A small but statistically real gain on the heavier routes — roughly ten times short of the bar.

Correction: an earlier revision reported these as "indistinguishable from zero" (+0.1 / +0.6 ms). Those runs were on a busier host; the identical harness on a quiet host resolves a ~3–5 ms gain the noise was hiding. The conclusion is unchanged, but "no effect" was a measurement artifact.

ad open end to end, real iOS simulator (n=150/arm)

Route	load	baseline	compile-cache	delta	95% CI	verdict
ad open Settings	7.7→5.7	1232.9 ms	1226.5 ms	-6.4 ms	[-10.0, -3.8]	faster

Also a coherence check: -6.4 ms end to end matches the -5.1 ms CLI-side gain, which is what should happen if only CLI startup is affected and the remaining ~1.2 s of device work is untouched.

Run on a dedicated throwaway simulator (every booted sim was claimed by another session; the device-claim guard correctly refused them). A first attempt was discarded, not reported — it was launched while the new sim was still doing first-boot work and Spotlight was indexing it, driving load to 120 on 12 cores.

Cold compile cache — a regression (n=60/arm, both arms wiped identically)

Route	load	baseline	compile-cache	delta	95% CI	verdict
--help	8.2→8.3	51.3 ms	62.6 ms	+11.3 ms	[+10.3, +12.4]	slower
open CLI-side path	8.3→8.3	62.3 ms	78.7 ms	+16.4 ms	[+15.9, +18.0]	slower

The cost of writing 664–848 kB of cache entries. Misses are routine: keyed per Node version, in os.tmpdir(), subject to tmp reaping.

Decision: no-ship

• The warm gain is real but an order of magnitude short of the >=50 ms bar; every interval lies far above the -50 ms ship threshold.
• Cold-cache runs are slower (+11 to +16 ms) — shipping trades a first-run regression for a warm gain ~3x smaller.
• The API is Stability 1.1 – Active Development at the repo's supported Node floor (added v22.8.0; repo requires >=22.12).

Limitation: cold OS page cache unmeasured

sudo purge requires a password unavailable non-interactively here, so only the warm-page-cache leg was measured. No claim is made about which way a cold page cache would move the comparison — a previous revision asserted it would penalize the compile-cache arm; that was speculation and has been removed. Settling it needs the same interleaved A/B on a host where page-cache eviction can run between samples, which would change the baseline arm's cost too.

Tests

pnpm lint, pnpm typecheck, pnpm build, pnpm format clean. bin/agent-device.mjs is byte-identical to main.

Part of #1961

[thymikee]

Not ready at aef6212b:

• perf(cli): measure module.enableCompileCache() for CLI cold start #1961 asks for user-visible cold/warm elapsed startup on ad --help and ad open. The retained writeup omits cold-cache evidence, substitutes session list, and applies the >=50 ms elapsed-time decision bar to child CPU time, which excludes compile-cache filesystem wait. Use interleaved/order-flipped elapsed samples on the requested routes under controlled load, or call this CPU-only evidence inconclusive and leave perf(cli): measure module.enableCompileCache() for CLI cold start #1961 open.
• enableCompileCache() was added in Node 22.8, not 22.1, and remains Active Development in the repo-minimum Node 22.12.
• -0.36 ms / +1.3% has contradictory signs while claiming same-sign replication.
• diagnose(1961): is not an allowed conventional commit prefix.

The docs-only scope is reasonable after these corrections.

[thymikee]

Thanks — all four findings addressed. Rebased onto 03c398406 (clean, docs-only). Taking them in turn:

1. Wrong metric / omitted cold evidence / substituted route → outcome (a): redone with elapsed samples

You were right that this was measuring the wrong quantity, and it mattered. The compile cache trades CPU for filesystem wait, and getrusage(RUSAGE_CHILDREN) excludes exactly that wait, so the CPU delta was not comparable to an elapsed bar. It flattered the change: --help showed -3.4 ms by CPU but +0.1 ms elapsed on the same binaries. Everything is now elapsed wall-clock.

The host had also quieted (load 7–10 vs 27–46 before), so interleaved elapsed sampling is now clean. Design: both arms in one loop, arm order flipped each iteration, median + bootstrap 95% CI on the median difference.

On the requested routes (warm, n=120/arm): --help +0.1 ms [-1.0, +0.9]; --version -0.4 ms [-1.6, +0.3].

ad open end to end, real iOS simulator (n=150/arm): -11.6 ms, CI [-26.0, +4.7]. This is powered for the question — at the observed pooled SD of 182 ms, resolving 50 ms needs ~102 samples/arm and 150 were collected, so a >=50 ms win is excluded rather than unproven. All three booted sims were claimed by other concurrent sessions (the device-claim guard correctly refused them), so I created and deleted a dedicated throwaway sim rather than disturb that work.

session list is dropped; it was not a valid stand-in and the doc now says so. open … --platform bogus still appears, but explicitly as a component of open — it loads the full CLI + open module graph (83 cache entries, the largest of any route) then fails validation before device work, isolating the only part a compile cache can touch. It is labelled as such, not as a substitute.

Cold evidence added, and it changed the conclusion's shape. Cold compile cache is a regression: --help +12.9 ms [+9.9, +14.6], open CLI-side path +22.7 ms [+18.3, +28.0] — the cost of writing 664–848 kB of entries. So the change trades a first-run regression for no warm gain.

Stated limitation: sudo purge needs a password unavailable non-interactively, so a cold OS page cache could not be simulated; all numbers are warm-page-cache. That is a separate axis from the cold/warm compile cache, which is fully controlled. A cold page cache would if anything penalize the compile-cache arm further.

One correction worth flagging: my first cold harness wiped the cache only before the compile-cache arm, leaving asymmetric filesystem state and producing a bogus +52.8 ms for open. Wiping before both arms gives the +22.7 ms above. The inflated figure never reached the doc.

2. Node version and stability — confirmed wrong, fixed

Verified against the Node docs: added in v22.8.0 (not 22.1), and Stability 1.1 – Active Development on the v22.x line matching the repo's engines: >=22.12. It reached 1.2 Release candidate only at v25.4.0. The doc now states the stability level and notes the supported floor is what governs a shipped change; it is also now a third reason in the no-ship rationale.

3. Contradictory -0.36 ms / +1.3% — removed

That line came from the CPU-time run and had mismatched signs. The whole CPU-time result set is gone, replaced by elapsed measurements where every delta is reported with its CI and consistent sign.

4. Commit prefix — fixed

No commitlint config in the repo, so I checked the enforced vocabulary against the last 400 commits on main: fix, refactor, test, feat, docs, perf, ci, chore, build. diagnose is not among them. Amended to docs(1961): and force-pushed (single commit, sole author, no other contributors on the branch).

Outcome

(a) — redone with elapsed samples on the named routes, cold-cache evidence included, cold page cache stated as a limitation. The no-ship decision now rests on evidence in the units the bar is stated in: warm CIs within ±2 ms of zero, ad open excluded at [-26.0, +4.7] ms with adequate power, and cold-cache runs measurably slower.

[thymikee]

The no-ship conclusion is supported, but the power claim at lines 156–159 (and repeated in the summary) is not. 2*(1.96*182/50)^2 ≈ 102 gives a 95% CI half-width of about 50 ms; it does not provide conventional 80% power to detect a 50 ms effect, and a median/interleaved design needs its own variance treatment. The observed bootstrap CI [-26.0, +4.7] ms already directly excludes the −50 ms ship threshold. Please remove the pooled-SD/sample-size claim and replace ‘with enough samples to have detected’ with the direct CI statement.

[thymikee]

You're right, and the correction makes the argument stronger. Fixed and pushed (rebased onto 04e4c23b9, docs(1961): prefix kept).

The error: 2*(1.96*182/50)^2 ≈ 102 is a precision calculation — it sizes a 95% CI half-width of ~50 ms. I labelled it "adequately powered," which is a different claim: conventional 80% power to detect a 50 ms effect needs roughly 2*(1.96+0.84)^2*SD^2/50^2, about twice that. And neither formula is valid as written here anyway, since the statistic is a median difference under an interleaved design, whose sampling variance is not the pooled sample SD. So the passage was wrong three ways over, and I've deleted it rather than repaired it.

The replacement is the direct empirical statement, as you asked: the bootstrap 95% CI on the median difference, [-26.0, +4.7] ms, lies entirely above the -50 ms ship threshold, so a >=50 ms improvement is excluded rather than merely unproven — resting on the variability actually observed in the run instead of an assumed model. No corrected power calculation substituted.

Swept the whole doc, not just 156–159. Three further spots restated the same overstatement in different words:

• the opening summary — "sampled past the point where a 50 ms effect would have shown" → now "an interval lying entirely above the -50 ms ship threshold";
• the ship/no-ship bullet — "with enough samples to have detected a 50 ms effect had one existed" → now states each interval lies above -50 ms and so directly excludes it;
• the warm-cache table note — "excludes a 50 ms effect by a factor of ~25", a precision ratio dressed as an exclusion → now "every interval lies within ±2 ms of zero, so all three exclude the -50 ms ship threshold outright".

The pooled-SD figure, the ~102 sample-size number, and every "power"/"adequately powered"/"detect" phrasing are gone; a grep for them across the doc now returns nothing. The remaining statistical claims are all direct CI-exclusion statements.

The conclusion is unaffected — if anything it's better supported, since the CI-based exclusion is empirical rather than model-dependent. No-ship still rests on: every measured CI excluding the threshold, the cold-cache regression (+13 to +23 ms), and Stability 1.1 at the repo's Node floor.

CI is green.

[thymikee]

The no-ship conclusion is supported, but this should not close #1961 as written: the issue explicitly requested cold and warm OS-file-cache measurements, while the retained record says every sample used a warm page cache. Change Closes #1961 to Part of #1961 unless that missing leg is measured. Also remove the unmeasured claim that a cold OS cache would necessarily penalize the compile-cache arm, and state route-specific load accurately: the end-to-end ad open run reached load 20, so interleaving reduces order drift but does not prove contention was absorbed. Retaining the minimal harness/raw samples would make the bootstrap record reproducible.

[github-actions]

Size Report

Metric	Base	Current	Diff
JS raw	2.37 MB	2.37 MB	0 B
JS gzip	794.8 kB	794.8 kB	0 B
npm tarball	916.5 kB	916.5 kB	0 B
npm unpacked	3.18 MB	3.18 MB	0 B

npm unpacked components

Component	Base	Current	Diff
JS / dist source	2.51 MB	2.51 MB	0 B
Apple runner source/project	564.2 kB	564.2 kB	0 B
macOS helper source	54.5 kB	54.5 kB	0 B
Android helper artifacts	0 B	0 B	0 B
Other package files	44.5 kB	44.5 kB	0 B

Startup median (7 runs, lower is better):

Scenario	Base	Current	Diff
CLI --version	26.9 ms	27.8 ms	+0.8 ms
CLI --help	79.3 ms	82.2 ms	+2.8 ms

Top changed chunks: no changes in the largest emitted chunks.

Top changed packed files

No changed packed files.

[thymikee]

All four addressed. Rebased onto 04e4c23b9; CI green (28/28, now the full suite since this adds a scripts/ file).

1. Scope → Part of #1961

Changed in the PR body and the commit trailer. You're right that the issue asked for cold and warm OS-file-cache legs and only warm was measured, so it stays open. The doc now says this in the summary and again in the decision section, rather than leaving it implied by a limitation note further down.

2. Removed the unmeasured claim

Deleted "a cold page cache would, if anything, penalize the compile-cache arm further." That was speculation dressed as a finding — I hadn't measured it. The replacement states plainly that the direction is untested, keeps the reasoning only as an explicitly-labelled hypothesis, and says what would settle it: the same interleaved A/B on a host where page-cache eviction can run between samples, which would change the baseline arm's cost too and could move the comparison either way.

3. Load stated per route, and the interleaving claim corrected

The doc now records the 1-minute load at the start and end of every route's block, in each results table, instead of one global "load 7–10" characterisation. And the reasoning is fixed: order-flipped interleaving removes order bias — neither arm systematically occupies the earlier slot — but it does not show contention was absorbed. That is called out where it matters most, and the doc notes this host is shared, so load was observable but never controllable.

4. Harness and raw samples retained

scripts/perf/compile-cache-ab.mjs plus docs/cli-compile-cache-startup-samples.json (25 kB, six runs). stats --in <file> recomputes every median and interval from the retained samples with a fixed bootstrap seed, so the published numbers reproduce exactly without re-running.

I could not retain samples for the previously-reported runs — the old harness printed summaries and never persisted the arrays — so rather than reconstruct anything I re-ran every leg with capture. That changed two results, both worth flagging:

The warm effect is real, not zero. On a quiet host (load 6–8), --help is -3.6 ms [-4.3, -2.9] and the open CLI-side path -5.1 ms [-5.6, -4.5] — intervals entirely below zero. The earlier "+0.1 ms / +0.6 ms, indistinguishable from zero" was noise from a busier host hiding a ~3–5 ms gain. End-to-end ad open also tightened to -6.4 ms [-10.0, -3.8], which is a useful coherence check: it matches the CLI-side gain, as it should if only CLI startup is affected and the remaining ~1.2 s of device work is untouched.

A mechanism claim I made was wrong. I had written that the compile cache "gives all of its CPU saving back as filesystem wait," inferred from -3.4 ms CPU against +0.1 ms elapsed. On a quiet host elapsed (-3.6 ms) agrees with CPU (-3.4 ms), so that pairing was reading a noise-masked measurement as a mechanism. Removed. The metrics genuinely diverge only in the cold case, which CPU time cannot see at all.

Cold-cache figures reproduced closely: --help +11.3 ms [+10.3, +12.4], open CLI-side +16.4 ms [+15.9, +18.0].

Conclusion unchanged

Still no-ship, and the trade is now stated more precisely: a real but ~10x-too-small warm gain (3–6 ms against a 50 ms bar) bought at the price of an +11 to +16 ms cold-cache regression, on an API at Stability 1.1 at our Node floor. Every interval lies far above the -50 ms ship threshold.

Two traps are recorded in "If revisited" so the next person doesn't repeat them: host load silently flipping the verdict, and cold-cache runs needing the wipe before both arms (wiping only the compile-cache arm inflated the open cold penalty to +52.8 ms, more than double the symmetric figure).

Cleanup: the throwaway simulators were deleted and no other session's devices were touched.

[thymikee]

The revised scope and cold-page-cache limitation are now accurate, the retained samples reproduce all published medians/CIs, and the no-ship conclusion is robust. Three small correctness fixes remain before readiness:

1. invoke() catches every execFileSync error and records the elapsed time as a valid sample, including unexpected nonzero exits, spawn errors, and the 300s timeout. Make expected exit behavior route-explicit, fail on every other error/timeout, and retain/assert per-sample outcome so the real ad open samples are proven successful.
2. The seeded bootstrap PRNG uses multiplication beyond JS safe-integer precision; its stream repeats after 10,466 outputs although each CI consumes millions. Use an integer-safe seeded generator such as Math.imul/xorshift and recompute the retained intervals.
3. The decision bullet says end-to-end ad open had “no significant difference,” contradicting the table’s -6.4 ms CI [-10.0, -3.8] and faster verdict.

All 28 checks are green and the corrected methodology need not change the decision.