Repository navigation
fix(oracle-gaps): retire a batch's copies from Vitest's state - #61
Draft
KyleAMathews wants to merge 21 commits into
Draft
KyleAMathews wants to merge 21 commits into
KyleAMathews wants to merge 21 commits into
Conversation
A check runs each mutant in a new copy of its test file. Vitest's state kept every file and test task it ever ran (filesMap by path, idMap by task id), so a check held every test of every copy: about 377k test cases on a four-file suite, which exhausted a 2 GB heap. Once a batch's units are plain data, its copies leave the state: their stale idMap entries go first (shape-guarded), then clearFiles replaces each path's file entry. Co-authored-by: Isaac <no-reply@databricks.com>
|
Important Draft PR not reviewedDraft PRs are not automatically reviewed by default.
To automatically review draft PRs, update your CodeRabbit configuration: reviews:
auto_review:
drafts: true
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. |
|
commit: |
KyleAMathews
marked this pull request as ready for review
November 13, 2026 18:11
KyleAMathews
marked this pull request as draft
November 13, 2026 18:21
The gap CLI's e2e driver sets ORACLE_GAPS_TEST_RETENTION on each run and records one line per batch: the count of Vitest state entries (filesMap, pathsSet, idMap) and Vite module nodes that still name the batch's retired copies. The law: each count is 0 after retirement. This commit adds the hook and the law, not the fix, so CI records the RED on the parent code. Co-authored-by: Isaac <no-reply@databricks.com>
… file entry Vitest's clearFiles replaces a copy's file entry, and leaves its path in pathsSet, one placeholder task in idMap, and Vite's module node and transform for the copy. The retention law (bounded retention, after each batch) fails on 46dfd3e in CI: each count is nonzero. retireCopies now removes the copy's filesMap key, pathsSet entry, and idMap tasks. retireModules invalidates each copy's module through Vite's API and removes it from every environment's graph maps. Both guard the shapes they touch and throw if one changes, so a Vitest or Vite that changes them fails loudly rather than leaking. Co-authored-by: Isaac <no-reply@databricks.com>
… log When the cold run exits 3, its stack is in its error.log. The incremental violation named only the exit code, so a failure printed no stack. It now appends the error log the driver already reads for a run that exits 3. Co-authored-by: Isaac <no-reply@databricks.com>
…rror log ce46f4b appends cold.stderr to the incremental violation when the cold run exits 3, but the incremental observation's cold run had no stderr, so the message stayed empty. It now carries the run's stderr, with its error.log, when it exits 3, and only then: a completed run's stderr holds timings, which differ between two runs of one history. Co-authored-by: Isaac <no-reply@databricks.com>
KyleAMathews
marked this pull request as ready for review
November 13, 2026 22:33
…s not retired A replaced context stayed reachable through indexOf and contextSpecs, which were never cleared for it, so its Vitest and Vite state stayed too. replace now drops both entries. A run cut off by a hang is not retired: its worker may still send updates, and its context is replaced by the caller. Also: the etag map is cleaned by node identity (Vite keys it by the etag), and the retention observation is the count of batches left with a retired copy's state, which is deterministic, not the per-batch lines. Co-authored-by: Isaac <no-reply@databricks.com>
KyleAMathews
marked this pull request as draft
November 13, 2026 22:58
…s size before the batch The hook now snapshots every structure a copy can leave state in (Vitest's filesMap, pathsSet, idMap, errorsSet, leakSet, and each Vite environment's url, id, etag and file maps and transformed nodes) before a batch, and the law requires each to return to its size after the batch's retirement. The law was tautological before: it counted the same predicate the deletion used. The retention observation is observed from every run kind (cold, daemon, stability, partial), and a run that computed a unit must have written a batch line. A check that ran more than one batch reaches a declared state. Co-authored-by: Isaac <no-reply@databricks.com>
…a batch line is owed by mutant results The totals now include the excess of indexOf and contextSpecs over the live contexts, which must be zero before each batch: a context replaced between batches is caught there, not by a single batch's before and after. The missing-line expectation read evidence.computed, which counts single-file runs that write no batch line, so it failed on ce6c93d. A run owes a batch line when a mutant has owner files, since such a mutant's units ran in batches. Without that check, a retirement that never runs writes no line and passes the count law vacuously: the no-retire ablation exited 0. Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
The cold check's store kept every computed unit, with its rows, for the whole run: 207 store entries and 103,585 row entries in the ce6c93d heap snapshot. A cold run serves no lookup (evidence.reused is 0 on every klona run), so the store now keeps nothing. The daemon's disk store is unchanged. The retention totals include the store's size, which must match before and after each batch. Paired identity runs (pre-change and this commit, both cut 0) give the same 391 results, and the baseline matches both. Co-authored-by: Isaac <no-reply@databricks.com>
…hes are logged evidence.batches counts every batch run (cut or not). Each batch writes one retention line, so each run's line count must equal batches: a run with no line for a batch it ran (the no-retire mutant) fails the law. A cut batch writes its before line, so the stale totals of a replaced context are checked there. Co-authored-by: Isaac <no-reply@databricks.com>
…t every cached module Vite's graph caches non-copy modules (a source or harness file, on first import), which legitimately grow during a batch; the before-and-after equality was wrong for Vite keys. The law now requires the copies' own module nodes to be zero after each settled batch, and keeps the before-and-after equality for Vitest's state, the store, and the stale context totals. Co-authored-by: Isaac <no-reply@databricks.com>
…on-copy nodes to one per file Vite's totals are named vite.<environment>.<map>, so the law tells them from Vitest's state by an explicit prefix the hook generates. After each settled batch, the copies' own module nodes must be 0, and each non-copy file (query stripped) has at most one node in each environment's graph: a file cached once per batch under a fresh id fails the law. Co-authored-by: Isaac <no-reply@databricks.com>
…o the completion law; the multi-batch state is declared A run with no report owes no batch line here: the completion law judges it, keyed on the model's prediction (a report unless a refusal is predicted), and fails it with its exit status and stderr. The multi-batch reach state is declared in the property's measured list. Co-authored-by: Isaac <no-reply@databricks.com>
…ched Co-authored-by: Isaac <no-reply@databricks.com>
… per package, counted as a delta A daemon keeps the environment of the call that started it, so a later daemon check wrote its retention lines to the hook file of the call that started the daemon, and its own (new) hook file was empty: the exact batch-line count failed on a correct run (retentionLines:daemon on the hang history). Counting side: every call for a package writes to one file, and a run's lines are the ones after its start. The production batch-line contract is unchanged. Co-authored-by: Isaac <no-reply@databricks.com>
…tory; a run keeps its own lines Replaces the delta: a daemon serves many calls and keeps the first caller's environment, so a line is attributed to the check that wrote it (its out directory, the test hook's only addition), and the run keeps the lines that name its own out directory. Two daemon checks at once against one hook file cannot mix. No production outcome changes. Co-authored-by: Isaac <no-reply@databricks.com>
…able names each run's batches and lines Diagnostic for the daemon line mismatch (the test hook only; no production outcome changes). Co-authored-by: Isaac <no-reply@databricks.com>
… retention lines are its own The table showed the cause: a daemon serves several checks of one test, and two of them shared an out directory, so a run kept lines written by an earlier check (batches 1, lines 4; a run with 0 batches kept 1). Production wrote one line per batch in every run. The counting side is wrong: each call gets its own directory, so the run filter sees only its lines. The debug table is removed (it was for this diagnosis). No production change. Co-authored-by: Isaac <no-reply@databricks.com>
…tories runCli gives each call Each call's out directory is numbered (out-<n>, out-daemon-<n>, ...), so the outputs of a run are its own. The law's list of own outputs names the fixed ones; the numbered ones match the same pattern the driver creates (OWN_NUMBERED). Without it, a run's out-25 writes read as strays. Co-authored-by: Isaac <no-reply@databricks.com>
…as given Repair, not widening (P7): the law listed fixed output names (out, out-daemon, out-stop, out-only), and numbered names were a pattern. Now each run's allowed output is the one path the model predicts (its --out, from cliArgs): runCli passes its numbered out, the stop law passes out-stop. A CLI that writes under any other directory, such as out-<n>-next, is a stray. The lists are removed. Co-authored-by: Isaac <no-reply@databricks.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The gap CLI kept Vitest and Vite state for every mutant copy after its batch had finished. Each copy is a new file, and Vitest kept its path in
filesMapandpathsSet, its file and test tasks inidMap, and Vite kept a module node and transform per copy. On the klona four-file suite (295 mutants, 2 contexts), a 2 GB heap runs out before the check ends. At the default heap, the child peaked at 3659 MB before this change.This PR removes a batch's copies from Vitest's and Vite's state once the batch ends. Each copy's
filesMapkey,pathsSetentry, andidMaptasks are removed, and its module nodes are invalidated through Vite's API and removed from each environment's graph maps. Every map touched is guarded, and the check throws if Vitest's or Vite's shape changes. Vitest'sclearFilesis not used, because it replaces the file entry but keeps the path and a placeholder task.Decisions
43483575797on the oracle commit46dfd3e, and GREEN is CI run43512235438on2de2b9e. The C ruling below (gap CLI, P9) covers the CLI-level probe as evidence for the heap effects, which the oracle cannot observe cheaply.results.jsonis the stable record (C ruling).retireCopiesthrows unlessfilesMap,idMap, andpathsSetare a Map, a Map, and a Set.retireModulesthrows unless each environment's graph has its id, url, etag, and file maps and aninvalidateModulemethod. Vitest 4.1.12 and 5.1.0 both have these shapes; 5.1.0 was checked from its source, not run.Evidence
filesMap,pathsSet, oridMap, and no module nodes in Vite's graphs. The e2e driver records one count line per batch (ORACLE_GAPS_TEST_RETENTION), and the check reports a violation for any nonzero count.43483575797, job132086037235(oracle commit46dfd3e, parent code). The job log readsoracle-gaps.report violated at checkpoint "retention" (law: bounded retention)with counts like{"filesMap":2,"pathsSet":2,"idMap":2,"moduleNodes":2}.43512235438on2de2b9e: success, and all 30 PR checks pass.results.jsonequals the baseline's after dropping timing fields and the output path. Both report 43 gaps (provisional).--expose-gc): first 101 MB, middle 831 MB, last 113 MB, maximum 1481 MB. Retention counts: 365 batches, all zero. The 3924eb6 reviewer probe climbed from 409 MB to 1801 MB with 365 batches and counts reaching about 525..worktrees/m-clearfiles) and no-module-step (.worktrees/m-nomodule), each on the same json.js sample 2 as a control run (the fix). The control has 0 of 290 batch rows nonzero. The clearFiles-only mutant has 290 of 290 nonzero (filesMap 2, pathsSet 2, idMap 2). The no-module-step mutant has 290 of 290 nonzero (moduleNodes 1 to 2). Both mutants fail the law. The clearFiles-only mutant's first run exited 3 from a ReferenceError in the mutant itself (itsstatereference), so that run is discarded and the corrected run is cited. Logs: /tmp/oom/mut-clearfiles2., /tmp/oom/mut-nomodule., /tmp/oom/control-fix.*.Limits
replace), which drops its Vitest instance and with it its state, so retirement after a cut-off batch leaves nothing behind. This has not been observed.--expose-gc, which adds a globalgc. A run with that flag gave 49 gaps and 1074 nondeterministic tests, against 43 gaps and none without it. The cause is provisionally the flag, pending the--expose-gc-only run. The heap numbers above are therefore provisional, and the outcome comparison is taken from runs without the flag.gc()changes klona's properties-test outcomes (49 gaps, 1074 nondeterministic tests with the probe, 43 gaps without it). Heap measurements and outcome comparisons are therefore taken in separate runs; outcomes are compared only without the probe.2de2b9e.This pull request and its description were written by Isaac.