Fix memory leak in experimental mutator engine's seen cache

[Copilot]

The experimental mutator engine (packages/compiler/src/experimental/mutators.ts) holds a module-level, never-cleared, strongly-referenced seen cache. Every mutateSubgraph/mutateSubgraphWithNamespace call inserts Type values into it, pinning the entire type-graph of every mutated program in memory for the process lifetime — heap grows unbounded for hosts that drive compile() in a loop (e.g. emitter test suites, versioning mutation via TCGC).

Why the cache can't simply be scoped per-engine

The obvious fix — moving the cache into createMutatorEngine so it dies with the engine — breaks recursive type graphs. Mutators such as @typespec/http's merge-patch transform call mutateSubgraph re-entrantly from inside their own mutate function, and each call spins up a fresh engine. The seen cache must be shared across those nested engines so that a self-referential type (e.g. model Resource { related?: Record<Resource> }) is cloned once instead of recursing forever. A per-engine cache regresses this into RangeError: Maximum call stack size exceeded (observed in the @typespec/http merge-patch and @typespec/samples visibility suites).

Changes

• Clear the cache when the outermost mutation completes. The seen cache stays module-scoped (so it is shared across nested re-entrant mutations and continues to break cycles), but a mutationDepth counter tracks re-entrancy and clears the cache once the outermost mutateSubgraph/mutateSubgraphWithNamespace call returns. The cache — and every Type it references — therefore becomes eligible for GC once the mutation pass completes, instead of living for the process lifetime. Mutations run synchronously, so the depth counter reliably identifies the outermost call, and clearing happens even if a mutator throws.
• CustomKeyMap.clear() — small helper used to reset the cache.
• Regression tests (packages/compiler/test/experimental/mutator.test.ts):
  • does not share the mutation cache across top-level calls — two independent top-level mutations of the same type/mutator produce distinct clones in distinct realms.
  • breaks cycles across re-entrant mutateSubgraph calls — a self-referential model run through a mutator that re-enters mutateSubgraph (mirroring merge-patch) must not overflow the stack. Verified to fail with RangeError: Maximum call stack size exceeded under a per-engine cache, confirming it guards the regression.

let mutationDepth = 0;

function runRootMutation<T>(fn: () => T): T {
  mutationDepth++;
  try {
    return fn();
  } finally {
    mutationDepth--;
    if (mutationDepth === 0) {
      // Outermost mutation done: release the cached type graph for GC while
      // still having shared it across any nested re-entrant mutations.
      seen.clear();
    }
  }
}

This fixes the leak without reintroducing unbounded recursion, and keeps separate top-level mutations independent (each gets a clean cache and its own realm).

[pkg-pr-new]

Open in StackBlitz

npm i https://pkg.pr.new/@typespec/compiler@11121

commit: 7e1949f

[azure-sdk-automation]

You can try these changes here

🛝 Playground	🌐 Website	🛝 VSCode Extension

[github-actions]

All changed packages have been documented.

• ✅ @typespec/compiler

Show changes

@typespec/compiler - fix ✏️

Fix memory leak in the experimental mutator engine where a module-level seen cache pinned the type graph of every mutated program in memory for the lifetime of the process. The cache is now cleared once the outermost mutation completes, so it can be garbage collected while still being shared across nested mutations (which is required for recursive type graphs to terminate).

[timotheeguerin]

Another option thinking if it was working before, would using the program as a key in a weakmap to cache all those 3 customkeymap and retrieve them work? This is the pattern we have been doing when needing this kind of "global" state

[xirzec]

I think the argument here is around should this cache be within a call to mutateSubgraph (the behavior in this PR) or within the lifetime of the Program? While we can certainly cache things for the duration of the Program, this is going to incur a much larger cost across a large spec that mutates many models.

Furthermore, because the cache is shared across calls a second call to mutateSubgraph gets a cache hit from the first call and you get back a type from the realm created by the first call that doesn't exist in the realm created by the second.

This is actually a flaw in the current design, if I'm understanding the intent of how mutateSubgraph is supposed to work, given its assertion from its ref doc that it @returns an object containing the mutated type and a nullable Realm in which the mutated type resides. See the test for does not share the mutation cache across top-level calls which would fail if we moved to a WeakMap on program.

[timotheeguerin]

that test is added exactly in this PR though no? this define this spec right here, isn't that incorrect with the original intent?

[timotheeguerin]

Actually didn't realized the test was added there