✨ feat(lang): add yield/generator support on the Tarn VM - #2322
Merged
harehare merged 52 commits intoSep 14, 2026
Conversation
harehare
added this pull request to stack #2323
September 10, 2026 12:52
harehare
force-pushed
the
add-yield-generator-support
branch
from
September 10, 2026 13:23
9b4a0d9 to
6640729
Compare
harehare
force-pushed
the
add-yield-generator-support
branch
from
September 10, 2026 13:59
164ad29 to
24339b8
Compare
Merging this PR will improve performance by 19.42%
|
| Benchmark | BASE |
HEAD |
Efficiency | |
|---|---|---|---|---|
| ⚡ | eval_compiled_foreach |
889.4 µs | 676.6 µs | +31.45% |
| ⚡ | eval_compiled_while |
9.5 ms | 7.6 ms | +25.49% |
| ⚡ | eval_compiled_array_fold |
436.6 µs | 393.2 µs | +11.03% |
| ⚡ | eval_compiled_dynamic_builtin_call |
1.7 ms | 1.5 ms | +11.02% |
| 🆕 | eval_compiled_generator_yield_loop |
N/A | 4.1 ms | N/A |
| 🆕 | eval_compiled_generator_yield_with_growing_array |
N/A | 78.9 ms | N/A |
Tip
Curious why performance improved? Comment @codspeedbot explain why performance improved on this PR, or directly use the CodSpeed MCP with your agent.
Comparing add-yield-generator-support (0b98a84) with remove-tarn-feature-tree-walker (dbde26f)
Footnotes
-
35 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩
harehare
force-pushed
the
add-yield-generator-support
branch
from
September 11, 2026 11:24
99ba38b to
131afe5
Compare
harehare
force-pushed
the
add-yield-generator-support
branch
from
September 11, 2026 13:49
742ca7b to
0b57a26
Compare
harehare
force-pushed
the
add-yield-generator-support
branch
2 times, most recently
from
September 12, 2026 10:26
045699c to
6ee3ad0
Compare
harehare
removed this pull request from stack #2323
September 13, 2026 05:20
harehare
added this pull request to stack #2336
September 13, 2026 05:21
harehare
force-pushed
the
add-yield-generator-support
branch
2 times, most recently
from
September 13, 2026 05:51
ebdbf68 to
044297f
Compare
harehare
force-pushed
the
add-yield-generator-support
branch
from
September 13, 2026 06:57
044297f to
9731190
Compare
Introduces `yield` inside `def`/`fn` bodies: a function whose body
directly contains `yield` becomes a generator, calling it returns a
coroutine instead of executing, and `next(stream)` drives it forward
one step at a time, returning `{ value, done }`.
- lexer/CST/AST: `yield` keyword, `yield: expr` / bare `yield`
- Tarn VM: coroutine state (Created/Suspended/Running/Completed/
Failed), `OpCode::Yield`/`Resume`, suspend/resume via a detached
frame stack + operand stack, reentrancy detection
- compiler: generator detection per def/fn, `yield outside function`
compile error, `next()` compiled to `OpCode::Resume`
- mq-hir: `yield` lowering and a `YieldOutsideFunction` diagnostic
- mq-check: type inference for `yield: value`
- mq-formatter: `yield` formatting
- mq-lint: `infinite_loop` recognizes `yield` as a loop exit
- editors (vscode, jetbrains, neovim, zed, playground): syntax
highlighting and snippets for `yield`
- docs: new Generators reference page
Groundwork for future lazy evaluation of foreach/selectors/pipelines;
`foreach`/selectors/pipelines do not yet consume streams lazily.
…sive calls A generator call that binds a missing default argument resumed through apply_pending without checking is_generator, so the coroutine's body ran inline and its first yield hit the top-level suspension assertion. A generator's fixed-arity self-recursive call compiled to the dedicated CallSelfExact0/1/2 opcodes, which skip generator detection entirely and which the bytecode verifier rejects for generator chunks, so recursive generators failed to compile.
Several types/functions introduced or touched by recent tarn/generator work (Ident, CoroutineHandle, Shared, array_mut, DictMap, Options) were already imported or reachable via `use super::*` in their files, so fully-qualifying them inline was redundant.
Fuse common bytecode operations to avoid operand-stack round trips.
…ines Two coroutines that capture each other through separate outer variables formed a strong reference cycle that neither coroutine's own self-reference check could detect, leaking both forever. Detect and break these pairwise mutual cycles by reusing the existing self-reference weakening machinery against the peer handle. Also fixes a related bug where a self-referencing coroutine nested inside a captured array/dict came back as `None` instead of its own coroutine on the next read, by making the cleared reference a resolvable `WeakCoroutine` marker instead of discarding it outright.
The direct-to-runtime-value JSON decoder accumulated object entries in a BTreeMap, sorting keys alphabetically instead of preserving document order — contradicting RuntimeValue::Dict's own insertion-ordered contract and breaking json_stringify/json_to_markdown_table round tripping. Decode straight into DictMap instead, keeping duplicate keys at their first position with the last value, same as before this regression was introduced.
The pairwise mutual-capture scan added in 118af24 ran on every yield regardless of whether the coroutine captured anything, adding a measured ~8-15% regression to generator suspend/resume benchmarks. A coroutine can only end up holding a peer that references it back through a variable shared at both creation sites, i.e. a capture, so skip the scan entirely when the current frames have none.
- Mutual-cycle "still reads its peer after breaking" test now checks both directions (a reads b, b reads a) via rstest instead of one. - JSON duplicate-key test now covers a duplicate in the middle, a key that only ever repeats, and multiple distinct duplicate keys.
mutually_capturing_peers only checked direct pairwise references, so a chain of 3+ mutually capturing coroutines (A -> B -> C -> A) leaked every coroutine, frame, and captured cell in the cycle once external handles were dropped. Replace the pairwise check with a graph traversal from each direct neighbor back to the suspending coroutine.
Inline module bodies are lowered into their enclosing scope with no distinct ScopeKind, so is_outside_function's scope walk reaches the enclosing function and misses the module boundary. Hir::errors() then disagreed with the compiler, which rejects the same yield. Add a symbol-parent-chain check that flags a Module ancestor reached before a Function ancestor.
harehare
force-pushed
the
add-yield-generator-support
branch
from
September 13, 2026 07:30
9731190 to
2f22777
Compare
next(arr)["value"] chained directly triggered a spurious overload resolution error in mq-check's type checker. Bind the step first, matching every other coroutine helper in builtin.mq.
…tin calls The CST lowers `f(x)["key"]` as `Call(f, [x, "key"])`, fusing the bracket key into the callee's argument list. User-defined calls already stripped trailing bracket keys before overload/arity checking; builtin calls did not, so `next(arr)["value"]` was type-checked as a 2-argument call to `next` (which only has 0- and 1-argument overloads) and spuriously failed with a "no matching overload" error. Add the same stripping for builtin calls, but only when the shorter overload's parameters are fully unconstrained (dynamic()/type variables): such a parameter list can only ever mismatch on argument count, so an excess of trailing String/Symbol children can only mean bracket-access keys. A concretely-typed overload (e.g. replace's `(string, string, string)`) is left alone so a real wrong-arity call like `replace(a, b, c, d)` still reports an overload error instead of silently treating the extra argument as a bracket key.
…alues Until now the only way to create a coroutine was writing a generator function with `yield`. Add `to_coroutine(value)`, the counterpart to `collect()`: it lazily yields the elements of an array or dictionary (entries, for a dict) as a coroutine, and returns an existing coroutine unchanged. This lets an eager array feed the coroutine-aware combinators (`map`, `take`, `skip`, ...) lazily without a hand-written generator.
Add --error-format <human|json> (default human, backward compatible). json renders the final uncaught error as a single-line JSON object via miette::JSONReportHandler instead of the colored graphical diagnostic, so CI and editor tooling can consume it without screen-scraping. Orthogonal to --stack-trace: since the stack trace is already folded into the error's Display text, --error-format=json --stack-trace works naturally, with the trace embedded in the JSON "message" field. main() and mq-dbg's main() now route the final error through Cli::report_error() instead of relying on miette::Result's implicit Debug-based rendering, and watch mode's per-iteration error printing uses the same path for consistent formatting.
…and add regression test
…ared argc The interpreter always pops one operand for Resume (next), or two when argc == 2 (send), regardless of the declared argc value. The verifier instead trusted argc directly as the required stack height, so a Resume(0) opcode would pass verification while the interpreter's unsafe unwrap_unchecked() pop still expected an operand, undermining the verifier's role as the safety basis for that unsafe code.
…oss-evaluation resume frame_or_coroutine took the token arena from execution.env.token_arena, which is the resuming evaluation's arena rather than the arena that matches the chunk pool actually executing. When a coroutine created by evaluation A is resumed by an unrelated evaluation B, and A's body creates and drives a child generator, the child was stamped with B's arena instead of A's, so its failures resolved Located token IDs against the wrong source. ExecutionContext now carries its own token_arena, decoupled from env. coroutine::resume overrides it to the resumed coroutine's own arena for the duration of driving its frames, so any child coroutine created mid-resume is stamped with the arena matching its own bytecode.
harehare
force-pushed
the
add-yield-generator-support
branch
from
September 13, 2026 13:44
c810c52 to
19108ac
Compare
Extends `each` to accept a coroutine, draining it via `next()` and invoking the callback on every yielded value, matching the existing coroutine support in `fold`/`any`/`all`.
Rust 1.98's clippy::all flags chunks_exact(2) in favor of as_chunks.
collect_output_coroutines only checked top-level values, so a coroutine inside an array or dict (e.g. `[g()]`) reached rendering unmaterialized and printed as the literal string "coroutine". Recursively walk arrays and dicts and rebuild only the containers that hold a coroutine.
Embedded coroutine helpers in the builtin module contain `yield` statements; the placement validation examined them alongside user symbols, which could misclassify builtin yields as errors. Skip symbols sourced from the embedded builtin module.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Introduces
yieldinsidedef/fnbodies: a function whose body directly containsyieldbecomes a generator, calling it returns a coroutine instead of executing, andnext(stream)drives it forward one step at a time, returning{ value, done }.yieldkeyword,yield: expr/ bareyieldOpCode::Yield/Resume, suspend/resume via a detached frame stack + operand stack, reentrancy detectionyield outside functioncompile error,next()compiled toOpCode::Resumeyieldlowering and aYieldOutsideFunctiondiagnosticyield: valueyieldformattinginfinite_looprecognizesyieldas a loop exityieldGroundwork for future lazy evaluation of foreach/selectors/pipelines;
foreach/selectors/pipelines do not yet consume streams lazily.Type of Change
Checklist
cargo fmtandcargo clippyand addressed any warningsjust test-alland all tests pass/docs, crateREADME.md) if neededAdditional Context