wasm: make JS WebAssembly.instantiate operand stack/heap tunable

[e-fu]

What

Makes the WASM operand stack and auxiliary heap for guests started via the
JavaScript WebAssembly.instantiate path tunable, via two new runtime-level options
:wasm_stack_size / :wasm_heap_size (both default 65536 — behavior is unchanged
unless a consumer opts in).

Why

The JS WebAssembly.instantiate path hardcoded a 64 KB operand stack (and 64 KB aux
heap) when starting an instance, while the native QuickBEAM.WASM NIF path already
accepts caller-supplied :stack_size / :heap_size. Guests with deep initialization
(e.g. Go GOOS=js) overflow the 64 KB operand stack at boot — the native path can boot
them, the JS path can't. This closes that parity gap.

The new values are threaded from the runtime/pool opts down to the instantiate site,
mirroring exactly how the existing :max_stack_size (the JS call stack, a separate
8 MB limit) is already plumbed. The standard instantiate(bytes, importObject) JS
signature stays spec-faithful — no extra JS argument; the limit comes purely from
per-runtime config.

How

• New wasm_stack_size / wasm_heap_size fields (default 65_536) on the two config
  carriers RuntimeData (types.zig) and PoolData (context_types.zig), and on the
  per-context ContextState (wasm_js.zig).
• Threaded through the single install chokepoint
  (worker.zig → wasm_js.install → ensure_context_state), the pool's
  PoolData → RuntimeData copy (context_worker.zig), the opts parsing
  (quickbeam.zig start_runtime + pool_start), and the Elixir Keyword.take
  allow-lists (runtime.ex, context_pool.ex).
• The instantiate site in wasm_js.zig now reads state.wasm_stack_size /
  state.wasm_heap_size instead of the 65_536, 65_536 literals.
• Opt parsing bounds-checks the u64 → u32 cast (std.math.cast): an out-of-range
  value returns a controlled error instead of trapping (Debug) / wrapping (ReleaseFast).
  Note: the pre-existing max_convert_depth / max_convert_nodes casts in the same
  function share this pattern and were left unchanged to keep this diff focused.
• Docs updated on QuickBEAM, QuickBEAM.Runtime, QuickBEAM.ContextPool, and
  QuickBEAM.WASM (clarifying the NIF path keeps its per-call :stack_size/:heap_size).

Tests

test/wasm_test.exs adds a small recursive rec(n) fixture (WAMR keeps call frames on
the stack_size buffer, so deep recursion is the canonical operand-stack-overflow repro)
and three regression assertions:

• default 64 KB → rec(10000) raises a …stack… error,
• QuickBEAM.start(wasm_stack_size: 8 MB) → rec(10000) returns {:ok, 0},
• the same through a QuickBEAM.ContextPool (exercises the pool PoolData → RuntimeData
  threading path, which the standalone test doesn't cover).

Verification

Built and tested in the CI-pinned container (OTP 27.0 / Elixir 1.18.3 / Zig 0.15.2,
MIX_ENV=test, Debug): 56 tests, 0 failures, mix compile clean. The diff also
compiles clean against this branch's base.

[Copilot]

Pull request overview

This PR makes the JS WebAssembly.instantiate guest WASM operand stack and aux heap sizes configurable at the runtime/pool level (:wasm_stack_size / :wasm_heap_size, defaulting to 65_536), bringing it in line with the native QuickBEAM.WASM path which already supports caller-supplied sizing.

Changes:

• Thread new runtime/pool options (:wasm_stack_size, :wasm_heap_size) through Elixir → Zig config carriers → JS WebAssembly.instantiate implementation.
• Replace hardcoded 65_536 stack/heap literals in the JS instantiate path with per-context state values.
• Add regression tests covering stack overflow at the default size and success with a raised :wasm_stack_size, including the ContextPool path.

Reviewed changes

Copilot reviewed 11 out of 11 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
test/wasm_test.exs	Adds JS WebAssembly.instantiate regression tests for default-stack overflow and raised :wasm_stack_size (runtime + pool).
lib/quickbeam/worker.zig	Passes runtime-configured WASM stack/heap sizes into the WASM JS install hook.
lib/quickbeam/wasm.ex	Clarifies docs: native WASM options vs JS instantiate runtime-level sizing.
lib/quickbeam/wasm_js.zig	Stores per-context WASM stack/heap sizes and uses them when starting managed instances from JS.
lib/quickbeam/types.zig	Extends RuntimeData with wasm_stack_size / wasm_heap_size defaults.
lib/quickbeam/runtime.ex	Allows :wasm_stack_size / :wasm_heap_size through to the runtime NIF opts.
lib/quickbeam/quickbeam.zig	Parses and bounds-checks wasm_stack_size / wasm_heap_size from Elixir opts for runtime + pool.
lib/quickbeam/context_worker.zig	Copies pool WASM sizing into each created context’s RuntimeData.
lib/quickbeam/context_types.zig	Extends PoolData with wasm_stack_size / wasm_heap_size defaults.
lib/quickbeam/context_pool.ex	Documents and forwards the new pool options to the NIF.
lib/quickbeam.ex	Documents new runtime options and corrects/aligns :max_stack_size default wording.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[dannote]

Thanks for the PR! I merged this locally onto current master and ran:

QUICKBEAM_BUILD=1 mix test test/wasm_test.exs --max-cases 1

The new ContextPool test crashes in WAMR:

panic: member access within misaligned address ... for type 'WASMBranchBlock'
...
wasm_interp_call_func_bytecode
wamr_bridge_call_typed
lib/quickbeam/wasm_js.zig:400: wasm_call_impl
lib/quickbeam/context_worker.zig:297: handle_ctx_eval

So I think this needs changes before merge. The option threading looks reasonable, but the new pooled deep-recursion case currently exposes a native panic path rather than a safe JS/WASM error.

Could you rebase onto current master and adjust the implementation or test so this fails safely?

[e-fu]

Thanks for the review. Addressed in the latest push (force-pushed — branch was rebased onto current master):

Stack test — fixed the BEAM-abort. The deep-recursion proof wasn't testing a stack overflow; it was tripping a pre-existing WAMR-under-UBSan defect. Under the Debug build (MIX_ENV=test → Zigler :env → UBSan on the vendored WAMR C), executing a guest whose call frame lands a WASMBranchBlock off WAMR's 4-byte-aligned csp_bottom raises a UBSan alignment trap that aborts the VM instead of a catchable error — independent of this feature, and frame-cell-count dependent (which is why the existing add tests pass by luck). The tests now prove :wasm_stack_size via a contrast on the already-safe add guest: a tiny stack overflows at frame allocation ("wasm operand stack overflow", before any branch block) and a generous one runs cleanly. Verified in Docker under the exact CI config (Debug + UBSan, no escape hatch): full wasm suite, 0 failures.

:wasm_heap_size — plumbed, but intentionally no behavioral test. I went to add one and found it isn't observable from the JS WebAssembly.instantiate path: the value sizes WAMR's host app heap, which only backs wasm_runtime_module_malloc/a guest-exported malloc (never reached by a plain instantiated module called from JS); it can't fail instantiation (clamped to APP_HEAP_SIZE_MAX = 1 GiB, guards only near UINT32_MAX); and it isn't reflected in memory.buffer.byteLength (app-visible pages, excluding the heap — the existing "exposes exported memory" test confirms a default 64 KiB heap still gives byteLength 65536). Any test would pass regardless of whether the value is honored, so I documented the rationale in a comment beside the stack tests rather than ship a false-green. The option is still threaded end-to-end and covered at the plumbing level by the same start/pool paths the stack tests exercise.

[e-fu]

Motivation — what this unblocks

For context on why the JS WebAssembly.instantiate path needs a tunable operand stack (it reads as abstract config otherwise):

This came out of a spike to run the lighter DEX's own signer in-BEAM via QuickBEAM (no Node). lighter signs with a zk_schnorr scheme that isn't EVM ECDSA — you can't reproduce it with ex_keccak/ex_secp256k1. The realistic path is to run lighter's actual signer, which ships as a Go GOOS=js/GOARCH=wasm module, on QuickBEAM's JS/WASM path.

The spike confirmed that's viable: lighter's Go js/wasm signer boots and runs CreateAuthToken end-to-end under QuickBEAM with the unmodified go.importObject (apis: :browser supplies crypto/TextDecoder/etc — no host-import patching), and it's deterministic under pinned RNG.

The one blocker was this: the JS instantiate path hardcoded a 64 KB WASM operand stack, and Go's deep init overflows it ("wasm operand stack overflow"). The native NIF path (QuickBEAM.WASM) already accepted stack_size/heap_size — so this is a parity gap between the two paths, not a new capability. This PR closes it by threading :wasm_stack_size / :wasm_heap_size through the Runtime/ContextPool config so a consumer (the lighter signer) can opt into a larger stack at runtime start.

Design choices that follow from that:

• Config option, not a 4th arg to the JS instantiate call — the size is a property of the runtime/context the guest runs in, matching how the NIF path already models it.
• Default stays 64 KB (unchanged behavior) — heavy guests opt in; nothing else pays for it.
• The stack knob is the one with teeth (Go deep-init overflow); heap is included for parity with the NIF path — see the test comment for why it has no behaviorally-observable effect on the JS path.