Skip to content

feat(cli,core,producer): ramp the parallel-DE router through the canary at 5% - #2840

Merged
vanceingalls merged 4 commits into
mainfrom
07-27-feat_producer_enable_parallel-de_router_by_default
Aug 8, 2026
Merged

feat(cli,core,producer): ramp the parallel-DE router through the canary at 5%#2840
vanceingalls merged 4 commits into
mainfrom
07-27-feat_producer_enable_parallel-de_router_by_default

Conversation

@vanceingalls

@vanceingalls vanceingalls commented Jul 28, 2026

Copy link
Copy Markdown
Collaborator

What

The parallel drawElement router becomes the default for eligible renders — ramped through the canary at 5%, not flipped on for everyone. Three things ship together:

  1. The kill switch actually works. isDeParallelRouterEnabled() parses false/0/off/no (case- and space-insensitive); unset or empty means default. The old !== "false" failed open on every spelling but the literal one.
  2. The circuit breaker survives a default-ON flag. It now writes an explicit "false" instead of delete-ing the var — under default-ON polarity, deleting means ON, so the breaker was a no-op on exactly the hosts it exists to protect.
  3. The rollout is a number someone chose. Enrolment gates the new behaviour; percentage: 0 in the registry is a full fleet-wide revert with no release.

Also removes the 25-render cap and the telemetry preconditions (see below).

Why the ramp

Default-ON without one is a ~17× exposure jump — from ~6% of eligible renders to all of them — landing on profiles today's trial population never covered: ≤4 CPUs (8.18% of eligible renders) and Docker (3.99%). ("Trial" is not a user opt-in — it arms automatically on the CLI render path. The enableDeParallelRouterTrial flag is call-site opt-in, excluding programmatic renderLocal consumers because it mutates process.env. So ~11% of installs route today without anyone choosing it.) 0.7.60–0.7.64 is the precedent: every unclamped render reverted for five consecutive releases and nobody noticed.

Worth being precise about what the canary buys, because it is easy to state wrongly: ~10.85% of rendering installs already route today, automatically. That number is emergent — the product of eligibility rules, the render cap and the breaker latch — so it drifts with fleet composition and cannot be switched off without shipping. The canary does not make exposure smaller. It makes it chosen and revertible.

Current soak (14d, ≥0.7.66)

installs engaged 1,466
never reverted 1,311
reverted exactly once 154
reverted more than once 1
per-render revert rate 2.79%
installs hitting ≥1 revert 10.57%

154 of 155 reverting installs stop at exactly one — the breaker working. Held steady as the population tripled from 434 installs, so that is a property of the mechanism, not a sample artifact.

What reverts actually are (routed renders, same window):

outcome renders installs
routed, clean 4,848 1,183
reverted — psnr 128 126
reverted — capture_error 18 18
reverted — blank 3 3

86% of reverts are the self-verify catching real visual damage. That is a per-render mechanism and is separate from the breaker — worth keeping straight, because it means removing the breaker would not risk corrupt output, only repeat cost.

What is removed, and why that is safe

  • The 25-render cap. Sampling logic for an experiment. Under a shipped default it would switch the feature off behind the user's back after 25 good renders.
  • The telemetry preconditions. The trial refused to arm without recordable telemetry — fair for an experiment, but as a default it means opting out of analytics silently costs you a slower renderer.

The breaker stays. Not caution — measurement. We cannot yet tell whether removing it is safe, because it censors the data: "154 of 155 stop at one revert" is the latch, not a natural repeat rate. Now that the canary gate exists, disabling the breaker for a slice is the honest experiment, and that is a follow-up.

Ramp discipline

Hold at 5% until PRINFRA-372 resolves: --workers auto crashes every worker ("detached Frame" / "Target closed") on macOS arm64 while --workers 1 is clean — and the router forces 3 workers. PRINFRA-339 (indefinite stalls) and PRINFRA-354 (silent hang) are also open. None of those emit render_complete, so neither the self-verify nor the breaker catches them — that is precisely the blast radius a ramp bounds.

At each step (5 → 25 → 100) split revert rate by cpu_count and is_docker. Baseline: routed renders fail at 2–3.4%, stable across versions.

The bucketer was validated before ramping — calibration at n=13,547: 9.62%/49.76% against 10%/50% targets, overrides and CI both attributable, sustained cohort flips at 0.10%, an order of magnitude under this feature's own revert rate.

Also replaces a test that could never pass

canary.test.ts pinned de-parallel-router to 0% with the note "ramp only alongside the per-install circuit breaker". Pinning 0 blocks the ramp permanently and never checks the wiring it names. It now asserts the wiring directly — a non-zero percentage is allowed only while the CLI render path gates on this canary and still consults the breaker.

Verification

core 1692, cli 2487 passing. Lint clean. Fault-injection: restoring the delete fails 3 tests, restoring the naive parse fails 1, removing the canary gate fails 2 CLI tests and the core wiring assertion.

Rebased onto main (was 308 behind; #2838 has since merged). #3095 closed as superseded — its useful piece is folded in here.

One pre-existing failure on main, not from this branch: audioPadTrim.realmedia times out at 5s, reproduced on clean origin/main.

Tracked: PRINFRA-384.

🤖 Generated with Claude Code

@vanceingalls
vanceingalls force-pushed the 07-27-feat_producer_enable_parallel-de_router_by_default branch from e9b6662 to 6a50c9f Compare July 28, 2026 08:11
@vanceingalls vanceingalls changed the title feat(producer): enable the parallel-DE router by default feat(producer): enable the parallel-DE router by default, behind a real circuit breaker Jul 28, 2026

@miga-heygen miga-heygen left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SSOT -- circuit breaker state machine audit

Traced every transition through the module-level state (deParallelRouterUserManaged, deParallelRouterUserManagedResolved, deParallelRouterBreakerTrippedThisProcess) and the on-disk deParallelRouterTrialFired flag. All five advertised transitions hold:

Initial state Event Result Verified
unset (fresh install) render succeeds stays ON (env untouched, producer default) yes
unset (fresh install) render falls back breaker writes explicit "false", persists deParallelRouterTrialFired: true yes
explicit "true" by user render falls back applyDeParallelRouterBreaker() is a no-op (user-managed latch), env stays "true", router stays ON yes
explicit "false" by user any render userValueEnablesDeParallelRouter() returns false, router OFF, breaker never consulted yes
breaker wrote "false" (prior process) user overrides with "true" user-managed latch fires, userValueEnablesDeParallelRouter() returns true, router ON yes

Cross-process persistence: the in-process latch (deParallelRouterBreakerTrippedThisProcess) guards within a process; the on-disk flag (deParallelRouterTrialFired via readConfigFresh) guards across processes. Both paths converge on writing process.env.HF_DE_PARALLEL_ROUTER = "false". The persistDeParallelRouterTrialFired retry loop (3 attempts, verify-after-write) handles concurrent-writer clobbers. Correct.

Kill switch parsing

isDeParallelRouterEnabled (producer, exported, takes env param) and userValueEnablesDeParallelRouter (CLI, private, reads process.env) are deliberately duplicated with identical logic. Both:

trim().toLowerCase()
undefined or "" -> true (ON)
"false" | "0" | "off" | "no" -> false (OFF)
everything else -> true (ON)

Tested spellings in isDeParallelRouterEnabled tests: "false", "FALSE", "False", "0", "off", "OFF", "no", "No", " false " (padded), "", " " (whitespace-only), unset. All produce the correct result. The naive === "true" / !== "false" pitfall that motivated this work is properly eliminated.

SSOT note (reviewed, accepted): The duplication between producer and CLI is documented with a rationale (producer is lazily loaded, CLI needs this on the startup path). The comment names the producer copy as source of truth and says "keep the two in sync." Acceptable trade-off -- flagging it here so a future editor knows to update both.

Side-effect invariant

The breaker writes process.env.HF_DE_PARALLEL_ROUTER = "false" exactly once on the first fallback (in maybeConsumeDeParallelRouterTrial). Subsequent renders in the same process short-circuit at step 3 of applyDeParallelRouterCircuitBreaker (the in-process latch) and call applyDeParallelRouterBreaker which is an idempotent re-write of "false". The disk write (writeConfig) also fires only on the fallback render (subsequent renders return routerActive = false, so maybeConsumeDeParallelRouterTrial exits at line 1). Correct.

Race conditions

Single-process: JS event loop -- no data race possible. Module-level flags are shared across all renderLocal calls in one process, but the manageDeParallelRouterBreaker opt-in polarity ensures only sequential call sites (single render, batch concurrency 1) use the mechanism. Batch concurrency >= 2 leaves the flag unset, so those renders never touch the breaker state. Correct.

Cross-process: two processes both reading deParallelRouterTrialFired: false, both tripping, both writing true -- idempotent, no corruption. The render counter can lose an increment under a race (documented, benign -- it's telemetry, not a decision input anymore). persistDeParallelRouterTrialFired verify-and-retry handles the clobber case for the safety-critical fired flag. Correct.

25-render cap removal

DE_PARALLEL_ROUTER_TRIAL_MAX_RENDERS = 25 is deleted. The only trip condition is now outcome === "reverted". The test that verified the old cap is rewritten to run 30 successful renders and assert the breaker never trips. The comment block replacing the constant explains why: under a shipped default, a render-count cap would silently disable the feature after 25 good renders. Correct and clean.

The deParallelRouterTrialRenderCount field is still incremented on every consumed render -- this is telemetry bookkeeping, not a decision input. Fine to keep.

Standards

No bare as T casts in the diff (only as const for literal types in test fixtures). No non-null assertions. Function signatures are clean. Comments are thorough and accurate -- every "review finding" annotation traces back to a real design constraint.

CI

Preflight (lint + format): pass
Preview parity: pass
Preview regression: pass
Player-perf: pass
Regression shards (1-8): in progress at time of review


Findings

1. Misleading user-facing message when explicit opt-in overrides breaker (nit)

When a user explicitly sets HF_DE_PARALLEL_ROUTER=true and a render reverts, reportDeParallelRouterBreakerTrip prints:

A frame failed verification, so parallel drawElement capture fell back to the screenshot path and is now off for this install.

But the user's explicit "true" wins over the breaker (applyDeParallelRouterBreaker is a no-op when deParallelRouterUserManaged), so the router is NOT off. The message is factually incorrect for this path. Low severity (the user explicitly opted in and probably knows what they're doing), but worth gating the message on !deParallelRouterUserManaged so it only prints when the breaker actually took effect.

Additionally, reportDeParallelRouterBreakerTrip fires on EVERY subsequent revert for this user (since routerActive stays true via the user's explicit value), producing the same misleading message each time. The first-revert-only semantics that apply to the non-user-managed path don't apply here.

2. Missing test: user explicit "true" survives a breaker trip (nit)

The state transition "explicit true -> render reverts -> breaker can't override -> env stays true" is logically correct (I traced it above) but has no dedicated test. The existing tests cover user-set "false" not being overridden, breaker writing "false" when unset, and the kill switch parsing. A test like this would lock down the "explicit user choice wins in both directions" invariant the comments describe:

it("does not override an explicit user opt-in even after a fallback", async () => {
  savedEnv.set("HF_DE_PARALLEL_ROUTER", process.env.HF_DE_PARALLEL_ROUTER);
  process.env.HF_DE_PARALLEL_ROUTER = "true";
  configState.disk = { /* fresh install */ };
  producerState.executeImpl = async (job) => {
    job.perfSummary = { drawElement: { parallelRouter: "reverted" } };
  };
  await renderLocal("/tmp/project", "/tmp/out.mp4", baseOptions);
  expect(process.env.HF_DE_PARALLEL_ROUTER).toBe("true");
});

3. events.test.ts -- gpu: false addition looks like a cross-PR fix (informational)

Two test calls to trackRenderComplete gain gpu: false. This is likely fixing a type error introduced by the base PR (#2838) that added a gpu property to the event payload. Not a problem, just noting it's a stacked-PR seam rather than a circuit-breaker change.


Verdict

Approve. The circuit breaker's state machine is correct and complete. The kill switch parsing closes the exact vulnerability the PR description identifies (naive !== "false" failing open for 0/off/no/FALSE). The 25-render cap removal is well-motivated and cleanly executed. The two nits above are low-severity edge cases -- finding 1 (misleading message for explicit opt-in users) is the only one I'd consider addressing before merge, and even that is a rare user path. The code is well-documented, the test coverage is strong, and the design trade-offs (deliberate duplication, latched user-managed detection) are clearly explained.

CI regression shards are still running. Confirm they pass before merge.

-- Miga

@miguel-heygen miguel-heygen left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Additive review at exact head 8d30cdb9af — Miga covered the state machine, kill-switch spellings, and user-message/test nits. One gap remains in the boundary between the producer parser and CLI breaker.

Strengths

  • packages/producer/src/services/renderOrchestrator.ts:1332-1339 centralizes default-on parsing and correctly handles the conventional OFF spellings.
  • packages/cli/src/commands/render.ts:1185-1207 preserves the persisted breaker across processes without consulting telemetry state.

blocker — set-but-empty enables the router but disables its circuit breaker

packages/cli/src/commands/render.ts:1172-1183 classifies any defined HF_DE_PARALLEL_ROUTER as user-managed (!== undefined). That disagrees with both parsers at render.ts:1148-1152 and renderOrchestrator.ts:1332-1339, where "" / whitespace explicitly mean “unset/default ON.”

Concrete failure: launch with HF_DE_PARALLEL_ROUTER= (or spaces), hit a verified fallback. The producer routes because empty means ON; applyDeParallelRouterCircuitBreaker returns true but latches deParallelRouterUserManaged = true; then applyDeParallelRouterBreaker() no-ops at render.ts:1154-1157. The install keeps retrying the failing router on later renders instead of receiving the advertised first-fallback protection. This is the central safety invariant of the PR.

Normalize before deciding ownership: empty/whitespace should remain breaker-managed, while a non-empty explicit value stays user-managed. Please pin it with a CLI-level regression that starts from HF_DE_PARALLEL_ROUTER="" (and whitespace), injects parallelRouter: "reverted", and asserts the env becomes "false" and the fired flag persists.

CI is still running eight regression shards; this finding is independent of that pending gate.

Verdict: REQUEST CHANGES
Reasoning: Default-on parsing is correct, but the CLI ownership latch exempts the parser's own empty/default state from the per-install breaker, leaving a documented default path unprotected after fallback.

— Magi

@vanceingalls
vanceingalls force-pushed the 07-27-feat_producer_enable_parallel-de_router_by_default branch from 8d30cdb to acdae81 Compare July 28, 2026 08:31

@james-russo-rames-d-jusso james-russo-rames-d-jusso left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed at 8d30cdb9.

Kill-switch parser at renderOrchestrator.ts:1333-1339 is clean — .trim().toLowerCase() up-front, then treats false | 0 | off | no as OFF, everything else including undefined/empty/whitespace as ON. Default-inversion trip parity with the breaker's process.env.HF_DE_PARALLEL_ROUTER = "false" write at render.ts:1156 — producer treats that as OFF via the same reader, so the trip signal reaches the router.

Persistence via ~/.hyperframes/config.json:deParallelRouterTrialFired written by persistDeParallelRouterTrialFired() at render.ts:1242-1251 with a fresh re-read + re-assert up to 3 times to survive concurrent clobber — solid pattern. In-memory latch (deParallelRouterBreakerTrippedThisProcess at render.ts:1101) set BEFORE the disk write so the decision holds if fs write fails. Test coverage is comprehensive across kill-switch spellings (renderOrchestrator.test.ts:1831-1850), 15 CLI-side scenarios (render.test.ts:700-1057) including 30-healthy-renders-no-trip, unwritable-config-in-process-latch, and concurrent-write-clobber-re-assertion.

Four concerns — none blocking, all worth thinking about before default-ON ships.

Blockers

None.

Concerns

1. "Trip only on real fallbacks" claim overstates what the predicate actually catches. The PR title says the breaker trips "only on real fallbacks" and the user-facing message at render.ts:1321-1323 says "A frame failed verification, so parallel drawElement capture fell back". But the trip predicate at render.ts:1297 is const fired = outcome === "reverted";, and "reverted" is stamped by renderOrchestrator.ts:2865 for the router-routed path on any shouldRetryViaPinnedFallback outcome (renderOrchestrator.ts:1475-1484 docstring 1442-1466) — OOM, worker crash, host-contention timeout, all trip the breaker permanently. The test at render.test.ts:893-909 ("worker crashed even after fallback") explicitly asserts this is intended.

Failure scenario: a user runs a large render on a hot machine, one worker gets OS-OOM-killed (not a router bug), retry succeeds via pinned fallback. Breaker persists trip=true. Every subsequent render on that install skips the router forever, even after the user upgrades RAM. Only manual HF_DE_PARALLEL_ROUTER=true or deleting ~/.hyperframes/config.json un-trips it.

Two paths to close: (a) narrow the trip predicate to just self-verify-failure (leave OOM/crash/timeout as one-render retry but not a permanent trip), or (b) update the PR title + user-facing message to accurately describe "any capture-stage retry via the pinned path" as the trip condition and consider automatic time-based un-trip.

2. One-shot deParallelRouterUserManaged latch clobbers mid-batch explicit overrides. render.ts:1078-1086 docstring claims "Their choice wins over the circuit breaker in BOTH directions." That's true only for the first applyDeParallelRouterCircuitBreaker call in the process. Programmatic wrapper scenario:

  1. Row 1: fresh env, applyDeParallelRouterCircuitBreaker latches deParallelRouterUserManaged = false.
  2. Programmatic wrapper sets process.env.HF_DE_PARALLEL_ROUTER = "true" between renders (documented at render.test.ts:979-981 as a legitimate scenario).
  3. Row 2: latch skipped (deParallelRouterUserManagedResolved === true), reads stale userManaged = false, router runs.
  4. Row 2 reverts → applyDeParallelRouterBreaker → writes "false" OVER the user's explicit "true" because the latch said false.

Not caught by any test — the test at render.test.ts:964-985 sets "false" (which matches what the breaker would write) and uses parallelRouter: "routed" (no revert, so the breaker write never fires). Fix: re-evaluate userValueEnablesDeParallelRouter(process.env.HF_DE_PARALLEL_ROUTER) at trip time in applyDeParallelRouterBreaker, not from a first-row latch.

3. CLI-copy of the kill-switch parser has no unit test. render.ts:1148-1152 (userValueEnablesDeParallelRouter) parses the same set of spellings as the producer, and the docstring at render.ts:1142-1147 warns: "Keep the two in sync — the producer copy is the source of truth." But only the producer copy has parser tests (renderOrchestrator.test.ts:1831-1850). A future refactor that adds "disabled" as a kill spelling to the producer but forgets the CLI copy would silently break trip-persistence for that spelling with no CI signal. Either add a parity test that imports both parsers and iterates the shared spelling list, or extract the parser to a shared module.

4. Disk-persisted trip is CLI-only. renderOrchestrator.ts:1333-1339 reads env only. Any future in-process or subprocess caller that invokes the producer via a path other than packages/cli/src/commands/render.ts bypasses the ~/.hyperframes/config.json trip check — that caller reads the router as default-ON regardless of prior CLI-side trips. Not exploitable today (no other caller), but the "per-install breaker" claim in the PR body is actually "per-install through the CLI." Worth noting explicitly in the docstring at render.ts:1120-1125 for future integrators (studio-server, aws-lambda, gcp-cloud-run).

Nits

  • Stale docs: packages/producer/tests/escape-hatch-fatal-fallback/src/README.md:33-35 still describes the router as opt-in via HF_DE_PARALLEL_ROUTER=true. Update to reflect default-ON + kill-switch semantics.
  • applyDeParallelRouterCircuitBreaker(options.quiet) at render.ts:837options.quiet is boolean | undefined; when undefined, !quiet inside is truthy, so the breaker's user message may print for programmatic callers who didn't explicitly set quiet. Non-critical since callers who opt in also set quiet in execute.ts:105,255.
  • The /tmp/pr-2840.diff was cut against 3da31e399 (pre-2838), not against 2838's HEAD, so the diff file conflates 2838's default-flip inline with 2840's parser tightening — easy to misread on GitHub's stack view. Reviewers using the diff file locally should check the git diff 6a50c9f42..HEAD numstat (24+/1- for renderOrchestrator.ts) to see the 2840-alone shape.

Questions

  • Is there an intentional reason NOT to auto-untrip after N healthy renders on a bumped-RAM/refactored machine? A 30-healthy-render green streak (already tested at render.test.ts:987-1010) would be a natural signal.
  • Was stop gating on telemetry reviewed with the dashboard-1807532 tripwire owner? Removing the telemetry gate means the "sustained revert >10%" signal now excludes the telemetry-opt-out cohort — the gate for that dashboard signal now has an unknown-size blind spot.

What I didn't verify

  • Local end-to-end with a real config file — trusting the test coverage.
  • Whether other consumers of ~/.hyperframes/config.json (studio, cli auth) could collide with concurrent writes to the trial-fired field during a batch render.

Review by Rames D Jusso

@vanceingalls

Copy link
Copy Markdown
Collaborator Author

Fixed in `0fed335aa`. Thanks both — the blocker is a genuine hole and in the same class as the bug this PR exists to close, just through a different door.

@miguel-heygen — blocker: set-but-empty exempts the install from its own breaker. Correct and confirmed. Ownership used `!== undefined` while both parsers normalize empty/whitespace to "unset → default ON", so `HF_DE_PARALLEL_ROUTER=` routed the render and latched `deParallelRouterUserManaged`, making `applyDeParallelRouterBreaker()` a no-op on fallback. The install then keeps retrying a failing router — losing precisely the first-fallback protection the PR is for. Ownership now uses the same normalization as the parsers:

deParallelRouterUserManaged = (process.env.HF_DE_PARALLEL_ROUTER ?? "").trim() !== "";

Pinned with the regression you asked for, parameterized over `""` and `" "`: both assert the env becomes `"false"` and the fired flag persists. Fault-injection verified — restoring !== undefined fails both.

@miga-heygen — nit 1: misleading message on explicit opt-in. Right, and the repeat-every-revert part is the worse half. The notice is now gated on the breaker having actually taken effect (fired && !deParallelRouterUserManaged), so an explicit opt-in no longer gets told the router is off when it is still on.

@miga-heygen — nit 2: missing explicit-"true"-survives-fallback test. Added, close to your sketch. That completes the "explicit user choice wins in both directions" pair — the opt-out half was already covered.

@miga-heygen — nit 3: gpu: false looked like a cross-PR seam. It was. Moved to #2838 where the gpu property originates; this branch no longer carries it. #2838's Typecheck failure was the same root cause and is fixed there.

69 CLI tests pass, typecheck/lint/fallow clean. Regression shards re-running on the new head.

miguel-heygen
miguel-heygen previously approved these changes Jul 28, 2026

@miguel-heygen miguel-heygen left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-reviewed at 0fed335aa25d3b9ac08e6b0f3dc7ea11369d634c against the head where I requested changes. The blocker is closed at the contract boundary: ownership now uses the same normalization as both router parsers (trim() !== ""), so absent, empty, and whitespace-only values remain breaker-managed while any explicit non-empty user choice still wins.

The regression coverage is executable rather than declarative: the parameterized empty/whitespace cases drive a real reverted render outcome and assert both the in-process false latch and the persisted fired flag. The explicit-true case locks the opposite polarity, and the trip message is now emitted only when the breaker actually took control. I ran the exact CLI render suite at this head: 69/69 passed. All current PR checks are green.

No blocking findings. The previously noted programmatic mid-process env-mutation/parser-duplication concerns remain follow-up-grade and are unchanged by this delta.

Verdict: APPROVE
Reasoning: The empty-value ownership mismatch is fixed using the parser contract itself, both default-like spellings and explicit override behavior are regression-pinned, targeted tests pass, and CI is green.

— Magi

Base automatically changed from 07-27-feat_producer_lower_parallel-de_router_floor_to_700_frames_power-state_telemetry to main July 28, 2026 12:24
@vanceingalls
vanceingalls dismissed miguel-heygen’s stale review July 28, 2026 12:24

The base branch was changed.

vanceingalls added a commit that referenced this pull request Jul 30, 2026
Adds a reusable staged-rollout primitive so a change can ship to a stable
slice of installs instead of all-or-nothing.

The gap it fills: the repo carries ~49 HF_*/PRODUCER_* booleans and every one
is binary — a feature is either off (and therefore unexercised on real
traffic) or on for everyone (and therefore a fleet-wide bet). The
parallel-drawElement router sat in that gap for weeks: default-off produced
almost no signal, and flipping it default-on would have exposed 100% of
eligible installs at once.

Shape:

- packages/core/src/canary.ts — pure evaluator. No fs, no network, no
  `process`; the caller supplies the unit id and overrides, so it imports
  cleanly into the CLI, producer, engine, studio-server, the browser-side
  studio bundle and the embeddable player. FNV-1a rather than node:crypto for
  the same reason.
- packages/core/src/canaryRegistry.ts — every rollout in one table (name,
  percentage, owner, description, sunsetAfter), so "what is rolling out, to
  whom, owned by whom" is answerable without grepping 49 env vars.
- packages/cli/src/telemetry/canary.ts — supplies the three things only the
  CLI knows: anonymousId, the HF_CANARY_<FEATURE> override, and is_ci.
  Day-to-day API is `isCanaryEnabled("name")`.

Three properties the tests pin, because getting them wrong is subtle:

- Slices are INDEPENDENT per feature: the bucket hashes `feature:unitId`, not
  the id alone. Bucketing on the id would hand every concurrent experiment to
  the same unlucky cohort and make two rollouts unreadable apart.
- Ramping is INCLUSIVE: `bucket < percentage`, so widening 10 -> 25 keeps the
  original cohort and before/after comparisons survive the ramp.
- It fails CLOSED: no unit id, unknown name, or CI install means not enrolled.
  A canary exists to bound blast radius, so "we don't know who this is" must
  never mean "enrol everyone".

Registry entries also carry a sunset date, and a test fails once one is past
due — a canary that outlives its rollout is a permanent fork of the product
with none of the review a permanent fork would get.

Ships with de-parallel-router registered at 0%: inert, and ready to ramp in a
patch release once #2840 lands.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

@miguel-heygen miguel-heygen left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-stamped exact head 0fed335aa25d3b9ac08e6b0f3dc7ea11369d634c. This is the same head I previously reviewed: empty/whitespace remains breaker-managed, explicit non-empty choices retain ownership, the persisted breaker semantics are pinned by regressions, and all exact-head checks are terminal green.

@vanceingalls
vanceingalls force-pushed the 07-27-feat_producer_enable_parallel-de_router_by_default branch from 0fed335 to 0e5562c Compare July 31, 2026 00:08
@vanceingalls

vanceingalls commented Jul 31, 2026

Copy link
Copy Markdown
Collaborator Author

Holding this until #2854 (canary rollouts) lands — sequencing decision, not a blocker on the code here.

Status check while noting it:

  • Rebased and clean. 0 behind main, 2 commits ahead, MERGEABLE. The conflict against renderOrchestrator.ts that v0.7.83/84 introduced is resolved.
  • The breaker now carries over for free. persistDeParallelRouterTrialFired()writeConfig()syncInstallState(), and on current main that mirrors the tripped bit into ~/.hyperframes/install-state.json (feat(cli): roll circuit-breaker state over across config wipes #2874, relocated by fix(cli): move install-state into the config dir so deleting it is a full reset #2904). No wiring change was needed here — the mirroring lives inside writeConfig, which this PR already calls. Worth knowing it only became true on rebase; the pre-rebase branch predated both.
  • Still REVIEW_REQUIRED. miguel's CHANGES_REQUESTED was resolved and dismissed, but nobody has approved since.

One thing to settle before merge: this ships default-on with the breaker; #2854 registers de-parallel-router as a canary at percentage: 0. Those are alternative rollout strategies, and if this lands default-on the registry entry becomes dead — you can't ramp a feature that is already on for everyone. So either this gets gated behind the canary and ramps, or the registry entry is dropped when this merges and the breaker is the whole mechanism.

My read is the latter: the soak evidence is a breaker result, not a cohort one — 0 of 141 installs reverted more than once on the floor-700 config (v0.7.78+, 30d), against 4 of 96 in the number quoted in the PR body. Refreshed figures, since the body's soak predates the floor change:

PR body (07-28) now
per-render revert 2.31% 3.17%
installs with ≥1 revert 9.8% 9.93%
installs reverting >1× 4 / 96 0 / 141
installs engaged 975 141

Two caveats worth reconciling before flipping the default: I can't reproduce the 975-install population (the poisoned pre-0.7.66 era had 1,733 installs at a 17.46% revert rate, uncomfortably close in magnitude), and every one of those 141 installs is a self-selected opt-in trial participant — default-on exposes everyone, which is the population the soak can't speak to.


⚠️ Superseded 2026-08-07 — the recommendation in this comment was not taken

This comment concluded: "the registry entry is dropped when this merges and the breaker is the whole mechanism." We did the opposite. This PR is now gated on the de-parallel-router canary and ramps 5 → 25 → 100; the registry entry is load-bearing, not dead.

The reasoning that changed it: default-ON is a ~17× exposure jump onto ≤4-CPU and Docker profiles the trial never covered, and 0.7.60–0.7.64 showed this subsystem can regress silently for five releases. A breaker bounds per-install damage after the fact; it does not bound how many installs discover a regression at once. Those are different guarantees and the ramp provides the second one.

The soak figures quoted here are also stale (975 installs / 9.8% / 2.31%). Current: 1,466 installs, 154 of 155 reverting installs stop at one, 2.79% per-render. See the PR description.

@vanceingalls

vanceingalls commented Aug 4, 2026

Copy link
Copy Markdown
Collaborator Author

VERDICT: WAITING

Window is ~4 hours since 0.7.92 published (2026-08-04). Install-share sample already crossed the ~900 threshold, but the 1-day-old observation is too fresh for Check 2 (drift) and one arm sits at the edge of Check 1 (accuracy). Cron reschedules for tomorrow 13:00 PT.

Observations at n=1620 (0.7.92+, since 2026-08-04 00:00Z, PostHog Hyperframes proj 356858)

Arm true installs share 95% Wilson CI Doc target Compatible with target?
canary-calibration-10 136 8.40% [7.14%, 9.85%] 10% No — upper CI 0.15pp below target
canary-calibration-50 784 48.40% [46.00%, 50.86%] 50% Yes
Overlap (both true) 64 3.95% [3.10%, 5.00%] ~5% (0.10×0.50) Yes

Flips (Check 3, per doc must be zero except HF_CANARY_ dev-toggle):* 3 installs flipped on calibration-10, 13 on calibration-50. Not yet split by dev-toggle rule-out — deferred until n grows and I can inspect the specific ids.

Cross-surface (Check 4): not yet measured — deferred to next fire when Studio + CLI events have accumulated enough same-id pairs to be meaningful.

Why not READY yet

  • Observation is 4 hours old. Check 2 (drift) is defined over 1/7/14/30-day windows per the doc. Even with n>900, one day of data doesn't distinguish a stable 8.4% from a converging 8.4→10% path.
  • calibration-10 at 8.40% [7.14, 9.85] is statistically below 10% at 95% CI, but the effect (−1.6pp) is small and could resolve with more installs. The doc doesn't quantify the tolerance for Check 1 — it says "install share must land on target" — so a borderline read this early is exactly what WAITING is for.
  • Independence looks fine: 3.95% overlap vs expected 4.07% under observed marginals (0.084 × 0.484). Consistent with the two flags being independent draws.
  • Flip attribution incomplete: 3+13 raw flippers need dev-toggle rule-out per the doc's Check 3 innocent explanation.

Why not BLOCKED yet

  • No check has failed in a way the doc does not accept. Check 1's borderline read is not a documented failure — it's a "on target" ambiguity that needs a longer window. Check 3's raw flippers might resolve into the doc's HF_CANARY_* exception. Check 2 not yet measurable.

Exact HogQL (re-runnable)

Install-share + independence:

WITH per_install AS (
  SELECT distinct_id,
    any(JSONExtractString(properties, '$feature/canary-calibration-10')) AS cal10,
    any(JSONExtractString(properties, '$feature/canary-calibration-50')) AS cal50
  FROM events
  WHERE timestamp >= toDateTime('2026-08-04 00:00:00')
    AND properties.cli_version >= '0.7.92'
    AND JSONExtractString(properties, '$feature/canary-calibration-10') != ''
  GROUP BY distinct_id
)
SELECT count() AS installs,
  sum(if(cal10='true',1,0)) AS cal10_true,
  sum(if(cal50='true',1,0)) AS cal50_true,
  sum(if(cal10='true' AND cal50='true',1,0)) AS both_true
FROM per_install

Flip count (Check 3, pre-attribution):

WITH per_install AS (
  SELECT distinct_id,
    countDistinct(JSONExtractString(properties, '$feature/canary-calibration-10')) AS cal10_uniq,
    countDistinct(JSONExtractString(properties, '$feature/canary-calibration-50')) AS cal50_uniq
  FROM events
  WHERE timestamp >= toDateTime('2026-08-04 00:00:00')
    AND properties.cli_version >= '0.7.92'
    AND JSONExtractString(properties, '$feature/canary-calibration-10') != ''
  GROUP BY distinct_id
)
SELECT count() AS total_installs,
  sum(if(cal10_uniq > 1, 1, 0)) AS cal10_flippers,
  sum(if(cal50_uniq > 1, 1, 0)) AS cal50_flippers
FROM per_install

Filters: cli_version >= '0.7.92', timestamp since 2026-08-04 00:00:00 UTC, PostHog project 356858 (Hyperframes).

Next fire: 2026-08-05 13:00 PT (20:00Z). Rechecking with ~24h more data.

— Via


⚠️ Superseded 2026-08-07 — the accuracy concern here was a measurement artifact

This read flagged calibration-10 at 8.40% with "upper CI 0.15pp below target" — i.e. a possible bucketer failure. That was almost certainly the CI-denominator trap, not a real deviation.

CI installs are excluded from percentage enrolment but still emit the assignment as "false", so leaving them in the denominator biases every canary low. The same mistake produced 9.22% / 48.64% on a later read — both "significantly under target" — and both landed on target once CI was removed.

At n = 13,547 with CI excluded: 9.62% and 49.76% against 10% / 50%. Accuracy passes. Independence passes (5.03% vs ~4.9% expected).

Any future read must exclude CI from the denominator. Since 0.7.96 the emitted canary_reason_* property makes this a property filter (reason IN ('in_cohort','out_of_cohort')) rather than a join on is_ci that someone has to remember.

@vanceingalls

vanceingalls commented Aug 5, 2026

Copy link
Copy Markdown
Collaborator Author

VERDICT: BLOCKED

Second-fire canary calibration read for the calibration-10 / calibration-50 bucketer, gating PR #2840 rewire.

Bottom line: Check 3 (stability) is dirty for the reason the doc pre-declared, and the defect is not standing still. Post-settled flip count has grown from 17 (this-morning's first read, recorded in docs/contributing/canary-rollouts.mdx) to 46 in this read, over the same overall window. The bucketer is not yet trustworthy to ramp a real feature on. Under the doc's own definition — "A flip that persists past that is not this race … By this check's own definition that is a real defect, not an accepted limit" — this blocks rewiring #2840 until the decision-reason emission ships and the table is re-read.

Window & filters

  • Window: 2026-08-04 00:00:00 UTC2026-08-06 00:00:00 UTC
  • First flag-bearing event: 2026-08-04 04:57:18 UTC (first 0.7.92 publish)
  • Last flag-bearing event in window: 2026-08-05 20:06:02 UTC
  • Version filter for accuracy / independence: cli_version >= '0.7.92' (excludes 18 pre-0.7.92 team source-builds; carried on cli_command, cli_command_result, render_observation, render_complete, render_error, cli_render_feedback, cli_error, init_template, check_report, browser_install)
  • CI filter: is_ci != 'true' (doc: "Exclude CI installs from the denominator. They are excluded from percentage enrolment but still emit the assignment as false")
  • Flag properties: $feature/canary-calibration-10, $feature/canary-calibration-50 — string "true" / "false"
  • PostHog project: Hyperframes, id 356858 (default project 55348 = HeyGen Prod returns zeros — data-trap)
  • Population summary: 16,984 distinct installs with any flag-bearing event; 15,674 after CI + >=0.7.92 filter

Check 1 — Accuracy (install-weighted, non-CI, >=0.7.92): PASS

WITH eligible AS (
  SELECT distinct_id
  FROM events
  WHERE timestamp >= toDateTime('2026-08-04 00:00:00')
    AND timestamp < toDateTime('2026-08-06 00:00:00')
    AND event IN ('cli_command','cli_command_result','render_observation','render_complete',
                  'render_error','cli_render_feedback','cli_error','init_template',
                  'check_report','browser_install')
    AND JSONExtractString(properties, 'cli_version') >= '0.7.92'
    AND JSONExtractString(properties, 'is_ci') != 'true'
  GROUP BY distinct_id
)
SELECT
  count()                                                          AS n_installs,
  countIf(any_c10 = 'true')                                        AS c10_true,
  countIf(any_c50 = 'true')                                        AS c50_true,
  round(countIf(any_c10 = 'true') * 100.0 / count(), 3)            AS c10_share_pct,
  round(countIf(any_c50 = 'true') * 100.0 / count(), 3)            AS c50_share_pct
FROM (
  SELECT distinct_id,
         any(JSONExtractString(properties, '$feature/canary-calibration-10')) AS any_c10,
         any(JSONExtractString(properties, '$feature/canary-calibration-50')) AS any_c50
  FROM events
  WHERE distinct_id IN (SELECT distinct_id FROM eligible)
    AND timestamp >= toDateTime('2026-08-04 00:00:00')
    AND timestamp < toDateTime('2026-08-06 00:00:00')
    AND JSONExtractString(properties, '$feature/canary-calibration-10') IN ('true','false')
  GROUP BY distinct_id
)
arm installs enrolled n_installs share Wilson 95% CI target
canary-calibration-10 1,495 15,674 9.538% [9.09%, 10.00%] 10%
canary-calibration-50 7,860 15,674 50.147% [49.37%, 50.92%] 50%

Both intervals contain the target — accuracy passes on install-share (the check that must land per the doc). Event-weighted skew is documented as a separate lens; install-share is authoritative.

Check 2 — Drift: not-failure-axis, per doc

Doc: "Check 2 will NOT come back flat, and that is expected rather than a defect. Per-install cohorts never flip, but a person who wipes their config gets a new id and a fresh roll." Alternatives (hardware fingerprinting, account identity) are explicitly rejected on record. Only two calendar days of window so far, so 1/7/14/30-day windows are not comparable yet; reporting deferred to a re-read after the decision-reason ships. Not a blocker.

Check 3 — Stability: DIRTY, real defect

All-install flip counts (matching the doc's methodology, no CI/version filter, so first-fire numbers are directly comparable):

SELECT
  countDistinct(distinct_id) AS n_installs_any,
  uniqIf(distinct_id, c10_flips) AS c10_flip_ids,
  uniqIf(distinct_id, c50_flips) AS c50_flip_ids
FROM (
  SELECT distinct_id,
         countDistinct(JSONExtractString(properties, '$feature/canary-calibration-10')) > 1 AS c10_flips,
         countDistinct(JSONExtractString(properties, '$feature/canary-calibration-50')) > 1 AS c50_flips
  FROM events
  WHERE timestamp >= toDateTime('2026-08-04 00:00:00')
    AND timestamp < toDateTime('2026-08-06 00:00:00')
    AND JSONExtractString(properties, '$feature/canary-calibration-10') IN ('true','false')
  GROUP BY distinct_id
)
population flips on canary-10 flips on canary-50
doc's first read (this morning; 9,702 installs) 40 107
this read (16,984 installs) 98 246

Minority-persistence breakdown (per-flip: for each flipping install, span from first minority-value event to last minority-value event):

-- For calibration-10 (identical query re-used with '-50' for the other row)
WITH per_install AS (
  SELECT
    distinct_id,
    countIf(JSONExtractString(properties, '$feature/canary-calibration-10') = 'true')  AS n_true,
    countIf(JSONExtractString(properties, '$feature/canary-calibration-10') = 'false') AS n_false,
    minIf(timestamp, JSONExtractString(properties, '$feature/canary-calibration-10') = 'true')  AS first_true,
    maxIf(timestamp, JSONExtractString(properties, '$feature/canary-calibration-10') = 'true')  AS last_true,
    minIf(timestamp, JSONExtractString(properties, '$feature/canary-calibration-10') = 'false') AS first_false,
    maxIf(timestamp, JSONExtractString(properties, '$feature/canary-calibration-10') = 'false') AS last_false
  FROM events
  WHERE timestamp >= toDateTime('2026-08-04 00:00:00')
    AND timestamp < toDateTime('2026-08-06 00:00:00')
    AND JSONExtractString(properties, '$feature/canary-calibration-10') IN ('true','false')
  GROUP BY distinct_id
),
flipped AS (
  SELECT
    distinct_id,
    if(n_true < n_false, n_true, n_false) AS minority_n,
    if(n_true < n_false, dateDiff('second', first_true,  last_true),
                         dateDiff('second', first_false, last_false)) AS minority_persist_sec
  FROM per_install
  WHERE n_true > 0 AND n_false > 0
)
SELECT
  multiIf(minority_persist_sec <= 5,     'le_5s (first-mint shaped)',
          minority_persist_sec <= 60,    '5-60s',
          minority_persist_sec <= 3600,  '1-60min',
          'gt_1h (post-settled)') AS bucket,
  count()                       AS n_installs,
  round(avg(minority_n), 1)     AS avg_minority_events
FROM flipped
GROUP BY bucket
ORDER BY bucket
minority persists canary-10 installs canary-10 avg minority events canary-50 installs canary-50 avg minority events
≤ 5s (first-mint shaped) 30 3.0 75 2.7
5-60s 23 9.1 48 9.1
1-60min 38 15.7 84 12.6
> 1h (post-settled) 7 50.7 39 101.8

Post-settled totals: 7 + 39 = 46 installs sustained a minority assignment past the first-mint window, up from 17 in the doc's first read at 9,702 installs. Even scaled by population growth (16,984 / 9,702 = 1.75×), the pace is holding, not decaying — the effect is genuine, not first-mint noise.

Neither accepted exception (dev-toggled HF_CANARY_CALIBRATION_* override; the first-mint seed race) covers a flip that persists 100+ minority events over hours. Per the doc: "A flip that persists past that is not this race … By this check's own definition that is a real defect, not an accepted limit."

Check 3 bonus — Independence: PASS

-- Actual overlap vs product-of-shares, on the same non-CI 0.7.92+ population
SELECT
  count()                                          AS n,
  countIf(c10 = 'true' AND c50 = 'true')           AS both_true,
  countIf(c10 = 'true')                            AS c10_true,
  countIf(c50 = 'true')                            AS c50_true,
  round(countIf(c10 = 'true' AND c50 = 'true') * 100.0 / count(), 3)                  AS overlap_actual_pct,
  round(countIf(c10 = 'true') * countIf(c50 = 'true') * 100.0 / (count() * count()), 3) AS overlap_expected_pct
FROM (
  -- same eligible + assignments CTEs as check 1
  SELECT distinct_id,
         any(JSONExtractString(properties, '$feature/canary-calibration-10')) AS c10,
         any(JSONExtractString(properties, '$feature/canary-calibration-50')) AS c50
  FROM events
  WHERE distinct_id IN (SELECT distinct_id FROM eligible)
    AND timestamp >= toDateTime('2026-08-04 00:00:00')
    AND timestamp < toDateTime('2026-08-06 00:00:00')
    AND JSONExtractString(properties, '$feature/canary-calibration-10') IN ('true','false')
  GROUP BY distinct_id
)
  • n = 15,684 (non-CI, >=0.7.92); both-true = 783
  • Actual overlap = 4.992%; expected (p1 × p2) = 4.785%
  • Delta = +0.207pp; z ≈ 1.22 (two-tailed p ≈ 0.22) — indistinguishable from independence at this window's power. First-read had 5.18% actual vs 4.71% expected (z=2.16); this read is closer to expectation, not further, so no cluster-shared-cohort signal.

Check 4 — Cross-surface (CLI vs Studio): sub-signal of check 3

SELECT
  countDistinct(distinct_id) AS n_installs_both_surfaces,
  uniqIf(distinct_id, cli_c10 = studio_c10)  AS agree_c10,
  uniqIf(distinct_id, cli_c10 != studio_c10) AS disagree_c10,
  uniqIf(distinct_id, cli_c50 = studio_c50)  AS agree_c50,
  uniqIf(distinct_id, cli_c50 != studio_c50) AS disagree_c50
FROM (
  SELECT distinct_id,
    any(if(startsWith(event, 'studio'), NULL, JSONExtractString(properties, '$feature/canary-calibration-10'))) AS cli_c10,
    any(if(startsWith(event, 'studio'), JSONExtractString(properties, '$feature/canary-calibration-10'), NULL)) AS studio_c10,
    any(if(startsWith(event, 'studio'), NULL, JSONExtractString(properties, '$feature/canary-calibration-50'))) AS cli_c50,
    any(if(startsWith(event, 'studio'), JSONExtractString(properties, '$feature/canary-calibration-50'), NULL)) AS studio_c50
  FROM events
  WHERE timestamp >= toDateTime('2026-08-04 00:00:00')
    AND timestamp < toDateTime('2026-08-06 00:00:00')
    AND JSONExtractString(properties, '$feature/canary-calibration-10') IN ('true','false')
  GROUP BY distinct_id
)
WHERE cli_c10 IN ('true','false') AND studio_c10 IN ('true','false')
  • n = 2,325 installs emitting on both surfaces
  • canary-10 CLI↔Studio disagreements: 12 (0.52%)
  • canary-50 CLI↔Studio disagreements: 34 (1.46%)

Attribution note: these are a strict subset of the check-3 flips. 46 post-settled flips overall, only 46 total CLI↔Studio disagreements — so most flips (>90% of the ≤ 60min buckets and effectively all the 1-60min bucket) live within a single surface, not across the CLI-launched-Studio boundary. That is consistent with the doc's leading unproven hypothesis — "a shared distinct_id across containers that each hold their own seed" — where each container-launched CLI process is a distinct "surface" from telemetry's point of view but reports the same distinct_id.

Power

Doc target for a bucketer read: p=0.10, ±2pp at 95% CI ≈ 900 installs. This window's n=15,674 non-CI 0.7.92+ installs clears that ceiling by ~17×; neither VERDICT: WAITING nor a "sample too small" reservation applies. Every observation above is at full power for its own check.

What blocks

Per the doc's "What passing looks like, and what cannot be fixed" — Checks 1 and 4 are properties of the design, and Check 3 should pass apart from the first-mint race. Check 3 does not pass, the persisting-minority flips are not first-mint shaped (mean 50.7 / 101.8 minority events, over 1+ hours), and the count is growing at the second read rather than settling. Doc-mandated next step: "emit the decision reason (forced_on / forced_off / in_cohort / out_of_cohort) so a deliberate override can be told apart from a genuine flip, then re-read this table."

Not proposing a fix in this comment (out of scope for a measurement read). PR #2840 stays held. When decision-reason emission ships and the table is re-read, if Check 3 passes, #2840 can be wired to a new canary-de-parallel-router and ramped 5 → 25 → 100 while watching revert-rate broken by cpu_count and is_docker (the two populations the current soak doesn't cover — ≤4-CPU at 13.2%, Docker at 6.2%).

Cron cron_1c411937 cancelled after this read.

— Via


⚠️ Superseded 2026-08-07 — this BLOCKED verdict no longer holds

Do not treat this comment as blocking. Two of its load-bearing claims did not survive attribution.

1. "The defect is not standing still" (17 → 46 post-settled flips) was a population effect, not a rate change. The rate has been flat across three reads at growing n: 0.14% → 0.12% → 0.10% of installs. The raw count grew because the fleet on 0.7.92+ grew. Comparing counts across windows with different denominators is what produced "growing".

2. Roughly 45% of those flips were never a bucketer defect. Once canary_reason_* shipped in 0.7.96, the sustained flips split cleanly: 32 of 71 showed false events that are exactly the CI ones — the documented exclude: is_ci rule meeting a distinct_id used both in and out of CI. The flip query had not excluded CI, the same trap that produced the false accuracy failure two comments up.

3. The innocent explanation this comment could not rule out is now ruled out. Zero forced_on / forced_off across every flip — no dev was toggling HF_CANARY_*.

Where it actually landed: ~14 genuinely unexplained sustained flips out of 13,547 installs (0.10%). That is an order of magnitude below this feature's own 2.79% revert rate, so cohort instability cannot meaningfully corrupt a ramp read.

The gating standard was also wrong in kind. The doc said check 3 "MUST be zero", and this comment applied it literally. A 0.10% stable edge does not justify blocking a 5% ramp that has a per-install breaker underneath it — the useful question is "can a ramp be trusted", not "is the number zero".

The remaining 38 (across both canaries) are a real open question and worth explaining, but they are not a blocker. See the PR description.

vanceingalls added a commit that referenced this pull request Aug 6, 2026
Review fixes.

BLOCKER: the doc quoted two different independence results for the same
window — the current table (5.03%, z = 1.06) and a stale closing paragraph
(5.18%, z = 2.16) from the earlier iteration. Deleted the paragraph rather
than updating it: the table already carries independence in the same shape as
the other primary results, and 'worth re-checking rather than acting on' no
longer fits at z = 1.06. It survived because an earlier edit re-wrapped the
line and a later string replace silently missed.

The reason enum has six wire values, not the four that carry attribution:
no_unit_id and excluded also land on real events, and excluded is what CI
installs emit — useful, since dropping them currently needs a join on is_ci.
Points at CanaryReason as the source of truth.

Also records that reason alone cannot settle the container hypothesis — it
reports the decision on both sides of a flip, not why the seeds diverged —
and names what would: bucket_seed_origin plus a machine fingerprint. Notes
that identity_persistence already ships and may corroborate it first.

Adds query provenance so the table can be reproduced, and states that the
#2840 hold is a sequencing decision rather than a code-enforced gate:
de-parallel-router is registered at 0% and unwired, and the live gate is the
env var plus the circuit breaker.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
pull Bot pushed a commit to ValidationExpression/hyperframes that referenced this pull request Aug 7, 2026
de_parallel_router is present on 95.4% of render_complete events and 0.83%
of render_error. Capture context itself survives failures fine (capture_mode
is on 98.6% of them), so this is not renders failing before capture — the
routing state specifically is being dropped.

Cause is ordering. deParallelRouter is assigned twice: once before the
capture-observability update, and again inside syncCapturePlan where routing
is actually resolved — including the 'reverted' case, which the earlier
assignment cannot know. The update in between recorded whatever was true
first, so a render that failed while routed reported no routing state at all.
The existing comment at the earlier call site says it is recorded there
precisely so hard failures carry it; that intent was correct and the value
just arrived too late.

This matters for the heygen-com#2840 ramp specifically. The per-install circuit breaker
only arms on a revert, which requires the render to finish and self-detect —
it cannot catch a crash or hang. Those are exactly the failure modes a
percentage ramp exists to bound, and they were the ones telemetry could not
see.

Also makes the ffprobe contract sweep resilient per entry. A dangling symlink
under packages/studio/data/projects threw ENOENT on stat and aborted the whole
traversal, so every package sorting after 'studio' — both studio-server
callers included — silently stopped being checked. main is currently red on
this. The manifest assertion is what caught it, which is what it was added
for.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
vanceingalls and others added 3 commits August 7, 2026 15:01
…r-install circuit breaker

The DE parallel router (HF_DE_PARALLEL_ROUTER) becomes default-ON. The soak
answered the safety question it was gated on: zero damaged frames shipped —
every fallback was the self-verification net catching a bad frame and
recovering on the screenshot path. Verify PSNR p10 sits flat near 40 dB
against a 32 dB floor. The residual 2.31% revert rate is an efficiency cost
(a revert forfeits the speedup, never the output), accepted in exchange for
parallelizing the >=700-frame band — roughly 80% of all DE capture
wall-clock, frame-weighted.

Default-ON is safe because the per-install circuit breaker stays underneath
it. That distinction matters: 9.8% of installs hit a revert, and they are
latched off permanently after the first one. Without the breaker those
installs would go from "one slow render, then protected" to "every eligible
render is slow".

The breaker, adapted for a default-ON flag:

- Writes an explicit HF_DE_PARALLEL_ROUTER=false and persists it to
  ~/.hyperframes/config.json, so the install stays off across processes.
  Absent no longer means off, so the switch has to be written, not unset.
- Trips only on a real fallback, never on render count — a healthy install
  keeps the speedup indefinitely.
- Independent of telemetry state: opting out of analytics must not cost a
  user the faster renderer. Telemetry governs reporting, not behavior.
- An explicit user value wins in both directions, latched before the breaker
  can write the var and make the two indistinguishable.
- The user is told when it trips and how to re-enable.

isDeParallelRouterEnabled() parses the kill switch properly: false/0/off/no
(case- and space-insensitive) disable; unset or empty is the default. A bare
`!== "false"` would silently ignore every spelling but one and hand parallel
DE to a user who asked for none.

Refs PRINFRA-384

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Ownership detection classified ANY defined HF_DE_PARALLEL_ROUTER as a user
choice, but both parsers read empty/whitespace as "unset -> default ON".
Launching with `HF_DE_PARALLEL_ROUTER=` therefore routed the render (empty
parses as ON) while exempting the install from its circuit breaker: after a
verified fallback applyDeParallelRouterBreaker() no-op'd, so the install kept
retrying the failing router instead of latching off. That is the exact
first-fallback protection this PR exists to provide, lost on a documented
default path. Ownership now uses the same normalization as the parsers.

Also: only announce a trip the breaker could act on. With an explicit user
opt-in the breaker is deliberately a no-op, so "now off for this install" was
factually wrong — and reprinted on every later revert, since the user's value
keeps the router active.

Tests: set-but-empty and whitespace both latch off and persist the fired flag
(fault-injection verified — restoring the old check fails both); explicit
"true" survives a fallback.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Rebased onto main (was 308 behind) and gated the new default-on behaviour on
the de-parallel-router canary, at 5%.

Default-ON without a ramp is a ~17x exposure jump: from ~6% of eligible
renders today to all of them, landing on profiles the opt-in trial never
covered (<=4 CPUs and Docker, ~12% of eligible renders between them).
0.7.60-0.7.64 is why that matters — every unclamped render reverted for five
consecutive releases and nobody noticed.

The gate reuses the breaker's own disarm: non-enrolled installs get an
explicit HF_DE_PARALLEL_ROUTER=false, because with default-ON polarity
deleting the var means ON. Setting the registry percentage to 0 is therefore
a full fleet-wide revert with no release.

Today's ~11% of installs routing is emergent — the product of eligibility
rules and a capped trial — so it drifts with fleet composition and cannot be
turned off without shipping. The point of the canary is that the number
becomes chosen and revertible, not that it is smaller.

Also replaces the registry test that pinned the percentage to 0. Its intent
was 'ramp only alongside the circuit breaker', but pinning 0 blocks the ramp
forever and never checks the wiring it names. It now asserts the wiring
directly, and fails if either the canary gate or the breaker consult is
removed.

Hold at 5% until PRINFRA-372 resolves: --workers auto crashes every worker on
macOS arm64 while --workers 1 is clean, and the router forces 3 workers.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@vanceingalls
vanceingalls force-pushed the 07-27-feat_producer_enable_parallel-de_router_by_default branch from 0e5562c to 4a25142 Compare August 7, 2026 22:11
@vanceingalls

vanceingalls commented Aug 7, 2026

Copy link
Copy Markdown
Collaborator Author

Rebased onto main (was 308 behind) and added the canary ramp. New head 4a2514232. #3095 closed as superseded — its useful piece is folded in here.

Why the ramp. This PR's own change is the risky part: default-ON takes routed renders from ~6% of eligible to all of them, a ~17x jump, onto profiles today's trial population never covered — ≤4 CPUs and Docker, ~12% of eligible renders between them. 0.7.60–0.7.64 is the precedent: every unclamped render reverted for five consecutive releases and nobody saw it.

The gate reuses this PR's own disarm mechanism — non-enrolled installs get an explicit HF_DE_PARALLEL_ROUTER=false, because with default-ON polarity deleting the var means ON, which is exactly the bug this PR fixed. Setting the registry percentage to 0 is a full fleet-wide revert with no release.

Worth being precise about what the canary buys, because I got this wrong at first: today ~10.85% of rendering installs already route, automatically. That number is emergent — the product of eligibility rules, the 25-render cap and the breaker latch — so it drifts with fleet composition and can't be switched off without shipping. The canary doesn't make exposure smaller; it makes it chosen and revertible.

Telemetry supporting the shape of this PR (14d, ≥0.7.94, routed renders):

outcome renders installs
routed, clean 4,848 1,183
reverted — psnr 128 126
reverted — capture_error 18 18
reverted — blank 3 3

~3% of routed renders revert and 86% of those are PSNR — self-verify catching real visual damage. That is the per-render protection, and it is separate from the breaker. Keeping the breaker (as this PR does) while dropping the cap is the right split: the cap is sampling logic, the breaker is the only thing stopping a damaged install from being damaged repeatedly.

Why the breaker should not also be removed yet. We cannot measure whether it is safe to remove — the breaker censors the data. "154 of 155 reverting installs stop at exactly one revert" is the latch, not a natural rate. Measuring the true repeat-revert rate needs the breaker disabled for a slice, which this canary now makes possible.

Hold at 5% until PRINFRA-372 resolves. --workers auto crashes every worker ("detached Frame" / "Target closed") on macOS arm64 while --workers 1 is clean — and the router forces 3 workers. PRINFRA-339 (indefinite stalls in parallel capture) and PRINFRA-354 (silent background-process hang) are also open, and none of those emit render_complete, so neither self-verify nor the breaker catches them.

Verification: core 1692, cli 2487 passing, lint clean. Gate mutation-tested at both layers — removing it fails 2 CLI tests and the core wiring assertion. One pre-existing failure on main (audioPadTrim.realmedia, 5s timeout), reproduced on clean origin/main.

At each ramp step, split revert rate by cpu_count and is_docker.


Correction 2026-08-07 — "opt-in trial" was the wrong phrase

I wrote "the opt-in trial" above. That understates today's exposure and I have corrected it in this comment and the PR description.

The trial is not a user opt-in — it arms automatically on the CLI render path (execute.ts sets enableDeParallelRouterTrial: true for single renders, and for batch whenever batchConcurrency <= 1). Nobody running hyperframes render chose it.

The flag is opt-in at the call site: it defaults off and only those two CLI sites set it, so programmatic renderLocal consumers are excluded — the mechanism mutates process.env.HF_DE_PARALLEL_ROUTER, which is unsafe under concurrent invocation. That polarity guards embedding contexts, not users.

The distinction matters for judging this PR: ~10.85% of rendering installs already route automatically. The ramp is not protecting users from a feature they opted into — it is governing exposure that is already happening without their choice.

@vanceingalls vanceingalls changed the title feat(producer): enable the parallel-DE router by default, behind a real circuit breaker feat(cli,core,producer): ramp the parallel-DE router through the canary at 5% Aug 8, 2026
It is not a user opt-in — execute.ts arms it automatically on the CLI render
path, so ~11% of installs already route without anyone choosing it. The
opt-in is at the CALL SITE: the flag defaults off and only the two CLI sites
set it, excluding programmatic renderLocal consumers because the mechanism
mutates process.env. That polarity guards embedding contexts, not users.

Calling it opt-in understates today's exposure, which changes how a reviewer
judges the ramp: it is not protecting users from a feature they chose, it is
governing exposure already happening without their choice.

Leaves the accurate uses alone — 'explicit user opt-in' means someone setting
HF_DE_PARALLEL_ROUTER themselves, and the call-site flag is genuinely opt-in.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@vanceingalls
vanceingalls merged commit 867eeab into main Aug 8, 2026
59 checks passed
@vanceingalls
vanceingalls deleted the 07-27-feat_producer_enable_parallel-de_router_by_default branch August 8, 2026 02:49
vanceingalls added a commit that referenced this pull request Aug 8, 2026
…anary gate

Deletes the `de-parallel-router` canary entry and the `isCanaryEnabled` guard
in render.ts together, leaving the producer's default-ON in place. Net effect
for users: the parallel drawElement router is on for everyone again.

## Why, and why not a ramp

Gating at 5% was itself the regression. Measured 2026-08-08, the day after
v0.7.101 shipped the canary: fleet router exposure fell from 3.13-4.25% of
non-CI renders to **0.13%**, roughly 25x, because out-of-cohort installs are
explicitly disarmed and #2840 deleted the everyone-armed trial in the same
change. 2,537 installs lost a feature they already had. Severity is speed
only, never output, and nothing is persisted to disk.

PR #2840's body claimed "the canary does not make exposure smaller; it makes
it chosen and revertible." That was true of the end state and false of the
first step. This lands the end state.

Entry and guard go together deliberately: at >=100 the evaluator
short-circuits ahead of the CI/seedless exclusions, so removing only the entry
would have flipped whatever still resolved false at deletion time, unstaged.

## Both stated blockers are void

- **≤4-CPU / Docker coverage gap.** Docker renders never use drawElement — 0
  of 4,281 across every CPU tier, software GL gates it out — and the router
  requires it. No percentage could ever expose Docker, so no ramp closes that
  gap. ≤4 CPUs yields ~42 drawElement candidates in three days.
- **PRINFRA-372.** Its signature has hits on 0.4.12, 0.4.37, 0.6.52, 0.6.93,
  0.6.109 and 0.6.110 — versions predating drawElement (v0.7.38) and therefore
  this router. It is real, still live on 0.7.101, and belongs to the
  screenshot/beginframe path. 11 reproduction runs across four configurations
  on the enriched profile (darwin/arm64 25.5.0) came back clean.

## Safety unchanged

The per-install circuit breaker and the per-render self-verify are untouched;
`HF_DE_PARALLEL_ROUTER=false` remains the user-facing kill switch. Post-canary
data at 14 days: >8 CPUs 3.02% revert (177/5,857), 5-8 CPUs 2.40% (6/250) —
consistent with the 2.75-3.16% baseline.

Revert path is now a code revert rather than a registry edit. That is the
trade this shape accepts in exchange for one release instead of two.

## Corrects two claims that shipped wrong

`~17x jump in exposure onto <=4 CPUs / Docker` overstated the reach, and
`~11% of installs already route` was an OUTCOME (the share clearing
eligibility and the old 25-render cap), not an exposure setting — read as a
rollout knob it inverts the arithmetic, which is how gating at 5% came to cut
exposure rather than ramp it. Both are recorded in render.ts so they are not
reintroduced.

## Tests

Removed the core wiring assertion and the two CLI canary-gating tests, which
pinned a gate that no longer exists. Added the inverse guarantee in its place:
an ordinary install must come out of the breaker with the var UNSET so the
producer default applies — writing "false" there is precisely what disarmed
the fleet at 5%.

core 1701 passing, cli 2491 passing, studio canary 29 passing. The 2 failures
in play.test.ts reproduce on clean origin/main and are unrelated (#3114 area).
oxlint and oxfmt clean.

Note: telemetry for this rollout stops with the entry — `$feature/canary-de-parallel-router`
and `canary_reason_de_parallel_router` are emitted from the registry, so the
`Ramp —` tiles and the exposure-floor alert on PostHog dashboard 1918875 go
blank once this ships. Watch drawElement engagement on 1807532 instead.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
vanceingalls added a commit that referenced this pull request Aug 8, 2026
…anary gate (#3120)

Deletes the `de-parallel-router` canary entry and the `isCanaryEnabled` guard
in render.ts together, leaving the producer's default-ON in place. Net effect
for users: the parallel drawElement router is on for everyone again.

## Why, and why not a ramp

Gating at 5% was itself the regression. Measured 2026-08-08, the day after
v0.7.101 shipped the canary: fleet router exposure fell from 3.13-4.25% of
non-CI renders to **0.13%**, roughly 25x, because out-of-cohort installs are
explicitly disarmed and #2840 deleted the everyone-armed trial in the same
change. 2,537 installs lost a feature they already had. Severity is speed
only, never output, and nothing is persisted to disk.

PR #2840's body claimed "the canary does not make exposure smaller; it makes
it chosen and revertible." That was true of the end state and false of the
first step. This lands the end state.

Entry and guard go together deliberately: at >=100 the evaluator
short-circuits ahead of the CI/seedless exclusions, so removing only the entry
would have flipped whatever still resolved false at deletion time, unstaged.

## Both stated blockers are void

- **≤4-CPU / Docker coverage gap.** Docker renders never use drawElement — 0
  of 4,281 across every CPU tier, software GL gates it out — and the router
  requires it. No percentage could ever expose Docker, so no ramp closes that
  gap. ≤4 CPUs yields ~42 drawElement candidates in three days.
- **PRINFRA-372.** Its signature has hits on 0.4.12, 0.4.37, 0.6.52, 0.6.93,
  0.6.109 and 0.6.110 — versions predating drawElement (v0.7.38) and therefore
  this router. It is real, still live on 0.7.101, and belongs to the
  screenshot/beginframe path. 11 reproduction runs across four configurations
  on the enriched profile (darwin/arm64 25.5.0) came back clean.

## Safety unchanged

The per-install circuit breaker and the per-render self-verify are untouched;
`HF_DE_PARALLEL_ROUTER=false` remains the user-facing kill switch. Post-canary
data at 14 days: >8 CPUs 3.02% revert (177/5,857), 5-8 CPUs 2.40% (6/250) —
consistent with the 2.75-3.16% baseline.

Revert path is now a code revert rather than a registry edit. That is the
trade this shape accepts in exchange for one release instead of two.

## Corrects two claims that shipped wrong

`~17x jump in exposure onto <=4 CPUs / Docker` overstated the reach, and
`~11% of installs already route` was an OUTCOME (the share clearing
eligibility and the old 25-render cap), not an exposure setting — read as a
rollout knob it inverts the arithmetic, which is how gating at 5% came to cut
exposure rather than ramp it. Both are recorded in render.ts so they are not
reintroduced.

## Tests

Removed the core wiring assertion and the two CLI canary-gating tests, which
pinned a gate that no longer exists. Added the inverse guarantee in its place:
an ordinary install must come out of the breaker with the var UNSET so the
producer default applies — writing "false" there is precisely what disarmed
the fleet at 5%.

core 1701 passing, cli 2491 passing, studio canary 29 passing. The 2 failures
in play.test.ts reproduce on clean origin/main and are unrelated (#3114 area).
oxlint and oxfmt clean.

Note: telemetry for this rollout stops with the entry — `$feature/canary-de-parallel-router`
and `canary_reason_de_parallel_router` are emitted from the registry, so the
`Ramp —` tiles and the exposure-floor alert on PostHog dashboard 1918875 go
blank once this ships. Watch drawElement engagement on 1807532 instead.
@vanceingalls

Copy link
Copy Markdown
Collaborator Author

Correcting the record on this PR (2026-08-09)

Three claims in the body above were falsified by post-release measurement.
Leaving them uncorrected would be actively misleading, since this PR is the
reference anyone reads before touching the router rollout.

1. "The canary does not make exposure smaller. It makes it chosen and revertible."

False as shipped. At 5% it made exposure ~25x smaller: fleet router
exposure fell from 3.13-4.25% of non-CI renders to 0.13% in the day after
v0.7.101. The reasoning assumed the canary gates the expansion on top of the
existing trial, but the implementation gates all routing — out-of-cohort
installs get an explicit HF_DE_PARALLEL_ROUTER="false", and this PR deleted
maybeEnableDeParallelRouterTrial() in the same change, so there was nothing
underneath. 2,537 installs lost a feature they already had.

The claim was accurate about the destination and wrong about the first step.
It became true in v0.7.103 (#3120), which removed the entry and the guard
together.

2. "~17x exposure jump onto <=4 CPUs (8.18%) and Docker (3.99%)."

Overstated the reach. Docker renders never use drawElement at all — 0 of
4,281 measured across every CPU tier, software GL gates it out — and the
router requires it, so no percentage could ever have exposed Docker. <=4 CPUs
yields ~42 drawElement candidates in three days. Those percentages counted
renders in those tiers, not renders the router can enter.

3. "Hold at 5% until PRINFRA-372 resolves... the router forces 3 workers."

Unsupported. The detached Frame / Target closed signature has hits on
0.4.12, 0.4.37, 0.6.52, 0.6.93, 0.6.109 and 0.6.110 — versions predating
drawElement capture (v0.7.38) and therefore this router. A crash occurring
where the router does not exist cannot require it. It is real and still live
on 0.7.101+, but it belongs to the screenshot/beginframe path. 11 reproduction
attempts across four configurations on the enriched profile (darwin/arm64
25.5.0) came back clean. See PRINFRA-372.

Also worth noting: the "~11% of installs already route today" figure is an
OUTCOME — the share that cleared eligibility and the 25-render cap — not an
exposure setting. The pre-#2840 trial armed every install. Reading 11% as
a rollout percentage is what made a 100%-to-5% cut look like a ramp.

What still stands

The kill-switch parse fix (false/0/off/no — the old !== "false"
failed open on every spelling but one), the explicit-"false" breaker write
under default-ON polarity, dropping the telemetry precondition, and the note
that the breaker censors its own repeat-revert data so disabling it for a
slice is the honest experiment. That last one is still the open follow-up.

Soak numbers held up: post-canary at 14 days, >8 CPUs 3.02% revert
(177/5,857) and 5-8 CPUs 2.40% (6/250), consistent with the 2.75-3.16%
baseline.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants