feat(cli,core,producer): ramp the parallel-DE router through the canary at 5% - #2840
Conversation
e9b6662 to
6a50c9f
Compare
miga-heygen
left a comment
There was a problem hiding this comment.
SSOT -- circuit breaker state machine audit
Traced every transition through the module-level state (deParallelRouterUserManaged, deParallelRouterUserManagedResolved, deParallelRouterBreakerTrippedThisProcess) and the on-disk deParallelRouterTrialFired flag. All five advertised transitions hold:
| Initial state | Event | Result | Verified |
|---|---|---|---|
| unset (fresh install) | render succeeds | stays ON (env untouched, producer default) | yes |
| unset (fresh install) | render falls back | breaker writes explicit "false", persists deParallelRouterTrialFired: true |
yes |
explicit "true" by user |
render falls back | applyDeParallelRouterBreaker() is a no-op (user-managed latch), env stays "true", router stays ON |
yes |
explicit "false" by user |
any render | userValueEnablesDeParallelRouter() returns false, router OFF, breaker never consulted |
yes |
breaker wrote "false" (prior process) |
user overrides with "true" |
user-managed latch fires, userValueEnablesDeParallelRouter() returns true, router ON |
yes |
Cross-process persistence: the in-process latch (deParallelRouterBreakerTrippedThisProcess) guards within a process; the on-disk flag (deParallelRouterTrialFired via readConfigFresh) guards across processes. Both paths converge on writing process.env.HF_DE_PARALLEL_ROUTER = "false". The persistDeParallelRouterTrialFired retry loop (3 attempts, verify-after-write) handles concurrent-writer clobbers. Correct.
Kill switch parsing
isDeParallelRouterEnabled (producer, exported, takes env param) and userValueEnablesDeParallelRouter (CLI, private, reads process.env) are deliberately duplicated with identical logic. Both:
trim().toLowerCase()
undefined or "" -> true (ON)
"false" | "0" | "off" | "no" -> false (OFF)
everything else -> true (ON)
Tested spellings in isDeParallelRouterEnabled tests: "false", "FALSE", "False", "0", "off", "OFF", "no", "No", " false " (padded), "", " " (whitespace-only), unset. All produce the correct result. The naive === "true" / !== "false" pitfall that motivated this work is properly eliminated.
SSOT note (reviewed, accepted): The duplication between producer and CLI is documented with a rationale (producer is lazily loaded, CLI needs this on the startup path). The comment names the producer copy as source of truth and says "keep the two in sync." Acceptable trade-off -- flagging it here so a future editor knows to update both.
Side-effect invariant
The breaker writes process.env.HF_DE_PARALLEL_ROUTER = "false" exactly once on the first fallback (in maybeConsumeDeParallelRouterTrial). Subsequent renders in the same process short-circuit at step 3 of applyDeParallelRouterCircuitBreaker (the in-process latch) and call applyDeParallelRouterBreaker which is an idempotent re-write of "false". The disk write (writeConfig) also fires only on the fallback render (subsequent renders return routerActive = false, so maybeConsumeDeParallelRouterTrial exits at line 1). Correct.
Race conditions
Single-process: JS event loop -- no data race possible. Module-level flags are shared across all renderLocal calls in one process, but the manageDeParallelRouterBreaker opt-in polarity ensures only sequential call sites (single render, batch concurrency 1) use the mechanism. Batch concurrency >= 2 leaves the flag unset, so those renders never touch the breaker state. Correct.
Cross-process: two processes both reading deParallelRouterTrialFired: false, both tripping, both writing true -- idempotent, no corruption. The render counter can lose an increment under a race (documented, benign -- it's telemetry, not a decision input anymore). persistDeParallelRouterTrialFired verify-and-retry handles the clobber case for the safety-critical fired flag. Correct.
25-render cap removal
DE_PARALLEL_ROUTER_TRIAL_MAX_RENDERS = 25 is deleted. The only trip condition is now outcome === "reverted". The test that verified the old cap is rewritten to run 30 successful renders and assert the breaker never trips. The comment block replacing the constant explains why: under a shipped default, a render-count cap would silently disable the feature after 25 good renders. Correct and clean.
The deParallelRouterTrialRenderCount field is still incremented on every consumed render -- this is telemetry bookkeeping, not a decision input. Fine to keep.
Standards
No bare as T casts in the diff (only as const for literal types in test fixtures). No non-null assertions. Function signatures are clean. Comments are thorough and accurate -- every "review finding" annotation traces back to a real design constraint.
CI
Preflight (lint + format): pass
Preview parity: pass
Preview regression: pass
Player-perf: pass
Regression shards (1-8): in progress at time of review
Findings
1. Misleading user-facing message when explicit opt-in overrides breaker (nit)
When a user explicitly sets HF_DE_PARALLEL_ROUTER=true and a render reverts, reportDeParallelRouterBreakerTrip prints:
A frame failed verification, so parallel drawElement capture fell back to the screenshot path and is now off for this install.
But the user's explicit "true" wins over the breaker (applyDeParallelRouterBreaker is a no-op when deParallelRouterUserManaged), so the router is NOT off. The message is factually incorrect for this path. Low severity (the user explicitly opted in and probably knows what they're doing), but worth gating the message on !deParallelRouterUserManaged so it only prints when the breaker actually took effect.
Additionally, reportDeParallelRouterBreakerTrip fires on EVERY subsequent revert for this user (since routerActive stays true via the user's explicit value), producing the same misleading message each time. The first-revert-only semantics that apply to the non-user-managed path don't apply here.
2. Missing test: user explicit "true" survives a breaker trip (nit)
The state transition "explicit true -> render reverts -> breaker can't override -> env stays true" is logically correct (I traced it above) but has no dedicated test. The existing tests cover user-set "false" not being overridden, breaker writing "false" when unset, and the kill switch parsing. A test like this would lock down the "explicit user choice wins in both directions" invariant the comments describe:
it("does not override an explicit user opt-in even after a fallback", async () => {
savedEnv.set("HF_DE_PARALLEL_ROUTER", process.env.HF_DE_PARALLEL_ROUTER);
process.env.HF_DE_PARALLEL_ROUTER = "true";
configState.disk = { /* fresh install */ };
producerState.executeImpl = async (job) => {
job.perfSummary = { drawElement: { parallelRouter: "reverted" } };
};
await renderLocal("/tmp/project", "/tmp/out.mp4", baseOptions);
expect(process.env.HF_DE_PARALLEL_ROUTER).toBe("true");
});3. events.test.ts -- gpu: false addition looks like a cross-PR fix (informational)
Two test calls to trackRenderComplete gain gpu: false. This is likely fixing a type error introduced by the base PR (#2838) that added a gpu property to the event payload. Not a problem, just noting it's a stacked-PR seam rather than a circuit-breaker change.
Verdict
Approve. The circuit breaker's state machine is correct and complete. The kill switch parsing closes the exact vulnerability the PR description identifies (naive !== "false" failing open for 0/off/no/FALSE). The 25-render cap removal is well-motivated and cleanly executed. The two nits above are low-severity edge cases -- finding 1 (misleading message for explicit opt-in users) is the only one I'd consider addressing before merge, and even that is a rare user path. The code is well-documented, the test coverage is strong, and the design trade-offs (deliberate duplication, latched user-managed detection) are clearly explained.
CI regression shards are still running. Confirm they pass before merge.
-- Miga
miguel-heygen
left a comment
There was a problem hiding this comment.
Additive review at exact head 8d30cdb9af — Miga covered the state machine, kill-switch spellings, and user-message/test nits. One gap remains in the boundary between the producer parser and CLI breaker.
Strengths
packages/producer/src/services/renderOrchestrator.ts:1332-1339centralizes default-on parsing and correctly handles the conventional OFF spellings.packages/cli/src/commands/render.ts:1185-1207preserves the persisted breaker across processes without consulting telemetry state.
blocker — set-but-empty enables the router but disables its circuit breaker
packages/cli/src/commands/render.ts:1172-1183 classifies any defined HF_DE_PARALLEL_ROUTER as user-managed (!== undefined). That disagrees with both parsers at render.ts:1148-1152 and renderOrchestrator.ts:1332-1339, where "" / whitespace explicitly mean “unset/default ON.”
Concrete failure: launch with HF_DE_PARALLEL_ROUTER= (or spaces), hit a verified fallback. The producer routes because empty means ON; applyDeParallelRouterCircuitBreaker returns true but latches deParallelRouterUserManaged = true; then applyDeParallelRouterBreaker() no-ops at render.ts:1154-1157. The install keeps retrying the failing router on later renders instead of receiving the advertised first-fallback protection. This is the central safety invariant of the PR.
Normalize before deciding ownership: empty/whitespace should remain breaker-managed, while a non-empty explicit value stays user-managed. Please pin it with a CLI-level regression that starts from HF_DE_PARALLEL_ROUTER="" (and whitespace), injects parallelRouter: "reverted", and asserts the env becomes "false" and the fired flag persists.
CI is still running eight regression shards; this finding is independent of that pending gate.
Verdict: REQUEST CHANGES
Reasoning: Default-on parsing is correct, but the CLI ownership latch exempts the parser's own empty/default state from the per-install breaker, leaving a documented default path unprotected after fallback.
— Magi
8d30cdb to
acdae81
Compare
james-russo-rames-d-jusso
left a comment
There was a problem hiding this comment.
Reviewed at 8d30cdb9.
Kill-switch parser at renderOrchestrator.ts:1333-1339 is clean — .trim().toLowerCase() up-front, then treats false | 0 | off | no as OFF, everything else including undefined/empty/whitespace as ON. Default-inversion trip parity with the breaker's process.env.HF_DE_PARALLEL_ROUTER = "false" write at render.ts:1156 — producer treats that as OFF via the same reader, so the trip signal reaches the router.
Persistence via ~/.hyperframes/config.json:deParallelRouterTrialFired written by persistDeParallelRouterTrialFired() at render.ts:1242-1251 with a fresh re-read + re-assert up to 3 times to survive concurrent clobber — solid pattern. In-memory latch (deParallelRouterBreakerTrippedThisProcess at render.ts:1101) set BEFORE the disk write so the decision holds if fs write fails. Test coverage is comprehensive across kill-switch spellings (renderOrchestrator.test.ts:1831-1850), 15 CLI-side scenarios (render.test.ts:700-1057) including 30-healthy-renders-no-trip, unwritable-config-in-process-latch, and concurrent-write-clobber-re-assertion.
Four concerns — none blocking, all worth thinking about before default-ON ships.
Blockers
None.
Concerns
1. "Trip only on real fallbacks" claim overstates what the predicate actually catches. The PR title says the breaker trips "only on real fallbacks" and the user-facing message at render.ts:1321-1323 says "A frame failed verification, so parallel drawElement capture fell back". But the trip predicate at render.ts:1297 is const fired = outcome === "reverted";, and "reverted" is stamped by renderOrchestrator.ts:2865 for the router-routed path on any shouldRetryViaPinnedFallback outcome (renderOrchestrator.ts:1475-1484 docstring 1442-1466) — OOM, worker crash, host-contention timeout, all trip the breaker permanently. The test at render.test.ts:893-909 ("worker crashed even after fallback") explicitly asserts this is intended.
Failure scenario: a user runs a large render on a hot machine, one worker gets OS-OOM-killed (not a router bug), retry succeeds via pinned fallback. Breaker persists trip=true. Every subsequent render on that install skips the router forever, even after the user upgrades RAM. Only manual HF_DE_PARALLEL_ROUTER=true or deleting ~/.hyperframes/config.json un-trips it.
Two paths to close: (a) narrow the trip predicate to just self-verify-failure (leave OOM/crash/timeout as one-render retry but not a permanent trip), or (b) update the PR title + user-facing message to accurately describe "any capture-stage retry via the pinned path" as the trip condition and consider automatic time-based un-trip.
2. One-shot deParallelRouterUserManaged latch clobbers mid-batch explicit overrides. render.ts:1078-1086 docstring claims "Their choice wins over the circuit breaker in BOTH directions." That's true only for the first applyDeParallelRouterCircuitBreaker call in the process. Programmatic wrapper scenario:
- Row 1: fresh env,
applyDeParallelRouterCircuitBreakerlatchesdeParallelRouterUserManaged = false. - Programmatic wrapper sets
process.env.HF_DE_PARALLEL_ROUTER = "true"between renders (documented atrender.test.ts:979-981as a legitimate scenario). - Row 2: latch skipped (
deParallelRouterUserManagedResolved === true), reads staleuserManaged = false, router runs. - Row 2 reverts →
applyDeParallelRouterBreaker→ writes"false"OVER the user's explicit"true"because the latch said false.
Not caught by any test — the test at render.test.ts:964-985 sets "false" (which matches what the breaker would write) and uses parallelRouter: "routed" (no revert, so the breaker write never fires). Fix: re-evaluate userValueEnablesDeParallelRouter(process.env.HF_DE_PARALLEL_ROUTER) at trip time in applyDeParallelRouterBreaker, not from a first-row latch.
3. CLI-copy of the kill-switch parser has no unit test. render.ts:1148-1152 (userValueEnablesDeParallelRouter) parses the same set of spellings as the producer, and the docstring at render.ts:1142-1147 warns: "Keep the two in sync — the producer copy is the source of truth." But only the producer copy has parser tests (renderOrchestrator.test.ts:1831-1850). A future refactor that adds "disabled" as a kill spelling to the producer but forgets the CLI copy would silently break trip-persistence for that spelling with no CI signal. Either add a parity test that imports both parsers and iterates the shared spelling list, or extract the parser to a shared module.
4. Disk-persisted trip is CLI-only. renderOrchestrator.ts:1333-1339 reads env only. Any future in-process or subprocess caller that invokes the producer via a path other than packages/cli/src/commands/render.ts bypasses the ~/.hyperframes/config.json trip check — that caller reads the router as default-ON regardless of prior CLI-side trips. Not exploitable today (no other caller), but the "per-install breaker" claim in the PR body is actually "per-install through the CLI." Worth noting explicitly in the docstring at render.ts:1120-1125 for future integrators (studio-server, aws-lambda, gcp-cloud-run).
Nits
- Stale docs:
packages/producer/tests/escape-hatch-fatal-fallback/src/README.md:33-35still describes the router asopt-in via HF_DE_PARALLEL_ROUTER=true. Update to reflect default-ON + kill-switch semantics. applyDeParallelRouterCircuitBreaker(options.quiet)atrender.ts:837—options.quietisboolean | undefined; when undefined,!quietinside is truthy, so the breaker's user message may print for programmatic callers who didn't explicitly setquiet. Non-critical since callers who opt in also set quiet inexecute.ts:105,255.- The
/tmp/pr-2840.diffwas cut against3da31e399(pre-2838), not against 2838's HEAD, so the diff file conflates 2838's default-flip inline with 2840's parser tightening — easy to misread on GitHub's stack view. Reviewers using the diff file locally should check thegit diff 6a50c9f42..HEADnumstat (24+/1- forrenderOrchestrator.ts) to see the 2840-alone shape.
Questions
- Is there an intentional reason NOT to auto-untrip after N healthy renders on a bumped-RAM/refactored machine? A 30-healthy-render green streak (already tested at
render.test.ts:987-1010) would be a natural signal. - Was
stop gating on telemetryreviewed with the dashboard-1807532 tripwire owner? Removing the telemetry gate means the "sustained revert >10%" signal now excludes the telemetry-opt-out cohort — the gate for that dashboard signal now has an unknown-size blind spot.
What I didn't verify
- Local end-to-end with a real config file — trusting the test coverage.
- Whether other consumers of
~/.hyperframes/config.json(studio, cli auth) could collide with concurrent writes to the trial-fired field during a batch render.
|
Fixed in `0fed335aa`. Thanks both — the blocker is a genuine hole and in the same class as the bug this PR exists to close, just through a different door. @miguel-heygen — blocker: set-but-empty exempts the install from its own breaker. Correct and confirmed. Ownership used `!== undefined` while both parsers normalize empty/whitespace to "unset → default ON", so `HF_DE_PARALLEL_ROUTER=` routed the render and latched `deParallelRouterUserManaged`, making `applyDeParallelRouterBreaker()` a no-op on fallback. The install then keeps retrying a failing router — losing precisely the first-fallback protection the PR is for. Ownership now uses the same normalization as the parsers: deParallelRouterUserManaged = (process.env.HF_DE_PARALLEL_ROUTER ?? "").trim() !== "";Pinned with the regression you asked for, parameterized over `""` and `" "`: both assert the env becomes `"false"` and the fired flag persists. Fault-injection verified — restoring @miga-heygen — nit 1: misleading message on explicit opt-in. Right, and the repeat-every-revert part is the worse half. The notice is now gated on the breaker having actually taken effect ( @miga-heygen — nit 2: missing explicit- @miga-heygen — nit 3: 69 CLI tests pass, typecheck/lint/fallow clean. Regression shards re-running on the new head. |
miguel-heygen
left a comment
There was a problem hiding this comment.
Re-reviewed at 0fed335aa25d3b9ac08e6b0f3dc7ea11369d634c against the head where I requested changes. The blocker is closed at the contract boundary: ownership now uses the same normalization as both router parsers (trim() !== ""), so absent, empty, and whitespace-only values remain breaker-managed while any explicit non-empty user choice still wins.
The regression coverage is executable rather than declarative: the parameterized empty/whitespace cases drive a real reverted render outcome and assert both the in-process false latch and the persisted fired flag. The explicit-true case locks the opposite polarity, and the trip message is now emitted only when the breaker actually took control. I ran the exact CLI render suite at this head: 69/69 passed. All current PR checks are green.
No blocking findings. The previously noted programmatic mid-process env-mutation/parser-duplication concerns remain follow-up-grade and are unchanged by this delta.
Verdict: APPROVE
Reasoning: The empty-value ownership mismatch is fixed using the parser contract itself, both default-like spellings and explicit override behavior are regression-pinned, targeted tests pass, and CI is green.
— Magi
The base branch was changed.
Adds a reusable staged-rollout primitive so a change can ship to a stable
slice of installs instead of all-or-nothing.
The gap it fills: the repo carries ~49 HF_*/PRODUCER_* booleans and every one
is binary — a feature is either off (and therefore unexercised on real
traffic) or on for everyone (and therefore a fleet-wide bet). The
parallel-drawElement router sat in that gap for weeks: default-off produced
almost no signal, and flipping it default-on would have exposed 100% of
eligible installs at once.
Shape:
- packages/core/src/canary.ts — pure evaluator. No fs, no network, no
`process`; the caller supplies the unit id and overrides, so it imports
cleanly into the CLI, producer, engine, studio-server, the browser-side
studio bundle and the embeddable player. FNV-1a rather than node:crypto for
the same reason.
- packages/core/src/canaryRegistry.ts — every rollout in one table (name,
percentage, owner, description, sunsetAfter), so "what is rolling out, to
whom, owned by whom" is answerable without grepping 49 env vars.
- packages/cli/src/telemetry/canary.ts — supplies the three things only the
CLI knows: anonymousId, the HF_CANARY_<FEATURE> override, and is_ci.
Day-to-day API is `isCanaryEnabled("name")`.
Three properties the tests pin, because getting them wrong is subtle:
- Slices are INDEPENDENT per feature: the bucket hashes `feature:unitId`, not
the id alone. Bucketing on the id would hand every concurrent experiment to
the same unlucky cohort and make two rollouts unreadable apart.
- Ramping is INCLUSIVE: `bucket < percentage`, so widening 10 -> 25 keeps the
original cohort and before/after comparisons survive the ramp.
- It fails CLOSED: no unit id, unknown name, or CI install means not enrolled.
A canary exists to bound blast radius, so "we don't know who this is" must
never mean "enrol everyone".
Registry entries also carry a sunset date, and a test fails once one is past
due — a canary that outlives its rollout is a permanent fork of the product
with none of the review a permanent fork would get.
Ships with de-parallel-router registered at 0%: inert, and ready to ramp in a
patch release once #2840 lands.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
miguel-heygen
left a comment
There was a problem hiding this comment.
Re-stamped exact head 0fed335aa25d3b9ac08e6b0f3dc7ea11369d634c. This is the same head I previously reviewed: empty/whitespace remains breaker-managed, explicit non-empty choices retain ownership, the persisted breaker semantics are pinned by regressions, and all exact-head checks are terminal green.
0fed335 to
0e5562c
Compare
|
Holding this until #2854 (canary rollouts) lands — sequencing decision, not a blocker on the code here. Status check while noting it:
One thing to settle before merge: this ships default-on with the breaker; #2854 registers My read is the latter: the soak evidence is a breaker result, not a cohort one — 0 of 141 installs reverted more than once on the floor-700 config (v0.7.78+, 30d), against 4 of 96 in the number quoted in the PR body. Refreshed figures, since the body's soak predates the floor change:
Two caveats worth reconciling before flipping the default: I can't reproduce the 975-install population (the poisoned pre-0.7.66 era had 1,733 installs at a 17.46% revert rate, uncomfortably close in magnitude), and every one of those 141 installs is a self-selected opt-in trial participant — default-on exposes everyone, which is the population the soak can't speak to.
|
VERDICT: WAITINGWindow is ~4 hours since Observations at n=1620 (0.7.92+, since 2026-08-04 00:00Z, PostHog Hyperframes proj 356858)
Flips (Check 3, per doc must be zero except HF_CANARY_ dev-toggle):* 3 installs flipped on Cross-surface (Check 4): not yet measured — deferred to next fire when Studio + CLI events have accumulated enough same-id pairs to be meaningful. Why not READY yet
Why not BLOCKED yet
Exact HogQL (re-runnable)Install-share + independence: WITH per_install AS (
SELECT distinct_id,
any(JSONExtractString(properties, '$feature/canary-calibration-10')) AS cal10,
any(JSONExtractString(properties, '$feature/canary-calibration-50')) AS cal50
FROM events
WHERE timestamp >= toDateTime('2026-08-04 00:00:00')
AND properties.cli_version >= '0.7.92'
AND JSONExtractString(properties, '$feature/canary-calibration-10') != ''
GROUP BY distinct_id
)
SELECT count() AS installs,
sum(if(cal10='true',1,0)) AS cal10_true,
sum(if(cal50='true',1,0)) AS cal50_true,
sum(if(cal10='true' AND cal50='true',1,0)) AS both_true
FROM per_installFlip count (Check 3, pre-attribution): WITH per_install AS (
SELECT distinct_id,
countDistinct(JSONExtractString(properties, '$feature/canary-calibration-10')) AS cal10_uniq,
countDistinct(JSONExtractString(properties, '$feature/canary-calibration-50')) AS cal50_uniq
FROM events
WHERE timestamp >= toDateTime('2026-08-04 00:00:00')
AND properties.cli_version >= '0.7.92'
AND JSONExtractString(properties, '$feature/canary-calibration-10') != ''
GROUP BY distinct_id
)
SELECT count() AS total_installs,
sum(if(cal10_uniq > 1, 1, 0)) AS cal10_flippers,
sum(if(cal50_uniq > 1, 1, 0)) AS cal50_flippers
FROM per_installFilters: Next fire: 2026-08-05 13:00 PT (20:00Z). Rechecking with ~24h more data. — Via
|
|
VERDICT: BLOCKED Second-fire canary calibration read for the Bottom line: Check 3 (stability) is dirty for the reason the doc pre-declared, and the defect is not standing still. Post-settled flip count has grown from 17 (this-morning's first read, recorded in Window & filters
Check 1 — Accuracy (install-weighted, non-CI,
|
| arm | installs enrolled | n_installs | share | Wilson 95% CI | target |
|---|---|---|---|---|---|
canary-calibration-10 |
1,495 | 15,674 | 9.538% | [9.09%, 10.00%] |
10% |
canary-calibration-50 |
7,860 | 15,674 | 50.147% | [49.37%, 50.92%] |
50% |
Both intervals contain the target — accuracy passes on install-share (the check that must land per the doc). Event-weighted skew is documented as a separate lens; install-share is authoritative.
Check 2 — Drift: not-failure-axis, per doc
Doc: "Check 2 will NOT come back flat, and that is expected rather than a defect. Per-install cohorts never flip, but a person who wipes their config gets a new id and a fresh roll." Alternatives (hardware fingerprinting, account identity) are explicitly rejected on record. Only two calendar days of window so far, so 1/7/14/30-day windows are not comparable yet; reporting deferred to a re-read after the decision-reason ships. Not a blocker.
Check 3 — Stability: DIRTY, real defect
All-install flip counts (matching the doc's methodology, no CI/version filter, so first-fire numbers are directly comparable):
SELECT
countDistinct(distinct_id) AS n_installs_any,
uniqIf(distinct_id, c10_flips) AS c10_flip_ids,
uniqIf(distinct_id, c50_flips) AS c50_flip_ids
FROM (
SELECT distinct_id,
countDistinct(JSONExtractString(properties, '$feature/canary-calibration-10')) > 1 AS c10_flips,
countDistinct(JSONExtractString(properties, '$feature/canary-calibration-50')) > 1 AS c50_flips
FROM events
WHERE timestamp >= toDateTime('2026-08-04 00:00:00')
AND timestamp < toDateTime('2026-08-06 00:00:00')
AND JSONExtractString(properties, '$feature/canary-calibration-10') IN ('true','false')
GROUP BY distinct_id
)| population | flips on canary-10 | flips on canary-50 |
|---|---|---|
| doc's first read (this morning; 9,702 installs) | 40 | 107 |
| this read (16,984 installs) | 98 | 246 |
Minority-persistence breakdown (per-flip: for each flipping install, span from first minority-value event to last minority-value event):
-- For calibration-10 (identical query re-used with '-50' for the other row)
WITH per_install AS (
SELECT
distinct_id,
countIf(JSONExtractString(properties, '$feature/canary-calibration-10') = 'true') AS n_true,
countIf(JSONExtractString(properties, '$feature/canary-calibration-10') = 'false') AS n_false,
minIf(timestamp, JSONExtractString(properties, '$feature/canary-calibration-10') = 'true') AS first_true,
maxIf(timestamp, JSONExtractString(properties, '$feature/canary-calibration-10') = 'true') AS last_true,
minIf(timestamp, JSONExtractString(properties, '$feature/canary-calibration-10') = 'false') AS first_false,
maxIf(timestamp, JSONExtractString(properties, '$feature/canary-calibration-10') = 'false') AS last_false
FROM events
WHERE timestamp >= toDateTime('2026-08-04 00:00:00')
AND timestamp < toDateTime('2026-08-06 00:00:00')
AND JSONExtractString(properties, '$feature/canary-calibration-10') IN ('true','false')
GROUP BY distinct_id
),
flipped AS (
SELECT
distinct_id,
if(n_true < n_false, n_true, n_false) AS minority_n,
if(n_true < n_false, dateDiff('second', first_true, last_true),
dateDiff('second', first_false, last_false)) AS minority_persist_sec
FROM per_install
WHERE n_true > 0 AND n_false > 0
)
SELECT
multiIf(minority_persist_sec <= 5, 'le_5s (first-mint shaped)',
minority_persist_sec <= 60, '5-60s',
minority_persist_sec <= 3600, '1-60min',
'gt_1h (post-settled)') AS bucket,
count() AS n_installs,
round(avg(minority_n), 1) AS avg_minority_events
FROM flipped
GROUP BY bucket
ORDER BY bucket| minority persists | canary-10 installs | canary-10 avg minority events | canary-50 installs | canary-50 avg minority events |
|---|---|---|---|---|
| ≤ 5s (first-mint shaped) | 30 | 3.0 | 75 | 2.7 |
| 5-60s | 23 | 9.1 | 48 | 9.1 |
| 1-60min | 38 | 15.7 | 84 | 12.6 |
| > 1h (post-settled) | 7 | 50.7 | 39 | 101.8 |
Post-settled totals: 7 + 39 = 46 installs sustained a minority assignment past the first-mint window, up from 17 in the doc's first read at 9,702 installs. Even scaled by population growth (16,984 / 9,702 = 1.75×), the pace is holding, not decaying — the effect is genuine, not first-mint noise.
Neither accepted exception (dev-toggled HF_CANARY_CALIBRATION_* override; the first-mint seed race) covers a flip that persists 100+ minority events over hours. Per the doc: "A flip that persists past that is not this race … By this check's own definition that is a real defect, not an accepted limit."
Check 3 bonus — Independence: PASS
-- Actual overlap vs product-of-shares, on the same non-CI 0.7.92+ population
SELECT
count() AS n,
countIf(c10 = 'true' AND c50 = 'true') AS both_true,
countIf(c10 = 'true') AS c10_true,
countIf(c50 = 'true') AS c50_true,
round(countIf(c10 = 'true' AND c50 = 'true') * 100.0 / count(), 3) AS overlap_actual_pct,
round(countIf(c10 = 'true') * countIf(c50 = 'true') * 100.0 / (count() * count()), 3) AS overlap_expected_pct
FROM (
-- same eligible + assignments CTEs as check 1
SELECT distinct_id,
any(JSONExtractString(properties, '$feature/canary-calibration-10')) AS c10,
any(JSONExtractString(properties, '$feature/canary-calibration-50')) AS c50
FROM events
WHERE distinct_id IN (SELECT distinct_id FROM eligible)
AND timestamp >= toDateTime('2026-08-04 00:00:00')
AND timestamp < toDateTime('2026-08-06 00:00:00')
AND JSONExtractString(properties, '$feature/canary-calibration-10') IN ('true','false')
GROUP BY distinct_id
)- n = 15,684 (non-CI,
>=0.7.92); both-true = 783 - Actual overlap = 4.992%; expected (
p1 × p2) = 4.785% - Delta = +0.207pp; z ≈ 1.22 (two-tailed p ≈ 0.22) — indistinguishable from independence at this window's power. First-read had 5.18% actual vs 4.71% expected (z=2.16); this read is closer to expectation, not further, so no cluster-shared-cohort signal.
Check 4 — Cross-surface (CLI vs Studio): sub-signal of check 3
SELECT
countDistinct(distinct_id) AS n_installs_both_surfaces,
uniqIf(distinct_id, cli_c10 = studio_c10) AS agree_c10,
uniqIf(distinct_id, cli_c10 != studio_c10) AS disagree_c10,
uniqIf(distinct_id, cli_c50 = studio_c50) AS agree_c50,
uniqIf(distinct_id, cli_c50 != studio_c50) AS disagree_c50
FROM (
SELECT distinct_id,
any(if(startsWith(event, 'studio'), NULL, JSONExtractString(properties, '$feature/canary-calibration-10'))) AS cli_c10,
any(if(startsWith(event, 'studio'), JSONExtractString(properties, '$feature/canary-calibration-10'), NULL)) AS studio_c10,
any(if(startsWith(event, 'studio'), NULL, JSONExtractString(properties, '$feature/canary-calibration-50'))) AS cli_c50,
any(if(startsWith(event, 'studio'), JSONExtractString(properties, '$feature/canary-calibration-50'), NULL)) AS studio_c50
FROM events
WHERE timestamp >= toDateTime('2026-08-04 00:00:00')
AND timestamp < toDateTime('2026-08-06 00:00:00')
AND JSONExtractString(properties, '$feature/canary-calibration-10') IN ('true','false')
GROUP BY distinct_id
)
WHERE cli_c10 IN ('true','false') AND studio_c10 IN ('true','false')- n = 2,325 installs emitting on both surfaces
- canary-10 CLI↔Studio disagreements: 12 (0.52%)
- canary-50 CLI↔Studio disagreements: 34 (1.46%)
Attribution note: these are a strict subset of the check-3 flips. 46 post-settled flips overall, only 46 total CLI↔Studio disagreements — so most flips (>90% of the ≤ 60min buckets and effectively all the 1-60min bucket) live within a single surface, not across the CLI-launched-Studio boundary. That is consistent with the doc's leading unproven hypothesis — "a shared distinct_id across containers that each hold their own seed" — where each container-launched CLI process is a distinct "surface" from telemetry's point of view but reports the same distinct_id.
Power
Doc target for a bucketer read: p=0.10, ±2pp at 95% CI ≈ 900 installs. This window's n=15,674 non-CI 0.7.92+ installs clears that ceiling by ~17×; neither VERDICT: WAITING nor a "sample too small" reservation applies. Every observation above is at full power for its own check.
What blocks
Per the doc's "What passing looks like, and what cannot be fixed" — Checks 1 and 4 are properties of the design, and Check 3 should pass apart from the first-mint race. Check 3 does not pass, the persisting-minority flips are not first-mint shaped (mean 50.7 / 101.8 minority events, over 1+ hours), and the count is growing at the second read rather than settling. Doc-mandated next step: "emit the decision reason (forced_on / forced_off / in_cohort / out_of_cohort) so a deliberate override can be told apart from a genuine flip, then re-read this table."
Not proposing a fix in this comment (out of scope for a measurement read). PR #2840 stays held. When decision-reason emission ships and the table is re-read, if Check 3 passes, #2840 can be wired to a new canary-de-parallel-router and ramped 5 → 25 → 100 while watching revert-rate broken by cpu_count and is_docker (the two populations the current soak doesn't cover — ≤4-CPU at 13.2%, Docker at 6.2%).
Cron cron_1c411937 cancelled after this read.
— Via
⚠️ Superseded 2026-08-07 — this BLOCKED verdict no longer holdsDo not treat this comment as blocking. Two of its load-bearing claims did not survive attribution.
1. "The defect is not standing still" (17 → 46 post-settled flips) was a population effect, not a rate change. The rate has been flat across three reads at growing n: 0.14% → 0.12% → 0.10% of installs. The raw count grew because the fleet on
0.7.92+grew. Comparing counts across windows with different denominators is what produced "growing".2. Roughly 45% of those flips were never a bucketer defect. Once
canary_reason_*shipped in0.7.96, the sustained flips split cleanly: 32 of 71 showedfalseevents that are exactly the CI ones — the documentedexclude: is_cirule meeting adistinct_idused both in and out of CI. The flip query had not excluded CI, the same trap that produced the false accuracy failure two comments up.3. The innocent explanation this comment could not rule out is now ruled out. Zero
forced_on/forced_offacross every flip — no dev was togglingHF_CANARY_*.Where it actually landed: ~14 genuinely unexplained sustained flips out of 13,547 installs (0.10%). That is an order of magnitude below this feature's own 2.79% revert rate, so cohort instability cannot meaningfully corrupt a ramp read.
The gating standard was also wrong in kind. The doc said check 3 "MUST be zero", and this comment applied it literally. A 0.10% stable edge does not justify blocking a 5% ramp that has a per-install breaker underneath it — the useful question is "can a ramp be trusted", not "is the number zero".
The remaining 38 (across both canaries) are a real open question and worth explaining, but they are not a blocker. See the PR description.
Review fixes. BLOCKER: the doc quoted two different independence results for the same window — the current table (5.03%, z = 1.06) and a stale closing paragraph (5.18%, z = 2.16) from the earlier iteration. Deleted the paragraph rather than updating it: the table already carries independence in the same shape as the other primary results, and 'worth re-checking rather than acting on' no longer fits at z = 1.06. It survived because an earlier edit re-wrapped the line and a later string replace silently missed. The reason enum has six wire values, not the four that carry attribution: no_unit_id and excluded also land on real events, and excluded is what CI installs emit — useful, since dropping them currently needs a join on is_ci. Points at CanaryReason as the source of truth. Also records that reason alone cannot settle the container hypothesis — it reports the decision on both sides of a flip, not why the seeds diverged — and names what would: bucket_seed_origin plus a machine fingerprint. Notes that identity_persistence already ships and may corroborate it first. Adds query provenance so the table can be reproduced, and states that the #2840 hold is a sequencing decision rather than a code-enforced gate: de-parallel-router is registered at 0% and unwired, and the live gate is the env var plus the circuit breaker. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
de_parallel_router is present on 95.4% of render_complete events and 0.83% of render_error. Capture context itself survives failures fine (capture_mode is on 98.6% of them), so this is not renders failing before capture — the routing state specifically is being dropped. Cause is ordering. deParallelRouter is assigned twice: once before the capture-observability update, and again inside syncCapturePlan where routing is actually resolved — including the 'reverted' case, which the earlier assignment cannot know. The update in between recorded whatever was true first, so a render that failed while routed reported no routing state at all. The existing comment at the earlier call site says it is recorded there precisely so hard failures carry it; that intent was correct and the value just arrived too late. This matters for the heygen-com#2840 ramp specifically. The per-install circuit breaker only arms on a revert, which requires the render to finish and self-detect — it cannot catch a crash or hang. Those are exactly the failure modes a percentage ramp exists to bound, and they were the ones telemetry could not see. Also makes the ffprobe contract sweep resilient per entry. A dangling symlink under packages/studio/data/projects threw ENOENT on stat and aborted the whole traversal, so every package sorting after 'studio' — both studio-server callers included — silently stopped being checked. main is currently red on this. The manifest assertion is what caught it, which is what it was added for. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…r-install circuit breaker The DE parallel router (HF_DE_PARALLEL_ROUTER) becomes default-ON. The soak answered the safety question it was gated on: zero damaged frames shipped — every fallback was the self-verification net catching a bad frame and recovering on the screenshot path. Verify PSNR p10 sits flat near 40 dB against a 32 dB floor. The residual 2.31% revert rate is an efficiency cost (a revert forfeits the speedup, never the output), accepted in exchange for parallelizing the >=700-frame band — roughly 80% of all DE capture wall-clock, frame-weighted. Default-ON is safe because the per-install circuit breaker stays underneath it. That distinction matters: 9.8% of installs hit a revert, and they are latched off permanently after the first one. Without the breaker those installs would go from "one slow render, then protected" to "every eligible render is slow". The breaker, adapted for a default-ON flag: - Writes an explicit HF_DE_PARALLEL_ROUTER=false and persists it to ~/.hyperframes/config.json, so the install stays off across processes. Absent no longer means off, so the switch has to be written, not unset. - Trips only on a real fallback, never on render count — a healthy install keeps the speedup indefinitely. - Independent of telemetry state: opting out of analytics must not cost a user the faster renderer. Telemetry governs reporting, not behavior. - An explicit user value wins in both directions, latched before the breaker can write the var and make the two indistinguishable. - The user is told when it trips and how to re-enable. isDeParallelRouterEnabled() parses the kill switch properly: false/0/off/no (case- and space-insensitive) disable; unset or empty is the default. A bare `!== "false"` would silently ignore every spelling but one and hand parallel DE to a user who asked for none. Refs PRINFRA-384 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Ownership detection classified ANY defined HF_DE_PARALLEL_ROUTER as a user choice, but both parsers read empty/whitespace as "unset -> default ON". Launching with `HF_DE_PARALLEL_ROUTER=` therefore routed the render (empty parses as ON) while exempting the install from its circuit breaker: after a verified fallback applyDeParallelRouterBreaker() no-op'd, so the install kept retrying the failing router instead of latching off. That is the exact first-fallback protection this PR exists to provide, lost on a documented default path. Ownership now uses the same normalization as the parsers. Also: only announce a trip the breaker could act on. With an explicit user opt-in the breaker is deliberately a no-op, so "now off for this install" was factually wrong — and reprinted on every later revert, since the user's value keeps the router active. Tests: set-but-empty and whitespace both latch off and persist the fired flag (fault-injection verified — restoring the old check fails both); explicit "true" survives a fallback. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Rebased onto main (was 308 behind) and gated the new default-on behaviour on the de-parallel-router canary, at 5%. Default-ON without a ramp is a ~17x exposure jump: from ~6% of eligible renders today to all of them, landing on profiles the opt-in trial never covered (<=4 CPUs and Docker, ~12% of eligible renders between them). 0.7.60-0.7.64 is why that matters — every unclamped render reverted for five consecutive releases and nobody noticed. The gate reuses the breaker's own disarm: non-enrolled installs get an explicit HF_DE_PARALLEL_ROUTER=false, because with default-ON polarity deleting the var means ON. Setting the registry percentage to 0 is therefore a full fleet-wide revert with no release. Today's ~11% of installs routing is emergent — the product of eligibility rules and a capped trial — so it drifts with fleet composition and cannot be turned off without shipping. The point of the canary is that the number becomes chosen and revertible, not that it is smaller. Also replaces the registry test that pinned the percentage to 0. Its intent was 'ramp only alongside the circuit breaker', but pinning 0 blocks the ramp forever and never checks the wiring it names. It now asserts the wiring directly, and fails if either the canary gate or the breaker consult is removed. Hold at 5% until PRINFRA-372 resolves: --workers auto crashes every worker on macOS arm64 while --workers 1 is clean, and the router forces 3 workers. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
0e5562c to
4a25142
Compare
|
Rebased onto main (was 308 behind) and added the canary ramp. New head Why the ramp. This PR's own change is the risky part: default-ON takes routed renders from ~6% of eligible to all of them, a ~17x jump, onto profiles today's trial population never covered — ≤4 CPUs and Docker, ~12% of eligible renders between them. 0.7.60–0.7.64 is the precedent: every unclamped render reverted for five consecutive releases and nobody saw it. The gate reuses this PR's own disarm mechanism — non-enrolled installs get an explicit Worth being precise about what the canary buys, because I got this wrong at first: today ~10.85% of rendering installs already route, automatically. That number is emergent — the product of eligibility rules, the 25-render cap and the breaker latch — so it drifts with fleet composition and can't be switched off without shipping. The canary doesn't make exposure smaller; it makes it chosen and revertible. Telemetry supporting the shape of this PR (14d, ≥0.7.94, routed renders):
~3% of routed renders revert and 86% of those are PSNR — self-verify catching real visual damage. That is the per-render protection, and it is separate from the breaker. Keeping the breaker (as this PR does) while dropping the cap is the right split: the cap is sampling logic, the breaker is the only thing stopping a damaged install from being damaged repeatedly. Why the breaker should not also be removed yet. We cannot measure whether it is safe to remove — the breaker censors the data. "154 of 155 reverting installs stop at exactly one revert" is the latch, not a natural rate. Measuring the true repeat-revert rate needs the breaker disabled for a slice, which this canary now makes possible. Hold at 5% until PRINFRA-372 resolves. Verification: core 1692, cli 2487 passing, lint clean. Gate mutation-tested at both layers — removing it fails 2 CLI tests and the core wiring assertion. One pre-existing failure on main ( At each ramp step, split revert rate by
|
It is not a user opt-in — execute.ts arms it automatically on the CLI render path, so ~11% of installs already route without anyone choosing it. The opt-in is at the CALL SITE: the flag defaults off and only the two CLI sites set it, excluding programmatic renderLocal consumers because the mechanism mutates process.env. That polarity guards embedding contexts, not users. Calling it opt-in understates today's exposure, which changes how a reviewer judges the ramp: it is not protecting users from a feature they chose, it is governing exposure already happening without their choice. Leaves the accurate uses alone — 'explicit user opt-in' means someone setting HF_DE_PARALLEL_ROUTER themselves, and the call-site flag is genuinely opt-in. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…anary gate Deletes the `de-parallel-router` canary entry and the `isCanaryEnabled` guard in render.ts together, leaving the producer's default-ON in place. Net effect for users: the parallel drawElement router is on for everyone again. ## Why, and why not a ramp Gating at 5% was itself the regression. Measured 2026-08-08, the day after v0.7.101 shipped the canary: fleet router exposure fell from 3.13-4.25% of non-CI renders to **0.13%**, roughly 25x, because out-of-cohort installs are explicitly disarmed and #2840 deleted the everyone-armed trial in the same change. 2,537 installs lost a feature they already had. Severity is speed only, never output, and nothing is persisted to disk. PR #2840's body claimed "the canary does not make exposure smaller; it makes it chosen and revertible." That was true of the end state and false of the first step. This lands the end state. Entry and guard go together deliberately: at >=100 the evaluator short-circuits ahead of the CI/seedless exclusions, so removing only the entry would have flipped whatever still resolved false at deletion time, unstaged. ## Both stated blockers are void - **≤4-CPU / Docker coverage gap.** Docker renders never use drawElement — 0 of 4,281 across every CPU tier, software GL gates it out — and the router requires it. No percentage could ever expose Docker, so no ramp closes that gap. ≤4 CPUs yields ~42 drawElement candidates in three days. - **PRINFRA-372.** Its signature has hits on 0.4.12, 0.4.37, 0.6.52, 0.6.93, 0.6.109 and 0.6.110 — versions predating drawElement (v0.7.38) and therefore this router. It is real, still live on 0.7.101, and belongs to the screenshot/beginframe path. 11 reproduction runs across four configurations on the enriched profile (darwin/arm64 25.5.0) came back clean. ## Safety unchanged The per-install circuit breaker and the per-render self-verify are untouched; `HF_DE_PARALLEL_ROUTER=false` remains the user-facing kill switch. Post-canary data at 14 days: >8 CPUs 3.02% revert (177/5,857), 5-8 CPUs 2.40% (6/250) — consistent with the 2.75-3.16% baseline. Revert path is now a code revert rather than a registry edit. That is the trade this shape accepts in exchange for one release instead of two. ## Corrects two claims that shipped wrong `~17x jump in exposure onto <=4 CPUs / Docker` overstated the reach, and `~11% of installs already route` was an OUTCOME (the share clearing eligibility and the old 25-render cap), not an exposure setting — read as a rollout knob it inverts the arithmetic, which is how gating at 5% came to cut exposure rather than ramp it. Both are recorded in render.ts so they are not reintroduced. ## Tests Removed the core wiring assertion and the two CLI canary-gating tests, which pinned a gate that no longer exists. Added the inverse guarantee in its place: an ordinary install must come out of the breaker with the var UNSET so the producer default applies — writing "false" there is precisely what disarmed the fleet at 5%. core 1701 passing, cli 2491 passing, studio canary 29 passing. The 2 failures in play.test.ts reproduce on clean origin/main and are unrelated (#3114 area). oxlint and oxfmt clean. Note: telemetry for this rollout stops with the entry — `$feature/canary-de-parallel-router` and `canary_reason_de_parallel_router` are emitted from the registry, so the `Ramp —` tiles and the exposure-floor alert on PostHog dashboard 1918875 go blank once this ships. Watch drawElement engagement on 1807532 instead. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…anary gate (#3120) Deletes the `de-parallel-router` canary entry and the `isCanaryEnabled` guard in render.ts together, leaving the producer's default-ON in place. Net effect for users: the parallel drawElement router is on for everyone again. ## Why, and why not a ramp Gating at 5% was itself the regression. Measured 2026-08-08, the day after v0.7.101 shipped the canary: fleet router exposure fell from 3.13-4.25% of non-CI renders to **0.13%**, roughly 25x, because out-of-cohort installs are explicitly disarmed and #2840 deleted the everyone-armed trial in the same change. 2,537 installs lost a feature they already had. Severity is speed only, never output, and nothing is persisted to disk. PR #2840's body claimed "the canary does not make exposure smaller; it makes it chosen and revertible." That was true of the end state and false of the first step. This lands the end state. Entry and guard go together deliberately: at >=100 the evaluator short-circuits ahead of the CI/seedless exclusions, so removing only the entry would have flipped whatever still resolved false at deletion time, unstaged. ## Both stated blockers are void - **≤4-CPU / Docker coverage gap.** Docker renders never use drawElement — 0 of 4,281 across every CPU tier, software GL gates it out — and the router requires it. No percentage could ever expose Docker, so no ramp closes that gap. ≤4 CPUs yields ~42 drawElement candidates in three days. - **PRINFRA-372.** Its signature has hits on 0.4.12, 0.4.37, 0.6.52, 0.6.93, 0.6.109 and 0.6.110 — versions predating drawElement (v0.7.38) and therefore this router. It is real, still live on 0.7.101, and belongs to the screenshot/beginframe path. 11 reproduction runs across four configurations on the enriched profile (darwin/arm64 25.5.0) came back clean. ## Safety unchanged The per-install circuit breaker and the per-render self-verify are untouched; `HF_DE_PARALLEL_ROUTER=false` remains the user-facing kill switch. Post-canary data at 14 days: >8 CPUs 3.02% revert (177/5,857), 5-8 CPUs 2.40% (6/250) — consistent with the 2.75-3.16% baseline. Revert path is now a code revert rather than a registry edit. That is the trade this shape accepts in exchange for one release instead of two. ## Corrects two claims that shipped wrong `~17x jump in exposure onto <=4 CPUs / Docker` overstated the reach, and `~11% of installs already route` was an OUTCOME (the share clearing eligibility and the old 25-render cap), not an exposure setting — read as a rollout knob it inverts the arithmetic, which is how gating at 5% came to cut exposure rather than ramp it. Both are recorded in render.ts so they are not reintroduced. ## Tests Removed the core wiring assertion and the two CLI canary-gating tests, which pinned a gate that no longer exists. Added the inverse guarantee in its place: an ordinary install must come out of the breaker with the var UNSET so the producer default applies — writing "false" there is precisely what disarmed the fleet at 5%. core 1701 passing, cli 2491 passing, studio canary 29 passing. The 2 failures in play.test.ts reproduce on clean origin/main and are unrelated (#3114 area). oxlint and oxfmt clean. Note: telemetry for this rollout stops with the entry — `$feature/canary-de-parallel-router` and `canary_reason_de_parallel_router` are emitted from the registry, so the `Ramp —` tiles and the exposure-floor alert on PostHog dashboard 1918875 go blank once this ships. Watch drawElement engagement on 1807532 instead.
Correcting the record on this PR (2026-08-09)Three claims in the body above were falsified by post-release measurement. 1. "The canary does not make exposure smaller. It makes it chosen and revertible." False as shipped. At 5% it made exposure ~25x smaller: fleet router The claim was accurate about the destination and wrong about the first step. 2. "~17x exposure jump onto <=4 CPUs (8.18%) and Docker (3.99%)." Overstated the reach. Docker renders never use drawElement at all — 0 of 3. "Hold at 5% until PRINFRA-372 resolves... the router forces 3 workers." Unsupported. The Also worth noting: the "~11% of installs already route today" figure is an What still standsThe kill-switch parse fix ( Soak numbers held up: post-canary at 14 days, >8 CPUs 3.02% revert |
What
The parallel drawElement router becomes the default for eligible renders — ramped through the canary at 5%, not flipped on for everyone. Three things ship together:
isDeParallelRouterEnabled()parsesfalse/0/off/no(case- and space-insensitive); unset or empty means default. The old!== "false"failed open on every spelling but the literal one."false"instead ofdelete-ing the var — under default-ON polarity, deleting means ON, so the breaker was a no-op on exactly the hosts it exists to protect.percentage: 0in the registry is a full fleet-wide revert with no release.Also removes the 25-render cap and the telemetry preconditions (see below).
Why the ramp
Default-ON without one is a ~17× exposure jump — from ~6% of eligible renders to all of them — landing on profiles today's trial population never covered: ≤4 CPUs (8.18% of eligible renders) and Docker (3.99%). ("Trial" is not a user opt-in — it arms automatically on the CLI render path. The
enableDeParallelRouterTrialflag is call-site opt-in, excluding programmaticrenderLocalconsumers because it mutatesprocess.env. So ~11% of installs route today without anyone choosing it.) 0.7.60–0.7.64 is the precedent: every unclamped render reverted for five consecutive releases and nobody noticed.Worth being precise about what the canary buys, because it is easy to state wrongly: ~10.85% of rendering installs already route today, automatically. That number is emergent — the product of eligibility rules, the render cap and the breaker latch — so it drifts with fleet composition and cannot be switched off without shipping. The canary does not make exposure smaller. It makes it chosen and revertible.
Current soak (14d, ≥0.7.66)
154 of 155 reverting installs stop at exactly one — the breaker working. Held steady as the population tripled from 434 installs, so that is a property of the mechanism, not a sample artifact.
What reverts actually are (routed renders, same window):
86% of reverts are the self-verify catching real visual damage. That is a per-render mechanism and is separate from the breaker — worth keeping straight, because it means removing the breaker would not risk corrupt output, only repeat cost.
What is removed, and why that is safe
The breaker stays. Not caution — measurement. We cannot yet tell whether removing it is safe, because it censors the data: "154 of 155 stop at one revert" is the latch, not a natural repeat rate. Now that the canary gate exists, disabling the breaker for a slice is the honest experiment, and that is a follow-up.
Ramp discipline
Hold at 5% until PRINFRA-372 resolves:
--workers autocrashes every worker ("detached Frame" / "Target closed") on macOS arm64 while--workers 1is clean — and the router forces 3 workers. PRINFRA-339 (indefinite stalls) and PRINFRA-354 (silent hang) are also open. None of those emitrender_complete, so neither the self-verify nor the breaker catches them — that is precisely the blast radius a ramp bounds.At each step (5 → 25 → 100) split revert rate by
cpu_countandis_docker. Baseline: routed renders fail at 2–3.4%, stable across versions.The bucketer was validated before ramping — calibration at n=13,547: 9.62%/49.76% against 10%/50% targets, overrides and CI both attributable, sustained cohort flips at 0.10%, an order of magnitude under this feature's own revert rate.
Also replaces a test that could never pass
canary.test.tspinnedde-parallel-routerto 0% with the note "ramp only alongside the per-install circuit breaker". Pinning 0 blocks the ramp permanently and never checks the wiring it names. It now asserts the wiring directly — a non-zero percentage is allowed only while the CLI render path gates on this canary and still consults the breaker.Verification
core 1692, cli 2487 passing. Lint clean. Fault-injection: restoring the
deletefails 3 tests, restoring the naive parse fails 1, removing the canary gate fails 2 CLI tests and the core wiring assertion.Rebased onto main (was 308 behind; #2838 has since merged). #3095 closed as superseded — its useful piece is folded in here.
One pre-existing failure on main, not from this branch:
audioPadTrim.realmediatimes out at 5s, reproduced on cleanorigin/main.Tracked: PRINFRA-384.
🤖 Generated with Claude Code