Idempotency follow-ups: atomic keyed start(), post-completion dedupe, and attribute-based run lookup

[pranaygp]

Follow-up to the hook-based run idempotency work in #2015, #2373, and #2011. Those PRs ship the in-flight story: deterministic hook tokens as run idempotency keys, hook.getConflict() resolving with the conflicting Run, and code-driven conflict-handling strategies (reject, adopt result, inspect, signal via resumeHook(), supersede via cancel()). This issue tracks the structural gaps that remain before the idempotency story is rock-solid.

Where we stand vs. comparable frameworks

Idempotency is a lifecycle; each framework answers four questions:

Phase	Temporal	Inngest	DBOS	Workflow (today)
Admission (dedupe atomic with start?)	Yes — workflow ID enforced server-side at start	Yes — idempotency key dedupes scheduling	Yes — workflow ID is effectively a DB primary key	No — start() always creates a run; the claim happens inside the run body
In-flight conflict policy	Enum: Fail / UseExisting / TerminateExisting	None (first wins)	Implicit UseExisting	Code — getConflict() hands the duplicate the owner's Run
Post-completion memory	RejectDuplicate, bounded by namespace retention	Fixed 24h TTL per key	Durable record	None — hook released at terminal state
Result reuse	No (rejects; caller queries the closed run)	Yes, within window	Yes — same ID returns stored result	Only while the owner is running (conflict.returnValue)

Where we're ahead: in-flight expressiveness. Policy-as-code beats a static enum ("inspect the owner's status, then decide" or "forward this request's payload to the owner" are inexpressible as configuration). And because the duplicate run is itself durable, the conflict-handling logic gets retries, replay, and observability — in admission-time systems that logic lives in a crashable client.

Where we're behind: admission atomicity and post-completion memory. Notably, no framework offers unbounded memory — Temporal's reject-duplicate is retention-bounded, Inngest is 24h — so the target is a retention window, not forever.

Gaps

1. Admission isn't atomic (root cause of the rest). The duplicate run is created, billed, queued, and executed before discovering it's a duplicate, and routes need the resume-with-retry dance to bridge the start() → hook-registration window. The docs already note a native atomic start-and-hook-registration API is planned.
2. No post-completion dedupe. The hook is a lease and the lease dies with the run. A retried request arriving seconds after the owner completes starts fresh duplicate-sensitive work. (Temporal: retention-bounded reject; DBOS: returns the stored result.)
3. Attributes are writable but not queryable. Runs can set (experimental_setAttributes) and even seed (CreateWorkflowRunParams.attributes) attributes, but ListWorkflowRunsParams only filters by workflowName/status — nothing can find a run by attribute, so attributes can't yet serve as post-completion memory.
4. Optimistic-strategy races (acceptable, but inherent). Supersede (cancel-and-reclaim) can lose the reclaim to a third arrival (ABA-shaped; the documented retry loop handles it); signal-the-owner can hit HookNotFoundError if the owner completes mid-forward.
5. Flat token namespace. order:123 collides across unrelated workflows sharing a key scheme. Same property as Temporal; a documented prefix convention is probably sufficient.

Recommendations (in order)

1. P0 — atomic keyed start() with { run, created } return semantics and a retention-bounded uniqueness window. Closes gaps 1 and 2 at once and yields DBOS-style result reuse (created === false + completed → run.returnValue) while keeping policies in code: no policy enum — the caller inspects the existing run and decides, which subsumes both of Temporal's knobs (conflict policy + reuse policy).
2. P1 — runs.list attribute filtering in the World contract, then document the attribute pattern as the post-completion bridge: hook claim = in-flight mutex; attribute (idempotency: <key>) = retention-bounded memory; a duplicate that wins the token after the owner finished queries completed runs by attribute in a step and adopts the prior result via getRun(prior.runId).returnValue. Must be documented as advisory (the query and subsequent work aren't atomic — residual race in the just-completed window) and retention-bounded. Stays useful after keyed start lands, for richer queries.
3. Document the two patterns that work today so users aren't stranded: the entity pattern (a long-lived run per key looping for await on its hook — strict serialization at the cost of one perpetual run per key and deployment pinning) and the app-record pattern (store runId under the domain key in your own DB inside a step; replays resolve via getRun).
4. Non-goal: a standalone lease/TTL primitive or a dispose: false hook option. A claim that outlives its run is a leak generator; keyed start with a retention window does the same job with better semantics.

🤖 Generated with Claude Code

[pranaygp]

Cross-referencing this against the patterns registry in #1858, which ships six coordination components (semaphore, rate-limiter, circuit-breaker, debounce, batch-aggregator, singleton-run) plus kill-switch, child-workflows, and recurring-cron — all built on hook channels and deterministic tokens. They turn out to be a useful stress test: every gap in this issue shows up as a concrete contortion or race in pattern code, and the patterns surface two gaps this issue doesn't cover yet (bottom).

How the gaps manifest in the patterns

Gap 1 (admission isn't atomic) is the sendWithEnsure dance, hand-rolled five times: resume(token) → catch → start(coordinator) → setTimeout(500) → retry ×3. The 500ms wall-clock sleep exists purely to bridge the start → hook-registration window. It also forces a design contortion: coordinators must take no initial payload via start() args (the natural start(debounce, [key, firstPayload]) is unsafe because the conflict-loser run would drop the payload), so first payloads travel via resume like everything else.

Gap 1's cost side (duplicate runs are created/billed/queued before discovering they're duplicates) is the documented caveat in singleton-run: "a failed duplicate with HookConflictError in the run list is the mechanism working, not a bug." getConflict() (#2373) converts that into a clean return { dedupedTo: conflict.runId } — and awaiting it commits registration early, shrinking the same window the 500ms sleep papers over.

Gap 2 (no post-completion dedupe) is live in the stripe dunning pattern: a payment_failed webhook redelivery arriving after the run completed starts a fresh dunning timeline. The hook claim can't help — it died with the run.

At-least-once start()/resumeHook() from steps (adjacent to gap 1, worth naming explicitly):

• child-workflows' spawn step can start a child twice if the step crashes between the start() call and its recording. The parent consumes one completion; the duplicate's side effects still ran. Keyed start() with key = completionToken makes spawns exactly-once.
• recurring-cron's continue-as-new step has the same duplication — which would fork the schedule into two parallel loops forever. It's only saved because the successor's stop-hook token is generation-keyed (cron:<name>:<iteration>), so the duplicate dies on HookConflictError. That a self-perpetuating duplication is prevented accidentally, by a token chosen for a different purpose, is a good argument for keyed start being P0.

Per-pattern severity

Pattern	Sharpest edge today	Fixed by
batch-aggregator	resume ack'd then dropped during the flush window (see below); double-count from retried sender steps	channel close+drain / resume keys — not yet in this issue
child-workflows	duplicate child spawn on step retry	keyed start()
kill-switch	probe-then-start TOCTOU: losing caller keeps the loser's runId → its .signal never fires	re-probe after start (userland); keyed start (proper)
recurring-cron	schedule forking, accidentally guarded by the generation token	keyed start per generation
stripe	post-completion webhook redelivery restarts dunning	retention-window dedupe
singleton-run + all coordinators	loser-run noise, registration window	getConflict() now; keyed start / resumeOrStart properly
semaphore / rate-limiter / circuit-breaker	benign — waiter timeout + dispose() self-healing is sound, just adds latency	resumeOrStart removes the dance

Two gaps to add

1. Channel close + drain. A coordinator that decides to terminate (e.g. the aggregator flushing at MAX_ITEMS) has no way to atomically stop accepting resumes and then consume what's already buffered. Today an item resumed between the coordinator's last await hook and run termination is acknowledged to the sender and then silently dropped — the buffered payload queue dies with the run. The window is the full duration of the flush step. dispose() is close but inverted: it stops future events without exposing what's already buffered, and there's no non-blocking way to poll the buffer. Something like hook.close() → subsequent resumeHook rejects (senders re-ensure) → hook.drain()/iterator-until-empty yields the already-accepted backlog.

2. Idempotency keys on resumeHook(). A send from inside a step is at-least-once: if the step crashes after the resume lands, the retry delivers the payload again, and channel-consuming patterns (aggregator counts, semaphore acquires) see duplicates. Userland mitigation is carrying a dedupe ID in the payload and deduping in coordinator state — which works but every consumer must remember to do it. A resumeHook(token, payload, { idempotencyKey }) deduped server-side (retention-bounded, same window as keyed start) is the hook-delivery analog of the post-completion run dedupe this issue already tracks. The natural key exists for free in steps: stepId (+ a counter for multi-send steps), mirroring the documented step-idempotency guidance.

On the proposed atomic resumeOrStart

Emphatic +1 — it's the single missing primitive behind five of the six coordinators; every send* helper in #1858 is a racy userland approximation of it. One spec nuance from building those: it must deliver the payload exactly once whichever branch wins — a freshly started run receives the triggering payload as its first channel message, atomically with creation. That's strictly stronger than composing keyed-start + retry-resume in userland (which still has the registration window), and it's what lets coordinators accept initial payloads naturally again. Framing: keyed start() is the admission face, resumeOrStart is the delivery face of the same primitive.

🤖 Generated with Claude Code

[pranaygp]

Empirical reproduction: CI hit the gap-1 × gap-2 interaction on every slow runtime

#2387 (e2e coverage for the documented conflict-handling strategies) accidentally produced a measured reproduction of gaps 1 and 2 interacting, in its first CI run (27444182929): the adopt-the-owner's-result test timed out at exactly 90s on every slow matrix entry — nextjs-turbopack/webpack (all variants), sveltekit, nuxt, astro local-dev, and both postgres jobs — while passing locally every time.

The trace:

1. start(owner) → owner registers the hook (verified) → start(duplicate) → test resumes the owner once.
2. Owner completes; token released (gap 2: no memory that the work happened).
3. The duplicate's first invocation finally lands — its claim is evaluated now, not at start() time (gap 1) — finds the token free, and becomes a legitimate fresh owner.
4. It awaits a payload that was already consumed by the previous owner → stranded run, 90s timeout.

Two takeaways worth recording here:

• The window is not theoretical and not small. Locally the start→claim gap is sub-millisecond, so the duplicate always conflicts. On CI dev servers the gap was routinely wide enough to fit an entire owner lifecycle (registration → resume → completion → disposal) inside it — on every slow runtime, deterministically enough to fail 12 jobs in one run.
• The strand is what happens when admission and delivery are split. This is concrete support for the spec nuance in Idempotency follow-ups: atomic keyed start(), post-completion dedupe, and attribute-based run lookup #2376 (comment): resumeOrStart must deliver the triggering payload exactly once whichever branch wins. The CI hang is precisely the failure of the userland composition — the payload was delivered to the admission-time owner, while a different run won execution-time ownership. The same comment's sendWithEnsure 500ms-sleep contortion is the workaround for this exact window; CI just measured the window being wider than any constant sleep.

The test was fixed in 52370fb by gating the owner's completion on the duplicate's recorded hook_conflict event — correct for a test (it verifies the conflict-handling branch, so it must arrange the interleaving that reaches it), but a production application cannot arrange interleavings. Until keyed start() / resumeOrStart exist, the docs' guidance stands: pair workflow-side claims with route-side resume-with-retry (self-healing for the strand), or pass inputs via start() args and use the hook claim-only.

🤖 Generated with Claude Code