Performance Megathread - Is your workflow run slow? Look here first

[VaguelySerious]

This issue gathers all known performance issues that are NOT due to edge-cases.

If your workflow performs well in most cases, but hits performance problems in some cases, please open a separate issue, and optionally link it here.

Known bottlenecks

Replay invocation overhead

Between every step, the workflow enqueues an event replay. This means invocation overhead, queue overhead, and initial state fetching overhead on each step call.

• >=5.0.0-beta.5 removes 90%+ of this overhead ([core] Combine flow+step bundle and process steps eagerly #1338), but still spawns new invocation for:
  • Parallel workloads (e.g. await Promise.all calls)
  • Every ~2 minutes we create a new invocation to stay ahead of serverless function timeouts. This could be tightened (Inline step execution optimization on Vercel doesn't make use of full function timeout #1927)

Replay event fetching overhead

• Event replays need to fetch all event
  • This happens on a paginated endpoint, which cause multiple network roundtrips for non-trivial runs (Event pagination is inefficient for long runs #1928)
    • >=5.0.0-beta.5 ensures only new events are fetched in many cases, instead of re-fetching all events
    • Snapshot Runtime: QuickJS WASM VM with snapshot/restore for workflow execution #1300 replaces the paginated event endpoint with a compressed snapshot, further reducing the amount
  • Events come with user payloads, which the World backend may choose to resolve lazily, requiring further network roundtrips, usually when the user payload is large
    • >=5.0.0-beta.16 ([world-vercel] Switch event endpoints to v4 wire format #2055) reduces the network overhead for large payloads considerably
    • Keeping your step inputs/outputs small is the best way to reduce network overhead here at this point in time
    • Using DurableAgent/WorkflowAgent will almost certainly result in large step inputs/outputs as the full conversation needs to be passed to/from the LLM. This can be reduced to some extent.
      • Consider turning off the collectUIChunks option if don't need it
      • DurableAgent may be returning more data than necessary (DurableAgent/WorkflowAgent duplicates doStreamStep return size #1929)

Event payload overhead

• All step inputs, outputs, and hook inputs need to be serialized and loaded on every replay. Many common operations e.g. passing step outputs into new steps frequently when modifying a large data structure, accumulate copies of this data structure, which can easily lead to
  • Compressing data before storing them can reduce this overhead by 80% for most json/string based payloads. This should be handled by the World, and is WIP, see RFC: gzip-compress serialized payload refs (specVersion 5) #2394.
  • When a step output is passed as a step input, or similar re-use of payloads, the runtime should detect this and de-duplicate by passing references only. This requires further thinking and is not currently in progress.
    • Alternatively, the runtime could store diffs only, and merge them at replay time. This requires further thinking and is not currently in progress.

Cold starts

Workflow "use workflow" and "use step" functions get bundled separately (< 5.0.0-beta.5) or as one bundle (>= 5.0.0-beta.5), each of which can incur cold starts separate from the API bundles that start the workflows.

The best ways to reduce this overhead are:

• Reduce bundle size
  • Remove unnecessary dependencies
  • Consider turning off sourcemaps for steps, which often contribute the bulk of the bundle size (Allow disabling step sourcemap with new sourcemap option in builders #1842)
• Pre-warm functions
  • On Vercel, using Fluid compute (highly recommended) will intelligently pre-warm your functions (and reduce costs)
• Use >= 5.0.0-beta.5 to half the amount of bundles that need to be warm

Cross-region networking

Your app, workflow runtime, and DB should all run in the same network region.

As of May 5, 2026, workflows run against iad1 (us-east), so it's recommended to run your workflow-enabled app and the connected databases or other services in the same region. See #1931 for updates.

---

A note on pre-releases

Performance improvements go through a QA phase before being released as 5.x..x-beta releases currently. If you want the latest improvements now, considering using pre-releases (nightly build) by installing tarballs from https://workflow-tarballs.labs.vercel.dev/

[schnitzel112]

Quick question on the nightly tarballs from workflow-tarballs.labs.vercel.dev: are these considered safe enough to run against the hosted Vercel World, or are they currently only meant for self-hosted / other World backends?

Asking because the perf improvements would be relevant for us, but we're running on Vercel and don't want to hit incompatibilities between a nightly client and the prod World backend.

[shtefcs]

Quick question on the nightly tarballs from workflow-tarballs.labs.vercel.dev: are these considered safe enough to run against the hosted Vercel World, or are they currently only meant for self-hosted / other World backends?

Asking because the perf improvements would be relevant for us, but we're running on Vercel and don't want to hit incompatibilities between a nightly client and the prod World backend.

I am wondering the same.

[VaguelySerious]

@schnitzel112 @shtefcs Any given commit on main (nightly tarballs) are extensively e2e tested, but we can't guarantee that any particular commit won't negatively affect your setup when you upgrade. We've seen e.g. some compile-time issues or framework-specific issues in the past. Skew protection on Vercel should limit the impact of these sorts of bugs.

That said, if you try out a tarball for a specific commit and it works for you, it's very likely to keep working for you. Especially now for the v2 flow performance improvements, if you try running https://workflow-tarballs-hdqkpb1dh.labs.vercel.dev/workflow.tgz and it works for you, you can keep staying on that without any upgrade/downgrade issues later.

Otherwise, we do recommend using the 5.x.x-beta.x version, or 4.x.x if you need an SLA. We're prepping the above tarball to be 5.0.0-beta.5 in a few days.

[VaguelySerious]

Many stability improvements for the beta release just went out in 5.0.0-beta.7