Skip to content

Commit d7155d0

Browse files
committed
Various benchmark improvements.
Hello! I am the lead engineer of Cloudflare Workers. We've been analyzing this test all week, and it helped us find a lot of things we could optimize. Thanks for that! ------------------ This commit fixes two issues with the test itself: * The next.js benchmark used force-dynamic on Vercel but not Cloudflare. It should use force-dynamic on both. Ironically, this discrepency should have given Cloudflare an advantage, but we found that Open Next had significant performance bugs in the non-dynamic code path that actually made it worse. Among other things, streaming of the response body was essentially disabled in this mode. We are fixing those bugs, but obviously it's most fair for both platforms to use the same dynamic setting. * The react-ssr benchmark was not setting process.env.NODE_ENV to "production", so React was running in dev mode. When using a higher-level framework, this is normally handled by the framework, but the react-ssr benchmark seems to call lower-level libraries directly. Vercel normally sets this as an actual environment variable in prod, but Workers does not. (Maybe we should...) This commit also includes some housekeeping, which likely has little impact: * Update wrangler to latest on all benchmarks. A few of them were set to extremely outdated versions, though we're not aware of any specific issues. * Updated compatibility date on all benchmarks. We haven't seen this make a difference, but some compatibility dates were six months old and lots has changed in node-compat since then. * Set `minify: true` in all wranlger.jsonc files. We haven't observed much difference from this, but as some of the bundle sizes are fairly large it could improve cold start time slightly. ------------------ We found many performance issues in Open Next this week, and have fixed several. So, this commit also bumps the version number of Open Next to get those improvements. That said, this work is ongoing: we expect to land more improvements in the future. Open Next is not as mature as Next.js, it seems. Separately, Cloudflare has made some changes to our production environment which should significantly improve performance. In particular: * We corrected a problem where CPU-heavy requests would tend to queue up on a single worker instance per colo, causing excess latency when running concurrent CPU-heavy requests driven from a single client location. (That said, it is still possible for requests to be randomly assigned to the same isolate and block each other, but this should be less common now.) * We found that we had tuned the V8 garbage collector too far in the direction of favoring memory usage over execution speed. A small adjustment made a big difference in performance, especially in these tests which do a lot of memory allocation. These two changes are already live for all Workers. We'll have a blog post about all these changes later. ------------------ Finally, we have a few suggestions about how to run and interpret these benchmarks: * The "shitty sine benchmark" is indeed suffering from a missing optimization in Node, penalizing Vercel. [We are fixing it](nodejs/node#60153), but it will presumably take some time for this Node change to find its way to Vercel. In the meantime, we agree this benchmark is silly and shouldn't be included. * We think it is more appropriate to test with a Vercel instance using 1vcpu rather than 2. [The CTO of Vercel argues there should be no difference since the workload is fundamentally single-threaded](https://x.com/cramforce/status/1975656443954274780), and [he is publishing pricing comparisons on the assumption that only 1 vcpu was actually used](https://x.com/cramforce/status/1975652040195084395). These pricing comparisons are only fair if the assumption is correct. We honestly think he is correct, so we think to avoid any questions the test should be run with 1vcpu. (I realize this sounds like some sort of trick, but it isn't. We haven't had a chance to test the difference ourselves. I just honestly think the 2vcpu thing creates confusion that would be nice to avoid.) * This benchmark still contains a singificant "luck" factor in terms of what hardware you get assigned to. Cloudflare has several different generations of hardware in our fleet, and we would expect Vercel / AWS does as well. Different CPUs may have surprisingly different single-threaded performance (example: my 16-core Ryzen 9 9950X personal desktop is 1.7x faster than my 44-core Xeon w9-3575X corp workstation, for single-threaded workloads). Noisy neighbors can also have significant impact by consuming memory bandwidth that is shared by all tenants on the machine. We have seen runs of the test where Cloudflare wins across the board, and others where Vercel wins across the board, presumably as a result of this noise -- and it's not just Cloudflare's performance that varies, but also Vercel's. Note, though, that simply running more iterations of the benchmark does not correct for this "luck", because once instances are assigned to machines, they tend to stay on those machines. Additionally, noisy neighbor effects can be driven by other factors like time of day, regional load imbalances, etc., that don't go away with additional iterations. To get a better sense of the average speed on Cloudflare, we would recommend running tests from many different global locations to hit different Cloudflare colos and thus different machines, but admittedly that's a lot of work.
1 parent 98d815c commit d7155d0

File tree

9 files changed

+18
-9
lines changed

9 files changed

+18
-9
lines changed

next-bench/cf-edition/package.json

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -12,7 +12,7 @@
1212
"cf-typegen": "wrangler types --env-interface CloudflareEnv ./cloudflare-env.d.ts"
1313
},
1414
"dependencies": {
15-
"@opennextjs/cloudflare": "^1.3.0",
15+
"@opennextjs/cloudflare": "^1.10.1",
1616
"next": "15.4.6",
1717
"react": "19.1.0",
1818
"react-dom": "19.1.0"
@@ -27,6 +27,6 @@
2727
"eslint-config-next": "15.4.6",
2828
"tailwindcss": "^4",
2929
"typescript": "^5",
30-
"wrangler": "^4.41.0"
30+
"wrangler": "^4.42.2"
3131
}
3232
}

next-bench/cf-edition/src/app/bench/page.tsx

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,7 @@
11
import ComplexComponent from "./complex-component";
22

3+
export const dynamic = "force-dynamic";
4+
35
export default async function ComplexPage() {
46
console.log("rendering", Date.now());
57

next-bench/cf-edition/wrangler.jsonc

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -6,8 +6,9 @@
66
"$schema": "node_modules/wrangler/config-schema.json",
77
"name": "next-cf-bench",
88
"main": ".open-next/worker.js",
9-
"compatibility_date": "2025-03-01",
9+
"compatibility_date": "2025-10-01",
1010
"compatibility_flags": ["nodejs_compat", "global_fetch_strictly_public"],
11+
"minify": true,
1112
"assets": {
1213
"binding": "ASSETS",
1314
"directory": ".open-next/assets",

react-ssr-bench/cf-edition/package.json

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -12,6 +12,6 @@
1212
"react-dom": "^18.2.0"
1313
},
1414
"devDependencies": {
15-
"wrangler": "^3.110.1"
15+
"wrangler": "^4.42.2"
1616
}
1717
}

react-ssr-bench/cf-edition/wrangler.jsonc

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -6,10 +6,14 @@
66
"$schema": "node_modules/wrangler/config-schema.json",
77
"name": "react-ssr-cf",
88
"main": "worker.js",
9-
"compatibility_date": "2025-03-01",
9+
"compatibility_date": "2025-10-01",
1010
"compatibility_flags": [
1111
"nodejs_compat"
1212
],
13+
"minify": true,
14+
"define": {
15+
"process.env.NODE_ENV": "'production'"
16+
},
1317
"observability": {
1418
"enabled": true
1519
}

sveltekit-bench/cf-edition/package.json

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -28,7 +28,7 @@
2828
"tailwindcss": "^4.0.0",
2929
"typescript": "^5.0.0",
3030
"vite": "^7.0.4",
31-
"wrangler": "^4.41.0"
31+
"wrangler": "^4.42.2"
3232
},
3333
"pnpm": {
3434
"onlyBuiltDependencies": [

sveltekit-bench/cf-edition/wrangler.jsonc

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,8 @@
77
"directory": ".svelte-kit/cloudflare"
88
},
99
"compatibility_flags": ["nodejs_compat"],
10-
"compatibility_date": "2025-05-13",
10+
"compatibility_date": "2025-10-01",
11+
"minify": true,
1112
"observability": {
1213
"enabled": true
1314
}

vanilla-bench/cf-edition/package.json

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,6 @@
77
"deploy": "wrangler deploy"
88
},
99
"devDependencies": {
10-
"wrangler": "^3.110.1"
10+
"wrangler": "^4.42.2"
1111
}
1212
}

vanilla-bench/cf-edition/wrangler.jsonc

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -6,10 +6,11 @@
66
"$schema": "node_modules/wrangler/config-schema.json",
77
"name": "vanilla-ssr-cf",
88
"main": "worker.js",
9-
"compatibility_date": "2025-03-01",
9+
"compatibility_date": "2025-10-01",
1010
"compatibility_flags": [
1111
"nodejs_compat"
1212
],
13+
"minify": true,
1314
"observability": {
1415
"enabled": true
1516
}

0 commit comments

Comments
 (0)