Add dev server to speed up dev iteration

[ycw]

fix: #30829

Description

Crafted a server dedicated for three.js development, it supports routes redirection so that build step (bundling) is not needed during development. This PR is a POC, it proved the 3rd solution in #30829 is doable, its upside is that 1/ fast feedback, 2/ devtools can reflect /src as-is, Workspace and Local Overrides will work as expected (see demo), very intuitive for debugging. The downside is that fail-too-late, something works in dev phase may broke after build, one may spent hours on a solution and back to square one.

Try: npm run dev then open http://127.0.0.1:3333/examples/webgl_geometry_text.html, then bring up devtools and navigate to Sources tab, you should see JS are being served from /src in Page panel. Now go back to terminal, you should see how JS was redirected (those HTTP 307).

Demo - Leverage workspace for quick edit, feedback is instant:

upside1.mp4

Details (click to open)

The server supports https, custom port, loopback only, ?browse etc, it's intended to serve files without compression and always without caching, users don't need to think about cache-control gzip br etc.

If --ssl presents, certificate will be created by mkcert module (the only dependency of this server) and is stored at utils/dev/server.key and utils/dev/server.crt. Those files are .gitignore-ed. The certificate validity is set to 365 days, so the fingerprints for that domain will last one year in browser cache, users no need to add exception for self-signed certificate every time they enter the same domain. Server will automatically create new certificate when the existing one is 364-day-old, 1 day before expiration to avoid issue caused by time zone. If the .key or .crt is deleted or exceeded valid time range (e.g. modified system clock), server will automatically create new one. Simply put, users should not bother certificate.

If --port presents, server will listen on that given port number instead of the default 8080. No port sniffing, let-it-throw instead.

If --internal presents, server will listen only loopback interfaces, remote accessible interfaces are skipped. For safety all package.json scripts enabled this flag. Users can toggle it manually in there if needed.

If --dev presents, server will perform routes redirection, except for routes with ?browse query, which are served as-is; note that redirection is not 1:1, e.g. "/build/three.module.js" is mapping to "/src/Three.js", not "/src/three.module.js". If --dev is not there, server will never redirect /build to /src no matter routes with ?browse query or not.

Web UI is in bare style, server does not automatically open index.html for directory (otherwise I can't browse content in this directory), for QoL index.html (if exists in directory) will be pinned to the top, right below dotdot; Sections dotdot, index.html, directories, and files are separated by negative spaces, no zebra no lines no icons. There're no features for filtering, to find a file from a long directory, use browser search (ctrl f) instead. CLI UI is also in bare style, styling is implemented by node built-in styleText, only non 200 HTTP responses are highlighted.

[github-actions]

📦 Bundle size

Full ESM build, minified and gzipped.

	Before	After	Diff
WebGL	336.36 78.34	336.36 78.34	+0 B +0 B
WebGPU	533.31 148.2	533.31 148.2	+0 B +0 B
WebGPU Nodes	532.77 148.1	532.77 148.1	+0 B +0 B

🌳 Bundle size after tree-shaking

Minimal build including a renderer, camera, empty scene, and dependencies.

	Before	After	Diff
WebGL	465.37 112.21	465.37 112.21	+0 B +0 B
WebGPU	604.89 164.17	604.89 164.17	+0 B +0 B
WebGPU Nodes	559.88 153.6	559.88 153.6	+0 B +0 B

[Mugen87]

TBH, I'm not in favor of a custom server script and I don't think it's good for the project to maintain all this code. Strongly vote against this approach.

We should use a standard solution and not a custom one.

[ycw]

We should use a standard solution and not a custom one.

Oh, the server is just to evaluate the 3rd solution in #30829, I am not suggesting adding this toy server to the repo as I said in OP this is a POC, and Yes we should choose a well-maintained dev-server solution that suits our needs, I've written down some of them in "Details" at the very end in OP, it has been folded so please click on it to read its content.

Anyways I hope I've showed that proxying /src as-is is efficient and never ever out syncs /src, each page refresh must reflect the latest modification from /src as server indeed loads /src per se, no intermediate artifacts involved. I think this fixed what @gkjohnson is dissatisfied with.

Leave some numbers as reference, webgl_geometry_text contains 389 requests, it finished in ~0.75s using the toy server; if I upgrade the toy server to http/2, page would be finished in ~0.58s,, the toy server not even compress, nor cache... and it had outperformed all build creations on the same device ...

[gkjohnson]

The downside is that fail-too-late, something works in dev phase may broke after build, one may spent hours on a solution and back to square one.

Do we have an example of what kind of case this might be? If the src runs in a browser then it's valid JS and should be bundle-able with rollup. Maintainers of this project are also apparently already redirecting source maps to point to src for dev, anyway.

Crafted a server dedicated for three.js development, it supports routes redirection so that build step (bundling) is not needed during development.

A simple solution would be replacing the /build files with something like the following during dev:

three.module.js

export * from '../src/Three.js';

three.webgpu.js

export * from '../src/Three.WebGPU.js';

[ycw]

Do we have an example of what kind of case this might be?

Rollup does multiple tasks not just bundling, one is minification, three.js indeed has one (terse in this case), its prop mangling feature may cause issues #24729 #26403 ... which are still open. Other examples are not common, at least not common for a graphic library, e.g. using import specifier's hash/query to create different stores; advanced proxy; dynamic script injection etc.

A simple solution would be replacing the /build files with something like the following during dev:

I would not suggest replacing import specifiers manually, this is workaround not a solution. I would not suggest doing so by build tool either because build tool is the root issue causing bad dev experience in your case, we should fix this instead.

Three.js right now is using two entities to build up dev environment, one is build tool another is server, they're total disjointed, server serves build files and never tracks build procedure of the build tool. Due to this uncertainty, one needs to repeating refresh the page until the page "seems to be" in the latest state, or say, we're ensuring involved build files are all finished bundling. Rollup creates bundles per unit, a unit may depend on bundles created by other units, so we can predict that some out-sync issues here, when client requests bundle A, bundle A itself is in latest state, but its external dependency say bundle B is not found (when unit B had been started but yet finished) or outdated (unit B has not been started at all).

As you can see, the uncertainty comes from two disjointed entities, so one solution is that augmenting the server to track build tool process and notify users on webpage that when the process started and finished; Another solution (=this PR) is that bypassing the intermediate build files, let server serves from sources as-is so that users refresh the webpage must reflect latest state of sources (=no out-sync issues). You can think in a way that I'm "replacing the /build files" by server

It would be great if folks could test this solution and then sharing thoughts, simply clone https://github.com/ycw/three.js, switch to devserver branch, then run npm run dev or npm run dev-ssl. Make some changes in three.js sources, then refresh the webpage, etc.

@gkjohnson Oh, I misunderstood, it seems that you're talking about replacing the content of build files, and not bundle it up at all, so build files are eventually like the following:

// filename: three.module.js
export * from '../src/Three.js'   <<< I misunderstood that you are saying bundle it from here

This approach should work because build time become negligible, and source maps is unrelated because sources are indeed being import-ed. Just a note that devtools will show up /src alongside /build, better to clarify to users that this is intended e.g. adding comments in build files, so that existent of /build is reasonable upon debugging, especially for new contributors.

[ycw]

@gkjohnson see #30865