Dual ESM/CJS emit with tsc

[andrewbranch]

#54546 explored one approach of enabling dual ESM/CJS emit for packages with tsc alone. The idea was that instead of determining the module format of .ts files by looking for package.json files in its ancestor directories, we would look for them starting at the directory where that file’s .js output would be emitted (a subdirectory of outDir). That way, two tsconfig files could point to two different output directories, each pre-seeded with a package.json files that set the module format for the output generated by that tsconfig.

This approach has two main downsides:

1. It’s annoying to have to commit package.json files into your output directory while gitignoring everything else. Additionally, if used in combination with other tools (e.g. tsc for declaration emit but rollup for JS emit), other tools might wipe the output directory, deleting your package.json.
2. Thinking about the implications for projects that aren’t doing dual emit, if we determine the module format based only on the output file structure, we potentially fail to analyze the behavior of ts-node, or a bundler that cares about package.json "type". This can be a problem if a project compiles with tsc for publishing, but runs input .ts files directly during development (which is a scenario I think we should make a habit of considering). Ideally, we want a solution that ensures the module format of the output agrees with that of the input, unless the output format is being intentionally changed by config (presumably for purposes of dual emitting).

I think both of these can be solved by doing two things:

1. Introduce a compiler option that allows a package.json with { "type": "module" } or { "type": "commonjs" } (or blank, whatever) into the outDir, and use this config setting (instead of what’s already present in the outDir) to determine the module format of input files at a higher priority than any package.json files in a directory higher than outDir. This solves problem (1) above—no need to pre-seed or commit files in your build directory.
2. Whenever a package.json in a subfolder of the common source directory / rootDir that affects the computed module format of a file is seen, emit that package.json (or a stub of it with just "type"?) into the corresponding subfolder within outDir. This solves (2), and actually solves an issue that exists today, where tsc output can be invalid for Node without manually copying a package.json that occurs inside rootDir. (@rbuckton mentioned this in team chat one time, but it didn’t get much discussion.)

Emitting package.json files would be new territory for us, but I think it’s worth it for the problems it solves.

[rbuckton]

#54546 explored one approach of enabling dual ESM/CJS emit for packages with tsc alone. The idea was that instead of determining the module format of .ts files by looking for package.json files in its ancestor directories, we would look for them starting at the directory where that file’s .js output would be emitted (a subdirectory of outDir). That way, two tsconfig files could point to two different output directories, each pre-seeded with a package.json files that set the module format for the output generated by that tsconfig.

This isn't an approach I would recommend. At one point I was looking into using "stub" package.json files to set { "type": "module" } in a specific directory to support dual emit using the TypeScript API. At first glance, it seems like this might work as long as you only ever use relative imports, or import from packages in your node_modules. However, if you use an import map and # imports to other locations in the root of your package, those imports will no longer work.

For example, consider a structure like this:

/dist/cjs/package.json    # contains `{ "type": "commonjs" }`
/dist/cjs/index.js
/dist/cjs/index.d.ts
/dist/esm/package.json    # contains `{ "type": "module" }`
/dist/esm/index.js
/dist/esm/index.d.ts
/lib/compat.node.js       # Node-specific functionality (not a build output)
/lib/compat.browser.js    # Browser-specific functionality (not a build output)
/lib/compat.d.ts
/src/index.ts             # source to be built as a dual module
/package.json

and a root package.json like this:

{
  "type": "commonjs",
  "exports": {
    ".": {
      "require": "./dist/cjs/index.js",
      "import": "./dist/esm/index.js"
    }
  },
  "imports": {
    "#lib": {
      "types": "./lib/compat.d.ts",
      "node": "./lib/compat.node.js",
      "browser": "./lib/compat.browser.js"
    }
  }
}

You would define your index.ts file (that is built to the /dist/cjs and /dist/esm outputs) such that it imports from "#compat" to load javascript that is specific to the runtime environment:

import * from "#compat";
...

At compile time it seems like this will work, because there is no stub package.json file in the /src directory. As such, resolution walks up to the package.json in the root and can successfully resolve the "#compat" import. However, the build outputs in /dist/cjs and /dist/esm will fail to find that import at runtime because #-style import maps are specific to the nearest package.json.

I ran into this in https://github.com/esfx/esfx/tree/main/packages/equatable, which forced me to abandon that approach.

[rbuckton]

Instead, the approach I took was to just build .cjs outputs to /dist/cjs and then transform them to .mjs (including extension renaming) under /dist/esm.

I know we've been reticent to rewrite imports during emit, but I think building a foo.ts with dual emit to a foo.cjs and foo.mjs, and rewriting module specifiers to match the extension is probably the most reliable mechanism.

[andrewbranch]

At compile time it seems like this will work, because there is no stub package.json file in the /src directory. As such, resolution walks up to the package.json in the root and can successfully resolve the "#compat" import. However, the build outputs in /dist/cjs and /dist/esm will fail to find that import at runtime because #-style import maps are specific to the nearest package.json.

Central to my proposal is considering not just the package.json files that already exist on disk, but the ones that we know are going to exist due to emit from this feature. Accurate compile time format detection and module resolution is table stakes and is relatively easily achievable. However, I didn’t think about the fact that this would block the root-level imports and exports, and that is definitely unfortunate.

[rbuckton]

However, I didn’t think about the fact that this would block the root-level imports and exports, and that is definitely unfortunate.

As someone who would want to use this feature, this would make it completely unusable for me.

Honestly, I'd love to see package.json add some mechanism of specifying the default "type" for a subfolder. That way there would still only be a single package.json to refer to, and package.json is already read and interpreted as part of module lookup.

[andrewbranch]

That’s exactly what @DanielRosenwasser said 👀

[andrewbranch]

I do wonder if imports/exports has to be a show-stopper, though. If we emit a module-format-controlling package.json to the root of a given outDir, we could copy imports and exports, modifying relative paths in all the keys. That strikes me as simpler, safer, and more scoped to new code than trying to modify every emitted filename and every relative module specifier in the program.

[rbuckton]

I do wonder if imports/exports has to be a show-stopper, though. If we emit a module-format-controlling package.json to the root of a given outDir, we could copy imports and exports, modifying relative paths in all the keys. That strikes me as simpler, safer, and more scoped to new code than trying to modify every emitted filename and every relative module specifier in the program.

I'm not sure if there are other implications to using stub package.json files we would have to consider. I would expect that extension mangling would be far easier and far more reliable than trying to rewrite exports and imports.

There are a number of other considerations as well. Some packages expect to walk up to find package.json for version information to use in logging, for configuration information to use at runtime, etc. That would no longer work if they switched to this mode, or they'd have to rewrite that logic to cater to our behavior. There just seems to me to be a lot more that can go wrong with this approach.

[fatcerberus]

modifying relative paths

Doesn't this open the exact same can of worms as rewriting paths in import specifiers?

[andrewbranch]

No, it opens a can of worms which is a strict subset of the can of worms you mentioned.