Automatically create JS versions of our TS code in the docs #2638

cprecioso · 2025-04-08T09:32:07Z

Description

Part of #2455

This PR allows to just write TS code directly, and automatically transform it and appear with the JS/TS selector.

It does so by adding two plugins to Docusaurus' MDX handling:

autoJSCode will find any code block with the auto-js meta, transform it with ts-blank-space, and remove the extra blank space with prettier.
Example transformation

Input:
```
 ```ts auto-js title="hello.ts"
 const foo = (n: number) => 8 as string
 ```
```
Equivalent output (actual output is done in AST):
```
 <Tabs groupId="js-ts">
 <TabItem value="js" label="JavaScript">
 
 ```js title="hello.js"
 const foo = (n) => 8
 ```
 
 </TabItem>
 <TabItem value="ts" label="TypeScript">
 
 ```ts title="hello.ts"
 const foo = (n: number) => 8 as string
 ```
 
 </TabItem>
 </Tabs>
```
Why use `ts-blank-space`?

Basically, it's the transformer that does the fewest modifications to the input code, which I think is important to keep the overall visual structure of the code in the examples. It also helps to not introduce TS-only constructs like enums that, when automatically converted, introduce distracting noise.

I also feel validated in its approach since it uses the TypeScript compiler directly to do its job, and not a custom parser that might get outdated. As well, the blank-space-replacing method is used by SWC and Node.js in their TypeScript support.
codeWithHole is made to deal with some of our examples, that just omit some syntactically needed parts of the code with .... In order for the TypeScript transformation to work correctly, this plugin allows us to use an identifier named hole that will be replaced by ..., while still being syntactically correct.
Example transformation

Input:
```
 ```ts with-hole
 const foo = (n: number) => {
 	return {
 		name: "bar",
 		hole
 	}
 }
 ```
```
Equivalent output (actual output is done in AST):
```
 ```ts
 const foo = (n: number) => {
 	return {
 		name: "bar",
 		/* ... */
 	}
 }
 ```
```

With these two plugins, we can transform our examples to be less complex to author while keeping the option to read our docs in JS or TS. If there are any examples that can't be automatically converted, or that shouldn't be converted, we still retain the option to do it manually as we've been doing until now.

I converted two examples so you can see the output, please check them out.

Martinsos

Very cool! Seems like an awesome thing to me, much less duplication! And we can still do it manually if we really need to (we can right? Would we then use those Tabs again?).

I will let others check the details of the PR and approve it, to make sure it works fully with how we do things currently (e.g. if you toggle to JS, does it remmeber the toggle? Mostly checking if we lost anything with not using our Tab system anymore). Aha but I see now that we actually do still use Tabs, they are produced by the plugin -> ok that sounds good then!

One th ing that would certainly be valuable is more documentation on this -> we should mention in docs README.md that we are using this mechanism because it is a bit magic, so having a central place with some discoverability where we describe it would be valuable.

Also, these remark/ files, we should provide some top level header comments in them, let's say one at the top of each file, to explain a bit the motivation and what are they here for, as they are a bit unexpected.
Also I didn't review the code in them, I will let the others do that.

Awesome all together!

web/docs/advanced/apis.md

cprecioso · 2025-04-08T10:28:07Z

we should mention in docs README.md

top level header comments

Yes! I wanted to get this out to get some comments on the approach but I wanted to do this, I'll get on it

Martinsos · 2025-04-08T10:48:51Z

we should mention in docs README.md

top level header comments

Yes! I wanted to get this out to get some comments on the approach but I wanted to do this, I'll get on it

No worries, I assumed so!
Advice: If you want the people to take a look at the PR, but not do a deep review -> make a draft PR. That basically communicates exactly that, that you are looking for comments, but you not done yet. While real PR means it is ready for proper reviewing.

cprecioso · 2025-04-09T13:30:15Z

Can I get a review? 🙏🏼

infomiho

I love this idea. It will make our lives much easier.

Testing

I tried with some more complex examples:

wasp/web/docs/advanced/web-sockets.md

Line 69 in 24920c0

here the tricky part are the TS specific comments (which stay in the JS version), we'd probably need to rewrite those code blocks not to use those comments and move that info somewhere outside of the code block

Another example:

wasp/web/docs/data-model/entities.md

Line 48 in 9977e27

We show Prisma code with the language switch for completeness sake - it toggles between different variants of the text below. So I guess, here we'll just use the tabs directly, you addressed it in the README, that's great 👍

In this example:

wasp/web/docs/data-model/operations/queries.md

Line 106 in 4c2a935

The JS version still has the import statement (without anything imported):

For me, the next step would be - try to convert all the code blocks to see where the current approach fails and try to address those cases. For some of the cases we'll need to modify the code (e.g. to exclude Typescript specific comments), that's okay for me.

web/README.md

infomiho · 2025-04-11T11:24:07Z

web/README.md

+```
+~~~
+
+And it will automatically generate a JS and TS version with a selector to switch between them.


I'd the resulting code with Tabs and everything so the reader can understand what happens under the hood more clearly.

<Tabs groupId="js-ts"> <TabItem value="js" label="JavaScript"> ... </TabItem> </Tabs>

infomiho · 2025-04-11T11:25:36Z

web/README.md

+it syntactically valid.
+
+~~~md
+```ts title="src/apis.ts" auto-js with-hole


I'd also show the code here so it's clear what will be the result (the reader doesn't have to guess). Also, I'd maybe show the example without auto-js and then mention below, "you can combine these two tags, of course".

infomiho · 2025-04-11T11:27:49Z

web/src/remark/auto-js-code.js

+This file defines a plugin for the unified library that processes code blocks
+in Markdown documents. It looks for code blocks with a specific meta flag
+(`auto-js`) and replaces them with a pair of code blocks: one for JavaScript
+and one for TypeScript, as well as a tabbed interface to switch between them.
+This way we can author code examples in TypeScript and have them automatically
+converted to JavaScript for the documentation.


An example code conversion would be amazing here, it will let the reader know right away what this plugin does and you can eliminate much of the prose.

infomiho · 2025-04-11T11:29:01Z

web/src/remark/auto-js-code.js

+const { format } = require('prettier')
+const path = require('path')
+
+const ENABLED_META_FLAG = 'auto-js'


Suggested change

const ENABLED_META_FLAG = 'auto-js'

const META_FLAG_NAME = 'auto-js'

The const is capturing the meta flag name, it's not communicating the state (enabled).

infomiho · 2025-04-11T11:39:58Z

web/src/remark/auto-js-code.js

+      const tsMeta = newMeta
+      const tsLang = node.lang
+      const tsCode = await format(node.value, { parser: 'babel-ts' })


Which prettier config are we using here? I think it would make a lot of sense to use the .prettierrc that also in the web/ folder to keep the code blocks consistent with the code blocks that were written by hand.

infomiho · 2025-04-11T11:45:17Z

web/src/remark/code-with-hole.js

+const { visitParents } = require('unist-util-visit-parents')
+const { default: escapeStringRegexp } = require('escape-string-regexp')
+
+const ENABLED_META_FLAG = 'with-hole'


Suggested change

const ENABLED_META_FLAG = 'with-hole'

const META_FLAG_NAME = 'with-hole'

infomiho · 2025-04-11T11:45:42Z

web/src/remark/code-with-hole.js

+const wrapInWordBoundaries = (/** @type {string} */ reStr) => {
+  return String.raw`\b${reStr}\b`
+}
+
+const enabledMetaRegexp = new RegExp(
+  wrapInWordBoundaries(escapeStringRegexp(ENABLED_META_FLAG))
+)
+
+const holeIdentifierRegexp = new RegExp(
+  wrapInWordBoundaries(escapeStringRegexp(HOLE_IDENTIFIER))
+)


Maybe just use new RegExp directly since we know the regexes up front.

infomiho · 2025-04-11T11:45:59Z

web/src/remark/code-with-hole.js

+        if (!node.lang || !SUPPORTED_LANGS.has(node.lang))
+          throw new Error(`Unsupported language: ${node.lang}`)


I'd wrap the code block with curly braces.

infomiho · 2025-04-11T11:47:35Z

web/src/remark/code-with-hole.js

@@ -0,0 +1,51 @@
+// @ts-check


What do you think - should we name it with-ellipsis vs. with-hole to be more specific? Granted, it's hard to write with-ellipsis than with-hole that a downside definitely.

In this case I used the name hole because it is a common concept in ASTs where a part of the program is not done or not valid yet. I don't especially favor the name ellipsis since it is harder to spell and no more specific.

cprecioso · 2025-04-14T13:06:47Z

@infomiho

try to convert all the code blocks

Are we sure converting 230 occurrences of this pattern is a good use of the team's time? 😅

I was planning on us leaving stuff as it is; writing new stuff with this plugin, and converting on the go whenever we touch a file.

infomiho · 2025-04-14T13:13:01Z

I think it's a valid use of team time since it will speed us the docs authoring process. It took me a couple of minutes per code block to do it, so it might be a few hours max IMHO.

What I looking to achieve: find the the edge cases for which the plugin might not produce 1:1 JS code. While this is still hot, while you have the context, I think it'll be good to figure out and write down the process/limits of the plugin.

Let's do it like this: invest one hour to convert as many examples as possible and try going for as many unique examples as possible 🙂

cprecioso · 2025-04-16T15:01:24Z

Turning to draft until #2658 is merged

cprecioso requested review from Martinsos and infomiho April 8, 2025 09:32

cprecioso changed the title ~~Automatically create JS versions of our TS code~~ Automatically create JS versions of our TS code in the docs Apr 8, 2025

cprecioso linked an issue Apr 8, 2025 that may be closed by this pull request

Update docs for userSignupFields to be consistent between JS and TS code examples #2455

Open

cprecioso removed a link to an issue Apr 8, 2025

Update docs for userSignupFields to be consistent between JS and TS code examples #2455

Open

Martinsos reviewed Apr 8, 2025

View reviewed changes

web/docs/advanced/apis.md Outdated Show resolved Hide resolved

web/docs/advanced/apis.md Show resolved Hide resolved

cprecioso added 4 commits April 8, 2025 12:32

Automatically create JS versions of our TS code

3e59636

Apply two examples

e0ee478

Add file comments

793b50f

Refactor some code to make it more understandable

4002350

Add readme docs

3bd0591

cprecioso force-pushed the auto-js-docs branch from 50e0e81 to 3bd0591 Compare April 8, 2025 11:00

cprecioso self-assigned this Apr 8, 2025

infomiho requested changes Apr 11, 2025

View reviewed changes

cprecioso added 7 commits April 14, 2025 14:24

Comments in readme

fd19f6a

Fix admonition

b255f6b

Typo

56112f3

More fixes

b954e89

Simple changes

49d7321

Find the correct prettier confirm

1f68723

Extract functions

f41ef1e

cprecioso marked this pull request as draft April 16, 2025 15:01

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Automatically create JS versions of our TS code in the docs #2638

Automatically create JS versions of our TS code in the docs #2638

cprecioso commented Apr 8, 2025 •

edited

Loading

Martinsos left a comment

cprecioso commented Apr 8, 2025

Martinsos commented Apr 8, 2025

cprecioso commented Apr 9, 2025

infomiho left a comment •

edited

Loading

infomiho Apr 11, 2025

infomiho Apr 11, 2025

infomiho Apr 11, 2025

infomiho Apr 11, 2025

infomiho Apr 11, 2025

infomiho Apr 11, 2025

infomiho Apr 11, 2025

infomiho Apr 11, 2025

infomiho Apr 11, 2025

cprecioso Apr 14, 2025

cprecioso commented Apr 14, 2025

infomiho commented Apr 14, 2025

cprecioso commented Apr 16, 2025

	const ENABLED_META_FLAG = 'auto-js'
	const META_FLAG_NAME = 'auto-js'

	const ENABLED_META_FLAG = 'with-hole'
	const META_FLAG_NAME = 'with-hole'

		if (!node.lang \|\| !SUPPORTED_LANGS.has(node.lang))
		throw new Error(`Unsupported language: ${node.lang}`)

Automatically create JS versions of our TS code in the docs #2638

Are you sure you want to change the base?

Automatically create JS versions of our TS code in the docs #2638

Conversation

cprecioso commented Apr 8, 2025 • edited Loading

Description

Martinsos left a comment

Choose a reason for hiding this comment

cprecioso commented Apr 8, 2025

Martinsos commented Apr 8, 2025

cprecioso commented Apr 9, 2025

infomiho left a comment • edited Loading

Choose a reason for hiding this comment

Testing

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cprecioso commented Apr 14, 2025

infomiho commented Apr 14, 2025

cprecioso commented Apr 16, 2025

cprecioso commented Apr 8, 2025 •

edited

Loading

infomiho left a comment •

edited

Loading