Repository navigation
feat: assemble source-owned docs and record checked candidates - #355
Merged
Merged
Conversation
The first published build dropped docs/.cursor/ silently: actions/upload-artifact skips hidden files by default, so the assembled tree had 268 files and the `assembled` branch had 267. Editor configuration is not site content and should not be published, but it should not vanish by accident either. The assembler now excludes dot-prefixed paths deliberately and names what it dropped, so the assembled tree and the published tree are the same thing — which is the guarantee the whole pipeline rests on.
Two problems found while verifying the first publish to `assembled`. The published branch had 267 files where the assembler produced 268: actions/upload-artifact skips hidden files by default and silently dropped docs/.cursor/. The first attempt excluded dotfiles from the assembly instead, which was wrong — mint export carries that file today, so removing it would have quietly deleted a published file at cutover. The harness caught that, which is the strongest evidence so far that it works. The upload now includes hidden files, and the assembler publishes what it assembled. The harness itself was intermittently failing on identical trees. Mintlify renders the OpenAPI reference non-deterministically: response code blocks come out syntax-highlighted on one run and plain on the next, ~2 KB across ~78 fragments. One comparison passed, the next failed, on byte-identical input. A gate that fails at random is one people learn to route around. That subtree is now compared for presence but not for bytes, and only that subtree. It gives up nothing about the assembler, which passes openapi.yml through byte-identically and cannot influence one render differently from the other; a page appearing or disappearing is still caught. Verified: five consecutive comparisons pass, an authored-page change is caught, a reference page being removed is caught, and a reference page's contents are knowingly not.
* feat: anchor the tables pages
First area of the anchor pass. Every section heading in docs/tables now carries
an explicit `{#anchor}` — 114 of them — which is the identity Enterprise
overlays attach to from A5 and which survives the heading being reworded.
Anchors are taken from the ids the site already renders, not derived from the
heading text. That distinction turned out to matter: Mintlify applies smart
quotes before slugifying, so `## What's next?` renders as `what’s-next` with a
curly apostrophe, and a derived slug would have silently changed the id and
broken every existing deep link to it. Reading the ids back from `mint export`
makes the pass correct by construction rather than by reimplementing rules we
would have to keep in sync.
Headings that begin with a number need their period escaped. `### 1. Setup`
renders with its number until an explicit anchor is added, at which point
Mintlify re-parses the text, treats the number as an ordered list marker, and
drops it from both the heading and the table of contents. 117 headings across 17
pages start this way, so the pass would have quietly renumbered a good deal of
the site.
Verified: the exported site is unchanged. All six existing deep links into
tables still resolve.
* feat: anchor the remaining pages
Completes the anchor pass. 805 of 810 section headings across the authored pages
now carry an explicit `{#anchor}`, the identity Enterprise overlays attach to
from A5 and the one thing that survives a heading being reworded.
Three more cases the tables pilot had not reached, each of which would have
corrupted anchors silently:
Setext headings. Eleven h2s in the reranking pages are written as text over a
rule rather than with hashes. Mintlify gives them ids like any other heading, so
a parser that only saw hashes consumed the rendered ids out of order and handed
every later heading on the page the wrong anchor — plausible names attached to
the wrong sections. They are rewritten to hashes, which renders identically, and
a strict count check now refuses to write anything when source headings and
rendered ids disagree.
Headings below h4. Mintlify emits no id for h5, so there is nothing to read back
and nothing to preserve. The eight in the corpus, all API method names on one
page, keep their generated markup and stay unanchored.
Ampersands. Mintlify keeps `&` in a generated id but strips it from an explicit
anchor, so `observability-&-performance` cannot be written down: any anchor set
on those headings changes the id and breaks links to it. Five headings join two
words this way; they keep their generated id. If A5 needs to attach to one,
rewording the heading is the honest fix rather than silently moving it.
Verified: the exported site is unchanged.
* refactor: retitle headings whose anchors carried punctuation
Eighteen headings across twelve pages produced anchors containing `&`, `/`, or
curly quotes. Ampersands were the pressing case — Mintlify keeps `&` in a
generated id but strips it from an explicit anchor, so those five headings could
not be anchored at all — but slashes made anchors read like paths and curly
quotes made them non-ASCII, and neither belongs in a key that Enterprise
overlays will be written against.
The punctuation is incidental in every case, so the headings say the same thing
without it: "Observability & performance" becomes "and", "S3 / GCS / Azure Blob"
becomes a comma list, "What's next?" becomes "Next steps". Nothing linked to any
of the old anchors, so nothing breaks.
Underscores and the plus in `analyze_plan`, `explain_plan`, `max_pooling`,
`approx_mode`, `torch_col` and `100B+ row scale` are left alone. Those
characters are part of an API name or a quantity rather than punctuation, they
are safe in a URL fragment, and renaming them would misname the thing the
heading documents.
All 810 headings now carry an anchor. The twelve pages whose rendered output
changed are exactly the twelve retitled here.
* refactor: retitle the last headings whose anchors held identifiers
Six headings named an API identifier or a quantity directly — `analyze_plan`,
`explain_plan`, `approx_mode`, `torch_col`, `max_pooling`, `100B+ row scale` —
so their anchors carried an underscore or a plus.
Renaming the identifier would have misnamed what the section documents, so the
headings now describe what the section does and the identifier stays in the
prose, where it was already being used: `analyze_plan` appears 14 times in that
page, `explain_plan` 11, `max_pooling` 9. Nothing about the API is lost, and
the heading reads better for it.
Every anchor in the corpus is now plain `[a-z0-9-]`.
The pages describing the open-source client now live in that repository, beside the code, and are assembled from there. This repository keeps what it still owns — the Geneva pages until Stage G replaces them, the generated dataset cards, the OpenAPI spec, and the Enterprise pages until they move to sophon. Navigation is the interesting part. It is authored once, in the root that owns the pages, so that root can also be served on its own — a contributor previews the open-source documentation with `mint dev` and no second checkout. Everything this repository still holds is contributed as a fragment that says where each entry belongs: the chain of groups above it and the sibling it follows. Appending was not good enough, because sidebar order is what a reader navigates by, and appending dropped Geneva below Support and Datasets past Use Cases. `scripts/split_nav.py` produced that fragment by walking the original navigation rather than by hand, which is how the six entries stayed six rather than ninety-seven. A5 reuses it when the Enterprise pages move. Three things had to match exactly, and each was found by the harness: The `openapi` block names a spec file this repository owns, and Mintlify refuses to build at all when it is missing — so it is lifted out of the base navigation and restored by the fragment. Key order in `docs.json` is load-bearing. Mintlify hashes its CSS and JS bundles from those bytes, so moving `navigation` renamed every asset on every page. `ensure_ascii` likewise: the banner text carries an em-dash, and escaping it differently changed the same bytes. CI now checks out lancedb/lancedb at main and assembles twice, comparing the two. The site no longer exists as a single tree anywhere, so a direct comparison against `docs/` is not available; determinism is what the comparison can still establish. Verified: the assembled tree is byte-identical to the one it replaces.
#350 was merged into deploy-freeze while the freeze was in effect, so the fix is live but absent from main and would have been lost at cutover: the redirect for /hybrid-search and the removal of a redundant layers diagram from the Enterprise architecture page. Redirects are now owned by this repository's navigation fragment, so that is where the redirect goes. The diagram removal is applied as a patch rather than by taking the whole file from deploy-freeze, which would have reverted that page's anchors along with it. Anything else that lands on deploy-freeze during the freeze needs the same treatment — it is the branch production serves, and it does not flow back.
The eleven Enterprise pages move to sophon, and this repository assembles them from there. Its navigation fragment keeps only what it still owns: the Geneva group and the Datasets tab. Adding a third root exposed two assumptions in the assembler that only held while there were two. Both were caught by the collision guard rather than by review, which is the guard doing its job. Every root now ships a complete `docs.json` so it can be previewed on its own — that is how a contributor reads the open-source pages without this repository, and how the Enterprise pages are reviewed in isolation. Only the first reference root's is the published navigation; the others are ignored rather than treated as a conflict. Shared assets legitimately appear in more than one root, because each root needs them to render alone. Identical bytes are no longer an error. Differing bytes still are: the site would otherwise depend on the order roots are declared in.
Retarget the Geneva group at the new Feature engineering section. Two assembler fixes the restructure surfaced. Redirects now accumulate across roots instead of the last fragment replacing them, so a repo can redirect its own moved pages; a source claimed twice with different destinations is an error. A root's README is no longer copied, which was publishing it at a URL nothing links to. The restructure also dropped the set directive supplying the REST reference its spec, which assembled cleanly and rendered a group with no operations in it. Validation now fails when the navigation never names the spec.
Removes the 34 pages, their nav group, and the redirect whose destination they were. Feature engineering is now documented in lancedb and sophon.
The REST pages moved out of their own tab into Basics.
The REST group is gone from the sidebar, so nothing references the spec from navigation and the old guard would fail every build. Check the synced file is present instead, which is the invariant that still holds.
Robotics moves above the image groups and gains the LeRobotDataset guide, which is how you read the two datasets already in it.
Same wasted level as the other tabs: a group named Overview holding one page.
An overlay page replaces the reference page at the same path. The reference root owns every path and the navigation, so an overlay never adds or moves a page -- it only says more about one that exists. sophon switches to that role.
Carries #364 from deploy-freeze.
Nineteen modify/delete conflicts, all resolved as delete: main edited pages that this branch moves to lancedb/lancedb, and every one of those edits has already been ported there and verified line by line. Two of the nineteen are the snippet-generating tests, which move to docs/web-tests/ in the same repository. Three files merged cleanly that had to be removed as well -- training/data-loading.mdx and two images added by #344 and #354. A clean merge is not the same as a correct one when the branch's whole purpose is to delete the directory they land in. Keeps main's pin of mint@4.2.888, which exists to stop newer releases rendering nondeterministic OpenAPI examples -- the same byte-comparability the harness in this branch checks.
Proposed with lancedb/lancedb's newer docs branch (jack/docs-web b8c87c46), which has an open-source page at every published path. The Enterprise root moves from sophon/docs to its public docs/web folder; the rest of sophon/docs is internal notes (ci.md, vector-duplicate-pairs/) that this root published. The overlay contract is now enforced instead of assumed: - An overlay's docs.nav.json may only carry redirects. With the published Sophon fragment, every Enterprise page appeared twice in the sidebar. - An overlay replaces pages. Its other files, such as preview logos and stylesheets, must match the reference copy byte for byte, or a stale preview stylesheet would replace the site's. - A root marked `private` may contain no symlinks and no hidden paths, so nothing beside its public folder can be published. The assembler also prints the commit it read each root from, so local and CI builds name all three inputs together.
The Assemble workflow now checks out lancedb/lancedb and lancedb/sophon, each sparse to docs/web, and records the three resolved commits in the job summary and the assembled branch's commit. A manual run can name other refs to build a candidate. It validates the assembled tree and checks its links, anchors and redirects. The required Docs Check (`broken-links-and-openapi`) keeps its name but checks the assembled site with the pinned Mint 4.2.888. Checking docs/ on its own reported pages that now live in the other roots as missing. Reading lancedb/sophon needs a SOPHON_DOCS_TOKEN secret limited to Contents: read on that repository. On October 5 no such secret was visible to this repository, so these jobs have not run.
The private-root guard rejected symlinks inside the root but resolved the root's own path first. So a symlink at sophon/docs/web, or at sophon/docs above it, was followed, and whatever it pointed at was published as Enterprise pages. The configured path is now checked before it is resolved. Every directory from the root up to its checkout (the nearest real directory with a .git) must be a real directory. Above the checkout a symlink only moves the checkout, so it is allowed; with no checkout, every directory on the path is checked. scripts/tests/test_assemble.py covers that boundary: a symlinked root, parent or checkout directory, a symlink inside the root, a symlink above the checkout, hidden paths, overlay navigation and asset rules, orphan overlays, and a valid overlay. The previous assembler fails the five path-symlink cases. `make test-assemble` runs the tests, and so does a new CI job, which needs no secret.
The README led with previewing docs/ alone, and the assembler header, Makefile comment and Assemble workflow header still described one root assembled byte-identically to docs/, or a comparison against it. The workflow now compares two assemblies. The README's Development section now starts with building and checking the combined site, and notes that docs/ on its own does not resolve links into the other roots. The second export is renamed from direct.zip to assembled-again.zip.
`publish` needed only `assemble`, so a failing `test` job, the assembler's overlay and private-root guards, could not stop the assembled branch from being updated. It now needs both. The main-push condition and the sources line read from `assemble` are unchanged.
Restores the 34 Geneva pages and their four snippets exactly as published at deploy-freeze (113db11), the "Feature Engineering (Geneva)" group after the open-source Feature Engineering group, and the /geneva/udfs/built-in redirect. 6618fe2 removed them, but they document APIs the Function pages do not replace, so they stay published until the official Function launch. The snippets are frozen copies: the tests that generated them need the geneva package and did not move with the other example tests.
The assemble.yaml comment still promised that Stage G would replace the Geneva pages. They now stay published until the official Function launch, because the Function pages do not replace their APIs. The README says what this repository's docs/ holds and that the Geneva snippets are frozen copies, since nothing here can regenerate them.
#355 moved the example tests and their generator to lancedb/lancedb, but the README, the agent instructions, WRITING.md and the Makefile still sent contributors to tests/ and scripts/mdx_snippets_gen.py in this repository. They now give the command that regenerates docs/web/snippets beside the tests, and the py, ts, rs and snippets targets stop with that command instead of failing on missing folders. The four Geneva snippets stay frozen.
Every successful push to main force-pushed the assembled site to `assembled`, so merging this repository also published it, and nothing recorded which build a branch update came from. The Assemble workflow now only reads: each run checks the site as before, then records it as a candidate (the commit each root was read from, the refs it asked for and a SHA-256 for every file, named by a checksum over them) and uploads it with that record. The Publish workflow is dispatched by hand with a run ID and that checksum. It puts the exact candidate on `staging`, for a hosted preview of the combined site, or on `assembled`; it reads no source repository and assembles nothing. scripts/candidate.py checks the run, the record and every file, then pushes a new commit on top of the branch without forcing. Production also needs a successful run on main that read both producers' main, the production environment's approval and its deploy key. Publishing an earlier candidate again is the rollback; once its artifact has expired, its earlier publication on the branch is used after its files are checked against the checksum. The tests publish into disposable local repositories. The environment, the rules on `assembled`, the deploy key and MINTLIFY_CONTENT_DIR are settings outside the repository; README.md lists them.
Production checked only the producers a record happened to name, so a record with no producers, with Enterprise missing, or with another name in place of the expected pair passed the main-branch check and was published. Every record must now name exactly lancedb, enterprise and build, each a clean commit, and refs for exactly the two producers. Anything missing or unexpected is refused before any branch is written: when recording, when publishing a run's artifact, and when republishing an earlier publication from the branch history. Production still requires both producer refs to be main and the build commit to be the run's; staging still takes other valid refs. Run evidence or a record that is not the expected kind of JSON object is refused instead of raising.
GitHub's REST reference shows a run's workflow path qualified with the ref its file was read from, as in `.github/workflows/build.yml@main`, but the publisher accepted only the bare path, so an otherwise valid Assemble run reported that way could not be published. The file must still be exactly .github/workflows/assemble.yml. A ref, if given, must be a plain ref name and the run's own branch, bare or as refs/heads/..., or its commit. Another workflow, another repository's copy, a lookalike file, another branch's or commit's ref and malformed refs stay refused, as do failed and pull-request runs whatever their path.
The workflow and README said production is refused without the outside settings and that it waits for approval in the production environment. The workflow checks only that production runs from main, that the environment supplies the deploy key and that MINTLIFY_CONTENT_DIR is set. It cannot see whether the environment requires reviewers or admits only main, or whether the rules on `assembled` refuse other writers, so the comments and README now describe those as protections to install and verify separately. The README also says that a record must name exactly the three source commits and both producers' refs.
Two of the history cases left out a producer ref as well as a source, so the refs check alone refused them and a history path that skipped the sources check still passed. Each history case now breaks one field, and an artifact case drops the Enterprise source with both refs present.
brendanclement
changed the title
brendanclement
added a commit
to lancedb/lancedb
that referenced
this pull request
Oct 7, 2026
Move the public website pages into `docs/web` beside the SDK, with shared navigation and executable example sources in `docs/web-tests`. The combined website is assembled by [lancedb/docs#355](lancedb/docs#355), with six Enterprise replacements from [lancedb/sophon#7681](lancedb/sophon#7681). This includes Function guidance, runnable Quickstarts, regenerated snippets, production contact/demo fixes, the legacy SQL route, and the LeRobot redirect. LeRobot remains attributed to Geneva; all 34 Geneva pages remain in the docs repository until Function officially launches. Materialized-view guidance follows the full rebuild refresh introduced by #4373: local refresh writes the complete result, has no `full` argument, supports `source_version`, and no longer requires stable row IDs. Deployment SQL refresh options and stable-ID setup are qualified by version/configuration. Function input guidance distinguishes ordinary stored columns from computed inputs supported by newer deployments. **Validation:** independently accepted local checks include 95 Python tests (real Enterprise excluded), eight TypeScript tests, Rust Quickstart, snippet parity, typos, deterministic assembly, Mint validation and links/anchors/redirects. The combined baseline retains all 181 production routes and 17 legacy redirects. Examples ran locally against source-built Python SDK 0.40.0b13; existing GitHub workflows do not run `docs/web-tests`. New main at `069b2f7d` is 0.41.0b0 and adds computed Function inputs; that SDK was not exercised by the accepted tests. Latest prose corrections receive focused site validation. **Merge/rollout:** merge this and #7681 before #355's combined CI, which reads producer `main`. This is migration infrastructure: section overlays, source-owned skills and automatic versioned paired releases remain follow-ups. The current manual publisher is transitional. Hosted validation, a supported SDK/Enterprise release pair and a real Enterprise walkthrough remain pending. Production Mintlify watches `lancedb/docs:deploy-freeze`, directory `docs`; preserve that source through review. The separate SDK GitHub Pages reference workflow rebuilds on main merges. **Updated-head CI (9a50f8d):** [Typos](https://github.com/lancedb/lancedb/actions/runs/37654185443/job/112904990558), [PR title](https://github.com/lancedb/lancedb/actions/runs/37654179507/job/112904974595) and [Aikido](https://app.aikido.dev/featurebranch/scan/215386145?groupId=74229) pass. Critical findings in the example lockfiles were patched: all Sharp copies use 0.35.5, protobufjs 7.6.6, tar 7.5.22, quinn-proto 0.11.19, zerovec 0.11.8 and zerovec-derive 0.11.6. The SDK's product dependencies are unchanged. The revised lockfiles pass eight TypeScript examples, HuggingFace decode/resize/crop/rescale/normalize plus AVIF roundtrip, and Rust Quickstart. Aikido passing does not mean every advisory is cleared; npm still reports 29 high and five moderate advisories in the example dependency graph. No scanner bypass or suppression was added. The Function prose correction passes combined/standalone Mint, deterministic assembly, routes and assets. The revised 325-file site checksum is `175bec06864abb400dd292e667ba93448c0479b8359fd7bf6146afabf9040647`. This PR remains open for independent review; further merges are on hold. --------- Co-authored-by: Brendan Clement <brendan.clement@lancedb.com>
LuQQiu
approved these changes
Oct 8, 2026
This was referenced Oct 10, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Note:` Mintlify still serves from the deploy-freeze/docs branch, so merging this PR won’t change the live website.
This PR builds one documentation site from the sources in lancedb and Sophon, plus the Geneva, dataset and API content still owned by this repository. The OSS repo owns the shared navigation and Sophon replaces six Enterprise pages that were here.
This PR replaces automatic publishing on merge with a checked build artifact and a separate publish workflow. Each build records its source commits and file checksums, so we can test and publish the same content. The manual publishing step is temporary right now while we get this stood up and the goal is automatic publication with releases.
Both producer PRs are merged, and a read-only GitHub App is configured to read the docs from the sophon repo.
Next steps: