feat(ui): add llms.txt generation for npm packages #1382
+860
−5
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add AgentFile and LlmsTxtResult interfaces for llms.txt generation and export from the shared types barrel.
Add discoverAgentFiles, fetchAgentFiles, generateLlmsTxt, and handleLlmsTxt orchestrator for llms.txt generation from npm packages.
Serve llms.txt at /api/registry/llms-txt/[...pkg] following existing registry API patterns with cached event handler and SWR.
Cover discoverAgentFiles, fetchAgentFiles, and generateLlmsTxt including root files, directory scanning, graceful failures, scoped packages, and full/minimal output generation.
|
The latest updates on your projects. Learn more about Vercel for GitHub.
1 Skipped Deployment
|
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
Contributor
📝 WalkthroughWalkthroughThis PR implements end-to-end llms.txt generation for npm packages. It adds a utility module for discovering agent instruction files from jsDelivr CDN, fetching their content, and generating structured llms.txt documents with package metadata, README content, and agent instructions. A new middleware routes llms.txt requests at multiple levels (root, organisation-level, package-level with optional versioning). Type definitions and canonical redirects are updated to support the feature, along with comprehensive unit tests. Possibly related issues
Possibly related PRs
Suggested reviewers
🚥 Pre-merge checks | ✅ 1✅ Passed checks (1 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing touches
🧪 Generate unit tests (beta)
No actionable comments were generated in the recent review. 🎉 🧹 Recent nitpick comments
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Use Nitro server routes at /package/.../llms.txt instead of an API route with middleware rewriting. Single handler re-exported across four route files for unscoped, scoped, and versioned URL patterns.
…txt content Add createPackageLlmsTxtHandler factory for DRY route creation. handleLlmsTxt now accepts includeAgentFiles option to control whether agent instruction files are included (llms_full.txt) or omitted (llms.txt). Add handleOrgLlmsTxt for org-level package listings and generateRootLlmsTxt for the root /llms.txt discovery page. Simplify route handlers to single-line factory calls.
Add server middleware to handle llms.txt routes that Nitro's radix3 file-based router cannot resolve (parameterized intermediate segments don't match literal children). Handles versioned package paths, org-level package listings, and root /llms.txt discovery page. Remove broken versioned route files and add llms_full.txt routes.
Extend canonical redirect regexes with optional /llms.txt and /llms_full.txt suffix capture groups so shorthand URLs like /nuxt/llms.txt redirect to /package/nuxt/llms.txt. Add explicit /llms.txt root path skip to prevent it matching as a package name.
Add ISR rules for llms_full.txt and root /llms.txt routes in nuxt.config.ts. Add #server alias to vitest config for resolving server utility imports in unit tests.
Test route pattern inclusion, example links, base URL substitution, and trailing newline for the root /llms.txt discovery page output.
Vercel ISR glob rules (/package/**/llms.txt) create catch-all serverless functions that intercept requests before Nitro's file-based routes can resolve them, breaking scoped packages and versioned paths. Move all llms.txt/llms_full.txt handling into the middleware, remove ISR route rules, and delete file-based route files.
lukeocodes
changed the title
Fix strict TypeScript errors: add fallback for split()[0] possibly undefined, narrow regex match group types, use non-null assertions for Record lookups after in-check, and use Nuxt's auto-generated Packument type instead of @npm/types import.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
Adds
llms.txtandllms_full.txtsupport across all package URL patterns.Supported routes
/llms.txt/package/nuxt/llms.txt/package/nuxt/v/3.15.0/llms.txt/package/@deepgram/sdk/llms.txt/package/@deepgram/sdk/v/4.0.0/llms.txt/package/@deepgram/llms.txt/package/nuxt/llms_full.txt/nuxt/llms.txt→/package/nuxt/llms.txt/nuxt/v/3.15.0/llms.txt→/package/nuxt/v/3.15.0/llms.txtllms.txt— README + metadata onlyllms_full.txt— README + metadata + agent instruction files (CLAUDE.md, .cursorrules, etc.)Test plan
/llms.txtreturns 200 withtext/markdowncontent typellms_full.txtincludes agent instruction files,llms.txtdoes not/nuxt/llms.txt) redirect 301 to canonical paths