feat: use vlt registry as default for all package manager benchmarks#115
Open
vltbaudbot wants to merge 3 commits into
Open
feat: use vlt registry as default for all package manager benchmarks#115vltbaudbot wants to merge 3 commits into
vltbaudbot wants to merge 3 commits into
Conversation
Switch all non-registry benchmark variations to use the vlt registry (https://registry.vlt.io/vlt/npm/) as the default package registry instead of the public npm registry. This serves two purposes: 1. The vlt registry should outperform the default npm registry, so total benchmark suite time should decrease. 2. Running the full benchmark suite against the vlt registry provides continuous load/performance testing to keep us honest as usage grows. Registry benchmark variations (registry-clean, registry-lockfile) are NOT affected — they continue to compare registries as before. Changes: - New `scripts/setup-vlt-registry.sh`: writes vlt registry URL + auth token to project-level .npmrc. No-op when VLT_REGISTRY_AUTH_TOKEN is not set (graceful fallback to npm registry for local development). - `scripts/variations/common.sh`: each PM's setup now calls the registry script before install: - npm, yarn classic, pnpm, pacquet, bun, deno, aube, nx, turbo, vp, node: configured via .npmrc - berry, zpm: configured via npmRegistryServer/npmAuthToken in .yarnrc.yml - vlt: configured via --registry CLI flag + VLT_REGISTRY/VLT_TOKEN env vars (vlt doesn't read .npmrc for registry config) - `scripts/clean-helpers.sh`: clean_all() now calls clean_npmrc() to strip registry/auth lines between runs, preventing stale config.
…oding The registry URL must match/be tied to the auth token since all calls to the vlt registry are authenticated. Reading both VLT_REGISTRY_URL and VLT_REGISTRY_AUTH_TOKEN from environment (configured as GitHub secrets) ensures they always stay in sync. - Updated setup-vlt-registry.sh to read VLT_REGISTRY_URL from env - Updated common.sh to use VLT_REGISTRY_URL env var - Added VLT_REGISTRY_URL to workflow env vars (from secrets) - Both URL and token must be set for vlt registry to activate; if either is missing, falls back to default npm registry with a diagnostic warning
aube@1.13.0 (published 2026-05-13) has a broken arm64 binary — the @endevco/aube-linux-arm64 platform package declares bin/aubr but the file doesn't exist in the tarball, causing npm install to fail with ENOENT during preinstall. Rather than pinning to a specific version, make the aube install non-fatal so a broken aube release doesn't block the entire benchmark suite (all 41+ jobs). If aube fails to install, a warning is logged and the suite continues benchmarking all other package managers.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Switch all non-registry benchmark variations to use the vlt registry (
https://registry.vlt.io/vlt/npm/) as the default package registry instead of the public npm registry.Why
What's NOT changed
Registry benchmark variations (
registry-clean,registry-lockfile) are unaffected — they continue to compare registries (npm, vlt, AWS CodeArtifact, GitHub) as before. Those usescripts/registry/common.sh, not the modifiedscripts/variations/common.sh.How it works
Each package manager is configured differently:
.npmrcwithregistry=+_authToken(viasetup-vlt-registry.sh).yarnrc.ymlwithnpmRegistryServer+npmAuthToken--registryCLI flag +VLT_REGISTRY/VLT_TOKENenv vars.npmrc(installs use npm under the hood)Graceful fallback
When
VLT_REGISTRY_AUTH_TOKENis not set (e.g. local development), the registry setup is a no-op and all PMs fall back to the default npm registry. A warning is logged but execution continues.Files changed
scripts/setup-vlt-registry.sh(new): Standalone script that writes vlt registry config to project-level.npmrc. Idempotent, preserves existing non-registry config lines.scripts/variations/common.sh: Each PM'sBENCH_SETUP_*now calls the registry setup script. vlt's install command includes--registrywhen the vlt registry is active.scripts/clean-helpers.sh:clean_all()now callsclean_npmrc()to strip registry/auth lines between benchmark runs.