[Content Addressable] Bundler Local Cache and Lockfile

[OughtPuts]

ruby#9733

TL;DR

This PR updates the bundler install flow for locally cached gems to support content addressable gem naming.

Description

This PR builds on the Bundler remote-install support for content-addressed gems.

It writes the gem’s content address and platform to Gemfile.lock, then parses those values back into LazySpecification during a subsequent install. This lets Bundler identify the content-addressed .gem in vendor/cache and install it with bundle install --local, without contacting the remote source again.

The same content-addressed name is used for the installed gem directory in both remote and local installation paths, ensuring lockfile round-trips produce a consistent on-disk layout.

Tests

Updates made to lockfile_parser_spec, content_addressable_spec and ext_spec to cover the changes.

Tophatting

1. Start a local rubygems.org server, that contains a content-addressable gem.
2. Set your gemfile to point to the local server and a content-addressable gem that is available via that server.
3. Run bundle install from the directory that contains your gemfile, using a Ruby version compatible with the content addressable gem.
4. Confirm that the Gemfile.lock contains a content address:

GEM
  remote: http://localhost:3000/
  specs:
    ca_test_harriet (0.3.5-arm64-darwin-23) d26b82f6

PLATFORMS
  arm64-darwin-23

DEPENDENCIES
  ca_test_harriet (= 0.3.5)

CHECKSUMS
  ca_test_harriet (0.3.5-arm64-darwin-23)

BUNDLED WITH
  4.1.0.dev

5. Remove any locally installed gem files other than the cached files (normally in the /cache/ directory)
6. Re-run bundle install --local. Confirm the content addressable gem installs correctly still and that it uses the correct content addressable name.

I've performed this tophat locally and it's all looking good! ✅

[Copilot]

Pull request overview

This PR adds Bundler/RubyGems support for content-addressable gem artifacts end-to-end, including compact-index v2 test infrastructure, lockfile encoding/parsing, and ensuring local cache resolution works with content-addressed filenames.

Changes:

• Extend specification/name tuple objects to carry content_address, and update resolution & lockfile serialization/parsing accordingly.
• Add Artifice compact-index v2 helpers to model content-addressable gems in specs, including platform metadata injection.
• Add/adjust specs to cover lockfile formatting/parsing and local-cache round-trips for content-addressable gems.

Reviewed changes

Copilot reviewed 28 out of 28 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
spec/support/builders.rb	Build helper now supports producing content-addressable gem files (Ruby ABI–scoped).
spec/support/artifice/helpers/compact_index.rb	Checksum calculation updated to use spec.full_name gem filenames.
spec/support/artifice/helpers/compact_index_v2.rb	New compact-index v2 Artifice helper with content-addressable gem support and metadata injection.
spec/support/artifice/helpers/compact_index_cooldown.rb	Removed (superseded by v2 helper).
spec/support/artifice/compact_index_v2.rb	New Artifice activation entrypoint for the v2 helper.
spec/support/artifice/compact_index_cooldown.rb	Removed (tests now use compact_index_v2).
spec/other/ext_spec.rb	Add coverage for Gem::NameTuple#lock_name and Bundler::LazySpecification#to_lock with content addresses.
spec/install/gemfile/content_addressable_spec.rb	New integration specs for content-addressable install, caching, and lockfile round-trips.
spec/install/cooldown_spec.rb	Switch cooldown tests to use the compact-index v2 Artifice setup.
spec/bundler/remote_specification_spec.rb	Rename “platform” to “suffix” in test setup and add content-address expectation coverage.
spec/bundler/override_spec.rb	Ensure overrides stubs include content_address.
spec/bundler/lockfile_parser_spec.rb	Add parsing coverage for lockfile lines with a trailing content address token.
spec/bundler/endpoint_specification_spec.rb	Add coverage for CA suffix handling via platform metadata and remote fetching behavior.
lib/rubygems/specification.rb	Include content_address in Specification#name_tuple.
lib/rubygems/safe_marshal.rb	Permit Gem::NameTuple to safely marshal/unmarshal @content_address.
lib/rubygems/name_tuple.rb	Add content_address field on Gem::NameTuple.
lib/bundler/stub_specification.rb	Propagate content_address when building StubSpecifications.
lib/bundler/source/rubygems.rb	Preserve content address when swapping remote specs after download.
lib/bundler/rubygems_integration.rb	Populate spec content_address from Gem::Package when available.
lib/bundler/rubygems_gem_installer.rb	Call RubyGems’ content-address assignment hook when supported.
lib/bundler/rubygems_ext.rb	Compatibility shims for older RubyGems plus lock name behavior for content-addressed tuples.
lib/bundler/resolver.rb	De-dupe resolved specs including content_address to avoid collapsing distinct CA variants.
lib/bundler/remote_specification.rb	Add content_address and include it in full_name when applicable.
lib/bundler/match_platform.rb	Prefer compatible content-addressable candidates when available.
lib/bundler/lockfile_parser.rb	Parse trailing content-address token on spec lines and propagate into LazySpecification.
lib/bundler/lazy_specification.rb	Track content_address, include it in full_name, and emit it in lock output.
lib/bundler/fetcher.rb	Treat the 3rd token as a “suffix” and pass through to endpoint/remote specification constructors.
lib/bundler/endpoint_specification.rb	Add CA suffix parsing via metadata-provided platform and fetch remote specs using CA suffix when present.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

Pull request overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated 5 comments.

[Copilot]

Pull request overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated 1 comment.

[Copilot]

Pull request overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated 2 comments.

[OughtPuts]

This allows us to correctly find the CA gem spec by either the full_name OR the lock_name after the recent naming changes, when storing the checksum.

[Copilot]

Pull request overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated no new comments.

[jenshenny]

Looking nice! I just had a question on the way we're storing the CA and the platform in the lockfile.

I'm curious on why the platform is being stored for the lock name and have CA outside instead of the CA as the lock name and the platform outside. The CA would provide uniqueness and be clear that it's the suffix of the .gem file but I might be missing something.

[jenshenny]

Should we also check if platform is a non ruby platform so we don't attack a content address in the lockfile?

[jenshenny]

Is there a reason why it was chosen for the lock_name for a CA gem to be same as a fat gem - could we have the lock_name as number-sha to avoid generating specs_by_lock_name?

Then I think we can leverage passing in the content address for LazySpec#name_tuple to get the correct lock name.

[OughtPuts]

Is there a reason why it was chosen for the lock_name for a CA gem to be same as a fat gem - could we have the lock_name as number-sha to avoid generating specs_by_lock_name?

Yeah so as in my other comment about number-platform preserving existing parsing really - my worry would be that if we swap platform in for sha here it extends the blast radius when it's not so important to take that risk. At the moment having SHA after the lock name means it's optional trailing metadata which feels a bit more cautious to me and means we are safer with backwards compatibility.

Happy to sync on this design choice tomorrow to confirm we're aligned?

[jenshenny]

Are these respond_tos for backwards compat reasons? We can add shims to handle that in rubygems_ext!

[OughtPuts]

Looking nice! I just had a question on the way we're storing the CA and the platform in the lockfile.

I'm curious on why the platform is being stored for the lock name and have CA outside instead of the CA as the lock name and the platform outside. The CA would provide uniqueness and be clear that it's the suffix of the .gem file but I might be missing something.

Yeah I see what you mean. It's just because the regex in lockfile_parser already expects to parse the platform in that position, so it made most sense to me to leave platform where it was and add the CA afterwards as an additional placing.

I'm happy to be outvoted though if you feel strongly.

[jenshenny]

Yeah I see what you mean. It's just because the regex in lockfile_parser already expects to parse the platform in that position, so it made most sense to me to leave platform where it was and add the CA afterwards as an additional placing.

Ah I see! I still feel that switching the CA and the platform make most sense to me because it's consistent throughout the system that the content address is the version token for skinny gems and the platform is the version token for fat ones. Having that discrepancy in the lockfile is a bit confusing to me...

It's a bit more work to handle both in $4 of the regex, but we're doing the same in remote specification and storing it as a suffix or version token instead.