Add Homebrew JSON API and bottle proxy support - #254
Merged
Merged
Conversation
There was a problem hiding this comment.
Pull request overview
Adds first-class proxying for Homebrew’s JSON API and homebrew/core OCI bottles, building on the existing metadata cache + OCI artifact store to support offline use while preserving signed API payload bytes and validating digest-addressed OCI content before caching.
Changes:
- Adds
/homebrew/*handler to proxy Homebrew JSON API responses (including signed JWS) with optional metadata caching. - Extends OCI
/v2/*routing to support per-repository upstream registries (used to sendhomebrew/coreto GHCR by default). - Adds digest verification for cached OCI blobs/manifests and expands validator handling (ETag/Last-Modified, HEAD behavior) across metadata + OCI manifest responses.
Reviewed changes
Copilot reviewed 13 out of 13 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| README.md | Documents Homebrew client configuration and upstream chaining. |
| internal/server/server.go | Wires /homebrew routing and registers homebrew/core registry override for /v2. |
| internal/handler/homebrew.go | New Homebrew JSON API handler and homebrew/core OCI registry registration helper. |
| internal/handler/homebrew_test.go | Tests byte-for-byte signed response preservation, validators, offline stale serving, and path/method rejection. |
| internal/handler/handler.go | Improves validator usage, HEAD behavior, and adds artifact digest verification before caching. |
| internal/handler/handler_test.go | Updates mock storage hashing to SHA-256 to support new digest verification behavior. |
| internal/handler/container.go | Adds per-repository registry routing and digest verification for blob fetches; improves HEAD response header handling. |
| internal/handler/container_test.go | Adds tests for longest-prefix registry routing, credential non-forwarding, and digest mismatch non-caching. |
| internal/handler/container_manifest.go | Adds registry-scoped manifest cache keys, Last-Modified handling, conditional 304 support, and manifest digest verification. |
| internal/config/config.go | Adds configurable Homebrew API + artifact upstreams with defaults and env var support. |
| internal/config/config_test.go | Verifies defaults + YAML/env loading for new Homebrew upstream config. |
| docs/configuration.md | Documents new upstream config keys. |
| config.example.yaml | Adds example Homebrew upstream configuration entries. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
andrew
force-pushed
the
homebrew-proxy-247
branch
from
August 17, 2026 08:40
ad13676 to
2e672fc
Compare
andrew
force-pushed
the
homebrew-proxy-247
branch
from
September 2, 2026 20:24
7693321 to
5daabf3
Compare
Route Homebrew API HEAD requests through ProxyCached so a warm cache answers without an upstream call and stale entries are served when the upstream is unreachable. HEAD still reaches upstream as HEAD when metadata caching is disabled. Limit OCI manifest digest verification to sha256 references and Docker-Content-Digest headers so other digest algorithms are proxied instead of rejected, and log the failing expected value.
Compute real manifest digests in #280's fixture upstreams so the new verification accepts them, and add headerETag / headerLastModified to
The Homebrew API cache key does not include Accept, so replaying the client header could serve one representation under another; the API does not negotiate anyway. Compare If-None-Match with weak comparison, list splitting and "*" per RFC 7232 instead of string equality, and apply the same helper to the metadata and swift responders.
Move the configureScanning doc comment back to its function after the auto-merge stacked it on mountProtocolHandlers, and drop the second ETag/Last-Modified set in writeMetadataCachedResponse now that the pre-304 set covers both response paths.
andrew
force-pushed
the
homebrew-proxy-247
branch
from
September 3, 2026 09:23
54106fe to
211fc3e
Compare
pinguinfuss
added a commit
to pinguinfuss/proxy
that referenced
this pull request
Sep 6, 2026
Homebrew (git-pkgs#254) routes every API path through ProxyCached and so, since git-pkgs#304, fetches formula.jws.json (~33 MB plain, ~5 MB gzip) uncompressed on every refresh -- the case that motivated git-pkgs#305. Request gzip for the JSON API via proxyCachedWithEncoding: brew fetches every API download with curl --compressed and decodes Content-Encoding itself, so the compressed bytes and header are cached and served as-is and both hops stay compressed. The analytics endpoints are the one brew consumer fetched without --compressed; they stay on identity.
andrew
added a commit
that referenced
this pull request
Sep 15, 2026
…ding (#324) * fix(handler): fetch conda repodata gzip-compressed on both hops #304 made the ProxyCached path request Accept-Encoding: identity so the metadata cache stores upstream bytes verbatim. That is required for the signed / hash-pinned index ecosystems, but conda's repodata.json is large plain JSON: linux-64 repodata.json is ~441 MB uncompressed (over the metadata_max_size cap, so it 502s today) versus ~34 MB gzip. Replace the ProxyCached path's verbatim bool with an explicit acceptEncoding string ('' = leave unset / transparent, 'identity', or 'gzip'), reusing #304's existing store-and-replay of Content-Encoding unchanged. ProxyCached keeps its exported signature and continues to send identity, so the nine other ecosystems and helm/maven are untouched; only conda's repodata.json / current_repodata.json now request gzip. Setting Accept-Encoding explicitly disables Go's transparent decompression, so the compressed bytes and the Content-Encoding: gzip header are cached and replayed exactly as identity bytes are. conda, mamba and pixi solicit and decode gzip on .json URLs; repodata.json.bz2 stays identity. Fixes #305 * fix(handler): pass metadata content-encoding with the body it describes The adversarial review of the conda gzip route found a reachable regression: writeMetadataCachedResponse took Content-Encoding from a fresh cache-row read while cacheMetadataBlob skips the row write when Storage.Store fails. Under identity that was benign (the body was plain anyway), but on the new gzip route a disk-full or object-store outage served raw gzip bytes as Content-Type: application/json with no Content-Encoding and HTTP 200 -- conda, mamba and pixi fail to parse them, with no HTTP signal and only a Warn log, on every request until a cache write succeeds. fetchOrCacheMetadata now returns the encoding of the body it hands back (the upstream value on a fetch, the stored row's value on a TTL hit or stale fallback) and proxyCachedWithEncoding passes it to writeMetadataCachedResponse, so the header always describes the bytes actually written. cachedMeta drops its now-unused content_encoding field. helm and maven pass "" -- both fetch transparently, so their stored encoding was always empty and behaviour is unchanged. Also fixes a vacuous assertion in the new conda test: the upstream request counter incremented behind the availability gate, so the cached-replay block could never observe a refetch. * fix(handler): pin the stale-fallback content-encoding and drop a dead guard Follow-ups from the adversarial review of the #305 branch, limited to code this branch introduced: - proxyMetadataStream is only ever reached with an explicit Accept-Encoding (ProxyCached passes identity, conda passes gzip or identity), so the guard around the header set was unreachable; replace it with the plain one-token substitution of the former literal, which is the smallest change from main. - The stale-fallback return of fetchOrCacheMetadata (encoding taken from the cache row) was the one #305 return site no test pinned: replacing it with an empty encoding survived the whole suite. Add a conda test that expires the entry, fails the upstream, and asserts the stored gzip blob is served with Content-Encoding: gzip. Not changed, by scope: cacheMetadataBlob still discards the UpsertMetadataCache error (pre-existing on main). If Storage.Store succeeds and the row write fails, a later stale fallback or TTL hit can serve the gzip blob with the row's stale encoding; that needs a DB write failure plus a second event and is tracked separately. * fix(handler): restore the pre-existing cachedMeta content-encoding field The third adversarial review classified deleting cachedMeta.contentEncoding and its lookupCachedMeta populate as elective: neither line was created by this branch nor forced by the fix (writeMetadataCachedResponse now reads the encoding from its parameter and ignores the row value). Under the rule that pre-existing code this branch did not have to touch stays untouched, restore both as they are on main. No behaviour change. Residuals the review documented, unchanged by scope (both share one root cause: the encoding lives in the cache row and the bytes in the blob, and neither is written or read atomically): - cacheMetadataBlob discards the UpsertMetadataCache error, so after a successful gzip Store and a failed row write a later stale fallback or TTL hit can serve the gzip blob with the row's stale encoding. - During the one-time identity->gzip rollout, a request that read a pre-branch identity row, lost the upstream race to a request that stored the gzip blob, and then failed upstream serves the gzip bytes with no Content-Encoding for that one response; later requests self-heal. - helm and maven now pass an empty encoding; on main a spec-violating upstream that answered a transparent gzip request with an encoding Go does not decode (e.g. br) would have had that header replayed from the row. Degenerate; documented rather than changed. * fix(handler): keep conda's proxyCached and .bz2 route as on main Threading acceptEncoding through CondaHandler.proxyCached changed the form of two pieces of original code the fix did not need to touch: the repodata.json.bz2 route (method value rewritten as a closure) and proxyCached itself (new parameter, new call). Restore both exactly as on main; ProxyCached still sends identity, so the .bz2 route is unchanged in behaviour. handleRepodata's non-cooldown branch now derives the cache key inline and calls proxyCachedWithEncoding with gzip directly, so the only original conda.go line that changes is that one call. * fix(handler): keep writeMetadataCachedResponse and its callers as on main Adding a contentEncoding parameter to writeMetadataCachedResponse changed a signature that predates #304 and dragged its two pre-#304 callers (helm.go, maven.go) into the diff, even though #304 only ever added the cm.contentEncoding block inside the function body. Restore writeMetadataCachedResponse's doc and signature exactly as on main and make it a delegate that passes an empty encoding to a new unexported writeMetadataCachedResponseWithEncoding, which carries the original body with #304's block reading the parameter instead of the cache row. proxyCachedWithEncoding calls the sibling with the encoding returned alongside the body. helm.go and maven.go drop out of the diff; their behaviour is unchanged (both fetch transparently, so their stored encoding was always empty). Same split pattern as ProxyCached -> proxyCachedWithEncoding. * fix(handler): move the conda gzip change to its own branch The conda call site in handleRepodata predates #304 and #304 never touched it, so under the rule that this PR only corrects code and behaviour #304 introduced it does not belong here. Restore conda.go and conda_test.go as on main; the conda change continues on a stacked branch against its own issue. Replace the conda-route tests with tests that exercise proxyCachedWithEncoding directly, so this PR still pins its own plumbing: gzip is requested and the compressed bytes plus Content-Encoding are cached and replayed (cached and streaming paths), the header survives a metadata cache write failure, and the stale fallback keeps the stored encoding. * fix(homebrew): fetch the JSON API gzip-compressed on both hops Homebrew (#254) routes every API path through ProxyCached and so, since #304, fetches formula.jws.json (~33 MB plain, ~5 MB gzip) uncompressed on every refresh -- the case that motivated #305. Request gzip for the JSON API via proxyCachedWithEncoding: brew fetches every API download with curl --compressed and decodes Content-Encoding itself, so the compressed bytes and header are cached and served as-is and both hops stay compressed. The analytics endpoints are the one brew consumer fetched without --compressed; they stay on identity. * fix(handler): leave Accept-Encoding unset in proxyMetadataStream for an empty value fetchUpstreamMetadata treats an empty acceptEncoding as 'do not set the header'; proxyMetadataStream set it unconditionally, which would send an empty Accept-Encoding line if a caller ever passed . Guard it the same way so both paths agree. No caller passes today. * fix(handler): keep the metadata row and blob from describing different bytes Two ways the cache row could stop describing the stored blob once a caller requests gzip, both raised by the review of #324: - cacheMetadataBlob stored the blob and then discarded the UpsertMetadataCache error. After a successful gzip store and a failed row write, a later TTL hit or stale fallback served the gzip blob with the previous row's encoding. On a row-write failure, log it and delete the blob just written, so the next request refetches instead. - fetchOrCacheMetadata read the row once up front and reused it for the stale fallback. A request that read an identity row, lost the upstream race to a request that stored the gzip blob, and then failed upstream labelled the new blob with the old row. Re-read the row before falling back so the encoding matches the blob as it is now. Both only become harmful with an encoding change, which this branch introduces; the pre-existing validator-from-row read is tracked separately. * Drop unused cachedMeta.contentEncoding and fix stale doc reference The field was added by #304 and its only reader is replaced in this branch by the encoding parameter passed alongside the body. The proxyCachedWithEncoding comment named conda repodata, which was moved out of this branch in 5991d95; Homebrew is the caller that ships here. --------- Co-authored-by: Andrew Nesbitt <andrewnez@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds first-class Homebrew JSON API and bottle proxy support for #247.
Signed API responses are cached without rewriting.
homebrew/coremanifests and blobs are served offline from cache and checked against their content digests before cache records are written.The API and artifact upstreams default to
https://formulae.brew.sh/apiandhttps://ghcr.io. Both can be configured through YAML or environment variables, so one proxy can use another proxy as its upstream.Client setup and upstream configuration are documented in the README.
Closes #247