feat: configure recovery retention and verify Tuist Git hosting - #51
Merged
Merged
Conversation
Git compresses large fetch negotiation requests, which made the full Tuist history clone fail when compressed bytes reached Git unchanged. Decode them incrementally, limit decoded work, and reject malformed streams and compressed pushes before materializing a replica. Expose decoded volume and rejection reasons through metrics and structured logs. Verified with the full unit suite, lint, static type analysis, the real Git compatibility checks, and the two-node end-to-end suite. Claude's adversarial review and focused follow-up found no remaining correctness blockers.
Exercise protocol versions, compressed negotiation, shallow and partial clones, tags, atomic rejection, leases, deletion and mirrors against real Git clients. Require the expected rejection reason so transport failures cannot pass negative checks. Add an opt-in full-history Tuist rehearsal with frozen references and object sets, compaction epoch checks, and explicit eviction checks on both nodes. The real two-node suite passes all 58 examples. A captured Tuist source with 4,395 reference targets and 459,778 reachable objects passed the full import, compaction, cache reconstruction, shallow and partial clone rehearsal. Timings and operational limitations are recorded in the verification documentation.
Store per-repository recovery retention overrides in the durable index, default to forever, and expose administrator configuration and read-only storage reports. Protect current objects, retained recovery data and the complete snapshot chain; keep unknown uploads unclassified. Do not delete any objects. Validate snapshot digests and identity, distinguish damaged history from missing repositories, refresh current protection for same-epoch writers, and bound report history, inventory, response size, concurrency and duration. Preserve conservative clock-skew handling and document legacy-history migration limitations. Emit metrics, trace spans and correlated structured logs. Verified with 513 passing unit tests, lint, static type analysis, 58 passing real two-node end-to-end examples, and Helm lint and rendering. Claude reviewed the changes adversarially; the findings were addressed and its focused follow-up found no remaining correctness blockers.
pepicrft
marked this pull request as ready for review
September 27, 2026 12:42
Accumulate unique required, recovery, and expired pointers while walking the snapshot chain. Use a local registry for the per-node concurrency limit and check availability once against the revalidated index. Skip metadata reads for unchanged pointers and retain protection for newly published current objects. Make the listing budget type include a valid zero remaining budget. A local 800-snapshot, 50-shared-pack benchmark produced identical reports in three alternating runs per implementation. Median duration fell from 827 to 614 milliseconds; sampled report-process memory fell from 32.8 to 4.1 megabytes, and individual metadata reads fell from 150 to zero. Verified with 515 passing unit tests, lint, static type analysis, all 58 real two-node end-to-end examples, and a focused real storage retention check after settling on the final standard set representation. New property tests cover unchanged-index request costs and a racing writer reusing an expired pack.
Carry the decoded-byte ceiling with the request decoder rather than looking up configuration for every decompression buffer. Preserve the same streaming bounds, rejection responses, and decoded-volume metrics. Verified by the compressed-request unit tests, the full unit suite, lint, static type analysis, and the real Git compatibility checks.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed
Add configurable recovery retention with a deployment default and durable repository overrides. Administrators can request a read-only storage report that separates current objects, retained recovery objects, objects referenced exclusively by expired snapshots, and unclassified uploads. Retention defaults to forever. Automatic deletion is not implemented.
Add a Git compatibility suite and a repeatable full history Tuist import rehearsal. Fix streaming decoding of compressed fetch requests uncovered by that rehearsal.
Why
Before moving
tuist/tuistto the hosted instance, we need evidence that its history survives import, compaction, and reconstruction on either node. We also need visibility into storage growth and an explicit recovery policy before implementing collection.Root cause
Git compresses large fetch negotiation requests. Code previously forwarded those compressed bytes directly to Git, causing the full history Tuist clone to fail.
Approach
Decode gzip fetch requests incrementally with bounded buffers and a configurable decoded-byte ceiling. Reject unsupported encodings and compressed pushes before replica synchronization. Reject corrupt streams, trailing data, and truncated streams during decoding. The compatibility checks cover protocol versions 0 and 2, compressed requests, shallow and partial clones, tags, atomic pushes, explicit leases, deletion, and mirroring.
Retention reports follow the durable compaction snapshot chain, validate snapshot digests and repository identity, protect shared packs, sidecars, and all snapshot metadata, and revalidate the current index before returning. Same-epoch updates refresh current protection. Snapshot pointers are deduplicated during traversal; unchanged indexes need no individual metadata requests. Report concurrency uses a local registry, and object availability is checked once against the revalidated index. Reports bound history, listings, response size, concurrency, and duration. Unknown uploads remain unclassified. Reports never read pack bodies or delete objects. Operations expose traces, metrics, and structured logs.
Impact
Existing repositories inherit
CODE_HISTORY_RETENTION_DAYS, which defaults toforever; the chart exposesconfig.historyRetentionDays. Repository overrides survive compaction and cache reconstruction. The administrator endpoints document the reporting-only behavior and stable failure responses. Unverifiable legacy snapshot chains fail closed and require a separately verified migration, which is not implemented. Reports exceeding their resource limits are refused rather than returning partial eligibility.The Tuist rehearsal imported 4,395 branch and tag targets and 459,778 reachable objects. Three full clones matched the frozen source and passed integrity checks, including reconstruction after both node caches were evicted. This verifies a local captured source snapshot, not production traffic or GitHub collaboration features. The sampled repacking process used approximately 3.5 gigabytes of resident memory, so maintenance sizing needs to account for Git as well as Code.
Validation
mise exec -- mix test --max-cases 4: 515 passed.mise run lint: passed.mise run typecheck: passed.CODE_VERIFY_SOURCE="$PWD/tmp/verification-source/tuist.git" bash scripts/verification/large-repository.sh: passed. The repeated Tuist rehearsal asserted the epoch advanced from 1 to 2, verified eviction on both nodes, and checked full reconstruction, integrity, shallow clone, and partial clone with checkout.mise exec -- helm lint charts/codeand chart rendering with a 90-day default: passed.Claude performed an adversarial review and a focused follow-up. Findings were addressed, including missing-history status codes, preservation of snapshot links, bounded report costs, same-epoch races, compressed-request amplification, and stronger verification assertions. The follow-up found no remaining correctness blockers.
Performance and maintenance
A local filesystem benchmark compared the previous implementation at
7c3c0edwith the revised report using 800 snapshots, 50 shared packs, and 100 references. Three alternating runs of each implementation produced identical reports.The implementation accumulates unique pointers in standard sets instead of keeping one record per snapshot. New tests require zero metadata requests for unchanged indexes and verify that a racing writer moves an expired pack into current protection. Gzip decoding reads its configured size limit once per request.
The complete unit suite, lint, and static type analysis pass. The full end-to-end suite passed all 58 examples; the final set representation was also verified against the real storage retention example.