feat(object-store): upload large packs through S3 multipart

[pepicrft]

What changed

Packs larger than a configurable threshold are now uploaded to object storage with S3 multipart upload instead of a single PUT. S3 is Amazon Simple Storage Service, whose API every store Code supports implements.

• Code.ObjectStore.S3.put_file/4 routes files above CODE_S3_MULTIPART_THRESHOLD_BYTES (100 MiB by default) through InitiateMultipartUpload, one UploadPart per CODE_S3_MULTIPART_PART_SIZE_BYTES slice (64 MiB by default), and CompleteMultipartUpload, aborting the upload on any failure. Smaller files keep the existing single PUT.
• The part size is validated against S3's limits (5 MiB to 5 GiB) when the node boots, and again before any upload.
• A 409 status on a pack upload is now treated as a conflict instead of as "already stored", on both the multipart and single PUT paths (see Root cause).
• New telemetry event [:code, :object_store, :multipart_upload], exported to Prometheus as code_object_store_multipart_upload_count, _bytes and _parts. A failed abort is logged as a warning with operation=multipart_abort.
• docs/architecture.md and docs/operations.md describe the new behaviour, the two environment variables, the store requirements, and a recommended bucket lifecycle rule for incomplete uploads.
• The end-to-end stack runs both nodes with a 5 MiB threshold and part size, and a new spec/multipart_upload_spec.sh covers a large and a small push.

Why

A single S3 PUT cannot exceed 5 GiB, and docs/architecture.md listed multipart upload as not implemented. A repository whose pack grew past 5 GiB could not be pushed at all. Of the gaps the docs call out, this was the one with the clearest contract and the smallest blast radius: it stays inside the object store abstraction and does not touch how pushes are ordered. The ceiling is now the part size times S3's 10,000-part limit, about 625 GiB at the default part size.

Root cause

Two bugs in the first version of this change were found by the new tests and reviews rather than by production, and are worth recording.

CompleteMultipartUpload returns the final ETag (the object's entity tag) inside its XML body, not as a response header. The first version read the header, so every multipart push failed. The end-to-end suite caught this on its first run against RustFS.

The existing single PUT path mapped both 409 and 412 to :precondition_failed, and the multipart completion copied that. For packs, Code.WAL turns :precondition_failed into "already stored". But S3 returns 409 (ConditionalRequestConflict) after a concurrent write or delete of the same key, so a 409 is not evidence that the pack exists, and treating it as success could let the log reference a missing pack. put_file/4 now reports a precondition failure after a 409 only when a follow-up HEAD finds the object. Otherwise it returns {:error, {:conflict, 409}}, which fails the push for the client to retry. put/3, which the index compare-and-swap uses, is unchanged: its caller re-reads and retries anyway.

Approach

The invariant that matters is that pack writes are create-only and idempotent. The first version enforced that with a HEAD before initiating the upload. That leaves a window where two writers both see the key as absent and both complete. The fix is to put If-None-Match: * on CompleteMultipartUpload itself, which S3 evaluates atomically, and to keep the HEAD only to avoid re-sending a pack that is already stored. A 403 on that HEAD is treated as "unknown", because AWS returns 403 for absent keys to credentials without s3:ListBucket, and the conditional completion still arbitrates.

Everything after initiate either completes or aborts. That includes raised exceptions, for example the source file disappearing between parts, which are then re-raised unchanged. Only a killed process skips the abort, which is what the documented AbortIncompleteMultipartUpload lifecycle rule is for.

Preconditions that multipart cannot enforce atomically (if_match, or an ETag-valued if_none_match) are refused rather than silently dropped. No caller uses them today.

Parts stream off disk in the same 1 MiB sub-chunks as a single PUT, so the "never hold a pack in memory" rule still holds.

Two options were considered and left out. Retrying individual parts would be more resilient, but I have not confirmed that Req re-signs a retried request, and a retry of the whole upload is already idempotent. The first review round also asked for an end-to-end test of two writers racing on completion. That is covered by unit tests plus a manual probe (below) rather than a new spec, because forcing the race through git push is fragile.

The change went through three rounds of adversarial review with Pi and Codex. Their findings were: the abort was skipped when an exception was raised; completion was unconditional; part sizes were not validated; HEAD behaves differently under least-privilege credentials; non-* preconditions were dropped silently; and the 409 mapping above. All are fixed and covered by tests, and both reviewers consider the final state mergeable.

Impact

• Pushes whose pack exceeds 5 GiB now work. Packs up to 100 MiB behave exactly as before.
• Operators get two optional environment variables. A part size outside 5 MiB to 5 GiB stops the node at boot with a message naming the valid range.
• The store must honour If-None-Match: * on CompleteMultipartUpload. AWS S3 documents it and RustFS enforces it. MinIO, Cloudflare R2, Ceph and Tigris are unverified, which docs/operations.md states. A store that ignores the header is harmless for content-addressed packs. A store that rejects it would fail packs above the threshold.
• Operators should add an incomplete-upload lifecycle rule to the bucket (example in docs/operations.md).
• A 409 on a pack upload now fails the push instead of being reported as success when the pack is absent.

Validation

• mise run lint: format and Credo strict, no issues.
• mise run test: 484 tests pass, including 20 new Req-stubbed tests in test/code/object_store/s3_test.exs. They cover threshold routing and clamping, part boundaries, the conditional completion header, HEAD 200 and 403, a 412 and a 409 at completion, a failed part, a part with no ETag, a 200 response carrying an <Error> body, a missing upload identifier, a source vanishing mid-upload, abort failures, refused preconditions, invalid part sizes, and the size ceiling.
• mise run typecheck: no errors.
• mise run e2e: 56 examples, 0 failures, against RustFS with a 5 MiB threshold. That includes the new multipart spec and the existing 64 MiB replication example, which now uploads in 13 parts.
• Manual probe against the end-to-end RustFS: completing a multipart upload onto an existing key with If-None-Match: * returns 412 PreconditionFailed.

🤖 Generated with Claude Code