[GatewayJ]

Summary

Add opt-in object transfers through an independently built RDMA executable. The public S3 client owns HTTPS signing, request options, multipart management, file publication, and retry decisions. The private provider owns native payload delivery and operation reconciliation.

The private dependency is pinned to fa81f604902a510801b882322b7abb9539a005d4. The independent build helper seeds a missing lockfile from the public workspace and preserves an existing optional-workspace lockfile.

Verification

Validation on Rust 1.99.0 for head 7e2c52a3c9525470a80eaffe190827f917b605a1:

• CARGO_BUILD_JOBS=8 make -j8 pre-commit passed: formatting, all-target workspace Clippy with -D warnings, and workspace tests with 1,950 passed and 2 ignored.
• cargo test --package rustfs-cli --test golden --features golden -j8: 5 passed.
• python3 -m unittest discover -s scripts/tests -p test_build_rdma.py -v: 7 passed.
• git diff --check origin/main...HEAD passed.
• All 18 patches are equivalent according to git range-diff after rebasing onto debec391d60f67e02e16a7fa90a743b1c700ce19.
• The hosted Dependency Advisories check passed for this head.

Previously recorded native evidence includes 30 facade tests, all-target Clippy, and signed HTTPS CLI upload/download interoperability on Linux/aarch64 with software transport. The optional native executable was not rebuilt against the updated workspace dependencies during this rebase.

Hosted check snapshot

Checks for 7e2c52a: CI and Integration Tests. Dependency Advisories, Clippy, formatting, CLI compatibility, Golden tests, both Smoke jobs, documentation, protected-file checks, and MSRV have passed. Platform builds and workspace tests are still running; Full Integration was skipped.

Compatibility

Existing CLI commands retain HTTP as the default, including in the optional executable. Ordinary builds require no private submodule or native libraries. Configuration and output schemas and exit-code definitions are unchanged.

BREAKING Rust source API changes: downstream exhaustive matches on rc_core::Error must handle DataTransfer, and direct construction of Cli must provide data_transport.

Explicit RDMA requires an authenticated, certificate-verified HTTPS endpoint with matching gateway capabilities and the pinned native environment. SSE-C is unsupported by the current native provider; auto can use HTTPS before admission. Unknown write outcomes do not permit generic operation replay.

Review and limitations

• Correctness: external completion checks require the declared length and verified outcome; failed file downloads preserve the destination.
• Compatibility: default HTTP selection and command/output behavior have focused tests and passing workspace and Golden checks.
• Security: signing and credentials remain in the host S3 client; external control requires verified HTTPS and bounded envelopes.
• Concurrency and durability: partial IO and unknown outcomes prevent HTTP fallback and whole-operation replay; the private engine revision is unchanged.
• Simplicity: one shared CLI launcher serves both executables, and the native dependency stays outside the default workspace.
• Test coverage: public integration paths and build fixtures are covered; the updated optional dependency graph and physical hardware remain unverified here.
• Performance: physical RDMA/DPU throughput, latency, and multi-node recovery require separate acceptance.

Users can select --data-transport http or use the ordinary executable to disable external payload transport.

[Copilot]

🟡 Changes recommended

Error displays hide actionable verification and completion diagnostics, and uploads can declare an incorrect payload length when a file changes between metadata lookup and reading.

2 open issues

• Include the redacted source in transfer error display · New
• Use the buffer length for the external request payload · New

What changed in this PR

Adds opt-in RDMA object transfers through a separately built executable, while keeping HTTP as the default and private native dependencies outside the normal workspace.

Changes:

• Introduces transport selection, provider interfaces, signed HTTPS control exchanges, and typed transfer outcomes.
• Integrates external uploads and downloads with checksum verification, atomic file publication, and restricted fallback.
• Shares the CLI launcher and adds optional-build tooling, documentation, and regression tests.

File	Description
scripts/​tests/​test_build_rdma.py	Tests build preflight, arguments, and lockfile preservation.
scripts/​build-rdma.sh	Adds the explicit private-build helper.
README.md	Links optional-build documentation.
rdma/​Cargo.toml	Defines the independent native executable workspace.
docs/​rdma-build.md	Documents building, selection, and runtime limitations.
docs/​ADR/​0001-object-data-transport.md	Records transport ownership and retry boundaries.
crates/​s3/​src/​payload_checksum.rs	Adds streaming checksum verification.
crates/​s3/​src/​lib.rs	Registers transport and checksum modules.
crates/​s3/​src/​data_transport.rs	Defines provider interfaces and observed IO wrappers.
crates/​s3/​src/​client.rs	Integrates external transfers, signing, and regression tests.
crates/​s3/​Cargo.toml	Adds the checksum implementation dependency.
crates/​core/​src/​lib.rs	Exports transport policy and outcome types.
crates/​core/​src/​error.rs	Adds typed transfer errors and exit-code propagation.
crates/​core/​src/​data_transport.rs	Defines transport modes and outcome classifications.
crates/​core/​src/​admin/​kms_diagnostic.rs	Handles and clears transfer-error diagnostics.
crates/​cli/​tests/​help_contract.rs	Includes the new global transport option.
crates/​cli/​src/​main.rs	Delegates startup to the shared launcher.
crates/​cli/​src/​lib.rs	Exports shared launcher functions.
crates/​cli/​src/​launcher.rs	Preserves launch behavior and adds host cleanup support.
crates/​cli/​src/​commands/​rb.rs	Maps typed transfer errors to exit codes.
crates/​cli/​src/​commands/​mod.rs	Parses and applies global transport selection.
Cargo.toml	Excludes optional native workspaces.
Cargo.lock	Records the direct checksum dependency.
.gitmodules	Declares the private transport submodule.
.gitignore	Ignores optional-build artifacts and lockfile.

_{Review effort: Balanced}