Skip to content

rdma: send the descriptor as x-amz-rdma-token; drop the addr:size append - #244

Merged
harshavardhana merged 1 commit into
minio:mainfrom
harshavardhana:harsha/rdma-descriptor-token
Aug 7, 2026
Merged

rdma: send the descriptor as x-amz-rdma-token; drop the addr:size append#244
harshavardhana merged 1 commit into
minio:mainfrom
harshavardhana:harsha/rdma-descriptor-token

Conversation

@harshavardhana

@harshavardhana harshavardhana commented Aug 7, 2026

Copy link
Copy Markdown
Member

What

rdmaPut/rdmaGet no longer append :addr:size to the RDMA token. The token from cuMemObjGetRDMAToken is the cuObject descriptor, whose leading fields already carry the buffer address and transfer size:

<addr:16>:<size:8>:<rkey:8>:<lid:4>:<qp:6>:<gid_present:1>:<gid:32>

Appending :addr:size merely duplicated fields 0 and 1, so the descriptor is now sent verbatim and the unused buf argument is dropped from rdmaPut/rdmaGet.

Validation

Probed on a live mlx5 fabric (libcuobjclient 1.2.0), both GET and PUT:

token = "000073967e5ff000:00400000:01f2e973:0000:007a03:1:...ffff0f0f0fdf"
  field[0] == ptr  ? YES
  field[1] == size ? YES

Server

The server reads addr/size from the descriptor's own fields (miniohq/aistor#6758, #6759). The change is backward-compatible on the server side (it still accepts an appended suffix and takes it as an override), so this can land independently. Companion client changes: nixl ai-dynamo/nixl#1840, minio-rs.

Build verified via ci-rdma (full RDMA build needs the vcpkg + cuObject toolchain).

Summary by CodeRabbit

  • Bug Fixes
    • Improved RDMA PUT and GET operations by using the provided RDMA token directly.
    • Preserved existing retry behavior and token lifecycle handling for more reliable transfers.

The RDMA token from cuMemObjGetRDMAToken already carries the buffer address
(field 0) and transfer size (field 1) in its own leading fields -- verified
against libcuobjclient 1.2.0 for both GET and PUT -- so appending ':addr:size'
merely duplicated them. Send the descriptor verbatim and drop the now-unused
buf argument from rdmaPut/rdmaGet; the server reads addr/size from the
descriptor's own fields.
@coderabbitai

coderabbitai Bot commented Aug 7, 2026

Copy link
Copy Markdown

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: 94d1b4e8-63f7-4f7b-8704-a456ec027cb7

📥 Commits

Reviewing files that changed from the base of the PR and between 5590ba7 and 477d56b.

📒 Files selected for processing (1)
  • include/miniocpp/rdma.h

📝 Walkthrough

Walkthrough

Changes

RDMA token forwarding

Layer / File(s) Summary
Forward tokens through PUT and GET
include/miniocpp/rdma.h
rdmaPut and rdmaGet now use the supplied RDMA token for request signing. Their retry helpers pass the updated arguments. Local token reconstruction was removed.

Estimated code review effort: 2 (Simple) | ~10 minutes

Possibly related PRs

Poem

A rabbit hops through tokens bright,
PUT and GET now pass them right.
No rebuilt token in the stream,
Retries follow the same scheme.
“Sniff!” says Bun, “the flow is clean!”

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly summarizes the main RDMA token change and the removal of the address-and-size suffix.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@harshavardhana
harshavardhana merged commit 769130e into minio:main Aug 7, 2026
12 checks passed
@harshavardhana
harshavardhana deleted the harsha/rdma-descriptor-token branch August 7, 2026 02:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant