POC: agentInject agent-only checkpoint storage (do not merge) - #4
Closed
Ronkahn21 wants to merge 2 commits into
Closed
POC: agentInject agent-only checkpoint storage (do not merge)#4Ronkahn21 wants to merge 2 commits into
Ronkahn21 wants to merge 2 commits into
Conversation
Signed-off-by: Ron Kahn <rkahn@nvidia.com>
Adds an opt-in storage access mode where workload pods never mount the checkpoint PVC. The node agent mounts it (like agentMount) and, for restore, grafts the checkpoint dir into the target container's mount namespace via open_tree(OPEN_TREE_CLONE)+move_mount inside nsrestore, before CRIU reads it. A plain /proc/self/fd bind cannot cross mount namespaces (EINVAL); open_tree clones a detached mount that move_mount attaches in the container ns. Operator skips PVC injection into capture/restore pods (PVCName empty) and stamps the resolved access mode; the agent treats the stamped value as authoritative over its own config. Signed-off-by: Ron Kahn <rkahn@nvidia.com>
github-actions
Bot
added
documentation
Owner
Author
|
Superseded: reopened against upstream ai-dynamo/dynamo (main) for full CI. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
POC — do not merge
Proof-of-concept for an agent-only checkpoint storage mode (
accessMode: agentInject) where theworkload pod never mounts the checkpoint PVC. The node agent owns the PVC and grafts the checkpoint
into the restore container's mount namespace via
open_tree(OPEN_TREE_CLONE)+move_mount, so CRIUcan read it. Opened to exercise CI on the change; not intended to merge as-is.
What it does
accessMode: agentInject(agent config + operator config + Helm).PVCNameempty) andstamps the resolved access mode; the agent treats the stamped value as authoritative.
nsrestoregains--checkpoint-fd: the agent clones its own checkpoint dir into a detached mount(
open_tree), passes the fd viaexec.Cmd.ExtraFiles, andnsrestoregrafts it withmove_mountinside the container before CRIU restore.
agentMount(agent writes via its own mount).Validation done outside CI
/proc/self/fdbind isEINVALcross-ns;
open_tree+move_mountworks.open_tree+move_mount,criu restorefrom the grafted mount, process resumes.
Not covered here
See
plans/snapshot-migration/agent-inject-cross-ns-explainer.mdfor the kernel-level writeup.