Repository navigation
Conversation
|
@JaewonHur Busy week and I hope to get to review this here soon. I saw your comment and I am all for following the direction we take in this PR. Though it might be worthwhile to get started on the discussion on how we want to handle test cases and maybe work together on that piece? |
|
Hi @saehejkang, sorry for moving forward on my own. I just did minimal implementation today, but I'll definitely put our name together here. Further things to do might be:
|
| case stopped | ||
| /// The object is waiting to be restarted. | ||
| case restarting | ||
| /// The object is currently bootstrapped. |
There was a problem hiding this comment.
What does the bootstrapped status mean? The container or sandbox is stopped but ready to be started?
There was a problem hiding this comment.
bootstrapped means (by itself) the container is bootstrapped (i.e., bootstrap function has been successfully executed on that container).
Before this update, the state was same stopped whether it is bootstrapped or not. The only difference was SandboxClient is allocated or not---I personally felt that somewhat weird, but left there.
But after adding restarting state, it becomes more weird, since container state should go restarting (waiting for backOff), then stopped (after bootstrap) then running`.
BTW, now we split the RuntimeStatus to ContainerStatus and SandboxStatus, so it only applies to containers.
There was a problem hiding this comment.
While writing this comment, I found that I missed setting the container restarting in scheduler lol.
There was a problem hiding this comment.
bootstrapped means (by itself) the container is bootstrapped (i.e., bootstrap function has been successfully executed on that container).
I understand that this has to do with the SandboxService and SandboxClient. Except, I have yet to understand what is exactly the point of those two things?
BTW, now we split the RuntimeStatus to ContainerStatus and SandboxStatus, so it only applies to containers.
So it seems like Containers and the Sandbox are two separate entities with different capabilities. Goes back to my question above and how they differentiate.
There was a problem hiding this comment.
Container and Sandbox are separate entities with different lifecycle (though they are tightly coupled).
Sandbox is more for managing the lifecycle of VM.
Sandbox is only alive after the Container successfully executed bootstrap.
| return results | ||
| } | ||
|
|
||
| public func runRestartScheduler() throws { |
There was a problem hiding this comment.
Never thought I would see a scheduler again after OS in college 😆.
Solid foundation work was done with the restart policy and I am sure it could follow the same pattern?
What would be the use case for this? Could it be used to automatically stop trying to restart a container after a certain count?
Hopefully we can use something like #1176 to be a sort of final e2e test of all the policies. |
Thanks, if you make PR to my branch, let me merge it :)
Yes. Docker supports that functionality (https://docs.docker.com/engine/containers/start-containers-automatically/) |
|
@JaewonHur I have not forgotten about this PR. Just another busy week and I hope to come back to this and #1201 in the near future. |
|
@JaewonHur Back at it and here to help wrap this up! I know you are very busy so feel free to assign me to anything from the implementation steps. I think it is best to wrap up |
While this PR was sitting in draft, apple#1258 (by JaewonHur, reviewed by saehejkang) appeared with a substantial in-flight restart manager implementation. To avoid a future merge conflict on every field of this contract, the fork's SDK shape is reshaped to match apple#1258 verbatim before any reviewer sees this PR. Changes ------- RestartPolicy: struct-with-mode-and-maxRetries -> bare String-backed enum { no, onFailure, always }. Matches apple#1258 line-for-line. Drops the 'unless-stopped' mode and the 'maxRetries' field; both are intentionally deferred so this PR mirrors upstream's deliberately conservative initial scope. Follow-ups will land them after apple#1258 merges (CHAOS-1321c). ContainerCreateOptions: restartPolicy goes from optional to non-optional, defaulting to .no. Custom 'init(from:)' uses 'decodeIfPresent ?? .no' so older 'options.json' blobs (no field) still decode cleanly. This is the forward-compat guarantee the original PR description promised but did not actually implement. Flags.swift: '--restart' goes from 'String?' + a hand-rolled 'parseRestartPolicy' helper to '@option var restart: RestartPolicy = .no'. ArgumentParser handles validation natively via a new 'extension RestartPolicy: ExpressibleByArgument {}'; the parser is deleted entirely. As a side benefit, invalid input now produces ArgumentParser's standard 'Invalid value for --restart' error instead of being silently dropped. ContainerRun.swift / ContainerCreate.swift: thread 'managementFlags .restart' through directly. ContainerCreate was missing the wiring entirely in the original PR (the flag declared but never read on the create path) -- fixed here. Tests/ContainerResourceTests/RestartPolicyTests.swift (new): 8 tests covering bare-string encode/decode, every-case round-trip, unknown- string rejection, and the legacy 'options.json' forward-compat invariant (legacy blob without restartPolicy decodes with restartPolicy == .no). Verification ------------ swift build -> Build complete! (21.99s), exit 0. swift test --filter RestartPolicyTests -> 8 tests passed in 0.001s. LSP diagnostics -> clean across all 6 files. Upstream alignment matrix (this PR vs apple#1258) ----------------------------------------------------------- RestartPolicy shape bare enum match unless-stopped mode absent match on-failure:N retries absent match restartPolicy optionality non-optional match decodeIfPresent default .no match Flag binding ExpressibleByArg match RuntimeStatus -> ContainerStatus rename NOT in scope defer The RuntimeStatus rename and the .restarting/.bootstrapped state additions stay deferred -- they belong to the daemon-side enforcement work, which is what apple#1258 actually does. This PR remains data-shape only on the fork side; enforcement arrives when apple#1258 lands upstream and the fork bumps its sync. Refs CHAOS-1321, CHAOS-1385. Sponsors apple#1258, apple#286.
While this PR was sitting in draft, apple#1258 (by JaewonHur, reviewed by saehejkang) appeared with a substantial in-flight restart manager implementation. To avoid a future merge conflict on every field of this contract, the fork's SDK shape is reshaped to match apple#1258 verbatim before any reviewer sees this PR. Changes ------- RestartPolicy: struct-with-mode-and-maxRetries -> bare String-backed enum { no, onFailure, always }. Matches apple#1258 line-for-line. Drops the 'unless-stopped' mode and the 'maxRetries' field; both are intentionally deferred so this PR mirrors upstream's deliberately conservative initial scope. Follow-ups will land them after apple#1258 merges (CHAOS-1321c). ContainerCreateOptions: restartPolicy goes from optional to non-optional, defaulting to .no. Custom 'init(from:)' uses 'decodeIfPresent ?? .no' so older 'options.json' blobs (no field) still decode cleanly. This is the forward-compat guarantee the original PR description promised but did not actually implement. Flags.swift: '--restart' goes from 'String?' + a hand-rolled 'parseRestartPolicy' helper to '@option var restart: RestartPolicy = .no'. ArgumentParser handles validation natively via a new 'extension RestartPolicy: ExpressibleByArgument {}'; the parser is deleted entirely. As a side benefit, invalid input now produces ArgumentParser's standard 'Invalid value for --restart' error instead of being silently dropped. ContainerRun.swift / ContainerCreate.swift: thread 'managementFlags .restart' through directly. ContainerCreate was missing the wiring entirely in the original PR (the flag declared but never read on the create path) -- fixed here. Tests/ContainerResourceTests/RestartPolicyTests.swift (new): 8 tests covering bare-string encode/decode, every-case round-trip, unknown- string rejection, and the legacy 'options.json' forward-compat invariant (legacy blob without restartPolicy decodes with restartPolicy == .no). Verification ------------ swift build -> Build complete! (21.99s), exit 0. swift test --filter RestartPolicyTests -> 8 tests passed in 0.001s. LSP diagnostics -> clean across all 6 files. Upstream alignment matrix (this PR vs apple#1258) ----------------------------------------------------------- RestartPolicy shape bare enum match unless-stopped mode absent match on-failure:N retries absent match restartPolicy optionality non-optional match decodeIfPresent default .no match Flag binding ExpressibleByArg match RuntimeStatus -> ContainerStatus rename NOT in scope defer The RuntimeStatus rename and the .restarting/.bootstrapped state additions stay deferred -- they belong to the daemon-side enforcement work, which is what apple#1258 actually does. This PR remains data-shape only on the fork side; enforcement arrives when apple#1258 lands upstream and the fork bumps its sync. Refs CHAOS-1321, CHAOS-1385. Sponsors apple#1258, apple#286.
|
This is also quite nice when NFS/CIFS/SMB mounts fail and a container needs to restart for connection purposes. |
|
I reviewed this again from the downstream The latest fork work is now split into narrow, PR-shaped branches rather than the older integration commits:
I think the clean review stack is:
Two details I would keep explicit in the first slice:
For Would Apple prefer this PR branch to be refreshed into that smaller stacked shape, or would a new first PR for only the create-time API/CLI contract be easier to review? |
While this PR was sitting in draft, apple#1258 (by JaewonHur, reviewed by saehejkang) appeared with a substantial in-flight restart manager implementation. To avoid a future merge conflict on every field of this contract, the fork's SDK shape is reshaped to match apple#1258 verbatim before any reviewer sees this PR. Changes ------- RestartPolicy: struct-with-mode-and-maxRetries -> bare String-backed enum { no, onFailure, always }. Matches apple#1258 line-for-line. Drops the 'unless-stopped' mode and the 'maxRetries' field; both are intentionally deferred so this PR mirrors upstream's deliberately conservative initial scope. Follow-ups will land them after apple#1258 merges (CHAOS-1321c). ContainerCreateOptions: restartPolicy goes from optional to non-optional, defaulting to .no. Custom 'init(from:)' uses 'decodeIfPresent ?? .no' so older 'options.json' blobs (no field) still decode cleanly. This is the forward-compat guarantee the original PR description promised but did not actually implement. Flags.swift: '--restart' goes from 'String?' + a hand-rolled 'parseRestartPolicy' helper to '@option var restart: RestartPolicy = .no'. ArgumentParser handles validation natively via a new 'extension RestartPolicy: ExpressibleByArgument {}'; the parser is deleted entirely. As a side benefit, invalid input now produces ArgumentParser's standard 'Invalid value for --restart' error instead of being silently dropped. ContainerRun.swift / ContainerCreate.swift: thread 'managementFlags .restart' through directly. ContainerCreate was missing the wiring entirely in the original PR (the flag declared but never read on the create path) -- fixed here. Tests/ContainerResourceTests/RestartPolicyTests.swift (new): 8 tests covering bare-string encode/decode, every-case round-trip, unknown- string rejection, and the legacy 'options.json' forward-compat invariant (legacy blob without restartPolicy decodes with restartPolicy == .no). Verification ------------ swift build -> Build complete! (21.99s), exit 0. swift test --filter RestartPolicyTests -> 8 tests passed in 0.001s. LSP diagnostics -> clean across all 6 files. Upstream alignment matrix (this PR vs apple#1258) ----------------------------------------------------------- RestartPolicy shape bare enum match unless-stopped mode absent match on-failure:N retries absent match restartPolicy optionality non-optional match decodeIfPresent default .no match Flag binding ExpressibleByArg match RuntimeStatus -> ContainerStatus rename NOT in scope defer The RuntimeStatus rename and the .restarting/.bootstrapped state additions stay deferred -- they belong to the daemon-side enforcement work, which is what apple#1258 actually does. This PR remains data-shape only on the fork side; enforcement arrives when apple#1258 lands upstream and the fork bumps its sync. Refs CHAOS-1321, CHAOS-1385. Sponsors apple#1258, apple#286.
77760eb to
a94b2ff
Compare
Code Coverage
|
| public var volumes: [String] = [] | ||
|
|
||
| public func validate() throws { | ||
| if remove && restart != .no { |
There was a problem hiding this comment.
This is an example of the scattered validation I mentioned.
| } | ||
|
|
||
| let state = try self._getContainerState(id: id) | ||
| try await self.lock.withLock(logMetadata: ["acquirer": "\(#function)", "id": "\(id)"]) { context in |
There was a problem hiding this comment.
Just a performance note, I don't think it needs to block this PR.
kill and stop holding the lock will now block all other container operations, and these can take a while.
| ]) | ||
| } | ||
|
|
||
| let options = try getContainerCreationOptions(id: id) |
There was a problem hiding this comment.
This now runs before the state is marked stopped.
If it throws, the container stays running after its VM is gone.
There was a problem hiding this comment.
I think there's no case this throws.
Already existing ContainerCreateOptions doesn't have restartPolicy field, setting it default .no on deserialization.
Later creating ContainerCreateOptions with invalid fields throws validation error.
| try? await self.resetBackOff(id: id) | ||
| } | ||
| ) | ||
| try await stabilityMonitor?.track(id: id) { |
There was a problem hiding this comment.
The stability timer is registered before bootstrap and startProcess, while the PR description says the callback registers on a successful startProcess. If the VM takes longer than the timer to boot, resetBackOff sees a status other than running, skips the reset, and the backoff keeps doubling.
There was a problem hiding this comment.
VM boot taking longer than 10 seconds (i.e., stabilityCall) is abnormal, and I'd handle this later if this is not a critical issue.
| return ExitStatus(exitCode: 0) | ||
| } | ||
|
|
||
| try await bootstrap(id: id, stdio: [FileHandle?](repeating: nil, count: 3), dynamicEnv: [:]) |
There was a problem hiding this comment.
dynamicEnv: [:] is problematic. It means restart of containers using --ssh will be broken, as will anything else that needs to plumb in a value can change between container create and run.
There was a problem hiding this comment.
For this, we'd need in-memory structure to remember ssh env per container.
Though, we need on-disk structure if we need to remember that across daemon lifecycle.
There was a problem hiding this comment.
Do not support --ssh for restartable container.
Containers move to .bootstrapped state after assigning SandboxClient. Following invariant holds: 1. If .stopped, state.client = nil 2. If .bootstrapped, state.client != nil. It'd better to tightly couple the state and client.
Then, add `restarting` state to ContainerStatus
ContainersService spawn an unstructured Task for restart scheduler, which receives stopped container ids through a async queue and restart them if matching conditions. While it doesn't run under lock, it doesn't introduce a new level of race, as all the actual operations changing container state (e.g., `bootstrap`) are lock protected.
This reuses ExitMonitor, which was not initially for this. It might be better to come up with generic Monitor structure?
doStop cannot guarantee the container is stopped in short term.
The catch block's kill could throw out of the unstructured Task, which triggers NoUseUnstructuredThrowingTask under warnings-as-errors. Log the kill failure instead.
manualStopped was never cleared, so a stop followed by a start disabled the restart policy for good. Clear it in bootstrap. stop on a restarting container had no client to stop, so the scheduler still bootstrapped it after the backoff. Move it back to stopped so the scheduler's status check bails.
The CLITest framework was removed on main. Move the restart policy tests to ContainerFixture under Tests/IntegrationTests/Run as a Serial suite.
…rt policy Parse --restart as no, always, on-failure or on-failure:0 (unlimited retries, matching Moby). Reject Swift enum spellings and non-zero retry counts. Reject --rm combined with any restart policy other than no, both in the CLI flags and in ContainersService.create. Add tests for parsing, legacy options decoding, and the --rm conflict.
RestartPolicy.onFailure now carries a maximum retry count, where 0 means unlimited as in Docker/Moby. The restart scheduler counts automatic restarts and stops once the limit is reached; a start that does not come from the scheduler resets the count. Add parsing, validation, and integration tests for the retry limit.
Make ContainerCreateOptions.validate private and call it from the throwing init and from init(from:), so an invalid combination such as --rm with a restart policy cannot be constructed or decoded. Drop the now redundant checks in the CLI flags and ContainersService.create.
Replace the failable ExpressibleByArgument conformance with a transform closure so the specific RestartPolicy parse error reaches the user.
Update testStabilityCall for the longer backoff before run 8 (12.8s).
Only kill a container on restart failure once the scheduler has begun bootstrapping it. A failure in restart() (for example because the user already started the container) must leave it untouched.
kill previously marked the container as stopped by the user before signalling, for any signal and any process. Set the flag only after the kill succeeds, for the init process, and only for SIGKILL, SIGTERM, SIGINT and SIGQUIT.
Keep at most one restart task per container, cancel it when the user stops or kills the container, and bail out of bootstrap if the task was cancelled while waiting for the lock.
f189acb to
fdbf0aa
Compare
The SSH agent socket is passed to bootstrap per start through dynamicEnv, which the restart scheduler cannot supply. Fail bootstrap with an unsupported error when a container with ssh enabled has a restart policy other than no, rolling back to stopped so a scheduled restart does not linger in restarting. Also await restartTasks in kill, which accesses the actor-isolated property from the lock closure, and document --restart and its limitations in the command reference.
fdbf0aa to
585e484
Compare
A bootstrapped container is transient, and before restart policies it was just a stopped container with a runtime client. Do not expose it as a new public status. Represent it as stopped with a live client instead: - bootstrap leaves the status as stopped (also when coming from restarting), so stop tears the runtime down. - handleContainerExit only returns early for stopped containers without a client, so a failed start is still cleaned up by stop. - restart refuses containers that already have a client, so the scheduler cannot take over a container the user is starting.
kill failed with "no runtime client exists" on a restarting container and left the restart scheduled. A terminating signal to the init process now cancels the pending restart, as stop does: mark the container as stopped by the user, clear the backoff, and move it to stopped without signalling a process that does not exist. For any other container the kill is unchanged, and the user stop is still recorded only after the kill succeeds.
delete did not touch the restart task, so deleting a restarting container left its task running until the backoff ended, and a restart could race with the cleanup of a force-deleted one. Cancel the task in cleanUp, which every removal path goes through, so it stops before the container is gone.
The restarting status is new, and existing API clients do not know it. Degrade it to stopped in list, the only path that sends container snapshots to clients, so the API stays backward compatible. The degrade runs before the status filter, so a restarting container also matches a stopped filter, consistent with what clients see.
Closes #286.
The first milestone is the restart policies
no,on-failure[:<max-retries>]andalways, which behave like the Docker restart policies. The CLI exposes them as --restart. The design can be extended later to support auto-start (#158, #1201).CLI
--restart on container run and container create:
on-failure:-1,on-failure:abc,on-failure:) and counts with other policies (always:3).unless-stoppedis deliberately left out as we need an additional persistent state for remembering whether a container was manually stopped or not. Will figure out after this.Persistent and runtime state
ContainerCreateOptions (persistent)
alwaysor.onFailure(maximumRetryCount:). The retry limit is part of the policy, not a separate option.ContainerState (runtime)
A new restarting status is added to RuntimeStatus. A container that is going to restart enters restarting, waits for its backOff, then runs bootstrap followed by startProcess. bootstrapped is also added as an explicit status between stopped and running. Stopping or killing a container while it is restarting cancels the pending restart.
Restart logic (handleContainerExit)
no: stay stopped.on-failure[:n]: stay stopped on a start error, a manual stop, a zero exit code, or when restartCount has reached n (if n > 0). Otherwise restart.always: stay stopped on a start error or a manual stop. Otherwise restart.A restart scheduler task in ContainersService receives the ids of stopped containers through an async queue and restarts those that match the policy. The scheduler doesn't hold the lock. Every state-changing operation it performs, such as bootstrap, is lock-protected, so this adds no new race.
Stability call
A container that stays running for 10 seconds is marked stable, and its backOff is reset. A monitor (similar to ExitMonitor) registers the callback on a successful startProcess.