✨ Fail-fast and reap AgentRuns whose pods never start - #203
Conversation
An AgentRun whose sandbox pod never reaches a healthy Running state — ImagePullBackOff, CrashLoopBackOff, InvalidImageName, CreateContainerConfigError, or Pending-because-unschedulable — used to sit in Pending forever: nothing observed the pod-level error, so the run was neither usable nor eligible for cleanup. These are the "broken agent runs with error conditions on the pod" that pile up. The AgentRun reconciler now, on a not-yet-running pod: - Fails the run immediately when a container (init containers first, so a bad skill-loader image is caught) is stuck on a fatal waiting reason. The kubelet's own reason (ImagePullBackOff, CrashLoopBackOff, InvalidImageName, CreateContainerConfigError) is surfaced verbatim on the Succeeded condition so the vocabulary matches the pod (shares the reporting direction of konveyor#171). - Enforces an optional startup deadline: if the pod has not reached a running state within spec.startupDeadlineSeconds (or the controller default --agentrun-startup-deadline), the run is failed with StartupDeadlineExceeded. A requeue drives the deadline even when no further pod event arrives (e.g. a pod stuck unschedulable). Both set a terminal Failed phase + CompletionTime, so the run stops occupying an active slot and becomes eligible for TTL GC (konveyor#198). Closes konveyor#199 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: ibolton336 <ibolton@redhat.com>
|
Important
This repository does not receive automatic reviews because it has fewer than 10 stars. ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
ibolton336
changed the title
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: ibolton336 <ibolton@redhat.com>
djzager
left a comment
djzager
left a comment
There was a problem hiding this comment.
Approving — clean fail-fast logic with good envtest coverage. Two low-severity, non-blocking notes below.
| // the container cannot be configured, or it crashes on every start. They are | ||
| // settled states — kubelet has already retried into a back-off, or the | ||
| // config is permanently wrong — so the run is failed rather than waited on. | ||
| var fatalWaitingReasons = map[string]bool{ |
There was a problem hiding this comment.
Sandbox pods use RestartPolicyNever (pod template ~L677, per the #51 comment), so a crashing container terminates rather than entering a CrashLoopBackOff waiting state — that case is handled by the Sandbox Finished/setTerminalOutcome path instead. CrashLoopBackOff here is effectively unreachable for this workload. Harmless as a guard, but a one-line note that it's defending against a future restart-policy change would help a reader who knows these pods never restart.
| if pod != nil { | ||
| podPhase = string(pod.Status.Phase) | ||
| } | ||
| setRunSucceeded(run, metav1.ConditionUnknown, "PodNotRunning", |
There was a problem hiding this comment.
Nit: this branch reports the bare podPhase, while the deadline-configured branch above reports the richer podStartupDetail(pod). The Pending message an operator sees then depends on whether a deadline happens to be set (e.g. an unschedulable pod shows unschedulable: ... in one path, just Pending in the other). Consider using podStartupDetail(pod) here too for consistency.
2 participants
Problem
An
AgentRunwhose sandbox pod never reaches a healthyRunningstate —ImagePullBackOff,CrashLoopBackOff,InvalidImageName,CreateContainerConfigError, orPendingbecause it is unschedulable — sat inPendingforever. Nothing observed the pod-level error condition, so the runwas neither usable nor eligible for cleanup. These are the "broken agent runs
with error conditions on the pod" that pile up.
Change
The AgentRun reconciler now, on a not-yet-running pod:
containers first, so a bad skill-loader image is caught before the agent
container) for settled waiting reasons —
ImagePullBackOff,CrashLoopBackOff,InvalidImageName,CreateContainerConfigError. Thekubelet's own reason is surfaced verbatim on the
Succeededcondition, sothe reason vocabulary matches what an operator sees on the pod (aligns with
the reporting direction of Skill-loader failures and never-starting pods never reach AgentRun.status #171). Checked before the pod-phase gate, since a
container stuck in
CrashLoopBackOffcan still put the pod inRunning.running state within
spec.startupDeadlineSeconds(or the controller default--agentrun-startup-deadline), the run is failed withStartupDeadlineExceeded. A requeue drives the deadline even when no furtherpod event arrives (e.g. a pod stuck unschedulable). Both default to disabled;
fatal pod errors fail runs immediately regardless.
Both paths set a terminal
Failedphase +CompletionTime, so the run stopsoccupying an active slot and becomes eligible for TTL GC (#198).
API
AgentRunSpec.startupDeadlineSeconds(*int32,+optional,Minimum=0).--agentrun-startup-deadlinecontroller flag (default0= disabled).Tests
envtest integration coverage:
ImagePullBackOff→ runFailed, reasonImagePullBackOff, message names the container,CompletionTimeset.startupDeadlineSeconds→ runFailed,reason
StartupDeadlineExceeded.Closes #199
🤖 Generated with Claude Code