DRA: Handle permanent driver failures

[nojnhuh]

Enhancement Description

DRA drivers may encounter errors such that the devices allocated by kube-scheduler for a pod can never be successfully returned from the NodePrepareResources gRPC call to the driver. Currently, pods in that state will be continuously retried forever, wasting CPU cycles in the kubelet and DRA driver. This proposal describes a method to break that cycle of continuous retries that are known will fail.

/sig node
/wg device-management
/assign @nojnhuh
/cc @pohly @lauralorenz @SergeyKanzhelev

• One-line enhancement description (can be used as a release note): DRA: Handle permanent driver failures
• Kubernetes Enhancement Proposal: https://github.com/nojnhuh/enhancements/blob/5322-dra-perma-err/keps/sig-node/5322-dra-driver-permanent-failure/README.md
• Discussion Link:
  • 13 May 2025 SIG Node meeting
  • DRA: kubelet: support permanent and transient errors kubernetes#125542
• Primary contact (assignee): @nojnhuh
• Responsible SIGs: SIG Node
• Enhancement target (which target equals to which milestone):
  • Alpha release target (x.y):
  • Beta release target (x.y):
  • Stable release target (x.y):
• Alpha
  • KEP (k/enhancements) update PR(s):
    • KEP-5322: DRA: Handle permanent driver failures #5549
  • Code (k/k) update PR(s):
  • Docs (k/website) update PR(s):

Please keep this description up to date. This will help the Enhancement Team to track the evolution of the enhancement efficiently.

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[SergeyKanzhelev]

/remove-lifecycle stale

We started doing it with the sig storage recently: kubernetes/kubernetes#132933

If we can create a new error code from the DRA - we can use it for failing admission late. It off course may mean that the Pod will be scheduled again in the same place and fail the same way. So ideally DRA needs to return the perma failure only if it updated the health of resources.