Skip to content

feat(vllm): Tackle out of memory errors (EAI-8058) - #251

Open
r0x0r wants to merge 1 commit into
mainfrom
gpu-out-of-memory
Open

feat(vllm): Tackle out of memory errors (EAI-8058)#251
r0x0r wants to merge 1 commit into
mainfrom
gpu-out-of-memory

Conversation

@r0x0r

@r0x0r r0x0r commented Aug 13, 2026

Copy link
Copy Markdown
Collaborator

This pull request improves GPU selection and user guidance for ROCm and vLLM, especially in shared or containerized environments where the standard amd-smi tool may not be available. It adds a fallback for GPU VRAM telemetry, enhances user warnings and hints for out-of-memory (OOM) conditions, and ensures consistent messaging across CLI and engine surfaces. The changes also include comprehensive tests for the new logic.

GPU selection and VRAM telemetry improvements:

  • Added a fallback to read per-GPU VRAM usage from the amdgpu DRM sysfs counters (/sys/class/drm/card*/device/mem_info_vram_{total,used}) when amd-smi is not available, so --gpu auto and low-VRAM warnings work reliably in stripped-down containers and shared nodes. The device count for selection now also derives from these sysfs rows if amd-smi is missing. [1] [2] [3] [4]
  • Added tests to verify that auto-selection uses the sysfs fallback and that VRAM usage is parsed and assigned ordinals correctly. [1] [2]

User guidance and warnings for vLLM:

  • Introduced a shared constant VLLM_GPU_MEMORY_UTILIZATION_HINT for the recommended workaround when running out of memory on a shared/busy GPU, ensuring CLI and engine logs use consistent wording.
  • The serve summary now prints a note about the --gpu-memory-utilization workaround when vLLM is selected and the GPU is busy, both interactively and in the deployment summary. [1] [2] [3] [4] [5]
  • vLLM engine startup logs now append the same utilization hint if an OOM is detected, so users receive actionable advice post-failure.
  • Added tests to ensure the low-VRAM warning and vLLM utilization hint are paired correctly and only shown for vLLM engines. [1] [2]

Documentation:

  • Updated docs/vllm.md to explain the behavior on shared/busy GPUs, the fallback telemetry, and the recommended OOM workaround.

These improvements make GPU selection more robust in diverse environments and provide clear, actionable guidance to users encountering memory issues with vLLM.

@r0x0r
r0x0r requested a review from a team as a code owner August 13, 2026 13:06
…e documentation

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
@r0x0r
r0x0r force-pushed the gpu-out-of-memory branch from be4861d to 3101e06 Compare August 14, 2026 06:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant