VM on Xen - weird ISO issue #10581

tdtmusic2 · 2025-03-17T20:39:20Z

tdtmusic2
Mar 17, 2025

Hi everyone. Got a strange error today and it doesn't make sense as it's not related to the actual action that I want to do, or at least I don't think so, maybe someone else has an idea. The setup is simple, a xen host added in ACS 4.19.2, with a GPU in passtrough mode, a Windows VM deployed with a W11 iso attached and Windows PV as OS version. Setup goes all the way, although I noticed it's MUCH slower than deploying the same way but with Windows 11 as OS version instead of PV (and manually adding the virtio drivers for the disk controller in the setup). After the setup I wanted to detach the ISO to attach another one and I got this error. The only way to detach it was to stop the VM and do it. I tested with other iso's and it's the same behavior. The logs bellow don't seem to reveal much about the issue either. Any thoughts?

2025-03-17 16:13:19,472 INFO  [o.a.c.f.j.i.AsyncJobMonitor] (API-Job-Executor-12:ctx-5af21047 job-337) (logid:8eec4d00) Add job-337 into job monitoring
2025-03-17 16:13:19,474 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-12:ctx-5af21047 job-337) (logid:e034732e) Executing AsyncJobVO: {id:337, usene, instanceId: 49, cmd: org.apache.cloudstack.api.command.admin.iso.DetachIsoCmdByAdmin, cmdInfo: {"virtualmachineid":"066a0d43-a8e1-47a8-a302-d84a51df8d84",uZWE79gkLJARV7XWYGQoTHFX7Cg","httpmethod":"GET","ctxStartEventId":"623","ctxDetails":"{\"interface com.cloud.vm.VirtualMachine\":\"066a0d43-a8e1-47a8-a302-d84"ISO.DETACH"}, cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result: null, initMsid: 52242796961, completeMsid: null, lastUpdated: null}
2025-03-17 16:13:19,474 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (qtp146370526-15:ctx-0015a751 ctx-63f74fd1) (logid:c3b82fb4) submit async job-337, details: AsanceType: VirtualMachine, instanceId: 49, cmd: org.apache.cloudstack.api.command.admin.iso.DetachIsoCmdByAdmin, cmdInfo: {"virtualmachineid":"066a0d43-a8e1-47Id":"2","sessionkey":"uZWE79gkLJARV7XWYGQoTHFX7Cg","httpmethod":"GET","ctxStartEventId":"623","ctxDetails":"{\"interface com.cloud.vm.VirtualMachine\":\"066a0d":"2","cmdEventType":"ISO.DETACH"}, cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result: null, initMsid: 52242796961, completeMsid: nd: null, removed: null}
2025-03-17 16:13:19,474 DEBUG [c.c.a.ApiServlet] (qtp146370526-15:ctx-0015a751 ctx-63f74fd1) (logid:c3b82fb4) ===END===  10.84.84.1 -- GET  virtualmachineid=0chIso&response=json&sessionkey=uZWE79gkLJARV7XWYGQoTHFX7Cg
2025-03-17 16:13:19,487 DEBUG [o.a.c.s.i.TemplateDataFactoryImpl] (API-Job-Executor-12:ctx-5af21047 job-337 ctx-d5fbc440) (logid:e034732e) template 204 is alr
2025-03-17 16:13:19,492 ERROR [c.c.r.ResourceManagerImpl] (API-Job-Executor-12:ctx-5af21047 job-337 ctx-d5fbc440) (logid:e034732e) Host 11 does not have requioup of Nvidia Corporation TU104GL [Tesla T4] GPUs, vGPU Type: passthrough
2025-03-17 16:13:19,495 ERROR [c.c.a.ApiAsyncJobDispatcher] (API-Job-Executor-12:ctx-5af21047 job-337) (logid:e034732e) Unexpected exception while executing ohIsoCmdByAdmin
com.cloud.utils.exception.CloudRuntimeException: Host 11 does not have required GPU device or out of capacity. GPU group: Group of Nvidia Corporation TU104GL
        at com.cloud.resource.ResourceManagerImpl.getGPUDevice(ResourceManagerImpl.java:3469)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
        at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
        at com.sun.proxy.$Proxy200.getGPUDevice(Unknown Source)
        at com.cloud.hypervisor.HypervisorGuruBase.toVirtualMachineTO(HypervisorGuruBase.java:285)
        at com.cloud.hypervisor.XenServerGuru.implement(XenServerGuru.java:93)
        at com.cloud.template.TemplateManagerImpl.attachISOToVM(TemplateManagerImpl.java:1296)
        at com.cloud.template.TemplateManagerImpl.attachISOToVM(TemplateManagerImpl.java:1314)
        at com.cloud.template.TemplateManagerImpl.detachIso(TemplateManagerImpl.java:1175)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
        at org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:107)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
        at com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:52)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
        at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
        at com.sun.proxy.$Proxy236.detachIso(Unknown Source)
        at org.apache.cloudstack.api.command.user.iso.DetachIsoCmd.execute(DetachIsoCmd.java:109)
        at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:172)
        at com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:112)
        at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:654)
        at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
        at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
        at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:602)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:829)
2025-03-17 16:13:19,497 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-12:ctx-5af21047 job-337) (logid:e034732e) Complete async job-337, jobStatus:udstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":"530","errortext":"Host 11 does not have required GPU device or out of capacity. GPU grT4] GPUs, vGPU Type: passthrough"}
2025-03-17 16:13:19,498 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-12:ctx-5af21047 job-337) (logid:e034732e) Publish async job-337 complete on
2025-03-17 16:13:19,498 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-12:ctx-5af21047 job-337) (logid:e034732e) Wake up jobs related to job-337
2025-03-17 16:13:19,498 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-12:ctx-5af21047 job-337) (logid:e034732e) Update db status for job-337
2025-03-17 16:13:19,499 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-12:ctx-5af21047 job-337) (logid:e034732e) Wake up jobs joined with job-337 a
2025-03-17 16:13:19,500 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-12:ctx-5af21047 job-337) (logid:e034732e) Done executing org.apache.cloudsta job-337
2025-03-17 16:13:19,500 INFO  [o.a.c.f.j.i.AsyncJobMonitor] (API-Job-Executor-12:ctx-5af21047 job-337) (logid:e034732e) Remove job-337 from job monitoring

DaanHoogland · 2025-03-26T10:34:13Z

DaanHoogland
Mar 26, 2025
Collaborator

@tdtmusic2 , i cannot give you a direct answer. Can you tell me the exact version which that stacktrace came from? I'll have a look.

2 replies

tdtmusic2 Mar 27, 2025
Author

@tdtmusic2 , i cannot give you a direct answer. Can you tell me the exact version which that stacktrace came from? I'll have a look.

Hi Daan. You mean CS version? It's 4.19.2.0.

DaanHoogland Mar 27, 2025
Collaborator

@tdtmusic2 , can you check the contents of your host_gpu_groups table? The code is expecting to find content in it and doesn't.

In the logs above I see ...does not have requioup of... , the word requioup does not exist and does not occur anywhere in the code.

2025-03-17 16:13:19,492 ERROR [c.c.r.ResourceManagerImpl] (API-Job-Executor-12:ctx-5af21047 job-337 ctx-d5fbc440) (logid:e034732e) Host 11 does not have requioup of Nvidia Corporation TU104GL [Tesla T4] GPUs, vGPU Type: passthrough

the line should read something like:

"Host " + hostId + " does not have required GPU device or out of capacity. GPU group: " + groupName + ", vGPU Type: " + vgpuType;

You might also want to try restarting the management servers.

tdtmusic2 · 2025-03-27T10:25:38Z

tdtmusic2
Mar 27, 2025
Author

This is what's in the sql table, I guess it's as it should be. Just tried to deploy another instance now, with gpu passthrough, and while powered on I tried to detach the iso, got the same error. I restarted the management service before this.

What I don't undestand is what's the connection between available gpus (in my case there is only one, so I guess it IS out of capacity after being attached to the new vm) and ejecting a iso image.

2 replies

DaanHoogland Mar 27, 2025
Collaborator

So there is no host 11 in that table and this is what the system is looking for. I am not sure (yet) if this is a configuration error or whether the system should handle this.

tdtmusic2 Mar 27, 2025
Author

So there is no host 11 in that table and this is what the system is looking for. I am not sure (yet) if this is a configuration error or whether the system should handle this.

Host 11 was removed since then, now it's 12 and it's the same error message about it:

2025-03-27 12:22:14,670 ERROR [c.c.r.ResourceManagerImpl] (API-Job-Executor-3:ctx-77c18f69 job-415 ctx-8f3c75db) (logid:50f8449f) Host 12 does not have required GPU device or out of capacity. GPU group: Group of Nvidia Corporation TU104GL [Tesla T4] GPUs, vGPU Type: passthrough
2025-03-27 12:22:14,672 ERROR [c.c.a.ApiAsyncJobDispatcher] (API-Job-Executor-3:ctx-77c18f69 job-415) (logid:50f8449f) Unexpected exception while executing org.apache.cloudstack.api.command.admin.iso.DetachIsoCmdByAdmin
com.cloud.utils.exception.CloudRuntimeException: Host 12 does not have required GPU device or out of capacity. GPU group: Group of Nvidia Corporation TU104GL [Tesla T4] GPUs, vGPU Type: passthrough
        at com.cloud.resource.ResourceManagerImpl.getGPUDevice(ResourceManagerImpl.java:3469)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
        at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
        at com.sun.proxy.$Proxy200.getGPUDevice(Unknown Source)
        at com.cloud.hypervisor.HypervisorGuruBase.toVirtualMachineTO(HypervisorGuruBase.java:285)
        at com.cloud.hypervisor.XenServerGuru.implement(XenServerGuru.java:93)
        at com.cloud.template.TemplateManagerImpl.attachISOToVM(TemplateManagerImpl.java:1296)
        at com.cloud.template.TemplateManagerImpl.attachISOToVM(TemplateManagerImpl.java:1314)
        at com.cloud.template.TemplateManagerImpl.detachIso(TemplateManagerImpl.java:1175)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
        at org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:107)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
        at com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:52)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
        at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
        at com.sun.proxy.$Proxy236.detachIso(Unknown Source)
        at org.apache.cloudstack.api.command.user.iso.DetachIsoCmd.execute(DetachIsoCmd.java:109)
        at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:172)
        at com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:112)
        at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:654)
        at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
        at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
        at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:602)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:829)
2025-03-27 12:22:14,674 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-3:ctx-77c18f69 job-415) (logid:50f8449f) Complete async job-415, jobStatus: FAILED, resultCode: 530, result: org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":"530","errortext":"Host 12 does not have required GPU device or out of capacity. GPU group: Group of Nvidia Corporation TU104GL [Tesla T4] GPUs, vGPU Type: passthrough"}

harikrishna-patnala · 2025-03-28T04:39:54Z

harikrishna-patnala
Mar 28, 2025
Collaborator

@tdtmusic2 what I can see from the error in the logs is, CS is calling the below sql query, the output of it seems to be empty. Please check if that can give you any hint

SELECT hgg.*
FROM host_gpu_groups hgg
INNER JOIN vgpu_types vt ON hgg.id = vt.gpu_group_id
WHERE hgg.host_id = 12
AND hgg.group_name = 'Group of Nvidia Corporation TU104GL [Tesla T4] GPUs'
AND vt.vgpu_type = 'passthrough'
AND vt.remaining_capacity > 0
ORDER BY vt.remaining_capacity DESC, hgg.id;

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

VM on Xen - weird ISO issue #10581

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 3 comments 4 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

Select a reply

VM on Xen - weird ISO issue #10581

tdtmusic2 Mar 17, 2025

Replies: 3 comments · 4 replies

DaanHoogland Mar 26, 2025 Collaborator

tdtmusic2 Mar 27, 2025 Author

DaanHoogland Mar 27, 2025 Collaborator

tdtmusic2 Mar 27, 2025 Author

DaanHoogland Mar 27, 2025 Collaborator

tdtmusic2 Mar 27, 2025 Author

harikrishna-patnala Mar 28, 2025 Collaborator

tdtmusic2
Mar 17, 2025

Replies: 3 comments 4 replies

DaanHoogland
Mar 26, 2025
Collaborator

tdtmusic2 Mar 27, 2025
Author

DaanHoogland Mar 27, 2025
Collaborator

tdtmusic2
Mar 27, 2025
Author

DaanHoogland Mar 27, 2025
Collaborator

tdtmusic2 Mar 27, 2025
Author

harikrishna-patnala
Mar 28, 2025
Collaborator