Version 4.18 uses glusterfs10.4 as the primary storage and cannot create VM

[maic]

libvirt 8.0 ,ubuntu22.04
Management error log：

2023-06-06 06:38:29,674 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-1:ctx-5024afa9 job-26/job-28 ctx-b07c23a5) (logid:85ef2375) release mem from host: 1, old used: (1.00 GB) 1073741824,reserved: (0 bytes) 0, total: (61.84 GB) 66404003840; new used: (0 bytes) 0,reserved:(0 bytes) 0; movedfromreserved: false,moveToReserveredfalse
2023-06-06 06:38:29,674 INFO [c.c.v.VmWorkJobHandlerProxy] (Work-Job-Executor-2:ctx-9a279884 job-27/job-29 ctx-4d260ac7) (logid:acc6738a) Rethrow exception com.cloud.exception.InsufficientServerCapacityException: Unable to create a deployment for VM instance {"id":4,"instanceName":"s-4-VM","type":"SecondaryStorageVm","uuid":"2de14fc6-04bf-4c99-a207-2e082c2f8e79"}Scope=interface com.cloud.dc.DataCenter; id=1
2023-06-06 06:38:29,675 DEBUG [c.c.v.VmWorkJobDispatcher] (Work-Job-Executor-2:ctx-9a279884 job-27/job-29) (logid:acc6738a) Done with run of VM work job: com.cloud.vm.VmWorkStart for VM 4, job origin: 27
2023-06-06 06:38:29,676 ERROR [c.c.v.VmWorkJobDispatcher] (Work-Job-Executor-2:ctx-9a279884 job-27/job-29) (logid:acc6738a) Unable to complete AsyncJobVO: {id:29, userId: 1, accountId: 1, instanceType: null, instanceId: null, cmd: com.cloud.vm.VmWorkStart, cmdInfo: rO0ABXNyABhjb20uY2xvdWQudm0uVm1Xb3JrU3RhcnR9cMGsvxz73gIAC0oABGRjSWRMAAZhdm9pZHN0ADBMY29tL2Nsb3VkL2RlcGxveS9EZXBsb3ltZW50UGxhbm5lciRFeGNsdWRlTGlzdDtMAAljbHVzdGVySWR0ABBMamF2YS9sYW5nL0xvbmc7TAAGaG9zdElkcQB-AAJMAAtqb3VybmFsTmFtZXQAEkxqYXZhL2xhbmcvU3RyaW5nO0wAEXBoeXNpY2FsTmV0d29ya0lkcQB-AAJMAAdwbGFubmVycQB-AANMAAVwb2RJZHEAfgACTAAGcG9vbElkcQB-AAJMAAlyYXdQYXJhbXN0AA9MamF2YS91dGlsL01hcDtMAA1yZXNlcnZhdGlvbklkcQB-AAN4cgATY29tLmNsb3VkLnZtLlZtV29ya5-ZtlbwJWdrAgAESgAJYWNjb3VudElkSgAGdXNlcklkSgAEdm1JZEwAC2hhbmRsZXJOYW1lcQB-AAN4cAAAAAAAAAABAAAAAAAAAAEAAAAAAAAABHQAGVZpcnR1YWxNYWNoaW5lTWFuYWdlckltcGwAAAAAAAAAAHBwcHBwcHBwcHA, cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result: null, initMsid: 86169082791055, completeMsid: null, lastUpdated: null, lastPolled: null, created: Tue Jun 06 06:38:19 UTC 2023, removed: null}, job origin:27
com.cloud.exception.InsufficientServerCapacityException: Unable to create a deployment for VM instance {"id":4,"instanceName":"s-4-VM","type":"SecondaryStorageVm","uuid":"2de14fc6-04bf-4c99-a207-2e082c2f8e79"}Scope=interface com.cloud.dc.DataCenter; id=1
at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:1191)
at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:5376)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
at com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5500)
at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:620)
at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:568)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
2023-06-06 06:38:29,690 ERROR [c.c.v.VmWorkJobHandlerProxy] (Work-Job-Executor-1:ctx-5024afa9 job-26/job-28 ctx-b07c23a5) (logid:85ef2375) Invocation exception, caused by: com.cloud.exception.InsufficientServerCapacityException: Unable to create a deployment for VM instance {"id":3,"instanceName":"v-3-VM","type":"ConsoleProxy","uuid":"566eea1d-894a-4040-9580-13d1974c801d"}Scope=interface com.cloud.dc.DataCenter; id=1
2023-06-06 06:38:29,690 INFO [c.c.v.VmWorkJobHandlerProxy] (Work-Job-Executor-1:ctx-5024afa9 job-26/job-28 ctx-b07c23a5) (logid:85ef2375) Rethrow exception com.cloud.exception.InsufficientServerCapacityException: Unable to create a deployment for VM instance {"id":3,"instanceName":"v-3-VM","type":"ConsoleProxy","uuid":"566eea1d-894a-4040-9580-13d1974c801d"}Scope=interface com.cloud.dc.DataCenter; id=1

agent error log：

2023-06-06 04:26:53,401 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-2:null) (logid:192e1783) Attempting to remove storage pool 8778b476-66d3-3848-8f0e-718d85c322d4 from libvirt
2023-06-06 04:26:53,423 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-2:null) (logid:192e1783) Storage pool 8778b476-66d3-3848-8f0e-718d85c322d4 has no corresponding secret. Not removing any secret.
2023-06-06 04:26:53,764 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-2:null) (logid:192e1783) Storage pool 8778b476-66d3-3848-8f0e-718d85c322d4 was successfully removed from libvirt.
2023-06-06 04:26:54,063 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-3:null) (logid:192e1783) Trying to fetch storage pool 3b8ce946-c4a6-38d0-9484-1495a452e9d2 from libvirt
2023-06-06 04:26:54,098 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-3:null) (logid:192e1783) Trying to fetch storage pool 3b8ce946-c4a6-38d0-9484-1495a452e9d2 from libvirt
2023-06-06 04:26:54,166 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-3:null) (logid:192e1783) Creating volume 70169d30-1815-4d64-9e6f-16b6dece69ea from template 642b7a0f-5c87-4881-a27c-4fbaa2db34be in pool 3b8ce946-c4a6-38d0-9484-1495a452e9d2 (Gluster) with size (0 bytes) 0
2023-06-06 04:26:54,167 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-3:null) (logid:192e1783) Attempting to create volume 70169d30-1815-4d64-9e6f-16b6dece69ea (Gluster) in pool 3b8ce946-c4a6-38d0-9484-1495a452e9d2 with size (4.88 GB) 5242880000
2023-06-06 04:26:58,368 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-4:null) (logid:192e1783) Trying to fetch storage pool 3b8ce946-c4a6-38d0-9484-1495a452e9d2 from libvirt
2023-06-06 04:26:58,489 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-4:null) (logid:192e1783) Trying to fetch storage pool 3b8ce946-c4a6-38d0-9484-1495a452e9d2 from libvirt
2023-06-06 04:26:58,715 WARN [kvm.resource.LibvirtKvmAgentHook] (agentRequest-Handler-4:null) (logid:192e1783) Groovy script '/etc/cloudstack/agent/hooks/libvirt-vm-xml-transformer.groovy' is not available. Transformations will not be applied.
2023-06-06 04:26:58,716 WARN [kvm.resource.LibvirtKvmAgentHook] (agentRequest-Handler-4:null) (logid:192e1783) Groovy scripting engine is not initialized. Data transformation skipped.
2023-06-06 04:26:58,859 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-1:null) (logid:2b045fd6) Trying to fetch storage pool 3b8ce946-c4a6-38d0-9484-1495a452e9d2 from libvirt
2023-06-06 04:26:58,886 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-1:null) (logid:2b045fd6) Trying to fetch storage pool 3b8ce946-c4a6-38d0-9484-1495a452e9d2 from libvirt
2023-06-06 04:26:58,924 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-1:null) (logid:2b045fd6) Creating volume 019b0e20-8bcd-4852-bebe-681489271e2b from template 642b7a0f-5c87-4881-a27c-4fbaa2db34be in pool 3b8ce946-c4a6-38d0-9484-1495a452e9d2 (Gluster) with size (0 bytes) 0
2023-06-06 04:26:58,924 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-1:null) (logid:2b045fd6) Attempting to create volume 019b0e20-8bcd-4852-bebe-681489271e2b (Gluster) in pool 3b8ce946-c4a6-38d0-9484-1495a452e9d2 with size (4.88 GB) 5242880000
2023-06-06 04:26:59,805 WARN [resource.wrapper.LibvirtStartCommandWrapper] (agentRequest-Handler-4:null) (logid:192e1783) LibvirtException
org.libvirt.LibvirtException: internal error: process exited while connecting to monitor: [2023-06-06 04:26:59.440660 +0000] I [io-stats.c:3711:ios_sample_buf_size_configure] 0-primarys: Configure ios_sample_buf size is 1024 because ios_sample_interval is 0
2023-06-06T04:26:59.452732Z qemu-system-x86_64: -blockdev {"node-name":"libvirt-2-format","read-only":false,"cache":{"direct":true,"no-flush":false},"driver":"qcow2","file":"libvirt-2-storage"}: Could not read L1 table: Input/output error
at org.libvirt.ErrorHandler.processError(Unknown Source)
at org.libvirt.ErrorHandler.processError(Unknown Source)
at org.libvirt.Connect.domainCreateXML(Unknown Source)
at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.startVM(LibvirtComputingResource.java:1815)
at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtStartCommandWrapper.execute(LibvirtStartCommandWrapper.java:89)
at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtStartCommandWrapper.execute(LibvirtStartCommandWrapper.java:49)
at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:78)
at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1847)
at com.cloud.agent.Agent.processRequest(Agent.java:662)
at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:1082)
at com.cloud.utils.nio.Task.call(Task.java:83)
at com.cloud.utils.nio.Task.call(Task.java:29)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
2023-06-06 04:26:59,965 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-4:null) (logid:192e1783) Trying to fetch storage pool 3b8ce946-c4a6-38d0-9484-1495a452e9d2 from libvirt
2023-06-06 04:27:00,151 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-2:null) (logid:2b045fd6) Trying to fetch storage pool 3b8ce946-c4a6-38d0-9484-1495a452e9d2 from libvirt
2023-06-06 04:27:00,229 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-2:null) (logid:2b045fd6) Trying to fetch storage pool 3b8ce946-c4a6-38d0-9484-1495a452e9d2 from libvirt
2023-06-06 04:27:00,298 WARN [kvm.resource.LibvirtKvmAgentHook] (agentRequest-Handler-2:null) (logid:2b045fd6) Groovy script '/etc/cloudstack/agent/hooks/libvirt-vm-xml-transformer.groovy' is not available. Transformations will not be applied.
2023-06-06 04:27:00,298 WARN [kvm.resource.LibvirtKvmAgentHook] (agentRequest-Handler-2:null) (logid:2b045fd6) Groovy scripting engine is not initialized. Data transformation skipped.
2023-06-06 04:27:01,065 WARN [resource.wrapper.LibvirtStartCommandWrapper] (agentRequest-Handler-2:null) (logid:2b045fd6) LibvirtException
org.libvirt.LibvirtException: internal error: qemu unexpectedly closed the monitor: [2023-06-06 04:27:00.642128 +0000] I [io-stats.c:3711:ios_sample_buf_size_configure] 0-primarys: Configure ios_sample_buf size is 1024 because ios_sample_interval is 0
2023-06-06T04:27:00.653774Z qemu-system-x86_64: -blockdev {"node-name":"libvirt-2-format","read-only":false,"cache":{"direct":true,"no-flush":false},"driver":"qcow2","file":"libvirt-2-storage"}: Could not read L1 table: Input/output error
at org.libvirt.ErrorHandler.processError(Unknown Source)
at org.libvirt.ErrorHandler.processError(Unknown Source)
at org.libvirt.Connect.domainCreateXML(Unknown Source)
at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.startVM(LibvirtComputingResource.java:1815)
at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtStartCommandWrapper.execute(LibvirtStartCommandWrapper.java:89)
at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtStartCommandWrapper.execute(LibvirtStartCommandWrapper.java:49)
at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:78)
at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1847)
at com.cloud.agent.Agent.processRequest(Agent.java:662)
at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:1082)
at com.cloud.utils.nio.Task.call(Task.java:83)
at com.cloud.utils.nio.Task.call(Task.java:29)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
2023-06-06 04:27:01,232 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-2:null) (logid:2b045fd6) Trying to fetch storage pool 3b8ce946-c4a6-38d0-9484-1495a452e9d2 from libvirt
2023-06-06 04:27:05,501 INFO [kvm.resource.LibvirtConnection] (agentRequest-Handler-3:null) (logid:2b045fd6) No existing libvirtd connection found. Opening a new one
2023-06-06 04:27:05,505 WARN [kvm.resource.LibvirtConnection] (agentRequest-Handler-3:null) (logid:2b045fd6) Can not find a connection for Instance s-1-VM. Assuming the default connection.
2023-06-06 04:27:05,629 WARN [kvm.resource.LibvirtKvmAgentHook] (agentRequest-Handler-3:null) (logid:2b045fd6) Groovy script '/etc/cloudstack/agent/hooks/libvirt-vm-state-change.groovy' is not available. Transformations will not be applied.
2023-06-06 04:27:05,629 WARN [kvm.resource.LibvirtKvmAgentHook] (agentRequest-Handler-3:null) (logid:2b045fd6) Groovy scripting engine is not initialized. Data transformation skipped.
2023-06-06 04:27:06,292 INFO [kvm.resource.LibvirtConnection] (agentRequest-Handler-5:null) (logid:192e1783) No existing libvirtd connection found. Opening a new one
2023-06-06 04:27:06,295 WARN [kvm.resource.LibvirtConnection] (agentRequest-Handler-5:null) (logid:192e1783) Can not find a connection for Instance v-2-VM. Assuming the default connection.
2023-06-06 04:27:06,415 WARN [kvm.resource.LibvirtKvmAgentHook] (agentRequest-Handler-5:null) (logid:192e1783) Groovy script '/etc/cloudstack/agent/hooks/libvirt-vm-state-change.groovy' is not available. Transformations will not be applied.
2023-06-06 04:27:06,416 WARN [kvm.resource.LibvirtKvmAgentHook] (agentRequest-Handler-5:null) (logid:192e1783) Groovy scripting engine is not initialized. Data transformation skipped.
2023-06-06 04:27:16,651 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-1:null) (logid:2b045fd6) Trying to fetch storage pool 3b8ce946-c4a6-38d0-9484-1495a452e9d2 from libvirt
2023-06-06 04:27:16,735 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-1:null) (logid:2b045fd6) Trying to fetch storage pool 3b8ce946-c4a6-38d0-9484-1495a452e9d2 from libvirt
2023-06-06 04:27:16,788 WARN [kvm.resource.LibvirtKvmAgentHook] (agentRequest-Handler-1:null) (logid:2b045fd6) Groovy script '/etc/cloudstack/agent/hooks/libvirt-vm-xml-transformer.groovy' is not available. Transformations will not be applied.
2023-06-06 04:27:16,789 WARN [kvm.resource.LibvirtKvmAgentHook] (agentRequest-Handler-1:null) (logid:2b045fd6) Groovy scripting engine is not initialized. Data transformation skipped.
2023-06-06 04:27:17,662 WARN [resource.wrapper.LibvirtStartCommandWrapper] (agentRequest-Handler-1:null) (logid:2b045fd6) LibvirtException
org.libvirt.LibvirtException: internal error: process exited while connecting to monitor: [2023-06-06 04:27:17.231713 +0000] I [io-stats.c:3711:ios_sample_buf_size_configure] 0-primarys: Configure ios_sample_buf size is 1024 because ios_sample_interval is 0
2023-06-06T04:27:17.244399Z qemu-system-x86_64: -blockdev {"node-name":"libvirt-2-format","read-only":false,"cache":{"direct":true,"no-flush":false},"driver":"qcow2","file":"libvirt-2-storage"}: Could not read L1 table: Input/output error
at org.libvirt.ErrorHandler.processError(Unknown Source)
at org.libvirt.ErrorHandler.processError(Unknown Source)
at org.libvirt.Connect.domainCreateXML(Unknown Source)
at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.startVM(LibvirtComputingResource.java:1815)
at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtStartCommandWrapper.execute(LibvirtStartCommandWrapper.java:89)
at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtStartCommandWrapper.execute(LibvirtStartCommandWrapper.java:49)
at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:78)
at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1847)
at com.cloud.agent.Agent.processRequest(Agent.java:662)
at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:1082)
at com.cloud.utils.nio.Task.call(Task.java:83)
at com.cloud.utils.nio.Task.call(Task.java:29)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
2023-06-06 04:27:17,877 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-1:null) (logid:2b045fd6) Trying to fetch storage pool 3b8ce946-c4a6-38d0-9484-1495a452e9d2 from libvirt
2023-06-06 04:27:21,140 INFO [kvm.resource.LibvirtConnection] (agentRequest-Handler-4:null) (logid:2b045fd6) No existing libvirtd connection found. Opening a new one
2023-06-06 04:27:21,143 WARN [kvm.resource.LibvirtConnection] (agentRequest-Handler-4:null) (logid:2b045fd6) Can not find a connection for Instance s-1-VM. Assuming the default connection.
2023-06-06 04:27:21,263 WARN [kvm.resource.LibvirtKvmAgentHook] (agentRequest-Handler-4:null) (logid:2b045fd6) Groovy script '/etc/cloudstack/agent/hooks/libvirt-vm-state-change.groovy' is not available. Transformations will not be applied.
2023-06-06 04:27:21,264 WARN [kvm.resource.LibvirtKvmAgentHook] (agentRequest-Handler-4:null) (logid:2b045fd6) Groovy scripting engine is not initialized. Data transformation skipped.
2023-06-06 06:30:18,664 INFO [cloud.agent.Agent] (Agent-Handler-3:null) (logid:c3b9da98) Lost connection to host: 192.168.12.154. Attempting reconnection while we still have 0 commands in progress.
2023-06-06 06:30:18,666 INFO [utils.nio.NioClient] (Agent-Handler-3:null) (logid:c3b9da98) NioClient connection closed
2023-06-06 06:30:18,667 INFO [cloud.agent.Agent] (Agent-Handler-3:null) (logid:c3b9da98) Reconnecting to host:192.168.12.154
2023-06-06 06:30:18,667 INFO [utils.nio.NioClient] (Agent-Handler-3:null) (logid:c3b9da98) Connecting to 192.168.12.154:8250
2023-06-06 06:30:18,669 WARN [utils.nio.NioConnection] (Agent-Handler-3:null) (logid:c3b9da98) Unable to connect to remote: is there a server running on port 8250
2023-06-06 06:30:23,670 INFO [cloud.agent.Agent] (Agent-Handler-3:null) (logid:c3b9da98) Reconnecting to host:192.168.12.154
2023-06-06 06:30:23,671 INFO [utils.nio.NioClient] (Agent-Handler-3:null) (logid:c3b9da98) Connecting to 192.168.12.154:8250
2023-06-06 06:30:23,672 WARN [utils.nio.NioConnection] (Agent-Handler-3:null) (logid:c3b9da98) Unable to connect to remote: is there a server running on port 8250
2023-06-06 06:30:28,673 INFO [cloud.agent.Agent] (Agent-Handler-3:null) (logid:c3b9da98) Reconnecting to host:192.168.12.154
2023-06-06 06:30:28,673 INFO [utils.nio.NioClient] (Agent-Handler-3:null) (logid:c3b9da98) Connecting to 192.168.12.154:8250
2023-06-06 06:30:28,674 WARN [utils.nio.NioConnection] (Agent-Handler-3:null) (logid:c3b9da98) Unable to connect to remote: is there a server running on port 8250
2023-06-06 06:30:33,675 INFO [cloud.agent.Agent] (Agent-Handler-3:null) (logid:c3b9da98) Reconnecting to host:192.168.12.154
2023-06-06 06:30:33,675 INFO [utils.nio.NioClient] (Agent-Handler-3:null) (logid:c3b9da98) Connecting to 192.168.12.154:8250
2023-06-06 06:30:33,676 WARN [utils.nio.NioConnection] (Agent-Handler-3:null) (logid:c3b9da98) Unable to connect to remote: is there a server running on port 8250
2023-06-06 06:30:38,677 INFO [cloud.agent.Agent] (Agent-Handler-3:null) (logid:c3b9da98) Reconnecting to host:192.168.12.154
2023-06-06 06:30:38,677 INFO [utils.nio.NioClient] (Agent-Handler-3:null) (logid:c3b9da98) Connecting to 192.168.12.154:8250
2023-06-06 06:30:38,678 WARN [utils.nio.NioConnection] (Agent-Handler-3:null) (logid:c3b9da98) Unable to connect to remote: is there a server running on port 8250
2023-06-06 06:30:43,679 INFO [cloud.agent.Agent] (Agent-Handler-3:null) (logid:c3b9da98) Reconnecting to host:192.168.12.154
2023-06-06 06:30:43,679 INFO [utils.nio.NioClient] (Agent-Handler-3:null) (logid:c3b9da98) Connecting to 192.168.12.154:8250
2023-06-06 06:30:43,680 WARN [utils.nio.NioConnection] (Agent-Handler-3:null) (logid:c3b9da98) Unable to connect to remote: is there a server running on port 8250
2023-06-06 06:30:48,681 INFO [cloud.agent.Agent] (Agent-Handler-3:null) (logid:c3b9da98) Reconnecting to host:192.168.12.154
2023-06-06 06:30:48,681 INFO [utils.nio.NioClient] (Agent-Handler-3:null) (logid:c3b9da98) Connecting to 192.168.12.154:8250
2023-06-06 06:30:48,682 WARN [utils.nio.NioConnection] (Agent-Handler-3:null) (logid:c3b9da98) Unable to connect to remote: is there a server running on port 8250
2023-06-06 06:30:53,683 INFO [cloud.agent.Agent] (Agent-Handler-3:null) (logid:c3b9da98) Reconnecting to host:192.168.12.154
2023-06-06 06:30:53,683 INFO [utils.nio.NioClient] (Agent-Handler-3:null) (logid:c3b9da98) Connecting to 192.168.12.154:8250
2023-06-06 06:30:53,684 WARN [utils.nio.NioConnection] (Agent-Handler-3:null) (logid:c3b9da98) Unable to connect to remote: is there a server running on port 8250
2023-06-06 06:30:58,685 INFO [cloud.agent.Agent] (Agent-Handler-3:null) (logid:c3b9da98) Reconnecting to host:192.168.12.154
2023-06-06 06:30:58,685 INFO [utils.nio.NioClient] (Agent-Handler-3:null) (logid:c3b9da98) Connecting to 192.168.12.154:8250
2023-06-06 06:30:58,686 WARN [utils.nio.NioConnection] (Agent-Handler-3:null) (logid:c3b9da98) Unable to connect to remote: is there a server running on port 8250
2023-06-06 06:31:03,687 INFO [cloud.agent.Agent] (Agent-Handler-3:null) (logid:c3b9da98) Reconnecting to host:192.168.12.154
2023-06-06 06:31:03,688 INFO [utils.nio.NioClient] (Agent-Handler-3:null) (logid:c3b9da98) Connecting to 192.168.12.154:8250
2023-06-06 06:31:03,691 INFO [utils.nio.Link] (Agent-Handler-3:null) (logid:c3b9da98) Conf file found: /etc/cloudstack/agent/agent.properties
2023-06-06 06:31:03,956 INFO [utils.nio.NioClient] (Agent-Handler-3:null) (logid:c3b9da98) SSL: Handshake done
2023-06-06 06:31:03,956 INFO [utils.nio.NioClient] (Agent-Handler-3:null) (logid:c3b9da98) Connected to 192.168.12.154:8250
2023-06-06 06:31:04,038 INFO [utils.linux.KVMHostInfo] (Agent-Handler-1:null) (logid:c3b9da98) Fetching CPU speed from command "lscpu".
2023-06-06 06:31:04,081 INFO [utils.linux.KVMHostInfo] (Agent-Handler-1:null) (logid:c3b9da98) Command [lscpu | grep -i 'Model name' | head -n 1 | egrep -o '[[:digit:]].[[:digit:]]+GHz' | sed 's/GHz//g'] resulted in the value [2600] for CPU speed.
2023-06-06 06:31:04,132 ERROR [kvm.resource.LibvirtConnection] (Agent-Handler-1:null) (logid:c3b9da98) Connection with libvirtd is broken: invalid connection pointer in virConnectGetVersion
2023-06-06 06:31:04,166 INFO [kvm.storage.LibvirtStorageAdaptor] (Agent-Handler-1:null) (logid:c3b9da98) Attempting to create storage pool b18e7a00-0161-4211-afc9-47b25138a851 (Filesystem) in libvirt
2023-06-06 06:31:04,184 INFO [kvm.storage.LibvirtStorageAdaptor] (Agent-Handler-1:null) (logid:c3b9da98) Found existing defined storage pool b18e7a00-0161-4211-afc9-47b25138a851, using it.
2023-06-06 06:31:04,185 INFO [kvm.storage.LibvirtStorageAdaptor] (Agent-Handler-1:null) (logid:c3b9da98) Trying to fetch storage pool b18e7a00-0161-4211-afc9-47b25138a851 from libvirt
2023-06-06 06:31:04,246 INFO [cloud.agent.Agent] (Agent-Handler-2:null) (logid:c3b9da98) Process agent startup answer, agent id = 0
2023-06-06 06:31:04,247 INFO [cloud.agent.Agent] (Agent-Handler-2:null) (logid:c3b9da98) Set agent id 0
2023-06-06 06:31:04,248 INFO [cloud.agent.Agent] (Agent-Handler-2:null) (logid:c3b9da98) Startup Response Received: agent id = 0
2023-06-06 06:31:04,657 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-2:null) (logid:74bcff4d) Attempting to create storage pool 3b8ce946-c4a6-38d0-9484-1495a452e9d2 (Gluster) in libvirt
2023-06-06 06:31:04,659 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-2:null) (logid:74bcff4d) Found existing defined storage pool 3b8ce946-c4a6-38d0-9484-1495a452e9d2, using it.
2023-06-06 06:31:04,660 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-2:null) (logid:74bcff4d) Trying to fetch storage pool 3b8ce946-c4a6-38d0-9484-1495a452e9d2 from libvirt
2023-06-06 06:31:06,612 INFO [cloud.agent.Agent] (agentRequest-Handler-2:null) (logid:74bcff4d) Processing agent ready command, agent id = 1
2023-06-06 06:31:06,612 INFO [cloud.agent.Agent] (agentRequest-Handler-2:null) (logid:74bcff4d) Set agent id 1
2023-06-06 06:31:06,613 INFO [cloud.agent.Agent] (agentRequest-Handler-2:null) (logid:74bcff4d) Ready command is processed for agent id = 1
2023-06-06 06:31:06,780 INFO [cloud.agent.Agent] (agentRequest-Handler-4:null) (logid:74bcff4d) Processing agent ready command, agent id = 1
2023-06-06 06:31:06,780 INFO [cloud.agent.Agent] (agentRequest-Handler-4:null) (logid:74bcff4d) Set agent id 1
2023-06-06 06:31:06,782 INFO [cloud.agent.Agent] (agentRequest-Handler-4:null) (logid:74bcff4d) Processed new management server list: 192.168.12.154@static
2023-06-06 06:31:06,782 INFO [cloud.agent.Agent] (agentRequest-Handler-4:null) (logid:74bcff4d) Ready command is processed for agent id = 1
2023-06-06 06:31:08,957 INFO [cloud.agent.Agent] (Agent-Handler-3:null) (logid:c3b9da98) Connected to the host: 192.168.12.154
2023-06-06 06:38:17,436 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-1:null) (logid:546d879e) Trying to fetch storage pool 3b8ce946-c4a6-38d0-9484-1495a452e9d2 from libvirt
2023-06-06 06:38:17,464 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-1:null) (logid:546d879e) Asking libvirt to refresh storage pool 3b8ce946-c4a6-38d0-9484-1495a452e9d2
2023-06-06 06:38:21,868 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-4:null) (logid:acc6738a) Trying to fetch storage pool 3b8ce946-c4a6-38d0-9484-1495a452e9d2 from libvirt
2023-06-06 06:38:21,899 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-4:null) (logid:acc6738a) Trying to fetch storage pool 3b8ce946-c4a6-38d0-9484-1495a452e9d2 from libvirt
2023-06-06 06:38:21,952 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-4:null) (logid:acc6738a) Creating volume 482766f5-07e2-4546-a9a2-28002a3810cc from template 642b7a0f-5c87-4881-a27c-4fbaa2db34be in pool 3b8ce946-c4a6-38d0-9484-1495a452e9d2 (Gluster) with size (0 bytes) 0
2023-06-06 06:38:21,952 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-4:null) (logid:acc6738a) Attempting to create volume 482766f5-07e2-4546-a9a2-28002a3810cc (Gluster) in pool 3b8ce946-c4a6-38d0-9484-1495a452e9d2 with size (4.88 GB) 5242880000
2023-06-06 06:38:22,123 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-3:null) (logid:85ef2375) Trying to fetch storage pool 3b8ce946-c4a6-38d0-9484-1495a452e9d2 from libvirt
2023-06-06 06:38:22,157 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-3:null) (logid:85ef2375) Trying to fetch storage pool 3b8ce946-c4a6-38d0-9484-1495a452e9d2 from libvirt
2023-06-06 06:38:22,189 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-3:null) (logid:85ef2375) Creating volume b0664477-9c67-4366-a758-f02fdd34488d from template 642b7a0f-5c87-4881-a27c-4fbaa2db34be in pool 3b8ce946-c4a6-38d0-9484-1495a452e9d2 (Gluster) with size (0 bytes) 0
2023-06-06 06:38:22,190 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-3:null) (logid:85ef2375) Attempting to create volume b0664477-9c67-4366-a758-f02fdd34488d (Gluster) in pool 3b8ce946-c4a6-38d0-9484-1495a452e9d2 with size (4.88 GB) 5242880000
2023-06-06 06:38:23,223 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-5:null) (logid:85ef2375) Trying to fetch storage pool 3b8ce946-c4a6-38d0-9484-1495a452e9d2 from libvirt
2023-06-06 06:38:23,313 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-5:null) (logid:85ef2375) Trying to fetch storage pool 3b8ce946-c4a6-38d0-9484-1495a452e9d2 from libvirt
2023-06-06 06:38:23,315 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-1:null) (logid:acc6738a) Trying to fetch storage pool 3b8ce946-c4a6-38d0-9484-1495a452e9d2 from libvirt
2023-06-06 06:38:23,379 WARN [kvm.resource.LibvirtKvmAgentHook] (agentRequest-Handler-5:null) (logid:85ef2375) Groovy script '/etc/cloudstack/agent/hooks/libvirt-vm-xml-transformer.groovy' is not available. Transformations will not be applied.
2023-06-06 06:38:23,379 WARN [kvm.resource.LibvirtKvmAgentHook] (agentRequest-Handler-5:null) (logid:85ef2375) Groovy scripting engine is not initialized. Data transformation skipped.
2023-06-06 06:38:23,400 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-1:null) (logid:acc6738a) Trying to fetch storage pool 3b8ce946-c4a6-38d0-9484-1495a452e9d2 from libvirt
2023-06-06 06:38:23,426 WARN [kvm.resource.LibvirtKvmAgentHook] (agentRequest-Handler-1:null) (logid:acc6738a) Groovy script '/etc/cloudstack/agent/hooks/libvirt-vm-xml-transformer.groovy' is not available. Transformations will not be applied.
2023-06-06 06:38:23,426 WARN [kvm.resource.LibvirtKvmAgentHook] (agentRequest-Handler-1:null) (logid:acc6738a) Groovy scripting engine is not initialized. Data transformation skipped.
2023-06-06 06:38:24,458 WARN [resource.wrapper.LibvirtStartCommandWrapper] (agentRequest-Handler-1:null) (logid:acc6738a) LibvirtException
org.libvirt.LibvirtException: internal error: qemu unexpectedly closed the monitor: [2023-06-06 06:38:23.798188 +0000] I [io-stats.c:3711:ios_sample_buf_size_configure] 0-primarys: Configure ios_sample_buf size is 1024 because ios_sample_interval is 0
2023-06-06T06:38:23.809358Z qemu-system-x86_64: -blockdev {"node-name":"libvirt-2-format","read-only":false,"cache":{"direct":true,"no-flush":false},"driver":"qcow2","file":"libvirt-2-storage"}: Could not read L1 table: Input/output error
at org.libvirt.ErrorHandler.processError(Unknown Source)
at org.libvirt.ErrorHandler.processError(Unknown Source)
at org.libvirt.Connect.domainCreateXML(Unknown Source)
at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.startVM(LibvirtComputingResource.java:1815)
at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtStartCommandWrapper.execute(LibvirtStartCommandWrapper.java:89)
at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtStartCommandWrapper.execute(LibvirtStartCommandWrapper.java:49)
at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:78)
at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1847)
at com.cloud.agent.Agent.processRequest(Agent.java:662)
at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:1082)
at com.cloud.utils.nio.Task.call(Task.java:83)
at com.cloud.utils.nio.Task.call(Task.java:29)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
2023-06-06 06:38:24,689 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-1:null) (logid:acc6738a) Trying to fetch storage pool 3b8ce946-c4a6-38d0-9484-1495a452e9d2 from libvirt
2023-06-06 06:38:24,840 WARN [resource.wrapper.LibvirtStartCommandWrapper] (agentRequest-Handler-5:null) (logid:85ef2375) LibvirtException
org.libvirt.LibvirtException: internal error: qemu unexpectedly closed the monitor: [2023-06-06 06:38:24.217794 +0000] I [io-stats.c:3711:ios_sample_buf_size_configure] 0-primarys: Configure ios_sample_buf size is 1024 because ios_sample_interval is 0
2023-06-06T06:38:24.228789Z qemu-system-x86_64: -blockdev {"node-name":"libvirt-2-format","read-only":false,"cache":{"direct":true,"no-flush":false},"driver":"qcow2","file":"libvirt-2-storage"}: Could not read L1 table: Input/output error
at org.libvirt.ErrorHandler.processError(Unknown Source)
at org.libvirt.ErrorHandler.processError(Unknown Source)
at org.libvirt.Connect.domainCreateXML(Unknown Source)
at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.startVM(LibvirtComputingResource.java:1815)
at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtStartCommandWrapper.execute(LibvirtStartCommandWrapper.java:89)
at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtStartCommandWrapper.execute(LibvirtStartCommandWrapper.java:49)
at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:78)
at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1847)
at com.cloud.agent.Agent.processRequest(Agent.java:662)
at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:1082)
at com.cloud.utils.nio.Task.call(Task.java:83)
at com.cloud.utils.nio.Task.call(Task.java:29)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
2023-06-06 06:38:24,861 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-5:null) (logid:85ef2375) Trying to fetch storage pool 3b8ce946-c4a6-38d0-9484-1495a452e9d2 from libvirt
2023-06-06 06:38:28,948 INFO [kvm.resource.LibvirtConnection] (agentRequest-Handler-2:null) (logid:acc6738a) No existing libvirtd connection found. Opening a new one
2023-06-06 06:38:28,951 WARN [kvm.resource.LibvirtConnection] (agentRequest-Handler-2:null) (logid:acc6738a) Can not find a connection for Instance s-4-VM. Assuming the default connection.
2023-06-06 06:38:29,071 WARN [kvm.resource.LibvirtKvmAgentHook] (agentRequest-Handler-2:null) (logid:acc6738a) Groovy script '/etc/cloudstack/agent/hooks/libvirt-vm-state-change.groovy' is not available. Transformations will not be applied.
2023-06-06 06:38:29,072 WARN [kvm.resource.LibvirtKvmAgentHook] (agentRequest-Handler-2:null) (logid:acc6738a) Groovy scripting engine is not initialized. Data transformation skipped.
2023-06-06 06:38:29,088 INFO [kvm.resource.LibvirtConnection] (agentRequest-Handler-4:null) (logid:85ef2375) No existing libvirtd connection found. Opening a new one
2023-06-06 06:38:29,091 WARN [kvm.resource.LibvirtConnection] (agentRequest-Handler-4:null) (logid:85ef2375) Can not find a connection for Instance v-3-VM. Assuming the default connection.
2023-06-06 06:38:29,208 WARN [kvm.resource.LibvirtKvmAgentHook] (agentRequest-Handler-4:null) (logid:85ef2375) Groovy script '/etc/cloudstack/agent/hooks/libvirt-vm-state-change.groovy' is not available. Transformations will not be applied.
2023-06-06 06:38:29,208 WARN [kvm.resource.LibvirtKvmAgentHook] (agentRequest-Handler-4:null) (logid:85ef2375) Groovy scripting engine is not initialized. Data transformation skipped.
2023-06-06 06:38:51,939 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-3:null) (logid:85ef2375) Trying to fetch storage pool 3b8ce946-c4a6-38d0-9484-1495a452e9d2 from libvirt
2023-06-06 06:38:52,029 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-3:null) (logid:85ef2375) Trying to fetch storage pool 3b8ce946-c4a6-38d0-9484-1495a452e9d2 from libvirt
2023-06-06 06:38:52,096 WARN [kvm.resource.LibvirtKvmAgentHook] (agentRequest-Handler-3:null) (logid:85ef2375) Groovy script '/etc/cloudstack/agent/hooks/libvirt-vm-xml-transformer.groovy' is not available. Transformations will not be applied.
2023-06-06 06:38:52,096 WARN [kvm.resource.LibvirtKvmAgentHook] (agentRequest-Handler-3:null) (logid:85ef2375) Groovy scripting engine is not initialized. Data transformation skipped.
2023-06-06 06:38:52,235 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-1:null) (logid:acc6738a) Trying to fetch storage pool 3b8ce946-c4a6-38d0-9484-1495a452e9d2 from libvirt
2023-06-06 06:38:52,289 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-1:null) (logid:acc6738a) Trying to fetch storage pool 3b8ce946-c4a6-38d0-9484-1495a452e9d2 from libvirt
2023-06-06 06:38:52,334 WARN [kvm.resource.LibvirtKvmAgentHook] (agentRequest-Handler-1:null) (logid:acc6738a) Groovy script '/etc/cloudstack/agent/hooks/libvirt-vm-xml-transformer.groovy' is not available. Transformations will not be applied.
2023-06-06 06:38:52,334 WARN [kvm.resource.LibvirtKvmAgentHook] (agentRequest-Handler-1:null) (logid:acc6738a) Groovy scripting engine is not initialized. Data transformation skipped.
2023-06-06 06:38:53,383 WARN [resource.wrapper.LibvirtStartCommandWrapper] (agentRequest-Handler-1:null) (logid:acc6738a) LibvirtException
org.libvirt.LibvirtException: internal error: process exited while connecting to monitor: [2023-06-06 06:38:52.841775 +0000] I [io-stats.c:3711:ios_sample_buf_size_configure] 0-primarys: Configure ios_sample_buf size is 1024 because ios_sample_interval is 0
2023-06-06T06:38:52.852779Z qemu-system-x86_64: -blockdev {"node-name":"libvirt-2-format","read-only":false,"cache":{"direct":true,"no-flush":false},"driver":"qcow2","file":"libvirt-2-storage"}: Could not read L1 table: Input/output error
at org.libvirt.ErrorHandler.processError(Unknown Source)
at org.libvirt.ErrorHandler.processError(Unknown Source)
at org.libvirt.Connect.domainCreateXML(Unknown Source)
at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.startVM(LibvirtComputingResource.java:1815)
at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtStartCommandWrapper.execute(LibvirtStartCommandWrapper.java:89)
at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtStartCommandWrapper.execute(LibvirtStartCommandWrapper.java:49)
at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:78)
at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1847)
at com.cloud.agent.Agent.processRequest(Agent.java:662)
at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:1082)
at com.cloud.utils.nio.Task.call(Task.java:83)
at com.cloud.utils.nio.Task.call(Task.java:29)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
2023-06-06 06:38:53,611 WARN [resource.wrapper.LibvirtStartCommandWrapper] (agentRequest-Handler-3:null) (logid:85ef2375) LibvirtException
org.libvirt.LibvirtException: internal error: qemu unexpectedly closed the monitor: [2023-06-06 06:38:52.972857 +0000] I [io-stats.c:3711:ios_sample_buf_size_configure] 0-primarys: Configure ios_sample_buf size is 1024 because ios_sample_interval is 0
2023-06-06T06:38:52.984004Z qemu-system-x86_64: -blockdev {"node-name":"libvirt-2-format","read-only":false,"cache":{"direct":true,"no-flush":false},"driver":"qcow2","file":"libvirt-2-storage"}: Could not read L1 table: Input/output error
at org.libvirt.ErrorHandler.processError(Unknown Source)
at org.libvirt.ErrorHandler.processError(Unknown Source)
at org.libvirt.Connect.domainCreateXML(Unknown Source)
at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.startVM(LibvirtComputingResource.java:1815)
at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtStartCommandWrapper.execute(LibvirtStartCommandWrapper.java:89)
at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtStartCommandWrapper.execute(LibvirtStartCommandWrapper.java:49)
at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:78)
at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1847)
at com.cloud.agent.Agent.processRequest(Agent.java:662)
at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:1082)
at com.cloud.utils.nio.Task.call(Task.java:83)
at com.cloud.utils.nio.Task.call(Task.java:29)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
2023-06-06 06:38:53,641 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-1:null) (logid:acc6738a) Trying to fetch storage pool 3b8ce946-c4a6-38d0-9484-1495a452e9d2 from libvirt
2023-06-06 06:38:53,652 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-3:null) (logid:85ef2375) Trying to fetch storage pool 3b8ce946-c4a6-38d0-9484-1495a452e9d2 from libvirt
2023-06-06 06:38:57,876 INFO [kvm.resource.LibvirtConnection] (agentRequest-Handler-5:null) (logid:acc6738a) No existing libvirtd connection found. Opening a new one
2023-06-06 06:38:57,876 INFO [kvm.resource.LibvirtConnection] (agentRequest-Handler-2:null) (logid:85ef2375) No existing libvirtd connection found. Opening a new one
2023-06-06 06:38:57,879 WARN [kvm.resource.LibvirtConnection] (agentRequest-Handler-5:null) (logid:acc6738a) Can not find a connection for Instance s-4-VM. Assuming the default connection.
2023-06-06 06:38:57,880 WARN [kvm.resource.LibvirtConnection] (agentRequest-Handler-2:null) (logid:85ef2375) Can not find a connection for Instance v-3-VM. Assuming the default connection.
2023-06-06 06:38:58,000 WARN [kvm.resource.LibvirtKvmAgentHook] (agentRequest-Handler-2:null) (logid:85ef2375) Groovy script '/etc/cloudstack/agent/hooks/libvirt-vm-state-change.groovy' is not available. Transformations will not be applied.
2023-06-06 06:38:58,000 WARN [kvm.resource.LibvirtKvmAgentHook] (agentRequest-Handler-2:null) (logid:85ef2375) Groovy scripting engine is not initialized. Data transformation skipped.
2023-06-06 06:38:58,499 WARN [kvm.resource.LibvirtKvmAgentHook] (agentRequest-Handler-5:null) (logid:acc6738a) Groovy script '/etc/cloudstack/agent/hooks/libvirt-vm-state-change.groovy' is not available. Transformations will not be applied.
2023-06-06 06:38:58,499 WARN [kvm.resource.LibvirtKvmAgentHook] (agentRequest-Handler-5:null) (logid:acc6738a) Groovy scripting engine is not initialized. Data transformation skipped.
2023-06-06 06:39:00,554 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-4:null) (logid:1b1f2c29) Trying to fetch storage pool 3b8ce946-c4a6-38d0-9484-1495a452e9d2 from libvirt
2023-06-06 06:39:00,588 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-1:null) (logid:ce0eeb00) Trying to fetch storage pool 3b8ce946-c4a6-38d0-9484-1495a452e9d2 from libvirt
2023-06-06 06:39:00,609 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-4:null) (logid:1b1f2c29) Attempting to remove volume b0664477-9c67-4366-a758-f02fdd34488d from pool 3b8ce946-c4a6-38d0-9484-1495a452e9d2
2023-06-06 06:39:00,639 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-1:null) (logid:ce0eeb00) Attempting to remove volume 482766f5-07e2-4546-a9a2-28002a3810cc from pool 3b8ce946-c4a6-38d0-9484-1495a452e9d2
2023-06-06 06:39:17,608 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-3:null) (logid:471848d3) Trying to fetch storage pool 3b8ce946-c4a6-38d0-9484-1495a452e9d2 from libvirt
2023-06-06 06:39:17,635 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-3:null) (logid:471848d3) Asking libvirt to refresh storage pool 3b8ce946-c4a6-38d0-9484-1495a452e9d2
2023-06-06 06:39:21,434 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-5:null) (logid:85ef2375) Trying to fetch storage pool 3b8ce946-c4a6-38d0-9484-1495a452e9d2 from libvirt
2023-06-06 06:39:21,467 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-5:null) (logid:85ef2375) Trying to fetch storage pool 3b8ce946-c4a6-38d0-9484-1495a452e9d2 from libvirt
2023-06-06 06:39:21,519 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-5:null) (logid:85ef2375) Creating volume bba87d20-244b-48c0-8bfb-613efb3d6bf9 from template 642b7a0f-5c87-4881-a27c-4fbaa2db34be in pool 3b8ce946-c4a6-38d0-9484-1495a452e9d2 (Gluster) with size (0 bytes) 0
2023-06-06 06:39:21,519 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-5:null) (logid:85ef2375) Attempting to create volume bba87d20-244b-48c0-8bfb-613efb3d6bf9 (Gluster) in pool 3b8ce946-c4a6-38d0-9484-1495a452e9d2 with size (4.88 GB) 5242880000
2023-06-06 06:39:21,782 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-4:null) (logid:acc6738a) Trying to fetch storage pool 3b8ce946-c4a6-38d0-9484-1495a452e9d2 from libvirt
2023-06-06 06:39:21,815 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-4:null) (logid:acc6738a) Trying to fetch storage pool 3b8ce946-c4a6-38d0-9484-1495a452e9d2 from libvirt
2023-06-06 06:39:21,863 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-4:null) (logid:acc6738a) Creating volume 2f48f71d-628e-41f7-8cd1-36cef891f59f from template 642b7a0f-5c87-4881-a27c-4fbaa2db34be in pool 3b8ce946-c4a6-38d0-9484-1495a452e9d2 (Gluster) with size (0 bytes) 0
2023-06-06 06:39:21,863 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-4:null) (logid:acc6738a) Attempting to create volume 2f48f71d-628e-41f7-8cd1-36cef891f59f (Gluster) in pool 3b8ce946-c4a6-38d0-9484-1495a452e9d2 with size (4.88 GB) 5242880000
2023-06-06 06:39:22,637 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-1:null) (logid:85ef2375) Trying to fetch storage pool 3b8ce946-c4a6-38d0-9484-1495a452e9d2 from libvirt
2023-06-06 06:39:22,744 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-1:null) (logid:85ef2375) Trying to fetch storage pool 3b8ce946-c4a6-38d0-9484-1495a452e9d2 from libvirt
2023-06-06 06:39:22,779 WARN [kvm.resource.LibvirtKvmAgentHook] (agentRequest-Handler-1:null) (logid:85ef2375) Groovy script '/etc/cloudstack/agent/hooks/libvirt-vm-xml-transformer.groovy' is not available. Transformations will not be applied.
2023-06-06 06:39:22,780 WARN [kvm.resource.LibvirtKvmAgentHook] (agentRequest-Handler-1:null) (logid:85ef2375) Groovy scripting engine is not initialized. Data transformation skipped.
2023-06-06 06:39:23,163 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-3:null) (logid:acc6738a) Trying to fetch storage pool 3b8ce946-c4a6-38d0-9484-1495a452e9d2 from libvirt
2023-06-06 06:39:23,208 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-3:null) (logid:acc6738a) Trying to fetch storage pool 3b8ce946-c4a6-38d0-9484-1495a452e9d2 from libvirt
2023-06-06 06:39:23,231 WARN [kvm.resource.LibvirtKvmAgentHook] (agentRequest-Handler-3:null) (logid:acc6738a) Groovy script '/etc/cloudstack/agent/hooks/libvirt-vm-xml-transformer.groovy' is not available. Transformations will not be applied.
2023-06-06 06:39:23,231 WARN [kvm.resource.LibvirtKvmAgentHook] (agentRequest-Handler-3:null) (logid:acc6738a) Groovy scripting engine is not initialized. Data transformation skipped.
2023-06-06 06:39:23,977 WARN [resource.wrapper.LibvirtStartCommandWrapper] (agentRequest-Handler-1:null) (logid:85ef2375) LibvirtException
org.libvirt.LibvirtException: internal error: process exited while connecting to monitor: [2023-06-06 06:39:23.608503 +0000] I [io-stats.c:3711:ios_sample_buf_size_configure] 0-primarys: Configure ios_sample_buf size is 1024 because ios_sample_interval is 0
2023-06-06T06:39:23.619092Z qemu-system-x86_64: -blockdev {"node-name":"libvirt-2-format","read-only":false,"cache":{"direct":true,"no-flush":false},"driver":"qcow2","file":"libvirt-2-storage"}: Could not read L1 table: Input/output error
at org.libvirt.ErrorHandler.processError(Unknown Source)
at org.libvirt.ErrorHandler.processError(Unknown Source)
at org.libvirt.Connect.domainCreateXML(Unknown Source)
at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.startVM(LibvirtComputingResource.java:1815)
at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtStartCommandWrapper.execute(LibvirtStartCommandWrapper.java:89)
at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtStartCommandWrapper.execute(LibvirtStartCommandWrapper.java:49)
at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:78)
at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1847)
at com.cloud.agent.Agent.processRequest(Agent.java:662)
at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:1082)
at com.cloud.utils.nio.Task.call(Task.java:83)
at com.cloud.utils.nio.Task.call(Task.java:29)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
2023-06-06 06:39:23,998 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-1:null) (logid:85ef2375) Trying to fetch storage pool 3b8ce946-c4a6-38d0-9484-1495a452e9d2 from libvirt
2023-06-06 06:39:24,604 WARN [resource.wrapper.LibvirtStartCommandWrapper] (agentRequest-Handler-3:null) (logid:acc6738a) LibvirtException
org.libvirt.LibvirtException: internal error: qemu unexpectedly closed the monitor: [2023-06-06 06:39:24.115477 +0000] I [io-stats.c:3711:ios_sample_buf_size_configure] 0-primarys: Configure ios_sample_buf size is 1024 because ios_sample_interval is 0
2023-06-06T06:39:24.125093Z qemu-system-x86_64: -blockdev {"node-name":"libvirt-2-format","read-only":false,"cache":{"direct":true,"no-flush":false},"driver":"qcow2","file":"libvirt-2-storage"}: Could not read L1 table: Input/output error
at org.libvirt.ErrorHandler.processError(Unknown Source)
at org.libvirt.ErrorHandler.processError(Unknown Source)
at org.libvirt.Connect.domainCreateXML(Unknown Source)
at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.startVM(LibvirtComputingResource.java:1815)
at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtStartCommandWrapper.execute(LibvirtStartCommandWrapper.java:89)
at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtStartCommandWrapper.execute(LibvirtStartCommandWrapper.java:49)
at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:78)
at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1847)
at com.cloud.agent.Agent.processRequest(Agent.java:662)
at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:1082)
at com.cloud.utils.nio.Task.call(Task.java:83)
at com.cloud.utils.nio.Task.call(Task.java:29)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
2023-06-06 06:39:24,801 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-3:null) (logid:acc6738a) Trying to fetch storage pool 3b8ce946-c4a6-38d0-9484-1495a452e9d2 from libvirt
2023-06-06 06:39:28,020 INFO [kvm.resource.LibvirtConnection] (agentRequest-Handler-5:null) (logid:acc6738a) No existing libvirtd connection found. Opening a new one
2023-06-06 06:39:28,023 WARN [kvm.resource.LibvirtConnection] (agentRequest-Handler-5:null) (logid:acc6738a) Can not find a connection for Instance s-6-VM. Assuming the default connection.
2023-06-06 06:39:28,141 WARN [kvm.resource.LibvirtKvmAgentHook] (agentRequest-Handler-5:null) (logid:acc6738a) Groovy script '/etc/cloudstack/agent/hooks/libvirt-vm-state-change.groovy' is not available. Transformations will not be applied.
2023-06-06 06:39:28,141 WARN [kvm.resource.LibvirtKvmAgentHook] (agentRequest-Handler-5:null) (logid:acc6738a) Groovy scripting engine is not initialized. Data transformation skipped.
2023-06-06 06:39:28,244 INFO [kvm.resource.LibvirtConnection] (agentRequest-Handler-2:null) (logid:85ef2375) No existing libvirtd connection found. Opening a new one
2023-06-06 06:39:28,247 WARN [kvm.resource.LibvirtConnection] (agentRequest-Handler-2:null) (logid:85ef2375) Can not find a connection for Instance v-5-VM. Assuming the default connection.
2023-06-06 06:39:28,366 WARN [kvm.resource.LibvirtKvmAgentHook] (agentRequest-Handler-2:null) (logid:85ef2375) Groovy script '/etc/cloudstack/agent/hooks/libvirt-vm-state-change.groovy' is not available. Transformations will not be applied.
2023-06-06 06:39:28,366 WARN [kvm.resource.LibvirtKvmAgentHook] (agentRequest-Handler-2:null) (logid:85ef2375) Groovy scripting engine is not initialized. Data transformation skipped.
2023-06-06 06:39:52,087 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-4:null) (logid:85ef2375) Trying to fetch storage pool 3b8ce946-c4a6-38d0-9484-1495a452e9d2 from libvirt
2023-06-06 06:39:52,166 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-4:null) (logid:85ef2375) Trying to fetch storage pool 3b8ce946-c4a6-38d0-9484-1495a452e9d2 from libvirt
2023-06-06 06:39:52,213 WARN [kvm.resource.LibvirtKvmAgentHook] (agentRequest-Handler-4:null) (logid:85ef2375) Groovy script '/etc/cloudstack/agent/hooks/libvirt-vm-xml-transformer.groovy' is not available. Transformations will not be applied.
2023-06-06 06:39:52,214 WARN [kvm.resource.LibvirtKvmAgentHook] (agentRequest-Handler-4:null) (logid:85ef2375) Groovy scripting engine is not initialized. Data transformation skipped.
2023-06-06 06:39:52,460 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-1:null) (logid:acc6738a) Trying to fetch storage pool 3b8ce946-c4a6-38d0-9484-1495a452e9d2 from libvirt
2023-06-06 06:39:52,525 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-1:null) (logid:acc6738a) Trying to fetch storage pool 3b8ce946-c4a6-38d0-9484-1495a452e9d2 from libvirt
2023-06-06 06:39:52,559 WARN [kvm.resource.LibvirtKvmAgentHook] (agentRequest-Handler-1:null) (logid:acc6738a) Groovy script '/etc/cloudstack/agent/hooks/libvirt-vm-xml-transformer.groovy' is not available. Transformations will not be applied.
2023-06-06 06:39:52,560 WARN [kvm.resource.LibvirtKvmAgentHook] (agentRequest-Handler-1:null) (logid:acc6738a) Groovy scripting engine is not initialized. Data transformation skipped.
2023-06-06 06:39:53,046 WARN [resource.wrapper.LibvirtStartCommandWrapper] (agentRequest-Handler-4:null) (logid:85ef2375) LibvirtException
org.libvirt.LibvirtException: internal error: qemu unexpectedly closed the monitor: [2023-06-06 06:39:52.584898 +0000] I [io-stats.c:3711:ios_sample_buf_size_configure] 0-primarys: Configure ios_sample_buf size is 1024 because ios_sample_interval is 0
2023-06-06T06:39:52.597012Z qemu-system-x86_64: -blockdev {"node-name":"libvirt-2-format","read-only":false,"cache":{"direct":true,"no-flush":false},"driver":"qcow2","file":"libvirt-2-storage"}: Could not read L1 table: Input/output error
at org.libvirt.ErrorHandler.processError(Unknown Source)
at org.libvirt.ErrorHandler.processError(Unknown Source)
at org.libvirt.Connect.domainCreateXML(Unknown Source)
at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.startVM(LibvirtComputingResource.java:1815)
at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtStartCommandWrapper.execute(LibvirtStartCommandWrapper.java:89)
at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtStartCommandWrapper.execute(LibvirtStartCommandWrapper.java:49)
at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:78)
at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1847)
at com.cloud.agent.Agent.processRequest(Agent.java:662)
at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:1082)
at com.cloud.utils.nio.Task.call(Task.java:83)
at com.cloud.utils.nio.Task.call(Task.java:29)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
2023-06-06 06:39:53,064 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-4:null) (logid:85ef2375) Trying to fetch storage pool 3b8ce946-c4a6-38d0-9484-1495a452e9d2 from libvirt
2023-06-06 06:39:53,827 WARN [resource.wrapper.LibvirtStartCommandWrapper] (agentRequest-Handler-1:null) (logid:acc6738a) LibvirtException
org.libvirt.LibvirtException: internal error: qemu unexpectedly closed the monitor: [2023-06-06 06:39:53.383481 +0000] I [io-stats.c:3711:ios_sample_buf_size_configure] 0-primarys: Configure ios_sample_buf size is 1024 because ios_sample_interval is 0
2023-06-06T06:39:53.394974Z qemu-system-x86_64: -blockdev {"node-name":"libvirt-2-format","read-only":false,"cache":{"direct":true,"no-flush":false},"driver":"qcow2","file":"libvirt-2-storage"}: Could not read L1 table: Input/output error
at org.libvirt.ErrorHandler.processError(Unknown Source)
at org.libvirt.ErrorHandler.processError(Unknown Source)
at org.libvirt.Connect.domainCreateXML(Unknown Source)
at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.startVM(LibvirtComputingResource.java:1815)
at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtStartCommandWrapper.execute(LibvirtStartCommandWrapper.java:89)
at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtStartCommandWrapper.execute(LibvirtStartCommandWrapper.java:49)
at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:78)
at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1847)
at com.cloud.agent.Agent.processRequest(Agent.java:662)
at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:1082)
at com.cloud.utils.nio.Task.call(Task.java:83)
at com.cloud.utils.nio.Task.call(Task.java:29)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
2023-06-06 06:39:54,012 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-1:null) (logid:acc6738a) Trying to fetch storage pool 3b8ce946-c4a6-38d0-9484-1495a452e9d2 from libvirt
2023-06-06 06:39:58,260 INFO [kvm.resource.LibvirtConnection] (agentRequest-Handler-5:null) (logid:acc6738a) No existing libvirtd connection found. Opening a new one
2023-06-06 06:39:58,263 WARN [kvm.resource.LibvirtConnection] (agentRequest-Handler-5:null) (logid:acc6738a) Can not find a connection for Instance s-6-VM. Assuming the default connection.
2023-06-06 06:39:58,381 WARN [kvm.resource.LibvirtKvmAgentHook] (agentRequest-Handler-5:null) (logid:acc6738a) Groovy script '/etc/cloudstack/agent/hooks/libvirt-vm-state-change.groovy' is not available. Transformations will not be applied.
2023-06-06 06:39:58,382 WARN [kvm.resource.LibvirtKvmAgentHook] (agentRequest-Handler-5:null) (logid:acc6738a) Groovy scripting engine is not initialized. Data transformation skipped.
2023-06-06 06:39:59,316 INFO [kvm.resource.LibvirtConnection] (agentRequest-Handler-3:null) (logid:85ef2375) No existing libvirtd connection found. Opening a new one
2023-06-06 06:39:59,319 WARN [kvm.resource.LibvirtConnection] (agentRequest-Handler-3:null) (logid:85ef2375) Can not find a connection for Instance v-5-VM. Assuming the default connection.
2023-06-06 06:39:59,438 WARN [kvm.resource.LibvirtKvmAgentHook] (agentRequest-Handler-3:null) (logid:85ef2375) Groovy script '/etc/cloudstack/agent/hooks/libvirt-vm-state-change.groovy' is not available. Transformations will not be applied.
2023-06-06 06:39:59,438 WARN [kvm.resource.LibvirtKvmAgentHook] (agentRequest-Handler-3:null) (logid:85ef2375) Groovy scripting engine is not initialized. Data transformation skipped.
2023-06-06 06:40:00,610 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-2:null) (logid:be555a18) Trying to fetch storage pool 3b8ce946-c4a6-38d0-9484-1495a452e9d2 from libvirt
2023-06-06 06:40:00,666 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-2:null) (logid:be555a18) Attempting to remove volume 2f48f71d-628e-41f7-8cd1-36cef891f59f from pool 3b8ce946-c4a6-38d0-9484-1495a452e9d2
2023-06-06 06:40:02,477 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-4:null) (logid:28d1b51d) Trying to fetch storage pool 3b8ce946-c4a6-38d0-9484-1495a452e9d2 from libvirt
2023-06-06 06:40:02,528 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-4:null) (logid:28d1b51d) Attempting to remove volume bba87d20-244b-48c0-8bfb-613efb3d6bf9 from pool 3b8ce946-c4a6-38d0-9484-1495a452e9d2
2023-06-06 07:09:40,324 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-1:null) (logid:9b577b71) Attempting to create storage pool 3b8ce946-c4a6-38d0-9484-1495a452e9d2 (Gluster) in libvirt
2023-06-06 07:09:40,345 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-1:null) (logid:9b577b71) Found existing defined storage pool 3b8ce946-c4a6-38d0-9484-1495a452e9d2, using it.
2023-06-06 07:09:40,346 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-1:null) (logid:9b577b71) Trying to fetch storage pool 3b8ce946-c4a6-38d0-9484-1495a452e9d2 from libvirt
2023-06-06 07:09:45,860 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-5:null) (logid:b6992438) Attempting to remove storage pool 3b8ce946-c4a6-38d0-9484-1495a452e9d2 from libvirt
2023-06-06 07:09:45,881 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-5:null) (logid:b6992438) Storage pool 3b8ce946-c4a6-38d0-9484-1495a452e9d2 has no corresponding secret. Not removing any secret.
2023-06-06 07:09:58,119 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-3:null) (logid:decefcf7) Attempting to remove storage pool 3b8ce946-c4a6-38d0-9484-1495a452e9d2 from libvirt
2023-06-06 07:09:58,141 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-3:null) (logid:decefcf7) Storage pool 3b8ce946-c4a6-38d0-9484-1495a452e9d2 has no corresponding secret. Not removing any secret.
2023-06-06 07:10:07,209 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-2:null) (logid:75913050) Attempting to create storage pool 3b8ce946-c4a6-38d0-9484-1495a452e9d2 (Gluster) in libvirt
2023-06-06 07:10:07,228 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-2:null) (logid:75913050) Found existing defined storage pool 3b8ce946-c4a6-38d0-9484-1495a452e9d2, using it.
2023-06-06 07:10:07,229 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-2:null) (logid:75913050) Trying to fetch storage pool 3b8ce946-c4a6-38d0-9484-1495a452e9d2 from libvirt
2023-06-06 07:18:44,808 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-4:null) (logid:d17785c6) Attempting to create storage pool 2f162451-afd6-358e-9eac-8d33bf2477d2 (NetworkFilesystem) in libvirt
2023-06-06 07:18:44,828 WARN [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-4:null) (logid:d17785c6) Storage pool 2f162451-afd6-358e-9eac-8d33bf2477d2 was not found running in

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template!

[kiranchavala]

Looks like Duplicate of #7603

[rajujith]

@maic not sure if anyone is using gluster with cloudstack, please refer #6866

[maic]

On the Centos 7 system, using glusterfs can create VMs under qemu 2.12 and libvirt 4.5, but on ubuntu 20.04 or 22.04 systems, using glusterfs with qemu greater than 2.12 and libvirt greater than 4.5 cannot create VMs

[tampler]

This is not related to GlusterFS. I had this with NFS and Linstor on Ubuntu 22.04. Fixed by removing all TLS related stuff in /etc/libvirt/qemu.conf

# All this should be removed
#vnc_tls=1
#vnc_tls_x509_verify=1
#vnc_tls_x509_cert_dir="/etc/pki/libvirt-vnc"

I think this is a config/documentation issue, which is Ubuntu 22.04 specific.

[DaanHoogland]

@maic closing this as a work around has been provided. Please reopen or create a new one of you feel this is still a problem.