Cloudstack wizard is getting stuck storage traffic on version 4.20 #10550

verendrakalsi · 2025-03-06T14:29:39Z

verendrakalsi
Mar 6, 2025

Ubuntu 22.04.5 LTS
Cloustack version :- 4.20

we have install cloudstack agent on 4 hosts and 1 hosts with cloudstack mgmt. After installing when we are trying to run the cloudstack mgmt wizard, it is getting stuck on the storage traffic section.

The Section creates the zone, pod, cluster, add the host, & Primary and secondary storage but after that nothing and it keep spinnig on the wizard at storage traffic section.

verendrakalsi · 2025-03-06T15:08:17Z

verendrakalsi
Mar 6, 2025
Author

its getting stuck here.

0 replies

verendrakalsi · 2025-03-06T17:59:32Z

verendrakalsi
Mar 6, 2025
Author

looks like some issue with monuting the secondary storage
Failed to create netfs mount: 10.10.10.12:/KVMSecondary1/template/tmpl/1/3 org.libvirt.LibvirtException: internal error: Child process (/usr/bin/mount -o nodev,nosuid,noexec 10.10.10.12:/KVMSecondary1/template/tmpl/1/3 /mnt/d702e2f4-ecde-318b-b04e-a3cf1bab9afd) unexpected exit status 32: mount.nfs: mounting 10.10.10.12:/KVMSecondary1/template/tmpl/1/3 failed, reason given by server: No such file or directory

0 replies

Pearl1594 · 2025-03-10T16:46:49Z

Pearl1594
Mar 10, 2025
Collaborator

@verendrakalsi Does /KVMSecondary1/ path exist on the secondary store: 10.10.10.12?

0 replies

verendrakalsi · 2025-03-10T17:06:45Z

verendrakalsi
Mar 10, 2025
Author

yes, it does. now this issue is resolved.

but now i am getting this error

2025-03-10 20:32:02,961 ERROR [cloud.agent.AgentShell] (main:[]) (logid:) Unable to start agent: javax.naming.ConfigurationException: Unable to configure LibvirtComputingResource
at com.cloud.agent.Agent.(Agent.java:196)
at com.cloud.agent.AgentShell.launchNewAgent(AgentShell.java:453)
at com.cloud.agent.AgentShell.launchAgentFromClassInfo(AgentShell.java:432)
at com.cloud.agent.AgentShell.launchAgent(AgentShell.java:416)
at com.cloud.agent.AgentShell.start(AgentShell.java:512)
at com.cloud.agent.AgentShell.main(AgentShell.java:542)

0 replies

Pearl1594 · 2025-03-10T22:29:23Z

Pearl1594
Mar 10, 2025
Collaborator

Can you please share the entire stacktrace. Thanks

0 replies

DaanHoogland · 2025-03-11T07:45:06Z

DaanHoogland
Mar 11, 2025
Collaborator

@verendrakalsi , the stacktrace as a "Cause", this is what @Pearl1594 is interested in.

Also can you add the log lines leading up to this stacktrace. Especially the ones containing LibvirtComputingResource or AgentShell?

0 replies

verendrakalsi · 2025-03-11T14:13:26Z

verendrakalsi
Mar 11, 2025
Author

this is what you are looking for ?

2025-03-11 14:11:17,825 INFO [cloud.agent.AgentShell] (main:[]) (logid:) Agent started
2025-03-11 14:11:17,830 INFO [cloud.agent.AgentShell] (main:[]) (logid:) Implementation Version is 4.20.0.0
2025-03-11 14:11:17,831 INFO [cloud.agent.AgentShell] (main:[]) (logid:) agent.properties found at /etc/cloudstack/agent/agent.properties
2025-03-11 14:11:17,870 INFO [cloud.agent.AgentShell] (main:[]) (logid:) Defaulting to using properties file for storage
2025-03-11 14:11:17,871 INFO [cloud.agent.AgentShell] (main:[]) (logid:) Defaulting to the constant time backoff algorithm
2025-03-11 14:11:17,895 INFO [cloud.utils.LogUtils] (main:[]) (logid:) log4j configuration found at /etc/cloudstack/agent/log4j-cloud.xml
2025-03-11 14:11:17,895 INFO [cloud.agent.AgentShell] (main:[]) (logid:) Using default Java settings for IPv6 preference for agent connection
2025-03-11 14:11:17,948 INFO [cloud.agent.Agent] (main:[]) (logid:) id is 0
2025-03-11 14:11:17,953 WARN [kvm.resource.LibvirtComputingResource] (main:[]) (logid:) Incorrect details for private Nic during initialization of ServerResourceBase
2025-03-11 14:11:17,953 ERROR [cloud.agent.AgentShell] (main:[]) (logid:) Unable to start agent: javax.naming.ConfigurationException: Unable to configure LibvirtComputingResource
at com.cloud.agent.Agent.(Agent.java:196)
at com.cloud.agent.AgentShell.launchNewAgent(AgentShell.java:453)
at com.cloud.agent.AgentShell.launchAgentFromClassInfo(AgentShell.java:432)
at com.cloud.agent.AgentShell.launchAgent(AgentShell.java:416)
at com.cloud.agent.AgentShell.start(AgentShell.java:512)
at com.cloud.agent.AgentShell.main(AgentShell.java:542)

0 replies

verendrakalsi · 2025-03-11T20:58:05Z

verendrakalsi
Mar 11, 2025
Author

2025-03-11T20:53:36,178 WARN [cloud.agent.Agent] (agentRequest-Handler-1:[]) Caught: com.cloud.utils.exception.CloudRuntimeException: Failed to get root directory from secondary storage URL [nfs://10.10.10.12/KVMSecondary1], using NFS version [null], due to [Unable to mount 10.10.10.12:/KVMSecondary1 at /mnt/SecStorage/bd7fe518-f806-3eee-92f9-0819535406ae due to mount.nfs: Connection timed out].
at org.apache.cloudstack.storage.resource.NfsSecondaryStorageResource.getRootDir(NfsSecondaryStorageResource.java:2675)
at org.apache.cloudstack.storage.resource.NfsSecondaryStorageResource.execute(NfsSecondaryStorageResource.java:2407)
at org.apache.cloudstack.storage.resource.NfsSecondaryStorageResource.executeRequest(NfsSecondaryStorageResource.java:298)
at com.cloud.storage.resource.PremiumSecondaryStorageResource.defaultAction(PremiumSecondaryStorageResource.java:62)
at com.cloud.storage.resource.PremiumSecondaryStorageResource.executeRequest(PremiumSecondaryStorageResource.java:58)
at com.cloud.agent.Agent.processRequest(Agent.java:686)
at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:1109)
at com.cloud.utils.nio.Task.call(Task.java:83)
at com.cloud.utils.nio.Task.call(Task.java:29)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: com.cloud.utils.exception.CloudRuntimeException: Unable to mount 10.10.10.12:/KVMSecondary1 at /mnt/SecStorage/bd7fe518-f806-3eee-92f9-0819535406ae due to mount.nfs: Connection timed out
at org.apache.cloudstack.storage.resource.NfsSecondaryStorageResource.attemptMount(NfsSecondaryStorageResource.java:3081)
at org.apache.cloudstack.storage.resource.NfsSecondaryStorageResource.mount(NfsSecondaryStorageResource.java:3032)
at org.apache.cloudstack.storage.resource.NfsSecondaryStorageResource.mountUri(NfsSecondaryStorageResource.java:3020)
at org.apache.cloudstack.storage.resource.NfsSecondaryStorageResource.getRootDir(NfsSecondaryStorageResource.java:2671)
... 12 more

2025-03-11T20:54:36,351 INFO [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-2:[]) Determined host 10.10.10.12 corresponds to IP 10.10.10.12

0 replies

DaanHoogland · 2025-03-12T09:47:05Z

DaanHoogland
Mar 12, 2025
Collaborator

@verendrakalsi , there still seems to be an issue with your NFS mount ; "Unable to mount 10.10.10.12:/KVMSecondary1 at /mnt/SecStorage/bd7fe518-f806-3eee-92f9-0819535406ae due to mount.nfs: Connection timed out"

5 replies

verendrakalsi Mar 12, 2025
Author

But i do see secondary storage mounted to the management server and i can access it

10.10.10.12:/KVMSecondary1 251G 454M 250G 1% /var/lib/cloudstack/mnt/253023765254217.61fa6fda
root@host:/home/kvmadmin# cd /var/lib/cloudstack/mnt/
root@host:/var/lib/cloudstack/mnt# ls
202872835190222.29e9619d 253023765254217.61fa6fda
root@host:/var/lib/cloudstack/mnt# cd 253023765254217.61fa6fda/
root@host:/var/lib/cloudstack/mnt/253023765254217.61fa6fda# ls
snapshots template volumes
root@host:/var/lib/cloudstack/mnt/253023765254217.61fa6fda#

verendrakalsi Mar 12, 2025
Author

Also now in ssystem VMs we are getting this error
/mnt/SecStorage/ca7ab41c-ce2a-3350-a7e9-88a4e20bc784/template/tmpl/1/4/dnld7827224589018529675tmp_, bytes=(0 bytes) 0, error=No route to host (Host unreachable), pct=0

weizhouapache Mar 12, 2025
Collaborator

can you log into ssvm and check if the secondary storage can be mounted ?

what are the IPs and routes in SSVM ?

verendrakalsi Mar 12, 2025
Author

i can see the secondary storage attached to the ssvm

root@s-9-VM:~# df -h
Filesystem Size Used Avail Use% Mounted on
udev 214M 0 214M 0% /dev
tmpfs 93M 492K 92M 1% /run
/dev/vda6 4.2G 1.3G 2.9G 32% /
tmpfs 231M 0 231M 0% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
/dev/vda1 366M 46M 321M 13% /boot
10.43.197.12:/KVMSecondary1 251G 455M 250G 1% /mnt/SecStorage/ca7ab41c-ce2a-3350-a7e9-88a4e20bc784
tmpfs 47M 0 47M 0% /run/user/0

verendrakalsi Mar 12, 2025
Author

default via 10.43.197.1 dev eth2
10.43.197.0/24 dev eth1 proto kernel scope link src 10.43.197.147
10.43.197.0/24 dev eth2 proto kernel scope link src 10.43.197.171
10.63.8.52 via 10.43.197.1 dev eth1
169.254.0.0/16 dev eth0 proto kernel scope link src 169.254.105.132

weizhouapache · 2025-03-12T20:49:52Z

weizhouapache
Mar 12, 2025
Collaborator

I wonder why the ssvm tried to mount

10.10.10.12:/KVMSecondary1

which has different IP as the secondary storage already mounted

10.43.197.12:/KVMSecondary1 251G 455M 250G 1% /mnt/SecStorage/ca7ab41c-ce2a-3350-a7e9-88a4e20bc784

9 replies

weizhouapache Mar 12, 2025
Collaborator

yes, we cannot upload the ISO still and it gives that error

Upload from local or from a URL ?

verendrakalsi Mar 12, 2025
Author

irrespective of local or URL.

for load, we are directly getting that error. But URL shows as if it is doing something but in the logs we get error

weizhouapache Mar 12, 2025
Collaborator

irrespective of local or URL.

for load, we are directly getting that error. But URL shows as if it is doing something but in the logs we get error

for upload from local, if you have not configured SSL certificate and domains, you can change global setting use.https.to.upload to false, restart mgmt server and destroy ssvm, then retry

for upload via URL, you can find the information in /var/log/cloud.log in ssvm

verendrakalsi Mar 12, 2025
Author

Those setting is already done, but we still have same issue.

weizhouapache Mar 14, 2025
Collaborator

Those setting is already done, but we still have same issue.

can you please check if the public IP of ssvm is accessible from your local server ?

I tested ok.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Cloudstack wizard is getting stuck storage traffic on version 4.20 #10550

{{title}}

Replies: 10 comments 14 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

Cloudstack wizard is getting stuck storage traffic on version 4.20 #10550

verendrakalsi Mar 6, 2025

Replies: 10 comments · 14 replies

verendrakalsi Mar 6, 2025 Author

verendrakalsi Mar 6, 2025 Author

Pearl1594 Mar 10, 2025 Collaborator

verendrakalsi Mar 10, 2025 Author

Pearl1594 Mar 10, 2025 Collaborator

DaanHoogland Mar 11, 2025 Collaborator

verendrakalsi Mar 11, 2025 Author

verendrakalsi Mar 11, 2025 Author

DaanHoogland Mar 12, 2025 Collaborator

verendrakalsi Mar 12, 2025 Author

verendrakalsi Mar 12, 2025 Author

weizhouapache Mar 12, 2025 Collaborator

verendrakalsi Mar 12, 2025 Author

verendrakalsi Mar 12, 2025 Author

weizhouapache Mar 12, 2025 Collaborator

weizhouapache Mar 12, 2025 Collaborator

verendrakalsi Mar 12, 2025 Author

weizhouapache Mar 12, 2025 Collaborator

verendrakalsi Mar 12, 2025 Author

weizhouapache Mar 14, 2025 Collaborator

verendrakalsi
Mar 6, 2025

Replies: 10 comments 14 replies

verendrakalsi
Mar 6, 2025
Author

verendrakalsi
Mar 6, 2025
Author

Pearl1594
Mar 10, 2025
Collaborator

verendrakalsi
Mar 10, 2025
Author

Pearl1594
Mar 10, 2025
Collaborator

DaanHoogland
Mar 11, 2025
Collaborator

verendrakalsi
Mar 11, 2025
Author

verendrakalsi
Mar 11, 2025
Author

DaanHoogland
Mar 12, 2025
Collaborator

verendrakalsi Mar 12, 2025
Author

verendrakalsi Mar 12, 2025
Author

weizhouapache Mar 12, 2025
Collaborator

verendrakalsi Mar 12, 2025
Author

verendrakalsi Mar 12, 2025
Author

weizhouapache
Mar 12, 2025
Collaborator

weizhouapache Mar 12, 2025
Collaborator

verendrakalsi Mar 12, 2025
Author

weizhouapache Mar 12, 2025
Collaborator

verendrakalsi Mar 12, 2025
Author

weizhouapache Mar 14, 2025
Collaborator