Skip to content

Conversation

@sureshanaparti
Copy link
Contributor

@sureshanaparti sureshanaparti commented Dec 24, 2025

Description

This PR addresses Clone VM issue with VMware 80u3, retries cloneVM task when any file access issue while cloning from volume or template and should succeed after few attempts.

This is another issue, recently identified with VMware 80u3 (other related issues addressed here: #10586)

Fixes #12035 #12328

Types of changes

  • Breaking change (fix or feature that would cause existing functionality to change)
  • New feature (non-breaking change which adds functionality)
  • Bug fix (non-breaking change which fixes an issue)
  • Enhancement (improves an existing feature and functionality)
  • Cleanup (Code refactoring and cleanup, that may add test cases)
  • Build/CI
  • Test (unit or integration test code)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

  • Major
  • Minor

Bug Severity

  • BLOCKER
  • Critical
  • Major
  • Minor
  • Trivial

Screenshots (if appropriate):

How Has This Been Tested?

How did you try to break this feature and the system with this change?

@sureshanaparti sureshanaparti linked an issue Dec 24, 2025 that may be closed by this pull request
@sureshanaparti
Copy link
Contributor Author

@blueorangutan package

@blueorangutan
Copy link

@sureshanaparti a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

@codecov
Copy link

codecov bot commented Dec 24, 2025

Codecov Report

❌ Patch coverage is 0% with 28 lines in your changes missing coverage. Please review.
✅ Project coverage is 16.23%. Comparing base (b394b5b) to head (c72cce8).
⚠️ Report is 1 commits behind head on 4.20.

Files with missing lines Patch % Lines
...m/cloud/hypervisor/vmware/mo/VirtualMachineMO.java 0.00% 28 Missing ⚠️
Additional details and impacted files
@@             Coverage Diff              @@
##               4.20   #12335      +/-   ##
============================================
- Coverage     16.23%   16.23%   -0.01%     
+ Complexity    13377    13373       -4     
============================================
  Files          5657     5657              
  Lines        498865   498879      +14     
  Branches      60545    60546       +1     
============================================
- Hits          80991    80984       -7     
- Misses       408843   408861      +18     
- Partials       9031     9034       +3     
Flag Coverage Δ
uitests 4.00% <ø> (ø)
unittests 17.09% <0.00%> (-0.01%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@blueorangutan
Copy link

Packaging result [SF]: ✖️ el8 ✖️ el9 ✔️ debian ✖️ suse15. SL-JID 16151

@kiranchavala
Copy link
Contributor

@blueorangutan package

@blueorangutan
Copy link

@kiranchavala a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

@blueorangutan
Copy link

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 16156

Copy link
Contributor

@kiranchavala kiranchavala left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, Tested with multicluster env

Was able to deploy a vm from a ova file

2025-12-26 07:48:12,899 DEBUG [o.a.c.s.i.TemplateDataFactoryImpl] (Work-Job-Executor-3:[ctx-880d0346, job-35/job-36, ctx-31015cc3]) (logid:aae472c7) template Template {"format":"OVA","id":202,"name":"ubuntu24","uniqueName":"202-2-979c3d6c-906b-3186-9fef-d05872b1fc3a","uuid":"1d8b5254-c5f4-4b1f-8997-7029bd904531"} with id 202 is already in store:ImageStore {"id":1,"name":"NFS:\/\/10.1.32.4\/acs\/secondary\/ref-trl-6142-v-Mu24-kiran-chavala\/ref-trl-6142-v-Mu24-kiran-chavala-sec1","uuid":"59a429d6-9367-45ed-b06f-af5d18201119"}, type: Image
2025-12-26 07:48:12,903 DEBUG [o.a.c.s.d.PrimaryDataStoreImpl] (Work-Job-Executor-3:[ctx-880d0346, job-35/job-36, ctx-31015cc3]) (logid:aae472c7) Not found (templateId:202poolId:1conf:null) in template_spool_ref, persisting it
2025-12-26 07:48:12,910 DEBUG [o.a.c.s.i.TemplateDataFactoryImpl] (Work-Job-Executor-3:[ctx-880d0346, job-35/job-36, ctx-31015cc3]) (logid:aae472c7) template Template {"format":"OVA","id":202,"name":"ubuntu24","uniqueName":"202-2-979c3d6c-906b-3186-9fef-d05872b1fc3a","uuid":"1d8b5254-c5f4-4b1f-8997-7029bd904531"} with id 202 is already in store:StoragePool {"id":1,"name":"ref-trl-6142-v-Mu24-kiran-chavala-esxi-pri1","poolType":"NetworkFilesystem","uuid":"f5a29843-7c46-3fb1-a0ea-844860f848f1"}, type: Primary
2025-12-26 07:48:12,911 DEBUG [o.a.c.s.v.VolumeServiceImpl] (Work-Job-Executor-3:[ctx-880d0346, job-35/job-36, ctx-31015cc3]) (logid:aae472c7) Found template Template {"format":"OVA","id":202,"name":"ubuntu24","uniqueName":"202-2-979c3d6c-906b-3186-9fef-d05872b1fc3a","uuid":"1d8b5254-c5f4-4b1f-8997-7029bd904531"} in storage pool StoragePool {"id":1,"name":"ref-trl-6142-v-Mu24-kiran-chavala-esxi-pri1","poolType":"NetworkFilesystem","uuid":"f5a29843-7c46-3fb1-a0ea-844860f848f1"} with VMTemplateStoragePool: TmplPool[3-202-1-null]
2025-12-26 07:48:12,912 DEBUG [o.a.c.s.v.VolumeServiceImpl] (Work-Job-Executor-3:[ctx-880d0346, job-35/job-36, ctx-31015cc3]) (logid:aae472c7) Acquire lock on VMTemplateStoragePool 3 with timeout 3600 seconds
2025-12-26 07:48:12,917 INFO  [o.a.c.s.v.VolumeServiceImpl] (Work-Job-Executor-3:[ctx-880d0346, job-35/job-36, ctx-31015cc3]) (logid:aae472c7) lock is acquired for VMTemplateStoragePool 3

2025-12-26 07:49:50,874 DEBUG [c.c.a.t.Request] (DirectAgent-482:[ctx-16173aa5]) (logid:aae472c7) Seq 2-8532350969030120521: Processing:  { Ans: , MgmtId: 32989492808056, via: 2(10.1.32.115), Ver: v1, Flags: 10, [{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"newData":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"size":"(3.50 GB) 3758096384","path":"ROOT-3","accountId":"0","id":"0","directDownload":"false","deployAsIs":"false","followRedirects":"false"}},"result":"true","wait":"0","bypassHostMaintenance":"false"}}] }
2025-12-26 07:49:50,874 DEBUG [c.c.a.t.Request] (Work-Job-Executor-3:[ctx-880d0346, job-35/job-36, ctx-31015cc3]) (logid:aae472c7) Seq 2-8532350969030120521: Received:  { Ans: , MgmtId: 32989492808056, via: 2(10.1.32.115), Ver: v1, Flags: 10, { CopyCmdAnswer } }
2025-12-26 07:49:50,887 DEBUG [o.a.c.s.v.VolumeObject] (Work-Job-Executor-3:[ctx-880d0346, job-35/job-36, ctx-31015cc3]) (logid:aae472c7) Updated {"name":"ROOT-3","uuid":"085a9cfe-ae78-4cee-9dd1-2f4697b2c643"} from {"encryptFormat":null,"format":"OVA","path":null,"poolId":1,"size":3758096384} to {"encryptFormat":null,"format":"OVA","path":"ROOT-3","poolId":1,"size":3758096384}


2025-12-26 07:50:17,958 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-29:[ctx-7c16c813, job-35, ctx-a9d8fb99]) (logid:aae472c7) Complete async job-35, jobStatus: SUCCEEDED, resultCode: 0, result: org.apache.cloudstack.api.response.UserVmResponse/virtualmachine/{"id":"18646751-7e92-45e6-b858-4ed896eaf58d","name":"VM-18646751

@kiranchavala
Copy link
Contributor

@blueorangutan test matrix

@blueorangutan
Copy link

@kiranchavala a [SL] Trillian-Jenkins matrix job (EL8 mgmt + EL8 KVM, Ubuntu22 mgmt + Ubuntu22 KVM, EL8 mgmt + VMware 7.0u3, EL9 mgmt + XCP-ng 8.2 ) has been kicked to run smoke tests

@weizhouapache
Copy link
Member

LGTM, Tested with multicluster env

Was able to deploy a vm from a ova file

2025-12-26 07:48:12,899 DEBUG [o.a.c.s.i.TemplateDataFactoryImpl] (Work-Job-Executor-3:[ctx-880d0346, job-35/job-36, ctx-31015cc3]) (logid:aae472c7) template Template {"format":"OVA","id":202,"name":"ubuntu24","uniqueName":"202-2-979c3d6c-906b-3186-9fef-d05872b1fc3a","uuid":"1d8b5254-c5f4-4b1f-8997-7029bd904531"} with id 202 is already in store:ImageStore {"id":1,"name":"NFS:\/\/10.1.32.4\/acs\/secondary\/ref-trl-6142-v-Mu24-kiran-chavala\/ref-trl-6142-v-Mu24-kiran-chavala-sec1","uuid":"59a429d6-9367-45ed-b06f-af5d18201119"}, type: Image
2025-12-26 07:48:12,903 DEBUG [o.a.c.s.d.PrimaryDataStoreImpl] (Work-Job-Executor-3:[ctx-880d0346, job-35/job-36, ctx-31015cc3]) (logid:aae472c7) Not found (templateId:202poolId:1conf:null) in template_spool_ref, persisting it
2025-12-26 07:48:12,910 DEBUG [o.a.c.s.i.TemplateDataFactoryImpl] (Work-Job-Executor-3:[ctx-880d0346, job-35/job-36, ctx-31015cc3]) (logid:aae472c7) template Template {"format":"OVA","id":202,"name":"ubuntu24","uniqueName":"202-2-979c3d6c-906b-3186-9fef-d05872b1fc3a","uuid":"1d8b5254-c5f4-4b1f-8997-7029bd904531"} with id 202 is already in store:StoragePool {"id":1,"name":"ref-trl-6142-v-Mu24-kiran-chavala-esxi-pri1","poolType":"NetworkFilesystem","uuid":"f5a29843-7c46-3fb1-a0ea-844860f848f1"}, type: Primary
2025-12-26 07:48:12,911 DEBUG [o.a.c.s.v.VolumeServiceImpl] (Work-Job-Executor-3:[ctx-880d0346, job-35/job-36, ctx-31015cc3]) (logid:aae472c7) Found template Template {"format":"OVA","id":202,"name":"ubuntu24","uniqueName":"202-2-979c3d6c-906b-3186-9fef-d05872b1fc3a","uuid":"1d8b5254-c5f4-4b1f-8997-7029bd904531"} in storage pool StoragePool {"id":1,"name":"ref-trl-6142-v-Mu24-kiran-chavala-esxi-pri1","poolType":"NetworkFilesystem","uuid":"f5a29843-7c46-3fb1-a0ea-844860f848f1"} with VMTemplateStoragePool: TmplPool[3-202-1-null]
2025-12-26 07:48:12,912 DEBUG [o.a.c.s.v.VolumeServiceImpl] (Work-Job-Executor-3:[ctx-880d0346, job-35/job-36, ctx-31015cc3]) (logid:aae472c7) Acquire lock on VMTemplateStoragePool 3 with timeout 3600 seconds
2025-12-26 07:48:12,917 INFO  [o.a.c.s.v.VolumeServiceImpl] (Work-Job-Executor-3:[ctx-880d0346, job-35/job-36, ctx-31015cc3]) (logid:aae472c7) lock is acquired for VMTemplateStoragePool 3

2025-12-26 07:49:50,874 DEBUG [c.c.a.t.Request] (DirectAgent-482:[ctx-16173aa5]) (logid:aae472c7) Seq 2-8532350969030120521: Processing:  { Ans: , MgmtId: 32989492808056, via: 2(10.1.32.115), Ver: v1, Flags: 10, [{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"newData":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"size":"(3.50 GB) 3758096384","path":"ROOT-3","accountId":"0","id":"0","directDownload":"false","deployAsIs":"false","followRedirects":"false"}},"result":"true","wait":"0","bypassHostMaintenance":"false"}}] }
2025-12-26 07:49:50,874 DEBUG [c.c.a.t.Request] (Work-Job-Executor-3:[ctx-880d0346, job-35/job-36, ctx-31015cc3]) (logid:aae472c7) Seq 2-8532350969030120521: Received:  { Ans: , MgmtId: 32989492808056, via: 2(10.1.32.115), Ver: v1, Flags: 10, { CopyCmdAnswer } }
2025-12-26 07:49:50,887 DEBUG [o.a.c.s.v.VolumeObject] (Work-Job-Executor-3:[ctx-880d0346, job-35/job-36, ctx-31015cc3]) (logid:aae472c7) Updated {"name":"ROOT-3","uuid":"085a9cfe-ae78-4cee-9dd1-2f4697b2c643"} from {"encryptFormat":null,"format":"OVA","path":null,"poolId":1,"size":3758096384} to {"encryptFormat":null,"format":"OVA","path":"ROOT-3","poolId":1,"size":3758096384}


2025-12-26 07:50:17,958 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-29:[ctx-7c16c813, job-35, ctx-a9d8fb99]) (logid:aae472c7) Complete async job-35, jobStatus: SUCCEEDED, resultCode: 0, result: org.apache.cloudstack.api.response.UserVmResponse/virtualmachine/{"id":"18646751-7e92-45e6-b858-4ed896eaf58d","name":"VM-18646751

as I remember, the issue appears very randomly.
it would better to register several tempates and deploy vms to see if the issue can be reproduced and the retry fixes the issue

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Unable to launch a vm on vmware cluster

4 participants