Improve migration of external VMware VMs into KVM cluster #8815

sureshanaparti · 2024-03-20T17:06:45Z

Description

This PR improves/optimises the migration/import performance of external VMware VMs into KVM cluster.

The improved migration process still uses the 'virt-v2v' for importing the VMware VMs, and uses exported OVF (of the VMware VM) as the source for the conversion to KVM instance, instead of the cloned VM (in earlier migration, accessing vCenter using 'virt-v2v' from KVM host). For this,

CloudStack would try to pick the supported host for conversion if one exists, when conversion host is not specified, Otherwise uses any random Up & Enabled host as conversion host. It first checks if the host supports conversion (with virt-v2v, and nbdkit for Ubuntu host) or not.
CloudStack would then export OVF template files of the source VMware VM (or Cloned VM - for Powered-On VMs, it creates a clone of the source VM on VMware) to the temporary conversion location (uses primary or secondary storage - only NFS pools are supported, defaulted to secondary storage). Any cloned VM and OVF template files are cleaned up after the migration, or if any errors.
CloudStack then delegates the conversion to a KVM host. The KVM host accesses/uses the exported OVF on the temporary conversion location as the source for the migration using 'virt-v2v' ova input. It creates the temporary disks and xml on the temporary conversion location, and later these are moved to the destination storage pools to import them. Any errors during virt-v2v conversion due to unsupported guest OS, or other reasons are handled, and the conversion operation is failed.

Noticed 70-90% decrease in the time taken for the migration with this approach (Some earlier linux VM migrations with 3-5 GB disks which took 30-35 mins, now takes 2-3 mins with this improvement).

There are no changes in the VMs supported on vCenter - Stopped & Running Linux VMs, and Stopped Windows VMs are supported as earlier. Additionally, Stopped VMs on standalone hosts are supported (can be imported by specifying its default datacenter name - ha-datacenter along with host and credentails).

Note: All 'virt-v2v' limitations are still applicable. CloudStack will not perform any checks about the guest OS compatibility for the virt-v2v library as indicated on: https://access.redhat.com/articles/1351473.

Doc PR: apache/cloudstack-documentation#388

Other improvements in the migration process:

MS checks the host for instance conversion support (with virt-v2v, and nbdkit for Ubuntu host), & windows guest conversion support (virtio-win package) before attempting migration (using CheckConvertInstanceCommand / CheckConvertInstanceAnswer).
MS checks ovftool available in the host to export OVF from vCenter directly, if not MS is used to export OVF
Auto selects any existing host with instance conversion capability if conversion host not specified (uses detail 'host.instance.conversion').
New Parameter: forcemstoimportvmfiles in importVM API, to force MS to import VM files/OVF if required. Added UI support for it.
New detail 'host.instance.conversion' with value 'true' is added to host_details table if the host supports instance conversion (needs agent restart after installing virt-v2v).
New host response parameter 'instanceconversionsupported' - true, indicating the instance conversion support.
UI shows Instance Conversion Supported - true in Host details if host supports conversion (with virt-v2v, and nbdkit for Ubuntu host).
UI shows Supported KVM hosts (with virt-v2v) for conversion on Import VM dialog.
Support for parallel import/download of OVF disk files on MS or KVM Host, using threads that are configurable using the global settings: threads.on.ms.to.import.vmware.vm.files or threads.on.kvm.host.to.import.vmware.vm.files respectively.

Also, fixes: #8632

Types of changes

Breaking change (fix or feature that would cause existing functionality to change)
New feature (non-breaking change which adds functionality)
Bug fix (non-breaking change which fixes an issue)
Enhancement (improves an existing feature and functionality)
Cleanup (Code refactoring and cleanup, that may add test cases)
build/CI

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Major
Minor

Bug Severity

Screenshots (if appropriate):

Host details (Instance Conversion Supported for KVM hosts with virt-v2v):

Import VM from VMware (Supported KVM hosts with virt-v2v):

Flag - to Force MS to import/download VM files (OVF):

VMwareMigrationToKVM_ForceMSFlag_Updated

Threads config - to import/download VM files

How Has This Been Tested?

Manually tested importing external VMware VMs with single and multiple disks(on different datastore) to KVM cluster in CloudStack.

How did you try to break this feature and the system with this change?

sureshanaparti · 2024-03-20T17:07:35Z

@blueorangutan package

blueorangutan · 2024-03-20T17:08:04Z

@sureshanaparti a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

codecov · 2024-03-20T17:15:09Z

Codecov Report

Attention: Patch coverage is 26.35135% with 109 lines in your changes are missing coverage. Please review.

Project coverage is 30.97%. Comparing base (308ed13) to head (3ebee84).
Report is 4 commits behind head on 4.19.

Files	Patch %	Lines
...ain/java/com/cloud/hypervisor/guru/VMwareGuru.java	0.00%	66 Missing ⚠️
.../storage/datastore/db/PrimaryDataStoreDaoImpl.java	0.00%	14 Missing ⚠️
.../apache/cloudstack/vm/UnmanagedVMsManagerImpl.java	66.66%	7 Missing and 5 partials ⚠️
...tack/engine/orchestration/NetworkOrchestrator.java	0.00%	7 Missing ⚠️
.../wrapper/LibvirtConvertInstanceCommandWrapper.java	70.58%	3 Missing and 2 partials ⚠️
.../java/com/cloud/hypervisor/HypervisorGuruBase.java	0.00%	2 Missing ⚠️
...m/cloud/hypervisor/vmware/mo/VirtualMachineMO.java	0.00%	2 Missing ⚠️
...va/com/cloud/agent/api/ConvertInstanceCommand.java	66.66%	1 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               4.19    #8815      +/-   ##
============================================
+ Coverage     30.92%   30.97%   +0.05%     
- Complexity    34290    34372      +82     
============================================
  Files          5355     5355              
  Lines        376634   376724      +90     
  Branches      54808    54823      +15     
============================================
+ Hits         116480   116705     +225     
+ Misses       244820   244661     -159     
- Partials      15334    15358      +24

Flag	Coverage Δ
simulator-marvin-tests	`24.85% <0.00%> (+0.10%)`	⬆️
uitests	`4.39% <ø> (ø)`
unit-tests	`16.56% <26.35%> (-0.01%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

blueorangutan · 2024-03-20T18:09:01Z

Packaging result [SF]: ✔️ el7 ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 8985

sureshanaparti · 2024-03-20T18:14:30Z

@blueorangutan test matrix

blueorangutan · 2024-03-20T18:16:03Z

@sureshanaparti a [SL] Trillian-Jenkins matrix job (centos7 mgmt + xenserver71, rocky8 mgmt + vmware67u3, centos7 mgmt + kvmcentos7) has been kicked to run smoke tests

blueorangutan · 2024-03-21T06:15:00Z

[SF] Trillian test result (tid-9541)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 41640 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr8815-t9541-kvm-centos7.zip
Smoke tests completed. 129 look OK, 0 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File

blueorangutan · 2024-03-21T07:25:13Z

[SF] Trillian test result (tid-9539)
Environment: xenserver-71 (x2), Advanced Networking with Mgmt server 7
Total time taken: 45973 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr8815-t9539-xenserver-71.zip
Smoke tests completed. 128 look OK, 1 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_02_trigger_shutdown	`Failure`	336.63	test_safe_shutdown.py

blueorangutan · 2024-03-21T07:39:32Z

[SF] Trillian test result (tid-9540)
Environment: vmware-67u3 (x2), Advanced Networking with Mgmt server r8
Total time taken: 46712 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr8815-t9540-vmware-67u3.zip
Smoke tests completed. 129 look OK, 0 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File

nvazquez

Code LGTM

sureshanaparti · 2024-03-22T06:32:01Z

@blueorangutan package

blueorangutan · 2024-03-22T06:34:03Z

@sureshanaparti a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2024-03-22T07:31:06Z

Packaging result [SF]: ✔️ el7 ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 9003

sureshanaparti · 2024-03-22T07:45:18Z

@blueorangutan test matrix

blueorangutan · 2024-03-22T07:46:03Z

@sureshanaparti a [SL] Trillian-Jenkins matrix job (centos7 mgmt + xenserver71, rocky8 mgmt + vmware67u3, centos7 mgmt + kvmcentos7) has been kicked to run smoke tests

blueorangutan · 2024-03-22T19:34:53Z

[SF] Trillian test result (tid-9553)
Environment: xenserver-71 (x2), Advanced Networking with Mgmt server 7
Total time taken: 41092 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr8815-t9553-xenserver-71.zip
Smoke tests completed. 128 look OK, 1 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_02_trigger_shutdown	`Failure`	336.82	test_safe_shutdown.py

blueorangutan · 2024-03-22T21:20:44Z

[SF] Trillian test result (tid-9554)
Environment: vmware-67u3 (x2), Advanced Networking with Mgmt server r8
Total time taken: 47384 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr8815-t9554-vmware-67u3.zip
Smoke tests completed. 128 look OK, 1 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_02_balanced_drs_algorithm	`Failure`	128.96	test_cluster_drs.py

DaanHoogland

clgtm

…alled in KVM host)

…mfiles', to be generic

…windows guest vms

…om VMware, instead of OVA file - this would further increase the migration performance (as it reduces the time for OVA preparation / archiving of the VM files into a single file)

… to 3 hrs

…g OVF from MS, and other changes below. - Skip clone for powered off VMs - Fixes to support standalone host (with its default datacenter) - Some code improvements

…to import vm files

sureshanaparti · 2024-06-26T13:19:57Z

@blueorangutan package

blueorangutan · 2024-06-26T13:22:03Z

@sureshanaparti a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2024-06-26T14:29:17Z

Packaging result [SF]: ✔️ el7 ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 10148

blueorangutan · 2024-06-26T18:02:13Z

[SF] Trillian test result (tid-10627)
Environment: vmware-70u3 (x2), Advanced Networking with Mgmt server r8
Total time taken: 49273 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr8815-t10627-vmware-70u3.zip
Smoke tests completed. 130 look OK, 1 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
ContextSuite context=TestClusterDRS>:setup	`Error`	0.00	test_cluster_drs.py

sureshanaparti · 2024-06-27T11:58:26Z

@blueorangutan package

blueorangutan · 2024-06-27T12:00:04Z

@sureshanaparti a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2024-06-27T13:07:23Z

Packaging result [SF]: ✔️ el7 ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 10167

vladimirpetrov

LGTM based on manual testing.

* Create/Export OVA file of the VM on external vCenter host, to temporary conversion location (NFS) * Fixed ova issue on untar/extract ovf from ova file "tar -xf" cmd on ova fails with "ovf: Not found in archive" while extracting ovf file * Updated VMware to KVM instance migration using OVA * Refactoring and cleanup * test fixes * Consider zone wide pools in the destination cluster for instance conversion * Remove local storage pool support as temporary conversion location - OVA export not possible as the pool is not accessible outside host, NFS pools are supported. * cleanup unused code * some improvements, and refactoring * import nic unit tests * vmware guru unit tests * Separate clone VM and create template file for VMware migration - Export OVA (of the cloned VM) to the conversion location takes time. - Do any validations with cloned VM before creating the template (and fail early). - Updated unit tests. * Check conversion support on host before clone vm / create template on vmware (and fail early) * minor code improvements * Auto select the host with instance conversion capability * Skip instance conversion supported response param for non-KVM hosts * Show supported conversion hosts in the UI * Skip persistence map update if network doesn't exist * Added support to export OVA from KVM host, through ovftool (when installed in KVM host) * Updated importvm api param 'usemsforovaexport' to 'forcemstodownloadvmfiles', to be generic * Updated hardcoded UI messages with message labels * Updated UI to support importvm api param - forcemstodownloadvmfiles * Improved instance conversion support checks on ubuntu hosts, and for windows guest vms * Use OVF template (VM disks and spec files) for instance conversion from VMware, instead of OVA file - this would further increase the migration performance (as it reduces the time for OVA preparation / archiving of the VM files into a single file) * OVF export tool parallel threads code improvements * Updated 'convert.vmware.instance.to.kvm.timeout' config default value to 3 hrs * Config values check & code improvements * Updated import log, with time taken and vm details * Support for parallel downloads of VMware VM disk files while exporting OVF from MS, and other changes below. - Skip clone for powered off VMs - Fixes to support standalone host (with its default datacenter) - Some code improvements * rebase fixes * rebase fixes * minor improvement * code improvements - threads configuration, and api parameter changes to import vm files * typo fix in error msg

boring-cyborg bot added component:api component:kvm component:vmware labels Mar 20, 2024

sureshanaparti self-assigned this Mar 20, 2024

sureshanaparti added this to the 4.19.1.0 milestone Mar 20, 2024

sureshanaparti requested a review from nvazquez March 20, 2024 17:45

sureshanaparti changed the title ~~Improve migration of VMware VMs into KVM cluster~~ Improve migration of external VMware VMs into KVM cluster Mar 20, 2024

sureshanaparti mentioned this pull request Mar 21, 2024

Migrate VMware VM to KVM improvement documentation apache/cloudstack-documentation#388

Merged

nvazquez approved these changes Mar 22, 2024

View reviewed changes

boring-cyborg bot added component:orchestration component:storage component:UI labels Mar 22, 2024

DaanHoogland approved these changes Mar 25, 2024

View reviewed changes

DaanHoogland added the status:needs-testing label Mar 25, 2024

sureshanaparti added 16 commits June 26, 2024 18:48

Skip persistence map update if network doesn't exist

97e9e5d

Added support to export OVA from KVM host, through ovftool (when inst…

70e7af5

…alled in KVM host)

Updated importvm api param 'usemsforovaexport' to 'forcemstodownloadv…

2dd2cd3

…mfiles', to be generic

Updated hardcoded UI messages with message labels

afcf445

Updated UI to support importvm api param - forcemstodownloadvmfiles

38ebaba

Improved instance conversion support checks on ubuntu hosts, and for …

6ca1000

…windows guest vms

Use OVF template (VM disks and spec files) for instance conversion fr…

5779b04

…om VMware, instead of OVA file - this would further increase the migration performance (as it reduces the time for OVA preparation / archiving of the VM files into a single file)

OVF export tool parallel threads code improvements

3f653b0

Updated 'convert.vmware.instance.to.kvm.timeout' config default value…

438da6d

… to 3 hrs

Config values check & code improvements

c402ca5

Updated import log, with time taken and vm details

2f14225

Support for parallel downloads of VMware VM disk files while exportin…

257e229

…g OVF from MS, and other changes below. - Skip clone for powered off VMs - Fixes to support standalone host (with its default datacenter) - Some code improvements

rebase fixes

a1926a2

rebase fixes

15ea501

minor improvement

4bff115

code improvements - threads configuration, and api parameter changes …

26d017c

…to import vm files

sureshanaparti force-pushed the vmware-to-kvm-migration-improvements branch from 3cb750d to 26d017c Compare June 26, 2024 13:18

github-actions bot removed the status:has-conflicts label Jun 26, 2024

typo fix in error msg

df7816a

vladimirpetrov approved these changes Jun 27, 2024

View reviewed changes

sureshanaparti merged commit 46f6725 into apache:4.19 Jun 27, 2024

DaanHoogland deleted the vmware-to-kvm-migration-improvements branch July 1, 2024 06:54

Improve migration of external VMware VMs into KVM cluster #8815

Improve migration of external VMware VMs into KVM cluster #8815

Uh oh!

Conversation

sureshanaparti commented Mar 20, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Types of changes

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

How did you try to break this feature and the system with this change?

Uh oh!

sureshanaparti commented Mar 20, 2024

Uh oh!

blueorangutan commented Mar 20, 2024

Uh oh!

codecov bot commented Mar 20, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

blueorangutan commented Mar 20, 2024

Uh oh!

sureshanaparti commented Mar 20, 2024

Uh oh!

blueorangutan commented Mar 20, 2024

Uh oh!

blueorangutan commented Mar 21, 2024

Uh oh!

blueorangutan commented Mar 21, 2024

Uh oh!

blueorangutan commented Mar 21, 2024

Uh oh!

nvazquez left a comment

Choose a reason for hiding this comment

Uh oh!

sureshanaparti commented Mar 22, 2024

Uh oh!

blueorangutan commented Mar 22, 2024

Uh oh!

blueorangutan commented Mar 22, 2024

Uh oh!

sureshanaparti commented Mar 22, 2024

Uh oh!

blueorangutan commented Mar 22, 2024

Uh oh!

blueorangutan commented Mar 22, 2024

Uh oh!

blueorangutan commented Mar 22, 2024

Uh oh!

DaanHoogland left a comment

Choose a reason for hiding this comment

Uh oh!

sureshanaparti commented Jun 26, 2024

Uh oh!

blueorangutan commented Jun 26, 2024

Uh oh!

blueorangutan commented Jun 26, 2024

Uh oh!

blueorangutan commented Jun 26, 2024

Uh oh!

sureshanaparti commented Jun 27, 2024

Uh oh!

blueorangutan commented Jun 27, 2024

Uh oh!

blueorangutan commented Jun 27, 2024

Uh oh!

vladimirpetrov left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

sureshanaparti commented Mar 20, 2024 •

edited

Loading

codecov bot commented Mar 20, 2024 •

edited

Loading