Xenserver/XCP-NG Inter-cluster VM migration with ROOT on managed storage

[tsinik-dw]

ISSUE TYPE

• Bug Report
• Improvement Request
• Enhancement Request

COMPONENT NAME

API, UI

CLOUDSTACK VERSION

4.16.0

CONFIGURATION

ACS 4.16.0
1 Zone
Cluster A: Two Xenserver 7.0 hosts
Cluster B: Two XCP-NG 8.2 hosts

Each Cluster has each own NFS primary storage
There is a Zone-wide Solidfire Storage (managed storage)

OS / ENVIRONMENT

VM-A1, on Cluster A, has a ROOT disk on managed storage

SUMMARY

The objective is to migrate VM-A1 from Cluster A to Cluster B

VM-A1 migration fails and throws the following message:

Migration failed
Migration of the VM: i-2-11-VM (ID: c87de296-e0e4-4052-81a2-53798cfadd6c) from host xen1 (ID: 7e374139-b522-44da-9ea5-2a03b369f787) to destination host xcp1 (ID: 99d6b992-d6b9-49f2-9cec-3e0394251664) doesn't involve migrating the volumes

STEPS TO REPRODUCE

1. Create VM on Xenserver 7.0 cluster with ROOT disk on managed storage
2. Migrate VM from Xenserver 7.0 Cluster to XCP-NG Cluster

EXPECTED RESULTS

VM should be migrated from xenserver cluster to xcp-ng cluster

ACTUAL RESULTS

VM-A1 migration fails and throws the following message:

Migration failed
Migration of the VM: i-2-11-VM (ID: c87de296-e0e4-4052-81a2-53798cfadd6c) from host xen1 (ID: 7e374139-b522-44da-9ea5-2a03b369f787) to destination host xcp1 (ID: 99d6b992-d6b9-49f2-9cec-3e0394251664) doesn't involve migrating the volumes

[nvazquez]

Hi @shwstppr, you have worked on inter-cluster migration, is it only supported on Vmware?

[shwstppr]

@nvazquez I've not tested it myself but I don't remember code conditions that prevent it on other hypervisors. Also, recently in issue #5894 it was reported to be working with NFS stores on 4.16 so this could be an issue only with managed storages

[nvazquez]

Hi @tsinik-dw are you attempting a live VM migration? In that case, do you get the hosts from the different cluster listed on the UI or are using the API? If not using the UI, can you check if the target host is listed on the output of the findSuitableHostsForMigration API?

[nvazquez]

Can you also please upload the management server logs for the migration attempt? @tsinik-dw

[nvazquez]

I suspect the issue is that the underlying API invoked is migrateVirtualMachineWithVolume instead of migrateVirtualMachine - can you please try using a client such as cloudmonkey the following API attempt?

migrate virtualmachine virtualmachineid=<VM_UUID> hostid=<HOST_UUID>

[tsinik-dw]

Hi, @nvazquez, I am setting up a test environment and reporting the results in a while along with the log.

[tsinik-dw]

Hi @nvazquez,

I tried a live VM migration from Xen to XCP. The destination host (XCP, Host 9 in log) is presented as expected in the UI along with the AutoSelect option. I checked the XCP host as destination and tried the migration.
In the log I can see that findHostsForMigration command is able to identify the XCP host as a possible destination but the migration fails. Management log attached live_intercluster_vm_migr_Xen70_to_Xcp82.txt

Your guess is right!... Indeed, the API invoked is migrateVirtualMachineWithVolume instead of migrateVirtualMachine.
But, when the same migration tried using cmk the VM migration fails with the following output:

(localcloud) 🐱 > migrate virtualmachine virtualmachineid=c87de296-e0e4-4052-81a2-53798cfadd6c hostid=73cd3fa4-01ce-4b55-970f-14efd0143aa1
cmd = org.apache.cloudstack.api.command.admin.vm.MigrateVMCmd
jobstatus = 2
jobprocstatus = 0
jobresultcode = 530
accountid = 48ae3b86-8032-11ec-8660-4626a7ce6ce6
userid = 48bd974a-8032-11ec-8660-4626a7ce6ce6
created = 2022-02-16T10:07:28+0200
completed = 2022-02-16T10:07:30+0200
jobid = be5805f5-2330-4033-a314-bde52e666da2
jobresulttype = object
jobresult = map[errorcode:534 errortext:Resource [Host:9] is unreachable: Host 9: Unable to prepare for migration due to Exception: com.cloud.utils.exception.CloudRuntimeException
Message: There are 0 VMs named i-2-11-VM
Stack: com.cloud.utils.exception.CloudRuntimeException: There are 0 VMs named i-2-11-VM
        at com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.prepareISO(CitrixResourceBase.java:4157)
        at com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixPrepareForMigrationCommandWrapper.execute(CitrixPrepareForMigrationCommandWrapper.java:59)
        at com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixPrepareForMigrationCommandWrapper.execute(CitrixPrepareForMigrationCommandWrapper.java:36)
        at com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixRequestWrapper.execute(CitrixRequestWrapper.java:122)
        at com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:1757)
        at com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:315)
        at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
        at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:829)

The corresponding management log during this cmk operation is attached too cmk_vm_migr_mgmt_log.txt

PS.1 I also tried to specify on cmk both hostid and storageid but got the (expected): "Only one of hostId and storageId can be specified". Log available if desired.
P.2 If I stop the VM on Xen cluster, then I can start it on the XCP cluster as normal. This is easy for my test environment, but (almost) impossible in production :-)

[weizhouapache]

@tsinik-dw
Hi, is global setting migrate.vm.across.clusters set to true ?

[tsinik-dw]

Hi @weizhouapache,
Yes, the migrate.vm.across.clusters is set to true.

Just checked (for fun) the migration with this global setting to false and got the exact same error message (:trollface:), if that is of any help.