KVM: if a volume is detached before OS has boot and migrated, admin not able to attach back

[borisstoyanov]

ISSUE TYPE

• Bug Report

COMPONENT NAME

Volumes, VM deployment

CLOUDSTACK VERSION

4.11

CONFIGURATION

OS / ENVIRONMENT

Observed with KVM, should be agnostic of distro and version. We tested on:

# virsh version
Compiled against library: libvirt 4.5.0
Using library: libvirt 4.5.0
Using API: QEMU 4.5.0
Running hypervisor: QEMU 1.5.3
# cat /etc/centos-release
CentOS Linux release 7.4.1708 (Core) 
# uname  -a
Linux <stripped> 3.10.0-693.11.1.el7.x86_64 #1 SMP Mon Dec 4 23:52:40 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

SUMMARY

When a user creates a VM without starting it, and then start and detach the volume before the guest OS has booted, post migration the volume cannot be attached back.

STEPS TO REPRODUCE

1. Create a VM with a data disk without starting it
2. Start the VM
3. Before the VM has booted, detach the disk
4. Migrate the data disk to another storage pool
5. Try to attach back the volume to the same VM

EXPECTED RESULTS

Volume being attached

ACTUAL RESULTS

Failed to attach volume test to VM VM-827b8faf-4ed6-414b-9c4f-29c7dcefd06c; org.libvirt.LibvirtException: XML error: target 'vdb' duplicated for disk sources '/mnt/610e0939-6665-311d-9401-bdcd354a275c/96912f59-95a5-458b-81c2-3e91c0687799' and '/mnt/610e0939-6665-311d-9401-bdcd354a275c/21d022d9-a2ba-4db0-acb4-e8f3044239c0'

It seems that the old record is not properly cleaned up when the volume is detached, and post migration we got a new path which creates a conflict and cloudstack is not able to attach the volume.

Workaround:
This could be escaped if the user waits enough for the Guest OS to boot. After that detaching seems to be cleaning properly the records.
Another way if the VM is already messed is to Stop and Start the VM, after that user is able to attach the volume.

[rohityadavcloud]

The limitation is known for years, this was the previous issue ticket discussing the limitation: https://issues.apache.org/jira/browse/CLOUDSTACK-2725

It still seems to happen even when KVM host has hotplugging capabilities:

# grep "CONFIG_HOTPLUG_PCI_ACPI=" /boot/config-`uname -r`
CONFIG_HOTPLUG_PCI_ACPI=y
# grep "CONFIG_HOTPLUG_PCI=" /boot/config-`uname -r`
CONFIG_HOTPLUG_PCI=y
# grep "CONFIG_MEMORY_HOTPLUG=" /boot/config-`uname -r`
CONFIG_MEMORY_HOTPLUG=y

[rohityadavcloud]

With qemu-ev 2.12, we see an error like "qemu-img: Could not open ‘/mnt/3aaac88e-61f6-3ab8-9fb0-79e9e025d097/5411f9c6-8212-4da6-b298-5b52935c0b5c': Failed to get shared "write" lockIs another process using the image?"

The issue is same as with stock qemu v1.5 on centos7, where disk is detached in CloudStack but still remains in the domain XML, for example:

    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2' cache='none'/>
      <source file='/mnt/05c30f6d-5725-3369-a8d6-b5fb37c7ba8f/eb391748-4269-471f-a014-cb0e5952cfb5'/>
      <backingStore type='file' index='1'>
        <format type='raw'/>
        <source file='/mnt/05c30f6d-5725-3369-a8d6-b5fb37c7ba8f/136b6808-666c-11e9-8569-f4d1082409e6'/>
        <backingStore/>
      </backingStore>
      <target dev='vda' bus='virtio'/>
      <serial>eb3917484269471fa014</serial>
      <alias name='virtio-disk0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </disk>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2' cache='none'/>
      <source file='/mnt/05c30f6d-5725-3369-a8d6-b5fb37c7ba8f/2079885e-683a-4379-b72b-80b98a1671bc'/>
      <backingStore/>
      <target dev='vdb' bus='virtio'/>
      <serial>2079885e683a4379b72b</serial>
      <alias name='virtio-disk1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </disk>

Workaround is to stop the VM, and start it which will fix the domain XML.

[DaanHoogland]

@borisstoyanov @rhtyd this is a bit questionable scenario:
...
2. Start the VM
3. Before the VM has booted, detach the disk
...
It seems to me not something that should be allowed on any system. how are we attacking this issue, if at all?

[rohityadavcloud]

This is a rare corner case, normally we would not need to detach a disk from a VM we just started.

[DaanHoogland]

I would say we shouldn't allow detach of a disk from a VM in state "Starting", @rhtyd

[weizhouapache]

@DaanHoogland I think so
If it is not disallowed, we should disallow it.

[DaanHoogland]

ok, should be a simple fix in the orchestration layer, I will try and find the time.

[rohityadavcloud]

@weizhouapache @DaanHoogland no, the problem is when the VM starts CloudStack immediate changes the state to Running. The problem is when CloudStack has updated state to Running, but the OS is yet to fully boot (stuck between say grub and starting OS).