Linstor root disk does not use size per offering or resize

[rohityadavcloud]

I've a 3-node kvm/Ubuntu 22.04 setup with ACS 4.18.0.0. Linstor that uses LVM_THIN on a 2TB nvme (on each of kvm1/2/3 nodes). I created a resource group cloudstack with placecount of 2, and add that as a zone-wide primary linstor storage in CloudStack with a specific storage tag linstor. Further I created disk and compute offerings with the same storage tag. I found the following issues:

1. I deployed a four test VMs with varying linstor specific offering, both using the compute offering and using the overiding disk offering for the root disk using the UI. In the created root disk for the VM, it uses the template's original size and doesn't resize the linstor volume based on the offering's disk size. The compute/disk offering specifies 100G as disk size.

# linstor v l
╭───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
┊ Node ┊ Resource                                ┊ StoragePool          ┊ VolNr ┊ MinorNr ┊ DeviceName    ┊ Allocated ┊ InUse  ┊      State ┊
╞═══════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════╡
┊ kvm1 ┊ cs-714a4f1f-93f4-4166-9b02-cf498c41aa07 ┊ nvme_pool            ┊     0 ┊    1003 ┊ /dev/drbd1003 ┊  2.20 GiB ┊ Unused ┊   UpToDate ┊
┊ kvm2 ┊ cs-714a4f1f-93f4-4166-9b02-cf498c41aa07 ┊ DfltDisklessStorPool ┊     0 ┊    1003 ┊ /dev/drbd1003 ┊           ┊ InUse  ┊   Diskless ┊
┊ kvm3 ┊ cs-714a4f1f-93f4-4166-9b02-cf498c41aa07 ┊ nvme_pool            ┊     0 ┊    1003 ┊ /dev/drbd1003 ┊  2.20 GiB ┊ Unused ┊   UpToDate ┊
┊ kvm1 ┊ cs-748e7f9d-7257-48dd-9efa-971396fee88f ┊ nvme_pool            ┊     0 ┊    1000 ┊ /dev/drbd1000 ┊  2.20 GiB ┊ Unused ┊   UpToDate ┊
┊ kvm2 ┊ cs-748e7f9d-7257-48dd-9efa-971396fee88f ┊ DfltDisklessStorPool ┊     0 ┊    1000 ┊ /dev/drbd1000 ┊           ┊ Unused ┊   Diskless ┊
┊ kvm3 ┊ cs-748e7f9d-7257-48dd-9efa-971396fee88f ┊ nvme_pool            ┊     0 ┊    1000 ┊ /dev/drbd1000 ┊  2.20 GiB ┊ Unused ┊   UpToDate ┊
┊ kvm1 ┊ cs-98ff59bc-3dd0-4efb-ba9a-381c6b5642c4 ┊ nvme_pool            ┊     0 ┊    1001 ┊ /dev/drbd1001 ┊  2.20 GiB ┊ Unused ┊   UpToDate ┊
┊ kvm2 ┊ cs-98ff59bc-3dd0-4efb-ba9a-381c6b5642c4 ┊ nvme_pool            ┊     0 ┊    1001 ┊ /dev/drbd1001 ┊  2.20 GiB ┊ Unused ┊   UpToDate ┊
┊ kvm3 ┊ cs-98ff59bc-3dd0-4efb-ba9a-381c6b5642c4 ┊ DfltDisklessStorPool ┊     0 ┊    1001 ┊ /dev/drbd1001 ┊           ┊ Unused ┊ TieBreaker ┊
┊ kvm1 ┊ cs-bdad90e3-2443-4d01-aef9-1f37db7b3a5a ┊ nvme_pool            ┊     0 ┊    1002 ┊ /dev/drbd1002 ┊  2.22 GiB ┊ InUse  ┊   UpToDate ┊
┊ kvm2 ┊ cs-bdad90e3-2443-4d01-aef9-1f37db7b3a5a ┊ DfltDisklessStorPool ┊     0 ┊    1002 ┊ /dev/drbd1002 ┊           ┊ Unused ┊   Diskless ┊
┊ kvm3 ┊ cs-bdad90e3-2443-4d01-aef9-1f37db7b3a5a ┊ nvme_pool            ┊     0 ┊    1002 ┊ /dev/drbd1002 ┊  2.22 GiB ┊ Unused ┊   UpToDate ┊
╰───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯

2. I also tried resizing the volume via CloudStack's volume -> resize, but that doesn't work either.

3. I tried to create data disk with the same offering, while the offering's size was 100G in the linstor v l I saw a new entries but with only 20.48MiB size:

┊ kvm1 ┊ cs-d4eea962-011c-411b-85d6-9a0bd26a1f44 ┊ DfltDisklessStorPool ┊     0 ┊    1006 ┊ /dev/drbd1006 ┊           ┊ InUse  ┊   Diskless ┊
┊ kvm2 ┊ cs-d4eea962-011c-411b-85d6-9a0bd26a1f44 ┊ nvme_pool            ┊     0 ┊    1006 ┊ /dev/drbd1006 ┊ 20.48 MiB ┊ Unused ┊   UpToDate ┊
┊ kvm3 ┊ cs-d4eea962-011c-411b-85d6-9a0bd26a1f44 ┊ nvme_pool            ┊     0 ┊    1006 ┊ /dev/drbd1006 ┊ 20.48 MiB ┊ Unused ┊   UpToDate ┊

4. For each VM deployment using the same VM template, in linstor I see multiple volume created. I'm not entirely sure if it's a bug, could you advise if it's expected that the current linstor plugin would create a volume per KVM host?

cc @rp- @Philipp-Reisner - pl advise if you need any further information, it's possible I've followed something wrong (and obvious?), users wouldn't be able to try/use Linstor with CloudStack. Is it supported with CloudStack with lvm (thin)? Or, should we be using zfs (thick/thin) or lvm (thick) with this? Thanks. (edit - changing the volume-definition manually and doing a stop/start fixed the size issue for me)

Misc:

# linstor --version
linstor-client 1.19.0; GIT-hash: 500970a65717de3eeba002ee32ceb668ebd654b1

# cat /proc/drbd
version: 9.2.5 (api:2/proto:86-122)
GIT-hash: b44520271e63d4b6f359a6642eb4d475b7cc04e0 build by root@kvm3, 2023-09-01 18:27:18
Transports (api:18): tcp (9.2.5)

ISSUE TYPE

• Bug Report

COMPONENT NAME

Linstor storage plugin

CLOUDSTACK VERSION

4.18.0.0

CONFIGURATION

Adv zone KVM with Ubuntu 22.04 using hwe kernel 6.2.0-31-generic

[sbrueseke]

Maybe I got you wrong, but as far as I understand you are using thin provisioning on linstor and your question is why you do not see 100GB allocated space, is that correct?
Linstor v l is showing you what really is allocated on the SrotagePool and not what the max of that volume is. If you would use lvm without thin it would show 100GB.

[rp-]

1. Yes, I found this bug last week and fixed it internal already.
   I'm currently working on more cloudstack improvements and will open a PR "soon"
2. resize should work, I'll investigate that on Monday
3. This might be normal with thin devices, as linstor v l only shows the allocated size. You might want to look at linstor vd l
4. It creates volumes depending on your replica count and maybe a diskless if cloudstack did start the VM on a non diskful host. Or didn't I get your question?

We currently support LVM-thin and ZFS (thick and thin)

[rohityadavcloud]

Thanks @sbrueseke @rp- , @rp- is right in case of LVM-thin, the allocated volume size seen in linstor vd l showed like 2.2GB (template size) instead of the root disk allocated size (100GB) in my case. Like I've mentioned briefly, I've workaround specifically by using set-size in linstor vd I could change the volume size to 100GB and do a VM stop/start. Resize didn't work for me though, I can try to test again.

@rp- on the last point, I wanted to learn how the linstor plugin caches the template. In my experimentation, I found the issue to be a case where a Linstor (resource) volume fails to be deleted (stuck in Destory state but not expunged in CloudStack's DB), CloudStack can mark it as removed in its database but the volume may not be deleted on the linstor side.

Update - I was able to do a resize on a running VM's root disk and it worked perfectly okay, I still had to do a stop/start on the VM for it to detect the change from the underlying raw block device (a simple VM reboot didn't work) though.

[hmadra]

This issue is valid.
When looking at the CS Gui, the Volume is assigned the new Size.
However, in linstor satellite note , when looking at lvs , the LSize is same as the template.

When logging into the VM, disk size is the same as the template, which in reality should be the overwritten root disk size.
For example ,
Lets say the template is 8GB .
When deploying the VM , we overwrite the Root disk size to 40 GB.
CS gui confirms that it is assigning 40G to the Root disk.

in Linstor, LV size is still 8GB (Thats ok)

However, when the VM is deployed, and we check the disk size using fdisk , the disk is 8GB and not 40GB.

I can safely stop the VM, resize the disk and start the VM with new size. However, this should not be needed, as it works fine with other storage plugins.

CS Version : 4.18

[rp-]

@hmadra

Yes, I found this bug last week and fixed it internal already.
I'm currently working on more cloudstack improvements and will open a PR "soon"

[DaanHoogland]

fixed in #7952