SystemVM(ConsoleProxy / Secstorage )not starting Services after upgrade from 4.16.1

[senft144]

ISSUE TYPE

• Bug Report

COMPONENT NAME

Assets systemvm

CLOUDSTACK VERSION

4.17.0 Ubuntu Apt Packages

CONFIGURATION

OS / ENVIRONMENT

Ubuntu 20.04 KVM

SUMMARY

New Service VMs are deployed and accessable vie SSH over host, but service (ConsoleProxy / Secstorage ) are not starting.
/usr/local/cloud/ is completly empty which is why the systemd cloud.service is not starting.

STEPS TO REPRODUCE

• be on Verison 4.16.1
• Update to 4.17.0
• Download new Template for 4.17.0 KVM
• Delete all old ServiceVMs
• Wait till manager auto-deploys ServiceVMs
• ServiceVMS stuck in "Starting"
• VMs will migtrate through the process is not ended successfully

EXPECTED RESULTS

SystemVMs should be deployed on any available host and services should be started.

ACTUAL RESULTS

System-VMs are stuck in state "starting" but are accessible via SSH Port 3922.
The Cloudstack-manager-process is trying all available host after an timeout, but no host is capable of starting the VMs.
No Error is found in any logs but no such file ...
cat /var/log/daemon.log | grep cloud
Jun 24 16:52:17 v-35-VM cloud-early-config[268]: Executing cloud-early-config.
Jun 24 16:52:17 v-35-VM cloud-early-config[313]: NOCHANGE: partition 2 could only be grown by 2015 [fudge=2048]
Jun 24 16:52:17 v-35-VM cloud-early-config[349]: NOCHANGE: partition 6 could only be grown by 2015 [fudge=2048]
Jun 24 16:52:17 v-35-VM cloud-early-config[371]: resize2fs 1.46.2 (28-Feb-2021)
Jun 24 16:52:17 v-35-VM cloud-early-config[371]: The filesystem is already 863488 (4k) blocks long. Nothing to do!
Jun 24 16:52:17 v-35-VM cloud-early-config[373]: mount: /media/cdrom0: no medium found on /dev/sr0.
Jun 24 16:52:17 v-35-VM cloud-early-config[268]: Scripts checksum detected: oldmd5=49ab1c8540447a3a584da18dcc89d6dd newmd5=.
Jun 24 16:52:17 v-35-VM cloud-early-config[384]: Bootstrapping systemvm appliance.
Jun 24 16:52:17 v-35-VM cloud-early-config[384]: Starting guest services for kvm.
Jun 24 16:52:17 v-35-VM cloud-early-config[468]: CONFIG_HOTPLUG_PCI=y
Jun 24 16:52:17 v-35-VM cloud-early-config[468]: CONFIG_HOTPLUG_PCI_ACPI=y
Jun 24 16:52:17 v-35-VM cloud-early-config[384]: acpiphp and pci_hotplug module already compiled in.
Jun 24 16:52:17 v-35-VM cloud-early-config[472]: Synchronizing state of qemu-guest-agent.service with SysV service script with /lib/systemd/systemd-sysv-install.
Jun 24 16:52:17 v-35-VM cloud-early-config[472]: Executing: /lib/systemd/systemd-sysv-install enable qemu-guest-agent
Jun 24 16:52:18 v-35-VM cloud-early-config[472]: The unit files have no installation config (WantedBy=, RequiredBy=, Also=,
Jun 24 16:52:18 v-35-VM cloud-early-config[472]: Alias= settings in the [Install] section, and DefaultInstance= for template
Jun 24 16:52:18 v-35-VM cloud-early-config[472]: units). This means they are not meant to be enabled using systemctl.
Jun 24 16:52:18 v-35-VM cloud-early-config[472]:
Jun 24 16:52:18 v-35-VM cloud-early-config[472]: Possible reasons for having this kind of units are:
Jun 24 16:52:18 v-35-VM cloud-early-config[472]: • A unit may be statically enabled by being symlinked from another unit's
Jun 24 16:52:18 v-35-VM cloud-early-config[472]: .wants/ or .requires/ directory.
Jun 24 16:52:18 v-35-VM cloud-early-config[472]: • A unit's purpose may be to act as a helper for some other unit which has
Jun 24 16:52:18 v-35-VM cloud-early-config[472]: a requirement dependency on it.
Jun 24 16:52:18 v-35-VM cloud-early-config[472]: • A unit may be started when needed via activation (socket, path, timer,
Jun 24 16:52:18 v-35-VM cloud-early-config[472]: D-Bus, udev, scripted systemctl call, ...).
Jun 24 16:52:18 v-35-VM cloud-early-config[472]: • In case of template units, the unit is meant to be enabled with some
Jun 24 16:52:18 v-35-VM cloud-early-config[472]: instance name specified.
Jun 24 16:52:18 v-35-VM cloud-early-config[384]: Received a new non-empty cmdline file from qemu-guest-agent.
Jun 24 16:52:18 v-35-VM cloud-early-config[384]: Booting from cloudstack, remove old configuration files in /etc/cloudstack/.
Jun 24 16:52:18 v-35-VM cloud-early-config[551]: umount: /media/cdrom: not mounted.
Jun 24 16:52:19 v-35-VM cloud-early-config[557]: net.ipv4.ip_forward = 1
Jun 24 16:52:19 v-35-VM cloud-early-config[557]: net.ipv4.conf.default.rp_filter = 0
Jun 24 16:52:19 v-35-VM cloud-early-config[557]: net.ipv4.conf.default.accept_source_route = 0
Jun 24 16:52:19 v-35-VM cloud-early-config[557]: net.ipv4.conf.default.arp_announce = 2
Jun 24 16:52:19 v-35-VM cloud-early-config[557]: net.ipv4.conf.default.arp_ignore = 2
Jun 24 16:52:19 v-35-VM cloud-early-config[557]: net.ipv4.conf.all.arp_announce = 2
Jun 24 16:52:19 v-35-VM cloud-early-config[557]: net.ipv4.conf.all.arp_ignore = 2
Jun 24 16:52:19 v-35-VM cloud-early-config[557]: net.ipv4.conf.all.accept_redirects = 0
Jun 24 16:52:19 v-35-VM cloud-early-config[557]: net.ipv4.conf.default.accept_redirects = 0
Jun 24 16:52:19 v-35-VM cloud-early-config[557]: net.ipv4.conf.all.send_redirects = 0
Jun 24 16:52:19 v-35-VM cloud-early-config[557]: net.ipv4.conf.default.send_redirects = 0
Jun 24 16:52:19 v-35-VM cloud-early-config[557]: net.ipv4.conf.all.secure_redirects = 0
Jun 24 16:52:19 v-35-VM cloud-early-config[557]: net.ipv4.conf.default.secure_redirects = 0
Jun 24 16:52:19 v-35-VM cloud-early-config[557]: net.ipv4.conf.all.promote_secondaries = 1
Jun 24 16:52:19 v-35-VM cloud-early-config[557]: net.ipv4.conf.default.promote_secondaries = 1
Jun 24 16:52:19 v-35-VM cloud-early-config[557]: net.ipv4.ip_nonlocal_bind = 1
Jun 24 16:52:19 v-35-VM cloud-early-config[557]: kernel.sysrq = 0
Jun 24 16:52:19 v-35-VM cloud-early-config[557]: kernel.core_uses_pid = 1
Jun 24 16:52:19 v-35-VM cloud-early-config[557]: kernel.panic = 10
Jun 24 16:52:19 v-35-VM cloud-early-config[557]: kernel.panic_on_oops = 1
Jun 24 16:52:19 v-35-VM cloud-early-config[557]: vm.panic_on_oom = 1
Jun 24 16:52:19 v-35-VM cloud-early-config[557]: net.ipv4.tcp_syncookies = 1
Jun 24 16:52:19 v-35-VM cloud-early-config[557]: net.ipv4.tcp_timestamps = 0
Jun 24 16:52:19 v-35-VM cloud-early-config[557]: net.ipv4.tcp_tw_reuse = 1
Jun 24 16:52:19 v-35-VM cloud-early-config[557]: net.ipv4.tcp_max_tw_buckets = 1000000
Jun 24 16:52:19 v-35-VM cloud-early-config[557]: net.core.somaxconn = 65535
Jun 24 16:52:19 v-35-VM cloud-early-config[557]: net.nf_conntrack_max = 1000000
Jun 24 16:52:19 v-35-VM cloud-early-config[557]: net.netfilter.nf_conntrack_max = 1000000
Jun 24 16:52:19 v-35-VM cloud-early-config[557]: net.ipv6.conf.all.disable_ipv6 = 1
Jun 24 16:52:19 v-35-VM cloud-early-config[557]: net.ipv6.conf.all.forwarding = 0
Jun 24 16:52:19 v-35-VM cloud-early-config[557]: net.ipv6.conf.all.accept_ra = 0
Jun 24 16:52:19 v-35-VM cloud-early-config[557]: net.ipv6.conf.all.accept_redirects = 0
Jun 24 16:52:19 v-35-VM cloud-early-config[557]: net.ipv6.conf.all.autoconf = 0
Jun 24 16:52:19 v-35-VM cloud-early-config[557]: vm.swappiness = 1
Jun 24 16:52:19 v-35-VM cloud-early-config[557]: vm.vfs_cache_pressure = 100
Jun 24 16:52:19 v-35-VM cloud-early-config[557]: vm.min_free_kbytes = 20480
Jun 24 16:52:19 v-35-VM cloud-early-config[384]: Configuring systemvm type=consoleproxy.
Jun 24 16:52:19 v-35-VM cloud-early-config[705]: base64: invalid input
Jun 24 16:52:19 v-35-VM cloud-early-config[559]: Setting up console proxy system vm.
Jun 24 16:52:19 v-35-VM cloud-early-config[716]: ifdown: interface eth0 not configured
Jun 24 16:52:19 v-35-VM cloud-early-config[742]: ifdown: interface eth1 not configured
Jun 24 16:52:19 v-35-VM cloud-early-config[768]: ifdown: interface eth2 not configured
Jun 24 16:52:19 v-35-VM cloud-early-config[799]: RTNETLINK answers: No such process
Jun 24 16:52:19 v-35-VM cloud-early-config[809]: 10.30.1.10
Jun 24 16:52:19 v-35-VM cloud-early-config[559]: Not setting up route of RFC1918 space to 10.3.0.1 because 10.30.1.10 is RFC1918..
Jun 24 16:52:19 v-35-VM cloud-early-config[559]: Setting up entry in hosts.
Jun 24 16:52:19 v-35-VM cloud-early-config[559]: Applying iptables rules.
Jun 24 16:52:19 v-35-VM cloud-early-config[559]: Configuring sshd.
Jun 24 16:52:19 v-35-VM cloud-early-config[812]: PING 10.93.64.1 (10.93.64.1): 56 data bytes
Jun 24 16:52:19 v-35-VM cloud-early-config[810]: PING 10.3.0.1 (10.3.0.1): 56 data bytes
Jun 24 16:52:19 v-35-VM cloud-early-config[811]: PING 10.30.1.1 (10.30.1.1): 56 data bytes
Jun 24 16:52:19 v-35-VM cloud-early-config[810]: 64 bytes from 10.3.0.1: icmp_seq=0 ttl=64 time=1.202 ms
Jun 24 16:52:19 v-35-VM cloud-early-config[812]: 64 bytes from 10.93.64.1: icmp_seq=0 ttl=63 time=1.336 ms
Jun 24 16:52:19 v-35-VM cloud-early-config[559]: cloud: disable rp_filter.
Jun 24 16:52:19 v-35-VM cloud-early-config[559]: disable rpfilter.
Jun 24 16:52:19 v-35-VM cloud-early-config[559]: cloud: enable_fwding = 0.
Jun 24 16:52:19 v-35-VM cloud-early-config[559]: enable_fwding = 0.
Jun 24 16:52:19 v-35-VM cloud-early-config[559]: Processors = 1 Enable service = 0.
Jun 24 16:52:19 v-35-VM cloud-early-config[384]: Finished setting up systemvm.
Jun 24 16:52:19 v-35-VM cloud-early-config[268]: Finished setting up systemvm.
Jun 24 16:52:22 v-35-VM systemd[1201]: cloud.service: Failed to locate executable /usr/local/cloud/systemvm/_run.sh: No such file or directory
Jun 24 16:52:22 v-35-VM systemd[1201]: cloud.service: Failed at step EXEC spawning /usr/local/cloud/systemvm/_run.sh: No such file or directory

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template!

[Pearl1594]

@senft144 has the cloudstack-agent package for ACS 4.17 been installed on the KVM hosts, based on the logs, it seems like it isn't updated.

[senft144]

@senft144 has the cloudstack-agent package for ACS 4.17 been installed on the KVM hosts, based on the logs, it seems like it isn't updated.

All Host were Updated to 4.17 and rebooted.

 dpkg -l | grep cloudstack
ii  cloudstack-agent                      4.17.0.0~focal                        all          CloudStack agent
ii  cloudstack-common                     4.17.0.0~focal                        all          A common package which contains files which are shared by several CloudStack packages

But i noticed one odd thing. The systevmtemplate seem to be still for 4.16.1

root@v-35-VM:~# cat /etc/cloudstack-release 
Cloudstack Release 4.16.1 Mon 31 Jan 2022 10:02:56 AM UTC

[Pearl1594]

Is the 4.17 systemvm template registered i.e., do you see it in the templates view in the UI? And what are the values of router.template.kvm & minreq.sysvmtemplate.version global setting post upgrade

[senft144]

i've been so paranoid, ive deleted the whole secondary-storage and uploaded the new template via the /usr/share/cloudstack-common/scripts/storage/secondary/cloud-install-sys-tmplt -m /export/secondary -u http://download.cloudstack.org/systemvm/4.17/systemvmtemplate-4.17.0-kvm.qcow2.bz2 -h kvm -F

These are the Values in Global Config:

Name	Description	Category	Value
router.template.kvm	Name of the default router template on KVM.	Advanced	SystemVM Template (KVM)
minreq.sysvmtemplate.version	What version should the Virtual Routers report	Advanced	4.10.0

But the strange thing is, the routervms are behaving as expected.
Only the ConsoleProxy and the SecondaryStorageVM are not starting.

[Pearl1594]

Just to confirm, the router VMs on restart with cleanup come up with the 4.17 template?
Do you see any exceptions /errors in the Management server logs?

[senft144]

Just to confirm, the router VMs on restart with cleanup come up with the 4.17 template? Do you see any exceptions /errors in the Management server logs?

Steps that i made to confirm this

• i've delete the routevm for an GuestNetwork
• tell the guestNW to restart via GUI
• routerVM deployed successfully
• testVM in Network gets IP via DHCP

its an shared network with only an DHCP-Service to provide IPs to the GuestVMs.
So could not really provide an errog from logs.

But an odd thing i noticed on the routeVM-Site in GUI->

[Pearl1594]

In an upgraded env, the systemvm template name should ideally be systemvm-kvm-4.17.0 for router.template.kvm and the minreq.sysvmtemplate.version should be 4.17.0. Can you please try restart the cloudstack-agent service on the KVM hosts and retry SystemVM creation..
The issue seems to be that it's picking the old systemVM template when re-deploying the systemVMs. Can you also return the result of the following DB query:
select * from vm_template where name = 'SystemVM Template (KVM)' and removed is null order by id desc limit 1\G and verify if this points to the 4.17 systemVM template

[senft144]

I'm sorry, in an act of stupidity and devestation, i've decided to newly deploy the hole thing with version 4.17.
But my guess would also be, that the process of upgrading the database didn't work out.
As an conclusion of this case, beeing able to manually delete the systemVM-templates would be great and maybe an manuell Database-Upgrade ?!

[Pearl1594]

You could just re-register the systemVM template - as a user template and then update/edit the template to system type , and also update the name such that it matches the global setting: router.template.kvm. Once this is done, you can re-create the systemVMs. Hopefully this works

[senft144]

Cant reproduce, Whole Cluster was deployed from scratch