cloud-init not running after boot

[MartinEmrich]

ISSUE TYPE

• Documentation Report

COMPONENT NAME

Documentation, cloud-init

CLOUDSTACK VERSION

4.18.1.0

CONFIGURATION

Hypervisor: XCP-ng 7.5

OS / ENVIRONMENT

VMs: Debian 12

SUMMARY

Cloud-init not starting, as datasource CloudStack is not detected.

STEPS TO REPRODUCE

• Create VM from Debian 12.2 netinst ISO
• install cloud-init
• configure cloud-init just as for Ubuntu (http://docs.cloudstack.apache.org/en/4.18.1.0/adminguide/templates/_cloud_init.html)
• prepare cloud-init (`cloud-init clean -l --seed --reboot)

EXPECTED RESULTS

(Assumption: Ubuntu configuration should closely-enough match Debian)
cloud-init should run and configure the OS.

ACTUAL RESULTS

cloud-init does not run at all.

Additional info:

After some research: (at least on Debian) cloud-init includes a "systemd generator" using ds-identify. That script tries to detect the correct datasource, but fails. If it does not detect any datasource, it will disable cloud-init completely, resulting in no cloud-init logs whatsoever.

Looking at the source code, it tries to determine whether it runs on CloudStack by using the DMI product_name field: https://github.com/canonical/cloud-init/blob/22.4.x/tools/ds-identify#L739-L743

But on XenServer (I bet on VMWare, Hyper-V, etc, too), this just returns a generic HVM domU. (I suspect the documentation was written only with KVM in mind, and the Cloudstack Agent for KVM somehow configures KVM to producte that DMI product_name)

While the latest version (http://docs.cloudstack.apache.org/en/latest/adminguide/templates/_cloud_init.html) does not, the 4.18.1.0 version of the documentation gives a hint for CentOS (although with a typo): http://docs.cloudstack.apache.org/en/4.18.1.0/adminguide/templates/_cloud_init.html

Editing the /etc/cloud/ds-identify.cfg file to override the datasource works on Debian 12, too. After adding that file with

datasource: CloudStack

cloud-init runs as expected.

Additional note: In https://bugs.launchpad.net/cloud-init/+bug/1892171/comments/4 , at least one cloud-init developer/contributor marks ds-identify.cfg as undocumented, meaning not a public stable interface.
It looks like the kernel command line is the official way to override the cloud-init datasource. (Idk though whether this comes from Cloudstack, especially in the HVM case, or if it has to be configured in the VM bootloader by the image author)

[MartinEmrich]

Out of curiosity: I looked at how Ubuntu (LTS 22.04 Server) handles cloud-init: After installing a fresh VM from the official CD-ROM:

• cloud-init comes pre-installed (on Debian, it has to be installed manually)
• it has all possible datasources enabled, including CloudStack
• it forces cloud-init to always run by setting the (again unofficial) policy: enabled in /etc/cloud/ds-identify.cfg

Thus Ubuntu will skip any autodetection and try the extensive list of Providers, including CloudStack.

[weizhouapache]

@MartinEmrich
I have created a PR to fix it in cloud-init canonical/cloud-init#4281

it has been merged into cloud-init 23.3.1. but unfortunately it caused some regression on other providers, it has been reverted in PR canonical/cloud-init#4501

[MartinEmrich]

@weizhouapache thanks, I indeed noticed the revert issue during my research.

Debian 12 comes with an older cloud-init 22.4, so I have the original version of ds-identify in either case.

For the documentation, I guess mentioning Debian alongside CentOS (and fixing the small typo in the path) at the ds-identify.cfg section is all that can be done now.

And unless there's another way for a VM to tell a XenServer/XCP-ng host managed by CloudStack apart from a standalone one, it would be hard to add a fire&forget logic to cloud-init/ds-identify either.

For now I get along well with that workaround with the configuration file.

[weizhouapache]

I've added a note to the document
https://cloudstack-documentation--383.org.readthedocs.build/en/383/adminguide/templates/_cloud_init.html#linux-with-cloud-init

[DaanHoogland]

@weizhouapache , is this an ACS issue?