v0.6.5-0.1 fails to install on Azure AKS 1.29.9

[MrPeacockNLB]

Description

After fixing #864 the installation of SysBox fails. I've a Node with SysBox v0.6.4 already installed. This works fine. But when I try to update to latest, it fails while it could not write to a read-only FS.

Detected Kubernetes version v1.29
No previous Sysbox version records found, proceed to upgrade to Sysbox v0.6.5-0 release.
Adding K8s taint "sysbox-runtime=not-running:NoSchedule" to node ...
node/aks-coderste16-39959049-vmss00001d modified
Adding K8s label "sysbox-runtime=installing" to node ...
node/aks-XXXXX-39959049-vmss00001d not labeled
Installing Sysbox dependencies on host ...
Copying shiftfs sources to host ...
Kernel version 5.15 is >= 5.15 and < 5.17
Deploying Sysbox installer helper on the host ...
Running Sysbox installer helper on the host (may take several seconds) ...
Stopping the Sysbox installer helper on the host ...
Removing Sysbox installer helper from the host ...
Stopping Sysbox ...
Installing Sysbox on host ...
sed: couldn't open temporary file /opt/sysbox/systemd/sedkZPHXr: Read-only file system

When downgrading back to v0.6.4 everything works as expected

[ctalledo]

Hi @MrPeacockNLB , thanks for reporting the issue.

@rodnymolina: I suspect the problem is caused by this commit:

commit 76c732f519ad8ae58dcf8f6fad6646bb4fa9d7ec
Author: Rodny Molina <[email protected]>
Date:   Fri Jul 5 11:00:56 2024 +0200

    k8s: Enhance logic to detect Sysbox's config-environment changes (sysctl)
    
    Signed-off-by: Rodny Molina <[email protected]>

Particularly this new code:

+function config_sysbox_env() {
+       # Set the sysbox edition in the sysbox-mgr and sysbox-fs systemd unit files.
+       sed -i "/^Environment=/ s|SYSBOX_EDITION=.*|SYSBOX_EDITION=${sysbox_edition}|" ${sysbox_artifacts}/systemd/sysbox-mgr.service
+       sed -i "/^Environment=/ s|SYSBOX_EDITION=.*|SYSBOX_EDITION=${sysbox_edition}|" ${sysbox_artifacts}/systemd/sysbox-fs.service
+}

Seems like /opt/sysbox/systemd is on a read-only file system on the Azure AKS node:

sed: couldn't open temporary file /opt/sysbox/systemd/sedkZPHXr: Read-only file system

Not sure why that is though; as a work-around we could we need to use a different host folder (i.e., not /opt/sysbox) to store the sysbox config files in AKS?

[rodnymolina]

Hmm, interesting, haven't ever seen /opt being placed in a RO partition in Ubuntu. Will try to repro when have a chance.

[MrPeacockNLB]

Let me know if I can test something...