Regarding BPF module in Ubuntu Base image

[pl853]

Describe the bug

Hi guys, really sorry that i have to post this here, but i couldnt find an ubuntu repo to ask this question.

We are enrolling defend 4 containers from Elastic Security on K8sS running on VKS 3.5. At first we had photonOS, but due to a missing LSM kernel config, which made enrolling Elastic D4C impossible, we decided to switch to Ubuntu as base image, because in our case monitoring > hardening. Now we are running into the problem that the BPF module is not loaded in the grub startup config. We try to add it in the grub config, but this is overwritten when rebooting the node.

Is there a way to bypass this, since we really need the security monitoring given the huge amount of supply chain attacks these days. Also, if there is a way to add BPF, do you guys know if this would warrent support?

Thanks in advance,

Kind regards

Reproduction steps

1. install D4C
2. start node
3. be sad because half of D4C telemtry is missing and you still can get hacked without you knowing
   ...

Expected behavior

1. install D4C
2. start node
3. be happy because D4C telemetry is complete and you know when you get hacked.
   ...

Additional context

No response

[dcasota]

Hi @pl853,

Thanks for reporting this. Unfortunately, explaining several current constellations of your case requires a more detailed response.

Edited on May 9th 2026: I've revisited my answer from yesterday and modified some parts.

A systemctl reboot does not revert /etc/default/grub, /boot/grub/grub.cfg, or anything else under /etc.
What's likely happening:

• You edited /boot/grub/grub.cfg directly. Apt's update-grub (triggered by an unattended-upgrades or kernel-update run that the reboot kicks off) regenerates that file from /etc/default/grub, dropping your edit.
• Or you edited /etc/default/grub but didn't run update-grub, so the cmdline at boot never actually changed.

In these cases try this:

sudo tee /etc/default/grub.d/99-bpf-lsm.cfg > /dev/null <<'EOF'
GRUB_CMDLINE_LINUX_DEFAULT="${GRUB_CMDLINE_LINUX_DEFAULT}
lsm=lockdown,capability,landlock,yama,apparmor,bpf"
EOF
sudo update-grub
sudo systemctl reboot

That survives reboots of the same VM.

Verification after reboot:

  cat /sys/kernel/security/lsm
  # expect: lockdown,capability,landlock,yama,apparmor,bpf

  grep -o 'lsm=[^ ]*' /proc/cmdline
  # expect: lsm=lockdown,capability,landlock,yama,apparmor,bpf

• Or what you saw wasn't really a reboot but a VKS node-replacement event (rolling upgrade, MachineDeployment rollout, scale-out). Those do destroy the rootfs. The "revert after reboot" you're seeing isn't a Photon OS or Ubuntu bug - it's the VKS / Cluster API immutable-node model. VKS nodes are provisioned from a VKr image; runtime changes to /etc/default/grub, /boot/…, or any file under /etc are lost the moment a node is replaced (upgrade, MachineDeployment rolling update, scale-out, sometimes a reconcile loop). The only stable place to put lsm=…,bpf is inside the VKr image itself, at bake time.

Legal

The Broadcom support contract for VKS covers VKS as shipped, including the published VKR node images. A node booted from a self-baked VKR is outside the supported VKR lineage; that node - and clusters built from it - are not entitled to support escalation through the VKS support path. Image Baker exists for OS-level customization, but using it crosses you into "self-supported" territory for the node lifecycle.
Hence, the advice is:
1. Support Case: File a Broadcom support case requesting lsm=…,bpf on the published Ubuntu VKR. That's the only path that's both supported and durable across node-replacement.
2. Ubuntu - Daemonset persistence: In the meantime, run a DaemonSet that injects the cmdline + reboots-once on each new node (uses stock VKR, keeps node-image support).
3. Custom-Baking Avoid custom-baking unless your needs go well beyond one cmdline param - it puts the cluster outside VKS support and creates a real maintenance liability which in fact is a business decision.
4. Support A custom-modified VKR is not VKS-supported.

Custom-bake mechanics, scripts, and FIPS/CVE caveats below if you decide to go that route anyway.

On Ubuntu

Ubuntu kernels already ship with CONFIG_BPF_LSM=y, so the only missing piece is having bpf on the boot cmdline of the VKr image. Bake it in once and every node - at every replacement - comes up with bpf in /sys/kernel/security/lsm. That solves your D4C visibility problem.

On Photon OS

The stock Photon OS 5.0 kernel (linux-6.12.78-5.ph5 in photon-updates, verified 2026-05 by pulling photon:5.0, installing linux, and grepping /boot/config-*) is built with # CONFIG_BPF_LSM is not set. The lsm= cmdline only chooses among LSMs the kernel was compiled with, so on Photon OS today no amount of cmdline tuning puts bpf into the active LSM list. The aspect of what should be default or not was a main topic in #1636 (comment).

The supported path is based on the Kubernetes Image Baker plugin and the VCF CLI:

• Building your own VKr with Image Baker
• Tools for building applications

Currently possible scripted way of a Kubernetes Image Baker workflow

I share the view that robustness is important and think it is valuable to follow the recommended approach. However, owning a custom VKr is explicitly unsupported. The supported path is a permanent maintenance commitment - that's a business decision, not a technical one.
The attached backed.zip offers a currently possible scripted way, including implementation ideas and taking into account CI issues, which are discussed further below.

Extract the baked.zip

Follow the included README.md. See the following description table to get an idea how to use the scripts.

  ┌────────────────────────────────────────────────────────┬───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
  │                       Want to …                        │                                                            Run                                                            │
  ├────────────────────────────────────────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
  │ Bake a Photon-5 + VKR 1.32.10 OVA into the current dir │ OS_TARGET=photon ARTIFACTS_DIR=$(pwd) ./setup-image-baker.sh                                                              │
  ├────────────────────────────────────────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
  │ Bake an Ubuntu-22.04 + VKR 1.32.10 OVA                 │ OS_TARGET=ubuntu ./setup-image-baker.sh                                                                                   │
  ├────────────────────────────────────────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
  │ Use a different VKR                                    │ VKR_K8S_VERSION=1.32.7 VKR_TAG=v1.32.7_vmware.1-fips-vkr.1 VKR_INNER_VERSION=v1.32.7+vmware.1-fips ./setup-image-baker.sh │
  ├────────────────────────────────────────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
  │ Custom kernel cmdline params                           │ KERNEL_PARAMS_LIST="lsm=… ipv6.disable=1" ./setup-image-baker.sh                                                          │
  ├────────────────────────────────────────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
  │ Scan an OVA (default = high-CVE gate + STIG)           │ ./scan-baked-ova.sh photon-…/photon-….ova photon-…                                                                        │
  ├────────────────────────────────────────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
  │ Skip STIG (no /dev/kvm)                                │ SKIP_STIG=1 ./scan-baked-ova.sh <ova> <name>                                                                              │
  ├────────────────────────────────────────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
  │ Inspect effective config                               │ source ./lib/common.sh && dump_config                                                                                     │
  └────────────────────────────────────────────────────────┴───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘

The latest Kubernetes Image Baker v3.6.2 plugin, tested in standalone mode (no vCenter context), has several bugs that either prevent the bake from completing or silently drop user-supplied kernelParameters. The baked.zip wraps vcf kr bake with a small set of bash scripts that work around them (pre-extracting cri-containerd.tar into the OS base image; stubbing the kubelet and registry units the bake expects. Because I've tested the scripts on WSL2 and encountered a WSL-specific issue, a workaround for the WSL host's /proc/cmdline JSON-encoding bug is in the bake's /tmp/export-system-info.sh; and a post-bake disk-repair step that patches /boot/photon.cfg directly because the bake writes kernelParameters only to /etc/default/grub).

A few things commonly missed when going down the custom-bake path

Image Baker is a build tool, not a security pipeline

Nothing in the documented Image Baker pipeline performs CVE scanning, SBOM generation, signature verification, or vulnerability assessment of the resulting OVA. The standard VKr images Broadcom publishes go through their internal security/CVE/FIPS pipeline before being signed and shipped to projects.packages.broadcom.com. VCF Operations 9.0.2 includes a Diagnostics framework that scans the running VCF environment for known CVEs in VMware components - that is not the same thing as scanning your custom-baked node OVAs. There is also no patch-in-place path for a baked OVA: the update model is rebake + redeploy.

For CVEs in baked-OVA filesystems, add an artifact scanner to the pipeline:

• Grype + Syft (Anchore) - SBOM-first, OSS, easy to gate on severity
• Trivy (Aqua) - similar profile, OSS
• Wiz / Prisma Cloud / Sysdig - enterprise, image scan + runtime

A reasonable shape e.g. with Grype + Syft:

  vcf kr bake → unpack OVA → mount VMDK → syft (SBOM) → grype (CVE scan) → gate → upload to content library

CVE scanners download their vulnerability database at scan time. In an air-gapped environment you have to mirror that database internally.

CVE scanning does not replace OS-hardening checks - it won't tell you whether AppArmor/SELinux is enforcing, whether the kernel cmdline actually contains lsm=…,bpf, or whether a debug shell was left enabled. For that, run a STIG/CIS profile against the same mounted rootfs (e.g. cinc-auditor / InSpec with the Photon OS or Ubuntu profile) as a parallel step in the pipeline.

FIPS

• FIPS in upstream VKrs is opt-in via a separate lineage (-fips-vkr.X tags).
• Don't bake a FIPS image yourself. FIPS 140-3 validation applies to a specific cryptographic module as built and tested. When you re-bake - even starting from a FIPS-tagged distribution image - you produce a new artifact that wasn't part of NIST's CMVP validation. The cryptographic binaries inside (validated OpenSSL / Go crypto) are still the same validated builds, but the composite OVA doesn't carry NIST's certificate. Once you re-bake, you have a FIPS-configured image, not a FIPS-validated one - different legally and contractually.
• The bake itself can break FIPS posture. Installing additional packages, changing the kernel cmdline (your BPF-LSM goal), or modifying systemd units can introduce non-FIPS crypto into the boot path or runtime. There is no validation step in vcf kr bake that checks whether the result is still FIPS-clean - it will happily produce a non-compliant image and exit zero.

The baked.zip also includes a elastic-d4c-on-vks-photon.md file with some information about D4C integration.

The safe path for FIPS workloads is to use Broadcom's pre-built FIPS VKr images (-fips-vkr.X) without re-baking.

This answer has been created using also [Anthropic ClaudeAI Chat + Opus 4.7, Anthropic Claude Code + Opus 4.7, xAI Grok Chat + Grok-4.3].

Kind regards,
Daniel

[pl853]

Thankyou for the extensive response! I'm going to look into it now!

[pl853]

We are going to submit a support ticket to enable BPF by default on Ubuntu base images. This would help Vmware too, given the fact that more security monitoring products (like crowdstrike and elasitc) will fullly work on Tanzu.

Do you have any knowledge on how long it usually takes to process such a request?

In the mean time we will, as you mentioned, do the following steps:

We create a deamonset that injects the cmdline + reboots-once on each new node (which uses stock VKR, keeps node-image support).

This deamonset will inject the following code in the grub boatloader config once a new node is enrolled

sudo tee /etc/default/grub.d/99-bpf-lsm.cfg > /dev/null <<'EOF'
GRUB_CMDLINE_LINUX_DEFAULT="${GRUB_CMDLINE_LINUX_DEFAULT}
lsm=lockdown,capability,landlock,yama,apparmor,bpf"
EOF

Then the deamonset runs the following 2 commands to apply the grub setting and reboot the node

sudo update-grub
sudo systemctl reboot

After that the BPF module is succesfully loaded and persists on reboot. The above will counter the issue described here:

Or what you saw wasn't really a reboot but a VKS node-replacement event (rolling upgrade, MachineDeployment rollout, scale-out). Those do destroy the rootfs. The "revert after reboot" you're seeing isn't a Photon OS or Ubuntu bug - it's the VKS / Cluster API immutable-node model. VKS nodes are provisioned from a VKr image; runtime changes to /etc/default/grub, /boot/…, or any file under /etc are lost the moment a node is replaced (upgrade, MachineDeployment rolling update, scale-out, sometimes a reconcile loop). The only stable place to put lsm=…,bpf is inside the VKr image itself, at bake time.

Due to the fact we are not editing the base image, we are still within the bound of VKS support, while we wait on the support ticket to be processed and the BPF module to be added.

[dcasota]

@bhllamoreaux Since the question of LSM defaults and CONFIG_BPF_LSM in the Photon OS kernel was already raised and discussed in #1636, I believe the optimal next step is to first obtain official feedback from the Photon OS team. This ensures customers follow the proper Broadcom/VMware workflow and technical governance before filing a support case or considering any 'Ubuntu - Daemonset persistence' and/or custom image baking.
Appreciate your help in keeping everything aligned with the established processes.

[pl853]

@dcasota thankyou again for your response. Yeh it might also be a problem that i'm here talking about the ubuntu base image in the photon base image issue section. For us being able to have BPF enabled in photonOS would be an even better option, but i would understand if this would be a no-go given the attack surface reduction argument discussed in #1636.

Therefore i think it would be better to enable the BPF module in the ubuntu base image, since ASR isnt a concern here.