Skip to content

feat: enable ABUpdate for UKI-based ACL images - #28

Open
bfjelds (bfjelds) wants to merge 32 commits into
aclmainfrom
user/bfjelds/ab-verity-partuuid-uki-addons-20260708
Open

feat: enable ABUpdate for UKI-based ACL images#28
bfjelds (bfjelds) wants to merge 32 commits into
aclmainfrom
user/bfjelds/ab-verity-partuuid-uki-addons-20260708

Conversation

@bfjelds

@bfjelds bfjelds (bfjelds) commented Jul 9, 2026

Copy link
Copy Markdown
Member

Summary

Enables Trident-driven A/B updates on UKI-based ACL images by giving /usr dm-verity a slot-agnostic boot path, installing/enabling Trident in the image, and expanding the ESP to support A/B UKI staging on arm64.

Grub-based ACL images have their partitions updated to match the UKI images, but Trident only supports A/B update for UKI images.

Changes

Verity: dedicated hash partitions + slot-agnostic boot

  • disk_layout_uki.json: add dedicated verity hash partitions HASH-A and HASH-B (type dps-usr-verity) paired to USR-A/USR-B via a new verity_hash cross-reference; renumber OEM/ROOT accordingly.
  • disk_util: add dps-usr-verity partition-type GUID resolution and the verity_hash layout key; add a separate-hash-partition verity mode (hash tree written to the dedicated partition instead of an inline hash-offset); emit the btrfs filesystem UUID and verity superblock UUID via new --fs_uuid/--verity_uuid flags; only the active (prioritized) slot is formatted at build time.
  • build_image_util.sh: capture the new UUID outputs and pass them to bootloader install.

UKI / systemd-boot (the boot path for ACL images). rpm/uki_install.sh:

  • The main UKI cmdline carries only slot-independent args (mount.usr=/dev/mapper/usr, root=LABEL=ROOT, ...).
  • Slot-specific verity identity is delivered via a systemd-stub addon (verity.addon.efi in <uki>.efi.extra.d/) built per slot with systemd.verity_usr_data=PARTUUID=<usr> and systemd.verity_usr_hash=PARTUUID=<hash> plus the slot root hash. Slot A is active by default; Trident switches slots by swapping the active addon. This lets one UKI boot either slot without re-signing.

GRUB (secondary bootloader path). grub_install.sh (declare --fs_uuid / --verity_uuid), rpm/grub_install.sh (read and inject the values), and rpm/grub.cfg (template @@FSUUID@@ / @@VERITYUUID@@): build the verity cmdline from systemd.verity_usr_data=UUID=<fs> and systemd.verity_usr_hash=UUID=<verity> instead of a hard-coded PARTUUID + hash-offset.

ESP capacity for A/B UKI staging

  • disk_layout_uki.json: increase the EFI System Partition from 128 MiB (262144 blocks) to 256 MiB (524288 blocks).
  • The arm64 UKI is approximately 82 MiB, compared with approximately 49 MiB on amd64. During an A/B update, the new UKI is staged on the ESP alongside the running UKI; the 128 MiB ESP runs out of space on arm64. The 256 MiB ESP accommodates two arm64 UKIs plus bootloader files and headroom, while remaining 4096-block aligned.

Trident enablement

  • rpm/package_catalog.yaml: add the trident package.
  • rpm/build_image_util.sh: enable tridentd.socket.

arm64 test-image boot reliability

  • vm_image_util.sh: add install_uki_timeout_addon, a UKI addon that raises systemd.default_device_timeout_sec=120. Scoped to the arm64 kola test image only; it fixes a CI-only QEMU-TCG emulation flake where udev cannot initialise the ESP/OEM/usr-verity devices within the default device timeout and the VM drops to an emergency shell. Production and amd64 boot behaviour are unchanged.

Docs

  • acl/docs/architecture.md: document the dedicated hash partition and slot-agnostic verity boot.

Required PRs

Validation

@bfjelds
bfjelds (bfjelds) changed the base branch from user/bfjelds/ab-trident-verity-hash-partitions-20260708 to aclmain July 13, 2026 20:38
@bfjelds bfjelds (bfjelds) changed the title feat: move verity kernel command line params into A/B PARTUUID-based UKI addons feat: enable ABUpdate for ACL Jul 13, 2026
@bfjelds
bfjelds (bfjelds) requested a lite review from Copilot July 14, 2026 18:09

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR enables Trident-driven A/B updates for Azure Container Linux (ACL) images by making /usr dm-verity boot slot-agnostic (primary UKI path via systemd-stub addons) and by updating the GRUB fallback cmdline construction to avoid hard-coding partition identity/inline hash offsets. It also adds Trident to the image and enables its socket.

Changes:

  • Introduce dedicated /usr dm-verity hash partitions (HASH-A / HASH-B) and plumb a verity_hash cross-reference through disk_util + the UKI disk layout.
  • Switch UKI/systemd-boot to a slot-independent main cmdline plus per-slot verity.addon.efi addons, and switch GRUB to UUID-based systemd.verity_usr_* parameters injected at build time.
  • Add trident to the RPM package catalog and enable tridentd.socket in the built image.

Reviewed changes

Copilot reviewed 10 out of 10 changed files in this pull request and generated 7 comments.

Show a summary per file
File Description
build_library/rpm/uki_install.sh Reworks UKI cmdline to be slot-agnostic and adds generation of per-slot verity addons.
build_library/rpm/rpm_install.sh Fixes punctuation in a log message.
build_library/rpm/package_catalog.yaml Adds the trident RPM to the package set.
build_library/rpm/grub.cfg Updates GRUB verity cmdline template to use injected FS/verity UUIDs instead of inline hash-offset logic.
build_library/rpm/grub_install.sh Injects FS UUID + verity UUID into GRUB template during provisioning.
build_library/rpm/build_image_util.sh Enables tridentd.socket via a sockets.target wants symlink.
build_library/disk_util Adds dps-usr-verity type resolution, supports separate hash partitions, and emits FS/verity UUIDs.
build_library/disk_layout_uki.json Adds HASH-A/HASH-B partitions and links USR-A/USR-B to them via verity_hash.
build_library/build_image_util.sh Plumbs new UUID output files into bootloader install arguments.
acl/docs/architecture.md Updates architecture docs for dedicated hash partitions / slot-agnostic verity boot.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread build_library/build_image_util.sh
Comment thread build_library/rpm/grub_install.sh
Comment thread build_library/rpm/grub_install.sh
Comment thread build_library/rpm/uki_install.sh
Comment thread build_library/rpm/uki_install.sh
Comment thread build_library/disk_util Outdated
Comment thread acl/docs/architecture.md Outdated

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 11 out of 11 changed files in this pull request and generated 2 comments.

Comment thread build_library/disk_util Outdated
Comment thread build_library/rpm/uki_install.sh Outdated
bfjelds (bfjelds) added a commit that referenced this pull request Jul 14, 2026
Copilot re-review follow-ups on PR #28:

- disk_util: the --fs_uuid output was only written in separate-hash-partition
  mode, so legacy inline-verity layouts produced no FS UUID file and now hard-
  fail in grub.cfg generation. Collapse the two near-identical verity branches
  into a single contextlib.ExitStack path that reads the FS UUID in both modes,
  removing the duplicated veritysetup/blkid logic.
- rpm/uki_install.sh: correct the stale UUID-validation comment (grub_install.sh
  now dies rather than warns; uki_install does not pass UUIDs through itself).

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 11 out of 11 changed files in this pull request and generated 2 comments.

Comment thread build_library/disk_util
Comment thread build_library/rpm/grub_install.sh

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 11 out of 11 changed files in this pull request and generated 1 comment.

Comment thread build_library/rpm/uki_install.sh Outdated

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 11 out of 11 changed files in this pull request and generated 1 comment.

Comment thread build_library/disk_util

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 11 out of 11 changed files in this pull request and generated no new comments.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 11 out of 11 changed files in this pull request and generated 5 comments.

Comment thread build_library/disk_util Outdated
Comment thread build_library/disk_util Outdated
Comment thread build_library/disk_util Outdated
Comment thread build_library/rpm/uki_install.sh
Comment thread build_library/rpm/grub_install.sh
bfjelds (bfjelds) added a commit that referenced this pull request Jul 20, 2026
disk_util: normalize root-hash/verity-uuid block indentation and use
`is not None` for root_hash.

uki_install.sh: validate UUID file content (non-empty), not just
existence, so a blank UUID file fails fast.

grub_install.sh: hard-fail when verity is enabled but no hash file is
provided, matching the sibling FS/verity UUID checks (a warn + empty
substitution produces an unbootable grub.cfg).

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 72f4b84a-8f5c-4df6-9f8e-fb4850c520e2

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 11 out of 11 changed files in this pull request and generated 1 comment.

Comment thread build_library/rpm/uki_install.sh Outdated

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 14 out of 14 changed files in this pull request and generated no new comments.

Suppressed comments (2)

Previously missed (2) — in code that hasn't changed since the last review.

acl/validate/validate_qemu.sh:56

  • net_info is assigned without local inside ensure_libvirt_network(). Because this file is sourced (not executed), that can leak net_info into the caller’s/global scope and cause hard-to-trace variable collisions later in the validation run.
    net_info="$(virsh net-info default 2>/dev/null)"

build_library/disk_layout_uki.json:13

  • The ESP resize comment says "128 MiB -> 256 MiB", but this file’s previous value was already 192 MiB (393216 blocks). Updating the comment to match the actual delta avoids confusion when auditing layout history.
                "_comment": "128 MiB -> 256 MiB (blocks are 512-byte sectors). UKI boot keeps kernel+initramfs on the ESP; the acl-t UKI rebuilt by Image Customizer and Trident A/B update addons both need more headroom than stock.",

@bfjelds

Copy link
Copy Markdown
Member Author

it looks like github-acl-pr-validation doesn't use mantle ref provided (which would have fixes for verity failures).

github-mantle-pr-validation has both mantle and acr fixes.

azldevel has both mantle and acr fixes as well.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 14 out of 14 changed files in this pull request and generated 3 comments.

Comment thread build_library/disk_layout.json
Comment thread build_library/disk_layout.json
Comment thread build_library/disk_layout.json
- generate_grub_hashes.py: fix PCR policy allowlist entry for the OEM
  partition, which moved from gpt6 to gpt8 in this PR's renumbered
  disk_layout.json. The expected measured GRUB command is the OEM
  search result substituted into "source (hd0,gptN)/grub.cfg"; it must
  track the actual partition number or secure/measured boot policy
  checks will fail against real hardware measurements.

- prod_image_util.sh: create_prod_tar() resolve ROOT and USR-A by GPT
  PARTLABEL via blkid instead of hardcoded partition numbers (p9/p3),
  which broke when ROOT moved from partition 9 to 11 in this PR.

- disk_util / disk_layout.json: add an explicit legacy_verity_offset
  opt-in flag. When set on a verity_hash-bearing partition, Verity()
  now writes the hash tree to both the dedicated hash partition and
  the legacy inline offset within the data partition (same salt, so
  both copies are byte-identical and share one root hash). This keeps
  Portage/bootengine boots working via the existing hardcoded inline
  hash-offset activation, without requiring any bootengine or GRUB
  cmdline changes, while RPM/UKI-style consumers of the dedicated
  HASH-A/HASH-B partitions are unaffected. Verified end-to-end with
  veritysetup that both write targets produce an identical root hash
  and both independently pass veritysetup open.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot AI review requested due to automatic review settings August 27, 2026 21:05

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 16 out of 16 changed files in this pull request and generated no new comments.

Suppressed comments (1)

Previously missed (1) — in code that hasn't changed since the last review.

build_library/disk_layout_uki.json:13

  • This layout history and sizing rationale are inaccurate: the previous value was 393216 sectors (192 MiB), not 128 MiB, and the A/B space pressure comes from staging a second arm64 UKI rather than from the comparatively small addon files. Please correct the comment so future ESP sizing changes are based on the actual baseline.
                "_comment": "128 MiB -> 256 MiB (blocks are 512-byte sectors). UKI boot keeps kernel+initramfs on the ESP; the acl-t UKI rebuilt by Image Customizer and Trident A/B update addons both need more headroom than stock.",

Copilot AI review requested due to automatic review settings August 28, 2026 16:53

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 16 out of 16 changed files in this pull request and generated no new comments.

Suppressed comments (2)

Previously missed (1) — in code that hasn't changed since the last review.

build_library/vm_image_util.sh:783

  • This condition does not actually restrict the workaround to QEMU-TCG: acl/build_rpm_image.sh:1302-1317 also sets INJECT_DOCKER_SYSEXT=true when building an Azure test image. Consequently, arm64 Azure UKI test images receive the 120-second timeout even though the PR says real-hardware/Azure behavior is unchanged. Include the VM image type in the guard so only the qemu_uefi test conversion gets this CI workaround.

This issue also appears on line 875 of the same file.

    if [[ "${ARCH}" != "arm64" ]] || [[ "${INJECT_DOCKER_SYSEXT:-false}" != "true" ]]; then

build_library/vm_image_util.sh:875

  • The GRUB guard has the same scope leak: INJECT_DOCKER_SYSEXT=true identifies both QEMU and Azure test images (acl/build_rpm_image.sh:1302-1317), so an arm64 Azure GRUB test image is patched despite this being documented as a QEMU-TCG-only workaround. Add the qemu_uefi image-type check here as well.
    if [[ "${ARCH}" != "arm64" ]] || [[ "${INJECT_DOCKER_SYSEXT:-false}" != "true" ]]; then

Deleting the multi-user.target.wants symlink for auditd.service is not
durable: Azure Linux ships a 90-default.preset with an explicit
"enable auditd.service" rule, so any later systemctl preset/preset-all
re-application (e.g. triggered by downstream image customization
installing more packages) recreates the symlink and the service starts
on the next boot.

Uninstall the audit and python3-audit packages instead, removing the
unit file itself so there is nothing left for a future preset pass to
enable. audit-libs is left in place since other packages link against
libaudit.so.1 directly.

rpm -e uses --noscripts, so %preun/%postun (which would run `systemctl
disable`) never execute, and the *.wants symlinks aren't part of the
package payload in the first place (systemctl/preset creates them, not
rpm). Removing the package alone leaves them as dangling symlinks,
which trips kola's dead-symlink content check. Explicitly remove both
possible .wants symlink locations after the uninstall.

This is a temporary workaround for trident-selinux's transitive
dependency on audit. Trident's fix (microsoft/trident#734) already
removed that dependency, but the fix hasn't been published to PMC yet
(PMC still serves trident 0.26.0). Once a trident release containing
the fix reaches PMC, this block should be removed.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot AI review requested due to automatic review settings August 28, 2026 18:24
@bfjelds
bfjelds (bfjelds) force-pushed the user/bfjelds/ab-verity-partuuid-uki-addons-20260708 branch from 49dcab5 to 1f81d57 Compare August 28, 2026 18:24

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 16 out of 16 changed files in this pull request and generated 5 comments.

Comment thread build_library/disk_layout.json
Comment thread build_library/disk_layout.json
Comment thread build_library/disk_layout.json
Comment thread build_library/prod_image_util.sh Outdated
Comment thread build_library/disk_util
create_prod_tar() runs under switch_to_strict_mode (set -u).
`local partdev rootdev usrdev` leaves rootdev/usrdev declared-but-unset,
and referencing an unset local variable -- even just to test it with
`[[ -z "${rootdev}" ]]` -- is itself an unbound-variable error under
set -u. If ROOT or USR-A can't be resolved by partition label, the
intended missing-partition check crashes on the check itself, before
it can reach the informative die_notrace message or detach the loop
device.

Initialize both to empty strings so the check works as intended.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 16 out of 16 changed files in this pull request and generated 1 comment.

Comment thread build_library/rpm/grub.cfg
Comment thread build_library/rpm/uki_install.sh Outdated

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 16 out of 16 changed files in this pull request and generated no new comments.

Comment thread build_library/rpm/uki_install.sh Outdated
Comment thread build_library/rpm/uki_install.sh
…l.slot cmdline

Rename the per-slot verity addon templates from verity-a/b.addon.efi to
slot-a/b.addon.efi, and install the active slot addon verbatim (no
rename) into the .extra.d directory. Add acl.slot=<a|b> to each slot cmdline
so the running kernel can identify its active slot.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Address Copilot review feedback on PR 64: slot-a.addon.efi/ looked
like a directory path due to a stray trailing slash. Replaced with a
comma so it reads as two alternative addon filenames.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
abupdate: use slot-*.addon.efi verbatim instead of verity*.addon.efi, add acl.slot

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 16 out of 16 changed files in this pull request and generated 1 comment.

Suppressed comments (1)

build_library/rpm/uki_install.sh:608

  • The live addon also needs Trident's canonical verity.addon.efi name. Trident stages the existing .extra.d contents verbatim and later writes the selected template as verity.addon.efi; leaving slot-a.addon.efi here means an update can load both the stale slot-A cmdline and the newly activated slot cmdline. Install the initial A template under the canonical active name.
    sudo cp "${template_dir}/slot-a.addon.efi" "${addon_dir}/slot-a.addon.efi"

Comment thread build_library/rpm/uki_install.sh
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants