You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Enables Trident-driven A/B updates on UKI-based ACL images by giving /usr dm-verity a slot-agnostic boot path and installing/enabling Trident in the image.
Grub-based ACL images have their partitions updated to match the UKI images, but Trident only supports A/B update for UKI images.
disk_layout_uki.json: add dedicated verity hash partitions HASH-A and HASH-B (type dps-usr-verity) paired to USR-A/USR-B via a new verity_hash cross-reference; renumber OEM/ROOT accordingly.
disk_util: add dps-usr-verity partition-type GUID resolution and the verity_hash layout key; add a separate-hash-partition verity mode (hash tree written to the dedicated partition instead of an inline hash-offset); emit the btrfs filesystem UUID and verity superblock UUID via new --fs_uuid/--verity_uuid flags; only the active (prioritized) slot is formatted at build time.
build_image_util.sh: capture the new UUID outputs and pass them to bootloader install.
UKI / systemd-boot (the boot path for ACL images).rpm/uki_install.sh:
The main UKI cmdline carries only slot-independent args (mount.usr=/dev/mapper/usr, root=LABEL=ROOT, ...).
Slot-specific verity identity is delivered via a systemd-stub addon (verity.addon.efi in <uki>.efi.extra.d/) built per slot with systemd.verity_usr_data=PARTUUID=<usr> and systemd.verity_usr_hash=PARTUUID=<hash> plus the slot root hash. Slot A is active by default; Trident switches slots by swapping the active addon. This lets one UKI boot either slot without re-signing.
GRUB (secondary bootloader path).grub_install.sh (declare --fs_uuid / --verity_uuid), rpm/grub_install.sh (read and inject the values), and rpm/grub.cfg (template @@FSUUID@@ / @@VERITYUUID@@): build the verity cmdline from systemd.verity_usr_data=UUID=<fs> and systemd.verity_usr_hash=UUID=<verity> instead of a hard-coded PARTUUID + hash-offset.
Trident enablement
rpm/package_catalog.yaml: add the trident package.
rpm/build_image_util.sh: enable tridentd.socket.
arm64 test-image boot reliability
vm_image_util.sh: add install_uki_timeout_addon, a UKI addon that raises systemd.default_device_timeout_sec=120. Scoped to the arm64 kola test image only; it fixes a CI-only QEMU-TCG emulation flake where udev cannot initialise the ESP/OEM/usr-verity devices within the default device timeout and the VM drops to an emergency shell. Production and amd64 boot behaviour are unchanged.
Docs
acl/docs/architecture.md: document the dedicated hash partition and slot-agnostic verity boot.
bfjelds (bfjelds)
changed the title
feat: move verity kernel command line params into A/B PARTUUID-based UKI addons
feat: enable ABUpdate for ACL
Jul 13, 2026
The reason will be displayed to describe this comment to others. Learn more.
Pull request overview
This PR enables Trident-driven A/B updates for Azure Container Linux (ACL) images by making /usr dm-verity boot slot-agnostic (primary UKI path via systemd-stub addons) and by updating the GRUB fallback cmdline construction to avoid hard-coding partition identity/inline hash offsets. It also adds Trident to the image and enables its socket.
Changes:
Introduce dedicated /usr dm-verity hash partitions (HASH-A / HASH-B) and plumb a verity_hash cross-reference through disk_util + the UKI disk layout.
Switch UKI/systemd-boot to a slot-independent main cmdline plus per-slot verity.addon.efi addons, and switch GRUB to UUID-based systemd.verity_usr_* parameters injected at build time.
Add trident to the RPM package catalog and enable tridentd.socket in the built image.
Reviewed changes
Copilot reviewed 10 out of 10 changed files in this pull request and generated 7 comments.
Show a summary per file
File
Description
build_library/rpm/uki_install.sh
Reworks UKI cmdline to be slot-agnostic and adds generation of per-slot verity addons.
build_library/rpm/rpm_install.sh
Fixes punctuation in a log message.
build_library/rpm/package_catalog.yaml
Adds the trident RPM to the package set.
build_library/rpm/grub.cfg
Updates GRUB verity cmdline template to use injected FS/verity UUIDs instead of inline hash-offset logic.
build_library/rpm/grub_install.sh
Injects FS UUID + verity UUID into GRUB template during provisioning.
build_library/rpm/build_image_util.sh
Enables tridentd.socket via a sockets.target wants symlink.
build_library/disk_util
Adds dps-usr-verity type resolution, supports separate hash partitions, and emits FS/verity UUIDs.
build_library/disk_layout_uki.json
Adds HASH-A/HASH-B partitions and links USR-A/USR-B to them via verity_hash.
build_library/build_image_util.sh
Plumbs new UUID output files into bootloader install arguments.
Copilot re-review follow-ups on PR #28:
- disk_util: the --fs_uuid output was only written in separate-hash-partition
mode, so legacy inline-verity layouts produced no FS UUID file and now hard-
fail in grub.cfg generation. Collapse the two near-identical verity branches
into a single contextlib.ExitStack path that reads the FS UUID in both modes,
removing the duplicated veritysetup/blkid logic.
- rpm/uki_install.sh: correct the stale UUID-validation comment (grub_install.sh
now dies rather than warns; uki_install does not pass UUIDs through itself).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
disk_util: normalize root-hash/verity-uuid block indentation and use
`is not None` for root_hash.
uki_install.sh: validate UUID file content (non-empty), not just
existence, so a blank UUID file fails fast.
grub_install.sh: hard-fail when verity is enabled but no hash file is
provided, matching the sibling FS/verity UUID checks (a warn + empty
substitution produces an unbootable grub.cfg).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 72f4b84a-8f5c-4df6-9f8e-fb4850c520e2
uki_install.sh runs under switch_to_strict_mode (set -e + ERR trap). With
jq -e, a missing/null .uuid makes jq exit 1, tripping the ERR trap with a
generic error before the explicit [[ -z || == null ]] check can emit the
clear "Partition has no uuid field" die message. Dropping -e lets jq emit
null (exit 0) so the explicit check produces the intended message. Also
matches the sibling matches=jq-r lookup which omits -e.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 72f4b84a-8f5c-4df6-9f8e-fb4850c520e2
Install a UKI addon setting systemd.default_device_timeout_sec=120 on the
arm64 kola test image only (gated on ARCH=arm64 and INJECT_DOCKER_SYSEXT=true).
Under kola's parallel QEMU-TCG emulation on aarch64, CPU contention can keep
udev from initialising the ESP/OEM/usr-verity devices within the default
initrd device timeout, dropping the VM to an emergency shell. Raising the
timeout via a test-image ESP addon fixes this. Verified against a parallel=24
repro: 18/18 boot-to-login vs 18/18 emergency shells at baseline.
Scoped to the test image, so production and amd64 boot behaviour is unchanged;
the failure is a CI-only emulation artefact that never occurs on real hardware.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 72f4b84a-8f5c-4df6-9f8e-fb4850c520e2
- install_uki_timeout_addon: require exactly one UKI on the ESP (die on
multiple) instead of silently picking one via sort -V | head, so the
addon can never land in the wrong <uki>.efi.extra.d. Mirrors the
single-UKI rule in rpm/uki_install.sh.
- RPM-mode tridentd.socket enable: hard-fail if tridentd.socket is absent.
trident is required for ACL images, so a missing unit is a broken image;
fail the build rather than leave a dangling wants-symlink.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 72f4b84a-8f5c-4df6-9f8e-fb4850c520e2
A present-but-empty verity_hash (e.g. "verity_hash": "") previously fell
through to legacy inline-verity mode instead of separate-hash-partition
mode, producing an invalid layout that is hard to debug. Strip and raise
InvalidLayout when the value is empty, consistent with the other malformed-
layout guards in this function.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 72f4b84a-8f5c-4df6-9f8e-fb4850c520e2
…ID wording
Under switch_to_strict_mode (set -e), a non-zero exit from `sudo ukify build`
aborted the script before the following existence check could emit the
intended die message or clean up the temp dir. Wrap every ukify invocation
(main UKI + firstboot/fips/kdump/debug/verity addons in uki_install.sh and
the timeout addon in vm_image_util.sh) in an explicit `if ! ...; then
rm -rf <tmp>; die; fi`, and clean up the temp dir on the produce-check path
too. Failures now surface with a clear message and no leaked temp dirs.
Also reword disk_util Verity's fs_uuid comment/exception from "btrfs FS UUID"
to "filesystem UUID": the blkid read is filesystem-agnostic. (ACL only
verities btrfs today, so this is wording-only.)
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 72f4b84a-8f5c-4df6-9f8e-fb4850c520e2
sudo ukify build can leave root-owned artifacts in the mktemp temp dirs, so a
plain `rm -rf` could fail and (under strict mode) abort before the intended
die. Use `sudo rm -rf` for every temp-dir cleanup in the UKI/addon builders
(uki_install.sh main UKI + firstboot/fips/kdump/debug/verity addons, and the
timeout addon in vm_image_util.sh), matching the existing sudo rm -rf
convention used throughout build_library. Each temp dir is assigned via
mktemp -d immediately before use, so the rm target is always a real path.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 72f4b84a-8f5c-4df6-9f8e-fb4850c520e2
GRUB's disk_layout.json previously used the legacy single-partition
inline-hash verity scheme. The UKI-side changes in this branch switched
grub.cfg's verity cmdline construction to UUID=-based hash identification
(dropping hash-offset=), which only works when the hash tree lives on its
own independently-discoverable partition.
Mirror disk_layout_uki.json's structure: add dedicated HASH-A/HASH-B
partitions (dps-usr-verity type) cross-referenced via verity_hash, so
blkid/udev can resolve systemd.verity_usr_hash=UUID=<verity_uuid> the same
way UKI does. ACL-GRUB still cannot do Trident A/B updates (no addon-swap
equivalent), but the partitioning/verity mechanics are now consistent
between the two boot paths instead of silently broken on GRUB.
ESP stays at 128MiB (no AB-staging need on GRUB). BIOS-BOOT partition is
kept (shared with Portage/flatcar builds; RPM/GRUB mode already skips
installing to it).
install_uki_timeout_addon already works around a CI-only QEMU-TCG flake on
arm64 kola test images: heavy parallel-VM CPU contention can prevent udev
from initialising ESP/OEM/usr-verity devices within the default initrd
device timeout, dropping the VM to an emergency shell. That fix was
UKI-only (built as a signed .efi addon dropped into <uki>.efi.extra.d/).
GRUB has no addon mechanism, so add install_grub_timeout_override: since
grub.cfg is plain text (not signed/embedded like a UKI), patch the
already-written grub.cfg copies on the test image's ESP directly via sed,
appending systemd.default_device_timeout_sec=120 to the shared
linux_cmdline. Same scoping as the UKI addon: arm64 test image only
(INJECT_DOCKER_SYSEXT=true), never applied to production images or amd64.
Addresses Copilot review comment on PR #28: install_grub_timeout_override
logged \ patched\ unconditionally even if the sed substitution silently
did not match (e.g. linux_cmdline format changed), leaving the timeout
override ineffective without any signal. Now verifies the substitution
took effect and dies with a clear message if not.
…check
virsh net-info default | grep -q could report a false pipeline failure
under set -o pipefail (inherited via sourcing from validate_common.sh):
grep -q exits as soon as it matches, which can SIGPIPE virsh while it is
still writing, making the whole pipeline non-zero even though the
pattern matched. Capture output first, then grep the captured string.
trident-selinux pulls in policycoreutils-python-utils (for SELinux
module install tooling), which transitively depends on the audit
package. Azure Linux audit RPM enables+starts auditd via its %post
systemd preset, unlike portage-mode builds which ship a blanket
disable-all preset. Remove the enable symlink so RPM-mode images
match upstream ACL behavior: auditd installed but disabled by
default.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
ACL images need trident-acl-agent (triggers AB-updates), not the
full trident package. Keep the auditd.service disable workaround in
build_image_util.sh regardless, since trident-acl-agent still
depends on trident, and defense-in-depth is warranted independent of
the trident.spec fix (microsoft/trident#734).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- generate_grub_hashes.py: fix PCR policy allowlist entry for the OEM
partition, which moved from gpt6 to gpt8 in this PR's renumbered
disk_layout.json. The expected measured GRUB command is the OEM
search result substituted into "source (hd0,gptN)/grub.cfg"; it must
track the actual partition number or secure/measured boot policy
checks will fail against real hardware measurements.
- prod_image_util.sh: create_prod_tar() resolve ROOT and USR-A by GPT
PARTLABEL via blkid instead of hardcoded partition numbers (p9/p3),
which broke when ROOT moved from partition 9 to 11 in this PR.
- disk_util / disk_layout.json: add an explicit legacy_verity_offset
opt-in flag. When set on a verity_hash-bearing partition, Verity()
now writes the hash tree to both the dedicated hash partition and
the legacy inline offset within the data partition (same salt, so
both copies are byte-identical and share one root hash). This keeps
Portage/bootengine boots working via the existing hardcoded inline
hash-offset activation, without requiring any bootengine or GRUB
cmdline changes, while RPM/UKI-style consumers of the dedicated
HASH-A/HASH-B partitions are unaffected. Verified end-to-end with
veritysetup that both write targets produce an identical root hash
and both independently pass veritysetup open.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Deleting the multi-user.target.wants symlink for auditd.service is not
durable: Azure Linux ships a 90-default.preset with an explicit
"enable auditd.service" rule, so any later systemctl preset/preset-all
re-application (e.g. triggered by downstream image customization
installing more packages) recreates the symlink and the service starts
on the next boot.
Uninstall the audit and python3-audit packages instead, removing the
unit file itself so there is nothing left for a future preset pass to
enable. audit-libs is left in place since other packages link against
libaudit.so.1 directly.
rpm -e uses --noscripts, so %preun/%postun (which would run `systemctl
disable`) never execute, and the *.wants symlinks aren't part of the
package payload in the first place (systemctl/preset creates them, not
rpm). Removing the package alone leaves them as dangling symlinks,
which trips kola's dead-symlink content check. Explicitly remove both
possible .wants symlink locations after the uninstall.
This is a temporary workaround for trident-selinux's transitive
dependency on audit. Trident's fix (microsoft/trident#734) already
removed that dependency, but the fix hasn't been published to PMC yet
(PMC still serves trident 0.26.0). Once a trident release containing
the fix reaches PMC, this block should be removed.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
create_prod_tar() runs under switch_to_strict_mode (set -u).
`local partdev rootdev usrdev` leaves rootdev/usrdev declared-but-unset,
and referencing an unset local variable -- even just to test it with
`[[ -z "${rootdev}" ]]` -- is itself an unbound-variable error under
set -u. If ROOT or USR-A can't be resolved by partition label, the
intended missing-partition check crashes on the check itself, before
it can reach the informative die_notrace message or detach the loop
device.
Initialize both to empty strings so the check works as intended.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…l.slot cmdline
Rename the per-slot verity addon templates from verity-a/b.addon.efi to
slot-a/b.addon.efi, and install the active slot addon verbatim (no
rename) into the .extra.d directory. Add acl.slot=<a|b> to each slot cmdline
so the running kernel can identify its active slot.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Address Copilot review feedback on PR 64: slot-a.addon.efi/ looked
like a directory path due to a stray trailing slash. Replaced with a
comma so it reads as two alternative addon filenames.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The systemd-boot/UKI bullet said ukify packs verity parameters into the
main kernel command line, contradicting the dm-verity section below it,
which documents the actual (newer) design: the main UKI cmdline is
slot-independent, and slot-specific verity identity is delivered via a
per-slot systemd-stub addon. Updated the bullet to match.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Address Copilot review: generate_grub_hashes.py hard-codes GRUB-specific
partition assumptions (OEM at GPT index 8, core.img/diskboot.img/grub.cfg
locations) that do not correspond to the UKI disk layout, and the script
ran unconditionally regardless of BOOTLOADER_MODE. UKI images have no
GRUB boot chain to measure, so skip generation entirely in UKI mode
rather than trying to compute a GRUB-shaped policy for a non-GRUB image.
Also guard the official-build pcr_policy cleanup step, since the zip
is no longer produced for UKI-mode builds.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Address Copilot review: the pcr_policy guard I added previously only
covered that one file, but the same rm -v also unconditionally removed
image_kernel and image_grub, both of which finish_image also skips
producing in UKI mode (standalone kernel copy and GRUB install are
both skipped there). Replace the partial fix with a uniform loop that
only removes artifacts that were actually produced.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
… images
generate_grub_hashes.py hardcoded a legacy Portage/Flatcar kernel cmdline
regex for the PCR-8 kernel_cmdline.config entry, which does not match the
actual RPM+GRUB verity cmdline (systemd-native mount.usr=/dev/mapper/usr
+ usrhash=/systemd.verity_usr_* tokens vs. the old PARTUUID/verity.usrhash
scheme). Add package_source_mode as a required 5th arg and branch the
pattern accordingly.
Also skip GRUB PCR-policy generation entirely for non-verity images in
finish_image() (disable_read_write=false): the real cmdline in that case
has no verity tokens at all, so no static pattern - RPM or Portage - can
ever match it.
Caveat: the new RPM pattern is derived from build_library/rpm/grub.cfg
source, not validated against a captured real PCR-8 measurement.
Nothing in either repo (azure-container-linux, azure-container-linux-mantle)
or in the mariner-org ADO org (acl-scripts, acl-pipelines, acl-baremetal,
polar, dev-sandbox) consumes pcr_policy.zip or its contents. Rather than
maintain hand-derived, unvalidated GRUB command/cmdline regexes for the
RPM+GRUB boot chain (which do not match the actual measured boot commands,
per prior review), stop generating this artifact for RPM entirely.
image_pcr_policy is now empty for PACKAGE_SOURCE_MODE=RPM in both
create_prod_image() and sbsign_prod_image(), which naturally short-circuits
the existing \[[ -n \ \\ ]]\ guards in finish_image()/sbsign_image().
This makes the UKI-mode and non-verity skip branches added there dead code
for RPM, so those are simplified back to their original unconditional form.
Portage/Flatcar PCR-policy generation is unchanged.
Trident 0.28 (which includes the trident-selinux audit dependency fix,
microsoft/trident#734) has been published to PMC.
- package_catalog.yaml: reference trident-acl instead of trident
- build_image_util.sh: remove the now-unneeded auditd removal
workaround
INJECT_DOCKER_SYSEXT=true is set for both QEMU and Azure test-image
builds. The arm64 device-init timeout workaround (120s) is only needed
to mitigate QEMU-TCG emulation CPU contention in CI and should not be
applied to Azure test images running on real hardware. Add an explicit
VM_IMG_TYPE == qemu_uefi check to both install_uki_timeout_addon and
install_grub_timeout_override.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
ROOT moved to partition 7 in disk_layout_uki.json (new HASH-A/HASH-B
verity partitions were inserted), but the nbd mount example still
referenced /dev/nbd0p5 as the fifth partition.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The reason will be displayed to describe this comment to others. Learn more.
Copilot review overview
🔵 Needs a closer look
It changes boot-chain and partition contracts, and the validator still has an unresolved network-recovery regression.
Review effort: Balanced Findings: None
Previously missed (1)
In code that hasn't changed since last review
Guard virsh network checks against set -e failures
acl/validate/validate_qemu.sh:57
This assignment runs as an unguarded command while the validator has set -e enabled. If virsh net-info default fails—for example, while the network is absent or transiently unavailable—the script exits before the existing start/recovery branch runs. Put both the assignment and activity check in the conditional so either failure reaches that branch.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Enables Trident-driven A/B updates on UKI-based ACL images by giving
/usrdm-verity a slot-agnostic boot path and installing/enabling Trident in the image.Grub-based ACL images have their partitions updated to match the UKI images, but Trident only supports A/B update for UKI images.
Changes
Verity: dedicated hash partitions + slot-agnostic boot
disk_layout_uki.json: add dedicated verity hash partitions HASH-A and HASH-B (typedps-usr-verity) paired to USR-A/USR-B via a newverity_hashcross-reference; renumber OEM/ROOT accordingly.disk_util: adddps-usr-veritypartition-type GUID resolution and theverity_hashlayout key; add a separate-hash-partition verity mode (hash tree written to the dedicated partition instead of an inline hash-offset); emit the btrfs filesystem UUID and verity superblock UUID via new--fs_uuid/--verity_uuidflags; only the active (prioritized) slot is formatted at build time.build_image_util.sh: capture the new UUID outputs and pass them to bootloader install.UKI / systemd-boot (the boot path for ACL images).
rpm/uki_install.sh:mount.usr=/dev/mapper/usr,root=LABEL=ROOT, ...).verity.addon.efiin<uki>.efi.extra.d/) built per slot withsystemd.verity_usr_data=PARTUUID=<usr>andsystemd.verity_usr_hash=PARTUUID=<hash>plus the slot root hash. Slot A is active by default; Trident switches slots by swapping the active addon. This lets one UKI boot either slot without re-signing.GRUB (secondary bootloader path).
grub_install.sh(declare--fs_uuid/--verity_uuid),rpm/grub_install.sh(read and inject the values), andrpm/grub.cfg(template@@FSUUID@@/@@VERITYUUID@@): build the verity cmdline fromsystemd.verity_usr_data=UUID=<fs>andsystemd.verity_usr_hash=UUID=<verity>instead of a hard-coded PARTUUID + hash-offset.Trident enablement
rpm/package_catalog.yaml: add thetridentpackage.rpm/build_image_util.sh: enabletridentd.socket.arm64 test-image boot reliability
vm_image_util.sh: addinstall_uki_timeout_addon, a UKI addon that raisessystemd.default_device_timeout_sec=120. Scoped to the arm64 kola test image only; it fixes a CI-only QEMU-TCG emulation flake where udev cannot initialise the ESP/OEM/usr-verity devices within the default device timeout and the VM drops to an emergency shell. Production and amd64 boot behaviour are unchanged.Docs
acl/docs/architecture.md: document the dedicated hash partition and slot-agnostic verity boot.Required PRs
Validation