[antsok]

Summary

Repair the plugin-owned CDI specification before health advertisement and allocation, and refuse workloads when device usability or repair cannot be established. Provide the trusted plugin with explicit hypervisor access through a separate, non-privileged bootstrap device-plugin DaemonSet.

Refs #13 and sokolaidev/maf-extensions#1423.

Changes

• Atomically repair missing, malformed or stale CDI specifications while leaving unrelated files untouched.
• Validate host-device identity and perform bounded KVM VM-creation and MSHV partition-creation checks before advertising healthy devices or allocating them. The MSHV check uses the backward-compatible ioctl arguments with kernel-default CPU/XSAVE features, then closes the partition without initializing it or allocating guest memory or vCPUs.
• Keep caller cancellation separate from shared health, broadcast health transitions to active streams, and bound registration retries.
• Require successful registration, an active health stream and a successful device check for readiness.
• Validate host KVM metadata in two CI phases. KIND startup may leave metadata unchanged or change only the GID to the node's own kvm group. Plugin testing and cleanup must preserve the validated metadata exactly. If startup validation does not complete, the cleanup check uses the original snapshot.
• Keep the existing KIND configuration unchanged.

Verification

• At 11c6f0a, hosted lint, build, device-plugin tests and Hyperlight guest execution passed. Integration Tests failed at the final metadata comparison because the host KVM GID differed.
• For the revised CI assertions at 791d799, actionlint, YAML parsing and git diff --check passed. The exact Bash assertions passed 24 fixture cases covering accepted startup transitions, dynamic node GIDs, unexpected changes to each metadata field, failed or malformed group lookup, missing/non-character devices, and cleanup after early failure.
• DCO passed for 791d799 and 0d3e036.
• For the MSHV probe at 0d3e036, Linux Go 1.25.14 race tests, vet, formatting after checkout newline normalization, amd64/arm64 builds, and git diff --check passed. Tests cover request layout, creation errors, partition/device descriptor cleanup, and rejection of /dev/null as an MSHV device. The non-hypervisor regression failed against the preceding implementation as expected.
• The opt-in MSHV live test disables test stubs and correctly fails when /dev/mshv is absent. Successful partition creation on a real MSHV host has not been verified.
• The PR Validation run for 791d799 required maintainer approval at the last check. Hosted validation for the new MSHV commit is pending.
• Earlier live validation, its tested commit, configuration and limitations remain documented in the existing validation report.

Notes for reviewers

Review the bootstrap device-access design, CDI repair and health transitions, and the two-phase metadata check. Reserve hyperlight.dev/device-access for trusted infrastructure; workloads continue to request hyperlight.dev/hypervisor.

MSHV now checks partition creation rather than only device identity and read/write access. Real MSHV validation is still pending; from device-plugin/, run TEST_MSHV_LIVE=1 go test -count=1 -run '^TestMSHVLivePartitionCreation ./... on an MSHV host. This does not test guest execution. Local assertions and earlier live evidence do not establish a passing hosted workflow for the current commit.

[antsok]

Standard AKS verification of upstream PR #14

Verified on 2026-09-24 using source 52bf3f203cfca36843afb36d18b0551b1f6ca00e and candidate OCI digest sha256:b89483a3ed73853afb7f5070d1b53e2c9a8d6f77b1e1ff1ce04cfdc602ada9a1.

One temporary, tainted Standard_D4ads_v5 node ran AKS 1.35.7, Ubuntu 24.04.5, kernel 6.8.0-1067-azure and containerd 2.3.3-2. Both candidate DaemonSets ran without privilege, with all capabilities dropped, RuntimeDefault seccomp and no service-account token. Runtime inspection confirmed zero effective capabilities. A temporary privileged diagnostic pod performed fault injection, kubelet restart and host-process checks; it was removed during cleanup. No RBAC or admission-policy changes were made. Two advertised slots permitted an existing guest plus a fresh validation guest; this is not a capacity benchmark.

Check	Live result
Real guest execution	12 positive application runs passed, including state reset, identity rejection and fresh execution after disposal. Applications ran as UID/GID 65534 under Restricted admission.
Missing, stale, malformed CDI	Automatic exact-content repair; observed recovery in 22.3–32.2 seconds including polling overhead. A fresh guest passed after each.
Non-regular CDI path	Readiness failed and advertised capacity became zero. Removing the obstruction recovered without restarting the plugin.
Read-only CDI storage	Missing-file repair failed deterministically; liveness remained healthy, readiness failed, and kubelet refused a new allocation. Restoring exact bytes recovered without restarting the plugin.
Missing host identity	Removing a synthetic character-device entry from an isolated host-device view withdrew capacity; restoration recovered without restarting the plugin. The real host device was untouched.
Existing unusable device	A per-container /dev/null mapping at /dev/kvm, with matching synthetic identity, failed KVM_GET_API_VERSION. Allocation was refused; restoring the normal manifest allowed fresh execution.
Registration failure	An isolated empty socket directory caused bounded registration timeouts and no readiness. Restoring the kubelet socket mount recovered.
Restarts	Plugin pod replacement preserved the guest. A real kubelet restart re-registered the plugin in 5.01 seconds and allowed fresh guest execution, but the supervised existing guest exited 70 during the restart.
Atomic replacement during allocation	6813 concurrent reads observed only complete stale or desired files, with zero read errors. Allocation-triggered repair was first observed after 2.93 seconds, before the next periodic check; the new guest passed.
Preservation	Unrelated CDI sentinel unchanged; real host KVM UID/GID, mode and device number unchanged. Continuous guests retained state across CDI, device-view and plugin-restart faults. Two runs exited successfully. The third was interrupted by the real kubelet restart and exited 70; its counter reached 246 before the interruption.
Cleanup	15 application pod UIDs checked: no surviving host processes, application pods or owner ledgers. Both temporary namespaces and the temporary node pool removed. Original node labels and original pod UIDs/restart counts unchanged.

Runtime image: sha256:03a33cee7cc4b7f58e777adc33fab7690cf87e5c13f5cdaaac465a7ce62f5339 (maf-sandbox-hyperlight 0.5.0, matched Hyperlight SDK/backend/guest 0.7.0 and Python 3.13.12). No running test resources remain.

The local harness needed two timing corrections: wait for asynchronous node-capacity publication, and exclude terminating DaemonSet pods when selecting the replacement. Their cleanup paths passed; the affected recovery/restart checks subsequently passed. No product source was changed. The downstream session supervisor deliberately retires a session when its controller stream closes or its lease expires. Exit 70 during kubelet restart is consistent with that contract; the exact retirement trigger was not captured. The existing session was not resumed. Plugin re-registration, fresh allocation and exact-owner cleanup passed. This observation does not establish a device-plugin recovery defect; preserving sessions across transport interruptions would be separate downstream lifecycle work.

Scope limits: one Standard AKS KVM node, no AKS Automatic/KIND/MSHV validation, no production capacity claim, and no physical host-device removal or host permission changes. The production upstream pin remains unchanged pending upstream acceptance. The exact-head PR Validation workflow still requires maintainer approval; these live results do not make CI green.

[Copilot]

Copilot review overview

🟡 Changes recommended

Deployment manifests still need capability dropping and an explicit termination-log path.

Get a fresh assessment by requesting another Copilot review.

Review effort: Lite
Findings: 2 · 2

Open (4)

• Drop all capabilities from the root plugin container · New
• Drop all capabilities from the root plugin container · New
• Drop all capabilities from the root bootstrap container · New
• Drop all capabilities from the root bootstrap container · New

What changed in this PR

This PR repairs CDI state and gates Hyperlight allocation on verified device readiness using scoped bootstrap access.

Changes:

• Adds bounded device probing, CDI reconciliation, cancellation-safe health tracking, and readiness checks.
• Adds bootstrap DaemonSets, host-device validation, regression tests, and CI verification.
• Requires capability dropping for plugin containers and an explicit termination-log path in the manifests.

File	Summary
docs/​architecture.md	Documents scoped access and readiness behavior.
device-plugin/​readiness_regression_test.go	Tests cancellation and host-device recovery.
device-plugin/​probe_linux.go	Adds bounded probing and device identity checks.
device-plugin/​main.go	Implements readiness, CDI repair, registration, and allocation.
device-plugin/​main_test.go	Adds CDI, readiness, registration, and bootstrap tests.
deploy/​manifests/​device-plugin.yaml	Updates DaemonSets with scoped access and health probes.
deploy/​local/​device-plugin.yaml	Mirrors deployment changes for local environments.
.github/​workflows/​pr-validation.yml	Adds readiness tests and device metadata verification.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[antsok]

Copilot review comments are addressed

[simongdavies]

@antsok Thanks very much for the PR, just a couple of minor comments, the main issue is the CI failure - Thanks !

[simongdavies]

Could we do something similar for mshv e.g. https://github.com/rust-vmm/mshv/blob/68af39fde2dd0c3bcf7b09ef088d897975ce6f00/mshv-ioctls/src/ioctls/system.rs#L356-L363

[antsok]

Implemented in 0d3e036. MSHV readiness now creates a non-isolated partition with the backward-compatible MSHV_CREATE_PARTITION arguments and immediately closes its descriptor, within the existing two-second helper deadline. It uses kernel-default CPU/XSAVE features and does not initialize the partition or allocate guest memory or vCPUs.

Code review is complete, and Linux race tests, vet, formatting, and amd64/arm64 builds pass. Regression tests cover ioctl failures, descriptor cleanup, and rejection of a non-hypervisor device; the opt-in live test disables test stubs. Actual MSHV partition creation remains unverified because neither my WSL nor Docker environment exposes /dev/mshv.

Could we use the main Hyperlight repository's Azure Linux MSHV runner pools to validate this change, or is another MSHV environment available for this repository? From device-plugin/, the check is TEST_MSHV_LIVE=1 go test -count=1 -run '^TestMSHVLivePartitionCreation$' ./....

[simongdavies]

Yes we could use one the hyperlight mshv node pools for that, they are available to all repos in the org, lets do that as a follow up.

[simongdavies]

I created #22 to track this.

[antsok]

Updated the CI check to accept KIND’s expected change of /dev/kvm’s group to the node’s own kvm GID, while requiring its permissions, owner and device numbers to remain unchanged. The check then requires exact metadata preservation through plugin testing and cleanup, without changing the existing KIND configuration.