Skip to content

fix: repair CDI and verify allocation health with scoped device access #14

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
simongdavies merged 7 commits into from
Oct 22, 2026
Merged

fix: repair CDI and verify allocation health with scoped device access #14

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
8ac6e3e
fix: reconcile CDI state and report allocation readiness
antsok Sep 24, 2026
f2dd612
fix: grant scoped device access and isolate readiness from cancellation
antsok Sep 24, 2026
b4d0d7f
fix: validate host devices and reap canceled probes before retry
antsok Sep 24, 2026
52bf3f2
test: share device plugin fixtures and error checks
antsok Sep 24, 2026
11c6f0a
fix: drop all capabilities from device plugin containers
antsok Sep 24, 2026
791d799
ci: allow KIND's expected KVM group change before plugin tests
antsok Oct 21, 2026
0d3e036
fix: verify MSHV partition creation before device readiness
antsok Oct 21, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
48 changes: 37 additions & 11 deletions .github/workflows/pr-validation.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,10 @@ jobs:
- name: Run linters
run: just lint-strict

- name: Test device readiness and recovery
working-directory: device-plugin
run: go test -race -count=1 -timeout=90s ./...

build:
name: Build
runs-on: ubuntu-latest
Expand Down Expand Up @@ -74,19 +78,12 @@ jobs:
go-version: '1.25'
cache-dependency-path: device-plugin/go.sum

- name: Enable KVM
- name: Check KVM
Comment thread
simongdavies marked this conversation as resolved.
Resolved
id: kvm-snapshot
run: |
echo 'KERNEL=="kvm", GROUP="kvm", MODE="0666", OPTIONS+="static_node=kvm"' | sudo tee /etc/udev/rules.d/99-kvm4all.rules
sudo udevadm control --reload-rules
sudo udevadm trigger --name-match=kvm

# Verify KVM is available
if [ ! -e /dev/kvm ]; then
echo "::error::KVM is not available on this runner"
exit 1
fi
test -c /dev/kvm
stat -c '%a:%u:%g:%t:%T' /dev/kvm > "$RUNNER_TEMP/kvm-before-kind"
ls -la /dev/kvm
echo "✓ KVM enabled"

- name: Install just
uses: extractions/setup-just@v2
Expand All @@ -104,6 +101,22 @@ jobs:
just local-up
echo "✓ KIND cluster created"

- name: Verify KIND KVM metadata transition
shell: bash
run: |
node=$(kind get nodes --name hyperlight)
kind_gid=$(docker exec "$node" getent group kvm | cut -d: -f3)
[[ "$kind_gid" =~ ^[0-9]+$ ]]
test -c /dev/kvm
stat -c '%a:%u:%g:%t:%T' /dev/kvm > "$RUNNER_TEMP/kvm-after-kind"
# KIND's systemd may assign its own kvm group to the bind-mounted device.
IFS=: read -r mode uid _ major minor < "$RUNNER_TEMP/kvm-before-kind"
printf '%s:%s:%s:%s:%s\n' "$mode" "$uid" "$kind_gid" "$major" "$minor" > "$RUNNER_TEMP/kvm-expected-kind"
if ! cmp -s "$RUNNER_TEMP/kvm-before-kind" "$RUNNER_TEMP/kvm-after-kind"; then
diff -u "$RUNNER_TEMP/kvm-expected-kind" "$RUNNER_TEMP/kvm-after-kind"
fi
cp "$RUNNER_TEMP/kvm-after-kind" "$RUNNER_TEMP/kvm-before-plugin"

- name: Build device plugin
run: |
just plugin-build
Expand All @@ -121,6 +134,7 @@ jobs:

- name: Wait for device plugin to be ready
run: |
kubectl rollout status daemonset/hyperlight-device-access -n hyperlight-system --timeout=120s
echo "Waiting for device plugin DaemonSet..."
kubectl rollout status daemonset/hyperlight-device-plugin -n hyperlight-system --timeout=120s

Expand All @@ -138,6 +152,7 @@ jobs:

- name: Run device plugin tests
run: |
kubectl exec -n hyperlight-system daemonset/hyperlight-device-plugin -- hyperlight-device-plugin --probe-hypervisor=kvm --probe-path=/dev/kvm
./scripts/test.sh plugin
./scripts/test.sh nodes
./scripts/test.sh kvm
Expand Down Expand Up @@ -184,3 +199,14 @@ jobs:
if: always()
run: |
just local-down || true

- name: Verify host KVM metadata after cleanup
if: always() && steps.kvm-snapshot.outcome == 'success'
run: |
baseline="$RUNNER_TEMP/kvm-before-kind"
if [ -f "$RUNNER_TEMP/kvm-before-plugin" ]; then
baseline="$RUNNER_TEMP/kvm-before-plugin"
fi
test -c /dev/kvm
stat -c '%a:%u:%g:%t:%T' /dev/kvm > "$RUNNER_TEMP/kvm-after"
diff -u "$baseline" "$RUNNER_TEMP/kvm-after"
99 changes: 92 additions & 7 deletions deploy/local/device-plugin.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -71,6 +71,8 @@ spec:
# KIND workaround: move termination log out of /dev
terminationMessagePath: /tmp/termination-log
env:
- name: HOST_DEVICE_DIR
value: /host-dev
# Number of concurrent hypervisor allocations per node.
# This is NOT physical devices - the hypervisor device is shared.
# Each allocation grants access to /dev/kvm or /dev/mshv.
Expand All @@ -83,38 +85,51 @@ spec:
- name: DEVICE_GID
value: "${DEVICE_GID}"
securityContext:
capabilities:
drop: [ALL]
privileged: false
readOnlyRootFilesystem: true
allowPrivilegeEscalation: false
runAsUser: 0 # Required: write access to /var/run/cdi and /var/lib/kubelet/device-plugins
volumeMounts:
- name: host-dev
mountPath: /host-dev
readOnly: true
# Kubelet device plugin socket directory
- name: device-plugin
mountPath: /var/lib/kubelet/device-plugins
# CDI spec directory
- name: cdi
mountPath: /var/run/cdi
# Host /dev to detect hypervisor device
- name: dev
mountPath: /dev
readOnly: true
resources:
requests:
memory: "32Mi"
cpu: "10m"
hyperlight.dev/device-access: "1"
Comment thread
antsok marked this conversation as resolved.
Resolved
limits:
memory: "64Mi"
cpu: "100m"
hyperlight.dev/device-access: "1"
livenessProbe:
exec:
command:
- test
- -S
- /var/lib/kubelet/device-plugins/hyperlight.sock
- hyperlight-device-plugin
- --health-check=liveness
initialDelaySeconds: 10
periodSeconds: 30
timeoutSeconds: 5
readinessProbe:
exec:
command:
- hyperlight-device-plugin
- --health-check=readiness
periodSeconds: 10
timeoutSeconds: 5
volumes:
- name: host-dev
hostPath:
path: /dev
type: Directory
- name: device-plugin
hostPath:
path: /var/lib/kubelet/device-plugins
Expand All @@ -123,6 +138,76 @@ spec:
hostPath:
path: /var/run/cdi
type: DirectoryOrCreate
terminationGracePeriodSeconds: 30

---
# Inventory only: kubelet grants the main plugin access to this one device.
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: hyperlight-device-access
namespace: hyperlight-system
labels:
app.kubernetes.io/name: hyperlight-device-access
app.kubernetes.io/part-of: hyperlight
spec:
selector:
matchLabels:
app.kubernetes.io/name: hyperlight-device-access
updateStrategy:
type: RollingUpdate
template:
metadata:
labels:
app.kubernetes.io/name: hyperlight-device-access
app.kubernetes.io/part-of: hyperlight
spec:
serviceAccountName: hyperlight-device-plugin
nodeSelector:
hyperlight.dev/enabled: "true"
priorityClassName: system-node-critical
containers:
- name: device-access
image: ${IMAGE}
imagePullPolicy: Always
args: ["--device-access"]
terminationMessagePath: /tmp/termination-log
Comment thread
antsok marked this conversation as resolved.
Resolved
securityContext:
capabilities:
drop: [ALL]
privileged: false
readOnlyRootFilesystem: true
allowPrivilegeEscalation: false
runAsUser: 0
volumeMounts:
- name: device-plugin
mountPath: /var/lib/kubelet/device-plugins
- name: dev
mountPath: /dev
readOnly: true
resources:
requests:
memory: "16Mi"
cpu: "10m"
limits:
memory: "32Mi"
cpu: "100m"
livenessProbe:
exec:
command: ["hyperlight-device-plugin", "--device-access", "--health-check=liveness"]
initialDelaySeconds: 10
periodSeconds: 30
timeoutSeconds: 5
readinessProbe:
exec:
command: ["hyperlight-device-plugin", "--device-access", "--health-check=readiness"]
periodSeconds: 10
timeoutSeconds: 5
volumes:
- name: device-plugin
hostPath:
path: /var/lib/kubelet/device-plugins
type: Directory
- name: dev
hostPath:
path: /dev
Expand Down
98 changes: 91 additions & 7 deletions deploy/manifests/device-plugin.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -53,6 +53,8 @@ spec:
image: ${IMAGE}
imagePullPolicy: Always
env:
- name: HOST_DEVICE_DIR
value: /host-dev
# Number of concurrent hypervisor allocations per node.
# This is NOT physical devices - the hypervisor device is shared.
# Each allocation grants access to /dev/kvm or /dev/mshv.
Expand All @@ -65,38 +67,51 @@ spec:
- name: DEVICE_GID
value: "${DEVICE_GID}"
securityContext:
capabilities:
drop: [ALL]
privileged: false
readOnlyRootFilesystem: true
allowPrivilegeEscalation: false
runAsUser: 0 # Required: write access to /var/run/cdi and /var/lib/kubelet/device-plugins
volumeMounts:
- name: host-dev
mountPath: /host-dev
readOnly: true
# Kubelet device plugin socket directory
- name: device-plugin
mountPath: /var/lib/kubelet/device-plugins
# CDI spec directory
- name: cdi
mountPath: /var/run/cdi
# Host /dev to detect hypervisor device
- name: dev
mountPath: /dev
readOnly: true
resources:
requests:
memory: "32Mi"
cpu: "10m"
hyperlight.dev/device-access: "1"
Comment thread
antsok marked this conversation as resolved.
Resolved
limits:
memory: "64Mi"
cpu: "100m"
hyperlight.dev/device-access: "1"
livenessProbe:
exec:
command:
- test
- -S
- /var/lib/kubelet/device-plugins/hyperlight.sock
- hyperlight-device-plugin
- --health-check=liveness
initialDelaySeconds: 10
periodSeconds: 30
timeoutSeconds: 5
readinessProbe:
exec:
command:
- hyperlight-device-plugin
- --health-check=readiness
periodSeconds: 10
timeoutSeconds: 5
volumes:
- name: host-dev
hostPath:
path: /dev
type: Directory
- name: device-plugin
hostPath:
path: /var/lib/kubelet/device-plugins
Expand All @@ -105,6 +120,75 @@ spec:
hostPath:
path: /var/run/cdi
type: DirectoryOrCreate
terminationGracePeriodSeconds: 30

---
# Inventory only: kubelet grants the main plugin access to this one device.
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: hyperlight-device-access
namespace: hyperlight-system
labels:
app.kubernetes.io/name: hyperlight-device-access
app.kubernetes.io/part-of: hyperlight
spec:
selector:
matchLabels:
app.kubernetes.io/name: hyperlight-device-access
updateStrategy:
type: RollingUpdate
template:
metadata:
labels:
app.kubernetes.io/name: hyperlight-device-access
app.kubernetes.io/part-of: hyperlight
spec:
serviceAccountName: hyperlight-device-plugin
nodeSelector:
hyperlight.dev/enabled: "true"
priorityClassName: system-node-critical
containers:
- name: device-access
image: ${IMAGE}
imagePullPolicy: Always
args: ["--device-access"]
securityContext:
capabilities:
drop: [ALL]
privileged: false
readOnlyRootFilesystem: true
allowPrivilegeEscalation: false
runAsUser: 0
volumeMounts:
- name: device-plugin
mountPath: /var/lib/kubelet/device-plugins
- name: dev
mountPath: /dev
readOnly: true
resources:
requests:
memory: "16Mi"
cpu: "10m"
limits:
memory: "32Mi"
cpu: "100m"
livenessProbe:
exec:
command: ["hyperlight-device-plugin", "--device-access", "--health-check=liveness"]
initialDelaySeconds: 10
periodSeconds: 30
timeoutSeconds: 5
readinessProbe:
exec:
command: ["hyperlight-device-plugin", "--device-access", "--health-check=readiness"]
periodSeconds: 10
timeoutSeconds: 5
volumes:
- name: device-plugin
hostPath:
path: /var/lib/kubelet/device-plugins
type: Directory
- name: dev
hostPath:
path: /dev
Expand Down
Loading
Loading