Detect and quarantine wedged vGPU VFs

[yummybomb]

Summary

Killing the process holding a vendor VFIO VF during guest driver init (~8s window) silently corrupts the VF: every later boot on it fails while sysfs, NVML, and the vGPU plugin all look healthy. #321 removed hypeman's own kill triggers; this layer detects the wedges external kills (OOM, operator, deploy restarts) still cause, and takes the VF out of placement.

Detection — guest agent report, no lifecycle change. On a wedged VF the guest driver is present, trying, and failing: its kernel loops NVRM: ... RmInitAdapter failed! in the guest kernel log. The guest agent watches /dev/kmsg for that line and reports it as a HYPEMAN-GPU-INIT-FAILED marker — the same guest→host channel as the existing HYPEMAN-* markers, landing in the instance's logs/app.log. A new VGPUSentinelController tails that file for every instance holding a vendor VFIO assignment and convicts on the marker; the full marker shape is required since a bare token could appear in echoed exec command lines. The agent only watches when an NVIDIA PCI function is present, throttles re-emission to one per 30s, and matches the full kernel-line shape, never the driver-build-specific (0x22:0x65:884) tuple. A no-driver image never produces the kernel line and can never be convicted. An image that skips the guest agent does not report — v1 accepts that coverage bound; the raw kernel line still reaches app.log for manual diagnosis.

Quarantine. Convictions persist to <data-dir>/gpu/vf-health.json (keyed by VF address, survives restarts) under the placement lock. One wedge produces one record no matter how many victim boots or controller restarts report it. The store fails closed on an unreadable state file: mutations are refused, and vGPU placement and advertised availability are disabled (with load retried on each attempt) rather than treating the empty in-memory set as healthy and returning quarantined VFs to rotation. A conviction whose persist fails is rolled back from memory so the next report retries it instead of reading as a repeat. Start archives the previous boot's serial log before persisting a new assignment (fatal on failure for GPU instances), so a scan can never replay the prior boot's report against a freshly assigned VF. A conviction burst (>3 in 15m) pauses auto-conviction so a systemic non-wedge init failure (e.g. a driver-mismatch rollout) can't quarantine the fleet; a paused conviction retries on the agent's next re-emission and lands once the window clears.

Placement.

• Quarantined VFs are excluded from selection and from advertised profile availability. Healthy siblings on the same card stay available.
• Cards are ordered by quarantined-VF count before load: a card with a wedged VF becomes overflow-only, so it drains toward the SR-IOV recovery cycle instead of staying warm.
• The final VF pick within the chosen card is randomized: the previous lowest-address tiebreak routed every first create on an idle host to the same VF, making one wedged VF look like a total GPU outage.

Observability. Conviction logs at error level; hypeman_instances_vgpu_sentinel_convictions_total (result=convicted|suppressed) and hypeman_instances_vgpu_quarantined_vfs gauge. The controller idles on hosts without the vendor VFIO framework.

GPU.md documents the new behavior and the manual recovery flow (DCGM quiesce → SR-IOV cycle → restart services → verification boot → clear the VF's entry in vf-health.json → restart hypeman immediately).

Out of scope (follow-up layer)

An operator force-cycle endpoint (DCGM quiesce → sriov-manage -d/-e → verification boot → clear quarantine). Until then recovery stays the documented manual runbook.

Testing

• Unit tests: kmsg record parsing/matching in the guest agent; marker pattern (incl. echoed-command and raw-kernel-line negative cases); tail behavior across partial lines, oversized lines, log archival, and new assignments; conviction retry on persist failure; brake pause-and-resume; already-quarantined rescans after controller restarts; quarantine store persistence, one-record-per-wedge idempotence, and load-failure refusal; placement exclusion / card bias / tiebreak; availability accounting.
• go test -race ./lib/devices ./lib/paths ./lib/providers ./lib/system/guest_agent and the vGPU-related lib/instances tests pass locally; go vet clean. The full lib/instances suite fails only on environment-dependent VM/network tests, identically on the unmodified base.
• The detection signal (kernel-line timing, serial-console delivery into app.log) and the recovery sequence were validated on real L40S hardware by deliberately wedging VFs; the guest-agent watcher and the controller have not yet run against a live wedge end-to-end — that run should happen on the dev GPU host before this merges.

---

Note

Medium Risk
Changes vGPU placement, advertised GPU capacity, and guest-agent reporting. A false conviction or store-load failure can shrink GPU inventory until operators clear vf-health.json and restart.

Overview
Automatically detects wedged vendor VFIO vGPU VFs (guest RmInitAdapter timeout with healthy sysfs) and takes them out of placement until an operator SR-IOV-cycles the parent GPU.

The guest agent watches /dev/kmsg on NVIDIA guests and emits a HYPEMAN-GPU-INIT-FAILED marker. A new VGPUSentinelController tails instance app.log for that marker, persists quarantines in gpu/vf-health.json (fail-closed on a corrupt store), and pauses auto-conviction if more than 3 hits land in 15 minutes.

Placement skips quarantined VFs, demotes their parent GPU to overflow-only, randomizes VF pick within a card (so one undetected wedge is not every first create), and excludes quarantined VFs from advertised availability. GPU instance start now fails if the previous boot log cannot be archived, so a stale marker cannot convict a new VF. Recovery remains a documented DCGM-quiesce + SR-IOV cycle + clear-file + restart; there is no operator API yet.

^{Reviewed by Cursor Bugbot for commit 9a90223. Bugbot is set up for automated code reviews on this repo. Configure here.}

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 7d54fb9. Configure here.}