openhcl: reseal VMGS hardware protector after live migration

[Ming-Wei Shih (mingweishih)]

Summary

Reseal the active VMGS datastore key after GET reports live migration completion, so a hardware protector bound to the source platform can be replaced with one bound to destination hardware.

• Wire NOTIFY_POST_LIVE_MIGRATION to a lifecycle-managed worker using bounded, coalesced notifications; the GET callback performs no hardware or storage I/O.
• Derive fresh hardware keys using the destination's local report SVN and the configured sealing policy, and seal the unchanged active DEK. No DEK rotation, remote key release, or GET protocol changes.
• Verify the candidate with fresh hardware derivations before and after persistence. Serialize the active-key comparison, protector write, and final flush through the VMGS broker.
• Retry failed event-triggered recovery with bounded exponential backoff and jitter, preserve events received during an attempt, and drain in-flight I/O before stop/teardown.
• Add typed runtime sealing helpers, regression tests, rate-limited diagnostics, and Guide documentation.

Scope and limitations

This is the event-driven change only. Recurring verification and ordinary startup checks are deferred to a separate follow-up. Memory-preserving live migration retains the resident worker, trusted runtime TCB floor, and pending recovery obligations. Serialized servicing/save/restore is unsupported for VMs with this worker: its save returns SaveError::NotSupported, which propagates to VM save, and restore rejects saved state. There is no saved-state reconstruction path or reconstruction-triggered durable rewrite; reconstructing a floor from destination reports or host-controlled VMGS would discard the trusted source floor.

A completely missed migration notification has no periodic recovery coverage in this PR. Recovery remains best effort: a crash after migration but before durable destination resealing may leave hardware-only VMGS unrecoverable. No encryption or sealing-policy downgrade is introduced.

Validation

Initial implementation validation (before subsequent TCB-floor and review changes) in a clean worktree using the OpenHCL entrypoint's normal crypto/TPM feature selection:

• cargo check --all-targets for the modified crates and openvmm_hcl
• cargo clippy --all-targets with -D warnings
• cargo doc --no-deps
• cargo nextest run --profile agent: 168 tests passed, 0 skipped
• cargo xtask fmt --fix: all passes succeeded

Coverage includes SNP/TDX protector verification, same-SVN hardware-secret changes, unchanged-DEK resealing, idle behavior without notifications, coalescing, retry backoff, events/migration during I/O, stale-key guards, flush failures, and stop/drain behavior and rejection of serialized save/restore.

Real-hardware live migration has not yet been tested.

[Copilot]

🔵 Needs a closer look

The change spans complex migration, hardware sealing, persistence, and lifecycle behavior, with review comments still outstanding.

Pull request overview

Adds event-driven VMGS hardware-protector resealing after live migration while preserving the active DEK and coordinating retries, persistence, and shutdown.

Changes:

• Adds migration notification handling and a lifecycle-managed resealing worker.
• Adds guarded VMGS broker operations and runtime sealing verification.
• Adds regression tests, diagnostics, dependency updates, and documentation.

File summaries

File	Summary
vm/vmgs/vmgs/src/vmgs_impl.rs	Exposes active-key retrieval and flushing.
vm/vmgs/vmgs/Cargo.toml	Updates VMGS package dependencies.
vm/vmgs/vmgs_broker/src/client.rs	Adds guarded key and write APIs.
vm/vmgs/vmgs_broker/src/broker/tests.rs	Tests broker guards and flush behavior. Nit (2 votes): remove the unused test_with_tracing::test import.
vm/vmgs/vmgs_broker/src/broker.rs	Implements serialized guarded writes.
vm/vmgs/vmgs_broker/Cargo.toml	Updates broker dependencies. Nit (1 vote): remove the unused test_with_tracing dev-dependency.
openhcl/underhill_core/src/worker.rs	Wires migration notifications and starts the reseal worker.
openhcl/underhill_core/src/lib.rs	Registers the hardware-reseal module.
openhcl/underhill_core/src/hardware_reseal/tests.rs	Tests notification, retry, I/O, and shutdown behavior.
openhcl/underhill_core/src/hardware_reseal.rs	Implements resealing, retries, persistence, and lifecycle handling.
openhcl/underhill_core/src/dispatch/mod.rs	Drains resealing during teardown.
openhcl/underhill_attestation/src/vmgs.rs	Shares VMGS protector parsing.
openhcl/underhill_attestation/src/runtime_sealing/tests.rs	Tests runtime sealing policies and failures.
openhcl/underhill_attestation/src/runtime_sealing.rs	Creates and verifies runtime protectors.
openhcl/underhill_attestation/src/lib.rs	Exports runtime sealing helpers.
openhcl/underhill_attestation/src/hardware_key_sealing.rs	Adds typed sealing errors and rate-limited diagnostics.
openhcl/underhill_attestation/Cargo.toml	Adds runtime sealing dependencies.
Guide/src/reference/architecture/openhcl/processes.md	Documents VMGS hardware resealing.
Cargo.lock	Updates dependency resolution.

Review details

Suppressed comments (1)

vm/vmgs/vmgs_broker/Cargo.toml:35

• This dev-dependency has no consumer: the only test_with_tracing reference in this crate is the unused import in broker/tests.rs, while every test uses #[async_test]. Remove this dependency along with that import so the workspace formatter does not delete it or leave unused test plumbing behind.

test_with_tracing.workspace = true

• Files reviewed: 18/19 changed files
• Comments generated: 1
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

🔵 Needs a closer look

Real-hardware live migration is untested, and the change includes complex lifecycle, retry, and persistence behavior.

Review details

• Files reviewed: 18/19 changed files
• Comments generated: 0 new
• Review effort level: Lite

[github-actions]

1 Petri tests failed (1 unstable)

[Copilot]

🔵 Needs a closer look

The migration path has unresolved reset behavior and lacks real source/destination hardware validation, requiring changes and final human review.

Review details

Suppressed comments (2)

openhcl/underhill_core/src/hardware_reseal.rs:224

• This path is the production handoff from the GET migration event to the destination TEE, but the tests only call MigrationNotification::notify() directly and use mock TEE/report behavior. Please add an end-to-end/manual validation that delivers NOTIFY_POST_LIVE_MIGRATION through GET and verifies resealing across a real source/destination hardware transition, or keep this as an explicit pre-approval requirement because a missed callback or report/SVN mismatch is not covered by the unit tests.

                let protector = runtime_sealing::create_protector(&*tee, &config, &key)?;
                // Validate with a second derivation, not the seal-time keys.
                anyhow::ensure!(
                    runtime_sealing::protector_matches(&*tee, &config, &protector, &key)?,
                    "hardware changed while constructing the protector"
                );
                Ok(protector)
            })

openhcl/underhill_core/src/hardware_reseal.rs:274

• StateUnit::reset is specified to return the unit to its initial state, but this only sets force_reseal, leaving prior failures/backoff timestamps intact and scheduling a hardware/storage attempt on the next start even when no migration notification is pending. Reset the scheduler itself so ordinary reset does not bypass the event-only trigger; the independent notification latch will still preserve a real pending event.

    async fn reset(&mut self) -> anyhow::Result<()> {
        self.schedule.force_reseal = true;
        Ok(())

• Files reviewed: 18/19 changed files
• Comments generated: 0 new
• Review effort level: Lite

[github-actions]

6 Petri tests failed (4 unstable)

[Copilot]

🔵 Needs a closer look

Reset scheduling behavior remains unresolved in the lifecycle-managed hardware resealing worker.

Review details

Suppressed comments (1)

openhcl/underhill_core/src/hardware_reseal.rs:273

• StateUnit::reset is required to reset the unit to its initial state, but this leaves the existing schedule untouched and sets force_reseal. A reset after a failed attempt therefore preserves its retry/backoff and causes a hardware/VMGS operation on the next start even without a migration notification, which conflicts with the event-only lifecycle. Please reset the schedule (and define how pending notifications should be handled) or make this an explicitly documented recovery trigger.

        self.schedule.force_reseal = true;

• Files reviewed: 18/19 changed files
• Comments generated: 0 new
• Review effort level: Lite

[github-actions]

2 Petri tests failed (0 unstable)

[Copilot]

🔵 Needs a closer look

This is security-critical hardware-sealing and migration logic, and real-hardware live migration remains unvalidated.

Review details

• Files reviewed: 25/26 changed files
• Comments generated: 0 new
• Review effort level: Lite

[Copilot]

🟡 Changes recommended

Critical final-flush and early-notification issues remain, along with a saved-state reconstruction mismatch.

Get a fresh assessment by requesting another Copilot review.

Review details

Suppressed comments (1)

openhcl/underhill_core/src/hardware_reseal.rs:292

• The PR description says saved-state reconstruction requests a durable rewrite to preserve an earlier failed-flush obligation, but this state unit unconditionally returns SaveError::NotSupported (and restore rejects all state). LoadedVm::save propagates that failure from state_units.save, so VMs with this worker cannot be serialized and no rewrite is requested. Either implement the described reconstruction path or update the scope/Guide to state that servicing is unsupported.

    async fn save(&mut self) -> Result<Option<SavedStateBlob>, SaveError> {
        // Memory-preserving migration retains the floor. Serialized servicing
        // must not reconstruct a new, potentially lower floor from VMGS/report.
        Err(SaveError::NotSupported)

• Files reviewed: 28/29 changed files
• Comments generated: 2
• Review effort level: Lite

[github-actions]

2 Petri tests failed (1 unstable)

[Copilot]

🟡 Changes recommended

A critical TDX TCB-floor issue and a moderate reset-triggered resealing issue remain unresolved.

Get a fresh assessment by requesting another Copilot review.

Review details

Suppressed comments (1)

openhcl/underhill_core/src/hardware_reseal.rs:285

• StateUnits::reset is used for an ordinary VM reset (openvmm/openvmm_core/src/worker/dispatch.rs:4271), so setting force_reseal unconditionally schedules a fresh hardware derivation, protector write, and flush after every reset even when the previous event already succeeded. That violates the event-driven/no-ordinary-verification behavior and adds unnecessary storage I/O; preserve only the existing retry obligation or a latched notification instead of forcing every reset to reseal.

        self.schedule.force_reseal = true;

• Files reviewed: 31/32 changed files
• Comments generated: 1
• Review effort level: Lite

[github-actions]

3 Petri tests failed (0 unstable)

[Copilot]

Copilot review overview

🔵 Needs a closer look

The event-driven hardware resealing and lifecycle-sensitive attestation changes require final human review.

Review effort: Lite
Findings: 1

Open (1)

• set_post_live_migration_callback only enqueues a control message; it does not install the…

Resolved since last review (1)

• TeeTcbSvn is defined in vm/x86/x86defs/src/tdx.rs:847-858 as minor SVN (byte 0), major SVN…

[Copilot]

Copilot review overview

🟡 Changes recommended

The SNP/Milan/Genoa TCB byte layout is incorrect; correct it and update the layout/ordering tests before approval.

Get a fresh assessment by requesting another Copilot review.

Review effort: Lite
Findings: 1

Open (1)

• Correct Milan/Genoa TCB SVN field layout · New

Resolved since last review (1)

• set_post_live_migration_callback only enqueues a control message; it does not install the…

[github-actions]

1 Petri tests failed (1 unstable)