From d89841fdeb9294ce96f2789d9b1a29404368c6d8 Mon Sep 17 00:00:00 2001 From: Kevin Wang Date: Tue, 22 Sep 2026 07:34:17 -0700 Subject: [PATCH 01/25] Expose GPU claim status in measured event --- CHANGELOG.md | 1 + docs/attestation-tdx.md | 3 +- docs/security/security-model.md | 12 ++++++-- docs/tutorials/attestation-verification.md | 2 +- dstack/dstack-util/src/system_setup.rs | 36 ++++++++++++++++++---- 5 files changed, 44 insertions(+), 10 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index a08edfc36..e05698858 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -8,6 +8,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## [Unreleased] ### Added +- dstack-util: the measured `gpu-attestation` event now exposes aggregate `dbgstat` and `secboot` values from the NVIDIA-signed claims. `dbgstat` is `enabled` if any attested GPU reports enabled, while `secboot` is true only if every attested GPU reports secure boot; the event schema version is bumped to 3. - sdk: every SDK ships two clients, one per guest-agent surface, and nothing else. `DstackClientV0` speaks the frozen v0.5.11 API on the unversioned paths -- every method a v0.5.x SDK had, including `verify()` (which calls the server `Verify` RPC) and `emit_event()` (which surfaces the agent's removal error verbatim). `DstackClientV1` speaks `dstack.guest.v1` at `/v1` and has exactly its six methods: `issue_cert`, `get_key`, `attest`, `attest_gpu`, `info`, `version`. They are transport mirrors, not a compatibility layer: neither translates a call to the other, and each one's method set is its surface's method set. **v1 derives different key material than v0 for the same inputs** -- see `docs/guest-api-v1.md` for the migration. `TappdClient` is untouched. The blockchain adapters (Ethereum/viem, Solana) remain **v0-only** and stay typed against the v0 `GetKeyResponse`. v1 has no chain-related surface, deliberately: it returns key material, and what an application builds from those bytes is its own business rather than something the SDKs model. diff --git a/docs/attestation-tdx.md b/docs/attestation-tdx.md index 14c20bcf5..f9744e4ba 100644 --- a/docs/attestation-tdx.md +++ b/docs/attestation-tdx.md @@ -43,7 +43,8 @@ For a GPU launch, any `init-script-hash` events are followed by evaluation, `gpu-attestation`. The `gpu-policy-hash` payload is `SHA-256(JCS(requirements.gpu_policy))`, using `{}` when the field is omitted. The `gpu-attestation` payload is JSON containing the verified device count, -CC/DevTools state, and `evidence_sha256`. +CC/DevTools state, aggregate signed-claim `dbgstat` and `secboot`, and +`evidence_sha256`. The guest-agent returns the complete `nvattest` record captured during boot from `/v1/Attest`, when the request sets `include_boottime_gpu_evidence`, so a verifier can fetch the quote and the GPU evidence in one round trip. `AttestResponse.boottime_gpu_evidence` is a list of `GpuEvidenceBundle` (`{vendor, format, evidence}`); the boot record is the bundle whose `vendor` is `nvidia` and whose `format` is `nvidia-nvattest-boottime-json-v1`, and its `evidence` is hex-encoded bytes that decode to the exact UTF-8 `nvattest` output. It is not trustworthy by itself. (`/v1/AttestGpu` runs a *fresh* attestation against a caller nonce and returns bundles tagged `nvidia-nvattest-collect-evidence-json-v1`, a deliberately distinct format that a boot-record verifier does not appraise; its result is not bound to the TD and must not be used as remote evidence; only the boot-time record below is.) After verifying the TDX quote and replaying the event log to RTMR3, hash the *decoded* bundle bytes — `SHA-256(hex_decode(bundle.evidence))`, never the JSON string as returned nor a re-serialized form — and require the result to equal the `gpu-attestation` event's `evidence_sha256`. See [GPU Security for AI Workloads](./security/security-model.md#gpu-security-for-ai-workloads) for the event schema, ordering, Rego example, and platform differences. diff --git a/docs/security/security-model.md b/docs/security/security-model.md index 6e7427cea..beed20f01 100644 --- a/docs/security/security-model.md +++ b/docs/security/security-model.md @@ -118,7 +118,7 @@ For every NVML-enumerated GPU, dstack calls `Device::is_cc_enabled()` and `Devic ### Dual Attestation -GPU workloads require verification of both hardware components. The CPU TEE quote verifies the CVM and its measured guest code. NVIDIA-signed evidence, checked against NVIDIA RIMs and certificate status by `nvattest`, verifies the GPU appraisal. After the optional policy and ready-state operations succeed, dstack emits a `gpu-attestation` launch event before `system-ready`. Its versioned payload records the number of appraised devices, asserted CC/DevTools state, and SHA-256 of the complete nvattest JSON output (claims and detached EAT). On TDX, both `gpu-policy-hash` and `gpu-attestation` are append-only RTMR3 events; they are never derived from application-controlled `report_data`. +GPU workloads require verification of both hardware components. The CPU TEE quote verifies the CVM and its measured guest code. NVIDIA-signed evidence, checked against NVIDIA RIMs and certificate status by `nvattest`, verifies the GPU appraisal. After the optional policy and ready-state operations succeed, dstack emits a `gpu-attestation` launch event before `system-ready`. Its versioned payload records the number of appraised devices, asserted CC/DevTools state, aggregate signed-claim debug and secure-boot status, and SHA-256 of the complete nvattest JSON output (claims and detached EAT). On TDX, both `gpu-policy-hash` and `gpu-attestation` are append-only RTMR3 events; they are never derived from application-controlled `report_data`. For a successful TDX GPU launch, the GPU-relevant RTMR3 event order is: @@ -135,15 +135,23 @@ boot-mr-done ```json { - "version": 2, + "version": 3, "provider": "nvidia", "devices": 1, "cc_mode": "on", "devtools": false, + "dbgstat": "disabled", + "secboot": true, "evidence_sha256": "" } ``` +`devtools` is the aggregate NVML state and is true if any device has DevTools +mode enabled. `dbgstat` and `secboot` summarize the NVIDIA-signed claims: +`dbgstat` is `enabled` if any device reports enabled, and `secboot` is true only +if every device reports secure boot. The complete per-device claims remain in +the boot-time evidence committed by `evidence_sha256`. + `/v1/Attest` returns that complete boot-time `nvattest` record when the request sets `include_boottime_gpu_evidence`; it runs no new attestation. `AttestResponse.boottime_gpu_evidence` is a list of `GpuEvidenceBundle`, each carrying `vendor`, `format`, and `evidence`. The boot record is the bundle whose `vendor` is `nvidia` and whose `format` is `nvidia-nvattest-boottime-json-v1`; its `evidence` is hex-encoded bytes that decode to the exact UTF-8 nvattest output as the agent read it from disk. The other format, `nvidia-nvattest-collect-evidence-json-v1`, comes from `/v1/AttestGpu` and is fresh evidence against a caller nonce; the two are deliberately distinct because a verifier for one does not appraise the other. To bind the API result to TDX evidence: verify the quote, replay the event log to the quote's RTMR3, require exactly one pre-`system-ready` `gpu-attestation` event, decode its JSON payload, select the boot-time bundle by `format`, and compare `evidence_sha256` with `SHA-256(hex_decode(bundle.evidence))`. Hash the decoded bytes, not the JSON string as returned and not a re-serialized form: parsing and re-serializing changes the digest and breaks the comparison. Only after this comparison should the verifier inspect the returned claims. This exact-byte comparison includes any whitespace or trailing newline in the decoded record. A verifier must replay the measured event log, require exactly one `gpu-policy-hash` event immediately after `compose-hash`, and compare its 32-byte payload with the expected policy digest (`SHA-256(JCS({}))` for the omitted/default policy). When MrConfigV3 includes `gpu_policy_hash`, it must match the same digest. When GPU protection is required, the verifier must also require exactly one pre-`system-ready` `gpu-attestation` event with `devices > 0` and, when applicable, the expected deployment count. The raw `attestation.out` file is not trusted by itself; if it is supplied for inspection, its digest must match the `gpu-attestation` event. diff --git a/docs/tutorials/attestation-verification.md b/docs/tutorials/attestation-verification.md index 9808ef94c..fa6b3da3c 100644 --- a/docs/tutorials/attestation-verification.md +++ b/docs/tutorials/attestation-verification.md @@ -486,7 +486,7 @@ These are the standard events you'll see in the log: | `compose-hash` | SHA-256 of docker compose config | Should match `tcb_info.compose_hash` | | `init-script-hash` | SHA-256 of one init script; repeated in configured order (maximum 5) | Should match the independently approved script bytes | | `gpu-policy-hash` | SHA-256 of the JCS-canonicalized GPU policy (default `{}`) | Should match the expected `requirements.gpu_policy` digest | -| `gpu-attestation` | Verified GPU state and digest of the boot-time `nvattest` JSON | Required for an attested GPU launch; verify as described below | +| `gpu-attestation` | Verified GPU state, aggregate `dbgstat`/`secboot`, and digest of the boot-time `nvattest` JSON | Required for an attested GPU launch; verify as described below | | `instance-id` | Unique instance identifier | Should match `instance_id` from response | | `boot-mr-done` | Boot measurements complete | Marker event | | `os-image-hash` | Guest OS image hash | Should match `tcb_info.os_image_hash` | diff --git a/dstack/dstack-util/src/system_setup.rs b/dstack/dstack-util/src/system_setup.rs index 66839b4fa..ca6691ca5 100644 --- a/dstack/dstack-util/src/system_setup.rs +++ b/dstack/dstack-util/src/system_setup.rs @@ -1356,7 +1356,7 @@ async fn do_sys_setup(stage0: Stage0<'_>) -> Result<()> { mod gpu { use super::*; - const EVENT_VERSION: u32 = 2; + const EVENT_VERSION: u32 = 3; const POLICY_ENTRYPOINT: &str = "data.policy.nv_match"; /// Bound Rego evaluation so a runaway application policy cannot hang boot. const POLICY_TIMEOUT: Duration = Duration::from_secs(10); @@ -1370,6 +1370,8 @@ mod gpu { devices: u32, cc_mode: &'static str, devtools: bool, + dbgstat: NvidiaGpuDebugStatus, + secboot: bool, evidence_sha256: String, } @@ -1386,7 +1388,17 @@ mod gpu { } pub(super) fn event(&self, devtools: bool) -> Result> { - attestation_event(&self.output, self.devices, devtools) + let dbgstat = if self + .parsed_claims + .iter() + .any(|claim| claim.dbgstat == NvidiaGpuDebugStatus::Enabled) + { + NvidiaGpuDebugStatus::Enabled + } else { + NvidiaGpuDebugStatus::Disabled + }; + let secboot = self.parsed_claims.iter().all(|claim| claim.secboot); + attestation_event(&self.output, self.devices, devtools, dbgstat, secboot) } pub(super) fn verify_claim_policy( @@ -1437,7 +1449,7 @@ mod gpu { dbgstat: NvidiaGpuDebugStatus, } - #[derive(Debug, Deserialize, PartialEq, Eq)] + #[derive(Debug, Clone, Copy, Deserialize, Serialize, PartialEq, Eq)] #[serde(rename_all = "lowercase")] enum NvidiaGpuDebugStatus { Disabled, @@ -1590,13 +1602,21 @@ mod gpu { Ok(()) } - fn attestation_event(stdout: &[u8], devices: u32, devtools: bool) -> Result> { + fn attestation_event( + stdout: &[u8], + devices: u32, + devtools: bool, + dbgstat: NvidiaGpuDebugStatus, + secboot: bool, + ) -> Result> { let event = GpuAttestationEvent { version: EVENT_VERSION, provider: "nvidia", devices, cc_mode: "on", devtools, + dbgstat, + secboot, evidence_sha256: hex::encode(sha256(stdout)), }; serde_json::to_vec(&event).context("failed to serialize GPU attestation event") @@ -1953,13 +1973,17 @@ mod gpu { fn event_commits_to_complete_nvattest_output() { let nonce = "22".repeat(32); let output = nvattest_output(&nonce, 1); - let event: Value = - serde_json::from_slice(&attestation_event(&output, 1, true).unwrap()).unwrap(); + let event: Value = serde_json::from_slice( + &attestation_event(&output, 1, true, NvidiaGpuDebugStatus::Enabled, false).unwrap(), + ) + .unwrap(); assert_eq!(event["version"], EVENT_VERSION); assert_eq!(event["devices"], 1); assert!(event.get("policy").is_none()); assert_eq!(event["cc_mode"], "on"); assert_eq!(event["devtools"], true); + assert_eq!(event["dbgstat"], "enabled"); + assert_eq!(event["secboot"], false); assert_eq!(event["evidence_sha256"], hex::encode(sha256(&output))); } From 63d78fc44282e25e846dd3f06ba615ad0d714dfc Mon Sep 17 00:00:00 2001 From: Kevin Wang Date: Tue, 22 Sep 2026 08:19:09 -0700 Subject: [PATCH 02/25] fix(os): drop foreign systemd presets so dstack's policy is terminal Debian's 90-systemd.preset sorts before 99-dstack-default.preset, so its enable rules shadowed the `disable *` catch-all and enabled units such as systemd-sysext and systemd-confext. Remove non-dstack presets in postinst (mkosi applies presets afterwards) and enable systemd-networkd-wait-online explicitly, since network-online.target relied on the distro preset. --- os/mkosi/mkosi.postinst | 6 ++++++ .../usr/lib/systemd/system-preset/80-dstack.preset | 2 ++ os/mkosi/tests/acceptance.sh | 5 +++++ 3 files changed, 13 insertions(+) diff --git a/os/mkosi/mkosi.postinst b/os/mkosi/mkosi.postinst index c7622e7e1..d247e2680 100755 --- a/os/mkosi/mkosi.postinst +++ b/os/mkosi/mkosi.postinst @@ -19,3 +19,9 @@ rm -f "$B"/etc/ssh/ssh_host_*_key "$B"/etc/ssh/ssh_host_*_key.pub # Build outputs are merged after package installation, so refresh the final # rootfs cache after manually staged NVIDIA and container-runtime libraries. ldconfig -r "$B" + +# systemd applies the first matching rule across all preset files sorted by +# name, so a distro preset such as Debian's 90-systemd.preset would shadow the +# `disable *` in 99-dstack-default.preset. Keep dstack's policy the only one. +find "$B"/usr/lib/systemd/system-preset "$B"/etc/systemd/system-preset \ + -maxdepth 1 -name '*.preset' ! -name '*dstack*' -print -delete 2>/dev/null || true diff --git a/os/mkosi/mkosi.skeleton/usr/lib/systemd/system-preset/80-dstack.preset b/os/mkosi/mkosi.skeleton/usr/lib/systemd/system-preset/80-dstack.preset index fbc884d3b..b51076f2a 100644 --- a/os/mkosi/mkosi.skeleton/usr/lib/systemd/system-preset/80-dstack.preset +++ b/os/mkosi/mkosi.skeleton/usr/lib/systemd/system-preset/80-dstack.preset @@ -1,5 +1,7 @@ # SPDX-License-Identifier: Apache-2.0 enable systemd-networkd.service +# The only unit WantedBy=network-online.target, which dstack-prepare waits on. +enable systemd-networkd-wait-online.service enable systemd-resolved.service enable chrony.service enable docker.service diff --git a/os/mkosi/tests/acceptance.sh b/os/mkosi/tests/acceptance.sh index 493315a44..e2114dacc 100755 --- a/os/mkosi/tests/acceptance.sh +++ b/os/mkosi/tests/acceptance.sh @@ -124,6 +124,11 @@ test ! -e "$D/../common/rootfs/wg-checker.service" # only meaningful with a terminal disable. Without it, every package pulled in # by Packages= would start at boot with no diff to 80-dstack.preset. grep -q '^disable \*$' "$D/mkosi.skeleton/usr/lib/systemd/system-preset/99-dstack-default.preset" +# ...and it is only terminal if no foreign preset sorts before it. +grep -q "! -name '\*dstack\*'" "$D/mkosi.postinst" || { + echo 'mkosi.postinst must drop foreign preset files' >&2; exit 1; } +grep -qx 'enable systemd-networkd-wait-online.service' \ + "$D/mkosi.skeleton/usr/lib/systemd/system-preset/80-dstack.preset" # The TEE simulator serves synthetic quotes; its preset must not ship in prod. if grep -rq 'dstack-tee-simulator' "$D/mkosi.skeleton/"; then echo 'simulator preset must live in the dev profile skeleton' >&2 From 0244500de7e279c52765af285b2a3f07affb8d04 Mon Sep 17 00:00:00 2001 From: Kevin Wang Date: Tue, 22 Sep 2026 08:19:59 -0700 Subject: [PATCH 03/25] fix(dstack-util): measure os-image-hash once across a KMS failover os-image-hash was extended inside the per-URL request, before the CA cert was parsed. If that parse failed, the next KMS URL extended it again and RTMR3 could no longer be replayed. Measure only after a request fully succeeds, and report every KMS's error instead of just the first. --- dstack/dstack-util/src/system_setup.rs | 42 +++++++++++--------------- 1 file changed, 18 insertions(+), 24 deletions(-) diff --git a/dstack/dstack-util/src/system_setup.rs b/dstack/dstack-util/src/system_setup.rs index b58fa9546..c45b95edf 100644 --- a/dstack/dstack-util/src/system_setup.rs +++ b/dstack/dstack-util/src/system_setup.rs @@ -2306,7 +2306,7 @@ impl<'a> Stage0<'a> { self.shared.dir.join(APP_KEYS) } - async fn request_app_keys_from_kms_url(&self, kms_url: String) -> Result { + async fn request_app_keys_from_kms_url(&self, kms_url: String) -> Result<(AppKeys, Vec)> { info!("Requesting app keys from KMS: {kms_url}"); let tmp_ca = { info!("Getting temp ca cert"); @@ -2340,9 +2340,6 @@ impl<'a> Stage0<'a> { .await .context("Failed to get app key")?; - emit_runtime_event("os-image-hash", &response.os_image_hash) - .context("failed to extend os-image-hash to the launch measurement")?; - let (_, ca_pem) = x509_parser::pem::parse_x509_pem(tmp_ca.ca_cert.as_bytes()) .context("Failed to parse ca cert")?; let x509 = ca_pem.parse_x509().context("Failed to parse ca cert")?; @@ -2362,34 +2359,31 @@ impl<'a> Stage0<'a> { tmp_ca_cert: tmp_ca.temp_ca_cert, }, }; - Ok(keys) + Ok((keys, response.os_image_hash)) } async fn request_app_keys_from_kms(&self) -> Result { if self.shared.sys_config.kms_urls.is_empty() { bail!("No KMS URLs are set"); } - let keys = 'out: { - let mut error = anyhow!("unknown error"); - for (i, kms_url) in self.shared.sys_config.kms_urls.iter().enumerate() { - let kms_url = kms_rpc_url(kms_url); - let response = self.request_app_keys_from_kms_url(kms_url.clone()).await; - match response { - Ok(response) => { - break 'out response; - } - Err(err) => { - warn!("Failed to get app keys from KMS {kms_url}: {err:?}"); - // Record the first error - if i == 0 { - error = err; - } - } + let mut errors = vec![]; + for kms_url in &self.shared.sys_config.kms_urls { + let kms_url = kms_rpc_url(kms_url); + match self.request_app_keys_from_kms_url(kms_url.clone()).await { + Ok((keys, os_image_hash)) => { + // Measured here, outside the per-URL request, so that a + // failover can never extend os-image-hash twice. + emit_runtime_event("os-image-hash", &os_image_hash) + .context("failed to extend os-image-hash to the launch measurement")?; + return Ok(keys); + } + Err(err) => { + warn!("Failed to get app keys from KMS {kms_url}: {err:?}"); + errors.push(format!("{kms_url}: {err:#}")); } } - return Err(error).context("Failed to get app keys from KMS"); - }; - Ok(keys) + } + bail!("Failed to get app keys from KMS: {}", errors.join("; ")) } fn verify_key_provider_id(&self, provider_id: &[u8]) -> Result<()> { From 1a78afb6c40b2612a8fb0edf74a894f67c57c1a6 Mon Sep 17 00:00:00 2001 From: Kevin Wang Date: Tue, 22 Sep 2026 09:07:42 -0700 Subject: [PATCH 04/25] fix(dstack-util): keep KMS failover error as a context layer, lowercase messages --- dstack/dstack-util/src/system_setup.rs | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/dstack/dstack-util/src/system_setup.rs b/dstack/dstack-util/src/system_setup.rs index c45b95edf..0d1d537d7 100644 --- a/dstack/dstack-util/src/system_setup.rs +++ b/dstack/dstack-util/src/system_setup.rs @@ -2378,12 +2378,12 @@ impl<'a> Stage0<'a> { return Ok(keys); } Err(err) => { - warn!("Failed to get app keys from KMS {kms_url}: {err:?}"); + warn!("failed to get app keys from KMS {kms_url}: {err:?}"); errors.push(format!("{kms_url}: {err:#}")); } } } - bail!("Failed to get app keys from KMS: {}", errors.join("; ")) + Err(anyhow!(errors.join("; "))).context("failed to get app keys from KMS") } fn verify_key_provider_id(&self, provider_id: &[u8]) -> Result<()> { From 3d3b390fa6765a4f44e62279bb0cfda1540fe226 Mon Sep 17 00:00:00 2001 From: Kevin Wang Date: Tue, 22 Sep 2026 08:20:28 -0700 Subject: [PATCH 05/25] fix(os): bound docker's wait for the guest agent and report failure The ExecStartPre loop spun forever (TimeoutStartSec=0) once the agent hit its start limit and was parked in failed, wedging the boot with nothing reported to the VMM. Exit on is-failed, bound the wait, and report boot.error to the host via OnFailure. --- os/common/rootfs/app-compose.service | 1 + os/common/rootfs/app-compose.sh | 5 +---- .../rootfs/docker.service.d/dstack-guest-agent.conf | 8 ++++++-- os/common/rootfs/dstack-boot-error@.service | 13 +++++++++++++ os/common/rootfs/dstack-guest-agent.service | 1 + .../recipes-core/dstack-guest/dstack-guest.bb | 1 + 6 files changed, 23 insertions(+), 6 deletions(-) create mode 100644 os/common/rootfs/dstack-boot-error@.service diff --git a/os/common/rootfs/app-compose.service b/os/common/rootfs/app-compose.service index b3979e255..a357d2836 100644 --- a/os/common/rootfs/app-compose.service +++ b/os/common/rootfs/app-compose.service @@ -2,6 +2,7 @@ Description=App Compose Service Wants=docker.service containerd.service containerd-stargz-grpc.service After=docker.service containerd.service containerd-stargz-grpc.service dstack-prepare.service dstack-guest-agent.service +OnFailure=dstack-boot-error@%n.service [Service] Type=oneshot diff --git a/os/common/rootfs/app-compose.sh b/os/common/rootfs/app-compose.sh index 4bd2dad79..3bb1f9585 100755 --- a/os/common/rootfs/app-compose.sh +++ b/os/common/rootfs/app-compose.sh @@ -126,10 +126,7 @@ start) docker-compose|nerdctl-compose) echo "Starting containers with runner=$runner snapshotter=$snapshotter" dstack-util notify-host -e "boot.progress" -d "starting containers" || true - if ! compose_start; then - dstack-util notify-host -e "boot.error" -d "failed to start containers" - exit 1 - fi + compose_start if [ "$runner" = docker-compose ]; then echo "Pruning unused Docker images and volumes" docker image prune -af diff --git a/os/common/rootfs/docker.service.d/dstack-guest-agent.conf b/os/common/rootfs/docker.service.d/dstack-guest-agent.conf index 2ca01dce3..e5528c5aa 100644 --- a/os/common/rootfs/docker.service.d/dstack-guest-agent.conf +++ b/os/common/rootfs/docker.service.d/dstack-guest-agent.conf @@ -5,7 +5,11 @@ [Unit] Wants=dstack-guest-agent.service After=dstack-guest-agent.service +OnFailure=dstack-boot-error@%n.service [Service] -TimeoutStartSec=0 -ExecStartPre=/bin/sh -c 'while ! systemctl is-active --quiet dstack-guest-agent.service || [ ! -S /var/run/dstack.sock ] || [ ! -S /var/run/tappd.sock ]; do sleep 1; done' +# The agent restarts itself on crashes; give up once it is parked in `failed` +# or never comes up, so OnFailure= reports the error instead of hanging boot. +# Its sockets come from dstack-guest-agent.socket and exist once it is active. +TimeoutStartSec=300 +ExecStartPre=/bin/sh -c 'until systemctl is-active -q dstack-guest-agent.service; do systemctl is-failed -q dstack-guest-agent.service && exit 1; sleep 1; done' diff --git a/os/common/rootfs/dstack-boot-error@.service b/os/common/rootfs/dstack-boot-error@.service new file mode 100644 index 000000000..3803c22cc --- /dev/null +++ b/os/common/rootfs/dstack-boot-error@.service @@ -0,0 +1,13 @@ +# SPDX-FileCopyrightText: © 2026 Phala Network +# +# SPDX-License-Identifier: Apache-2.0 + +# Pulled in via OnFailure=dstack-boot-error@%n.service to report a failed boot +# unit to the host. +[Unit] +Description=Report %i failure to the host +After=dstack-prepare.service + +[Service] +Type=oneshot +ExecStart=/bin/dstack-util notify-host -e boot.error -d "%i failed" diff --git a/os/common/rootfs/dstack-guest-agent.service b/os/common/rootfs/dstack-guest-agent.service index a88d395d9..47a856bb2 100644 --- a/os/common/rootfs/dstack-guest-agent.service +++ b/os/common/rootfs/dstack-guest-agent.service @@ -3,6 +3,7 @@ Description=dstack Guest Agent Service Requires=dstack-guest-agent.socket After=network.target tboot.service dstack-guest-agent.socket Before=docker.service +OnFailure=dstack-boot-error@%n.service [Service] OOMScoreAdjust=-1000 diff --git a/os/yocto/layers/meta-dstack/recipes-core/dstack-guest/dstack-guest.bb b/os/yocto/layers/meta-dstack/recipes-core/dstack-guest/dstack-guest.bb index c2beca373..3a7aee1d5 100644 --- a/os/yocto/layers/meta-dstack/recipes-core/dstack-guest/dstack-guest.bb +++ b/os/yocto/layers/meta-dstack/recipes-core/dstack-guest/dstack-guest.bb @@ -93,6 +93,7 @@ do_install() { install -m 0644 ${DSTACK_ROOTFS_FILES}/app-compose.service ${D}${systemd_system_unitdir} install -m 0644 ${DSTACK_ROOTFS_FILES}/dstack-gateway-checker.service ${D}${systemd_system_unitdir} install -m 0644 ${DSTACK_ROOTFS_FILES}/dstack-guest-agent.socket ${D}${systemd_system_unitdir} + install -m 0644 ${DSTACK_ROOTFS_FILES}/dstack-boot-error@.service ${D}${systemd_system_unitdir} install -m 0644 ${DSTACK_ROOTFS_FILES}/llmnr.conf ${D}${sysconfdir}/systemd/resolved.conf.d # Drop-ins the image ships are vendor configuration, so they belong # beside the units in ${systemd_system_unitdir}. /etc is the operator's From f5e9446782751b69a0f2b09a580d1b8828d0580a Mon Sep 17 00:00:00 2001 From: Kevin Wang Date: Sun, 20 Sep 2026 00:06:07 -0700 Subject: [PATCH 06/25] fix(os): give the guest journal a budget it can satisfy SystemKeepFree=1G asked for more free space than a 2 GB CVM's /var/log has, and journald folds keep-free into the limit, so the declared 128M budget collapsed to the 16M floor. The System* settings were inert anyway: nothing in the image creates /var/log/journal, so journald has always used /run/log/journal, where RuntimeMaxUse=0 left it with no budget at all. --- os/common/rootfs/journald.conf | 33 ++++++++++++++++++++++++++++----- 1 file changed, 28 insertions(+), 5 deletions(-) diff --git a/os/common/rootfs/journald.conf b/os/common/rootfs/journald.conf index 90a182d4d..8db4b50eb 100644 --- a/os/common/rootfs/journald.conf +++ b/os/common/rootfs/journald.conf @@ -1,7 +1,30 @@ [Journal] -SystemMaxUse=128M -SystemKeepFree=1G -SystemMaxFileSize=10M -SystemMaxFiles=10 -RuntimeMaxUse=0 +# The guest has no persistent journal, so say so and budget the one it has. +# +# Storage=auto means persistent only if /var/log/journal already exists, and +# nothing creates it: Debian's systemd package ships no /var/log entries +# (verified against systemd 257.13-1~deb13u1 from the pinned snapshot), its +# tmpfiles rule for that directory is `z` -- adjust if present, never create -- +# and no dstack recipe makes one. So journald has always written to +# /run/log/journal, where every System* setting below used to be inert and +# RuntimeMaxUse=0 was the only live one. journald reported the result as +# "Runtime Journal (...) is 8.0M, max 0B, 0B free": no budget at all. +# +# The System* settings would not have worked either. /var/log is an overlay +# whose upper layer is the /var/volatile tmpfs, so SystemKeepFree=1G asked for +# more free space than the whole filesystem has on a 2 GB CVM. journald folds +# keep-free into the limit -- MIN(MAX(used + free - keep_free, min_use), +# max_use) -- so the 128M budget collapsed to the 16M floor: "System Journal +# (...) is 8.0M, max 16.0M" on a 1 GiB /var/log, against "max 128.0M" on 8 GiB. +# +# Keeping the journal in /run rather than creating /var/log/journal is +# deliberate. Both are tmpfs and neither survives a reboot, but /var/volatile +# is also where /var/lib/docker's writable layer lives, and the images are what +# fills a small CVM. Every limit here is a size journald can satisfy in /run, +# and RuntimeKeepFree keeps its default 15% of the filesystem, which scales +# with the CVM instead of being a fixed floor larger than it. +Storage=volatile +RuntimeMaxUse=128M +RuntimeMaxFileSize=10M +RuntimeMaxFiles=10 ReadKMsg=no From 208d25a6a8e2f0b6756e094a604ff8f6ddd5b84d Mon Sep 17 00:00:00 2001 From: Kevin Wang Date: Tue, 22 Sep 2026 19:34:26 -0700 Subject: [PATCH 07/25] docs: explain that gateway_app_id may list several gateway app ids --- docs/auth-simple-operations.md | 2 ++ docs/onchain-governance.md | 4 +++- dstack/dstack-util/src/system_setup.rs | 5 +++++ 3 files changed, 10 insertions(+), 1 deletion(-) diff --git a/docs/auth-simple-operations.md b/docs/auth-simple-operations.md index 416df9e4c..6b8826d8f 100644 --- a/docs/auth-simple-operations.md +++ b/docs/auth-simple-operations.md @@ -227,6 +227,8 @@ The Gateway is a special app that routes traffic to other apps. Once deployed, a The `gatewayAppId` is returned in boot responses and used by KMS for key derivation. +To trust several gateways, join their App IDs with commas (`"0xaaa...,0xbbb..."`). CVMs accept a gateway whose App ID is contained in this value. + --- ## KMS Onboarding (Multi-Node) diff --git a/docs/onchain-governance.md b/docs/onchain-governance.md index 42099c069..777675399 100644 --- a/docs/onchain-governance.md +++ b/docs/onchain-governance.md @@ -98,6 +98,8 @@ GATEWAY_APP_ID= \ Output: `Set gateway app ID: ` +To trust more than one gateway (for example during a gateway migration), set the value to several app IDs joined by commas, e.g. `,`. A CVM accepts a gateway whose certificate carries any app ID contained in this string. + Add the gateway's compose hash to the whitelist. To compute the compose hash: ```bash @@ -155,7 +157,7 @@ The central governance contract that manages OS image whitelisting, app registra |----------|-------------| | `addOsImageHash(bytes32)` | Whitelist an OS image hash | | `removeOsImageHash(bytes32)` | Remove an OS image from whitelist | -| `setGatewayAppId(string)` | Set the trusted Gateway app ID | +| `setGatewayAppId(string)` | Set the trusted Gateway app ID(s) | | `registerApp(address)` | Register an app contract | | `deployAndRegisterApp(...)` | Deploy and register app in one transaction | | `isAppAllowed(AppBootInfo)` | Check if an app is allowed to boot | diff --git a/dstack/dstack-util/src/system_setup.rs b/dstack/dstack-util/src/system_setup.rs index 0d1d537d7..fdd9b774f 100644 --- a/dstack/dstack-util/src/system_setup.rs +++ b/dstack/dstack-util/src/system_setup.rs @@ -2215,6 +2215,11 @@ struct AppIdValidator { } impl AppIdValidator { + /// `allowed_app_id` may list several gateway app ids (e.g. + /// `","`), so the peer is accepted if its hex app id appears + /// anywhere in it. This relies on the KMS only issuing certificates with + /// 20-byte app ids (`ensure_app_id_len`), so a shorter id cannot match a + /// fragment. fn validate(&self, cert: Option) -> Result<()> { if self.allowed_app_id == "any" { return Ok(()); From 9f027f7627dd5b3d6983484ccc40033382e1896a Mon Sep 17 00:00:00 2001 From: Kevin Wang Date: Tue, 22 Sep 2026 19:36:01 -0700 Subject: [PATCH 08/25] fix(os): require dstack-prepare and the guest agent before the app --- os/common/rootfs/app-compose.service | 2 ++ 1 file changed, 2 insertions(+) diff --git a/os/common/rootfs/app-compose.service b/os/common/rootfs/app-compose.service index b3979e255..53e037a2f 100644 --- a/os/common/rootfs/app-compose.service +++ b/os/common/rootfs/app-compose.service @@ -1,5 +1,7 @@ [Unit] Description=App Compose Service +# After= alone is satisfied by a failed unit; app must not run without these. +Requires=dstack-prepare.service dstack-guest-agent.service Wants=docker.service containerd.service containerd-stargz-grpc.service After=docker.service containerd.service containerd-stargz-grpc.service dstack-prepare.service dstack-guest-agent.service From 1666f89f176272f786b9f81030c9dd833d31bd92 Mon Sep 17 00:00:00 2001 From: Kevin Wang Date: Tue, 22 Sep 2026 19:50:05 -0700 Subject: [PATCH 09/25] fix(os): let the runtime journal budget scale with guest memory --- os/common/rootfs/journald.conf | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/os/common/rootfs/journald.conf b/os/common/rootfs/journald.conf index 8db4b50eb..37ce1c82a 100644 --- a/os/common/rootfs/journald.conf +++ b/os/common/rootfs/journald.conf @@ -6,7 +6,7 @@ # (verified against systemd 257.13-1~deb13u1 from the pinned snapshot), its # tmpfiles rule for that directory is `z` -- adjust if present, never create -- # and no dstack recipe makes one. So journald has always written to -# /run/log/journal, where every System* setting below used to be inert and +# /run/log/journal, where every System* setting used to be inert and # RuntimeMaxUse=0 was the only live one. journald reported the result as # "Runtime Journal (...) is 8.0M, max 0B, 0B free": no budget at all. # @@ -14,17 +14,17 @@ # whose upper layer is the /var/volatile tmpfs, so SystemKeepFree=1G asked for # more free space than the whole filesystem has on a 2 GB CVM. journald folds # keep-free into the limit -- MIN(MAX(used + free - keep_free, min_use), -# max_use) -- so the 128M budget collapsed to the 16M floor: "System Journal -# (...) is 8.0M, max 16.0M" on a 1 GiB /var/log, against "max 128.0M" on 8 GiB. +# max_use) -- so the 128M budget collapsed to the 16M floor. # # Keeping the journal in /run rather than creating /var/log/journal is # deliberate. Both are tmpfs and neither survives a reboot, but /var/volatile # is also where /var/lib/docker's writable layer lives, and the images are what -# fills a small CVM. Every limit here is a size journald can satisfy in /run, -# and RuntimeKeepFree keeps its default 15% of the filesystem, which scales -# with the CVM instead of being a fixed floor larger than it. +# fills a small CVM. +# +# RuntimeMaxUse and RuntimeKeepFree stay at journald's defaults (10% and 15% +# of /run, itself 20% of RAM), so the budget scales with guest memory: a fixed +# 128M would take most of /run on a 512M CVM. The file limits cap it at 100M. Storage=volatile -RuntimeMaxUse=128M RuntimeMaxFileSize=10M RuntimeMaxFiles=10 ReadKMsg=no From 4558315539d250a2bbebd1eab07260381ff7b137 Mon Sep 17 00:00:00 2001 From: Kevin Wang Date: Tue, 22 Sep 2026 19:50:49 -0700 Subject: [PATCH 10/25] fix(os): trim journald.conf comment --- os/common/rootfs/journald.conf | 29 ++++------------------------- 1 file changed, 4 insertions(+), 25 deletions(-) diff --git a/os/common/rootfs/journald.conf b/os/common/rootfs/journald.conf index 37ce1c82a..763b6e1f3 100644 --- a/os/common/rootfs/journald.conf +++ b/os/common/rootfs/journald.conf @@ -1,29 +1,8 @@ [Journal] -# The guest has no persistent journal, so say so and budget the one it has. -# -# Storage=auto means persistent only if /var/log/journal already exists, and -# nothing creates it: Debian's systemd package ships no /var/log entries -# (verified against systemd 257.13-1~deb13u1 from the pinned snapshot), its -# tmpfiles rule for that directory is `z` -- adjust if present, never create -- -# and no dstack recipe makes one. So journald has always written to -# /run/log/journal, where every System* setting used to be inert and -# RuntimeMaxUse=0 was the only live one. journald reported the result as -# "Runtime Journal (...) is 8.0M, max 0B, 0B free": no budget at all. -# -# The System* settings would not have worked either. /var/log is an overlay -# whose upper layer is the /var/volatile tmpfs, so SystemKeepFree=1G asked for -# more free space than the whole filesystem has on a 2 GB CVM. journald folds -# keep-free into the limit -- MIN(MAX(used + free - keep_free, min_use), -# max_use) -- so the 128M budget collapsed to the 16M floor. -# -# Keeping the journal in /run rather than creating /var/log/journal is -# deliberate. Both are tmpfs and neither survives a reboot, but /var/volatile -# is also where /var/lib/docker's writable layer lives, and the images are what -# fills a small CVM. -# -# RuntimeMaxUse and RuntimeKeepFree stay at journald's defaults (10% and 15% -# of /run, itself 20% of RAM), so the budget scales with guest memory: a fixed -# 128M would take most of /run on a 512M CVM. The file limits cap it at 100M. +# Nothing creates /var/log/journal, so the journal lives in /run. Keep it there: +# /var/volatile is shared with docker's writable layer. +# RuntimeMaxUse/RuntimeKeepFree stay at their defaults (10%/15% of /run) so the +# budget scales with guest memory. Storage=volatile RuntimeMaxFileSize=10M RuntimeMaxFiles=10 From 866dec12edf7f6ff7056bf95579f384d230f2e6e Mon Sep 17 00:00:00 2001 From: Kevin Wang Date: Sun, 20 Sep 2026 00:12:00 -0700 Subject: [PATCH 11/25] feat(dstack-util): measure whether the data disk was encrypted `storage-fs` is a launch event; `dstack.storage_encrypted` was not, on the grounds (issue #618) that the kernel command line is measured into RTMR2. That holds on TDX legacy and TDX lite, which both recompute RTMR2 from the image's base command line, on SEV-SNP, where the command line is hashed into the launch measurement, and on AWS NitroTPM, where PCR12 is inside the bound boot_pcr_digest. It does not hold on GCP TDX: verify_os_image_hash_for_gcp_tdx compares PCR0 and the UKI Authenticode digest out of the PCR2 event log and nothing else -- no RTMR1/RTMR2, and no PCR11/PCR12, where systemd-stub measures a command line handed to it rather than the one inside the UKI. Emit `storage-encrypted` next to `storage-fs` so the answer rides the one channel every platform replays. The event ships with a new os_image_hash, so no deployment sees RTMR3 move without the image moving too. --- docs/aws-attested-instance-security-evaluation.md | 2 +- docs/tutorials/attestation-verification.md | 1 + dstack/dstack-util/src/system_setup.rs | 5 +++++ 3 files changed, 7 insertions(+), 1 deletion(-) diff --git a/docs/aws-attested-instance-security-evaluation.md b/docs/aws-attested-instance-security-evaluation.md index 4f3ca9682..3575da342 100644 --- a/docs/aws-attested-instance-security-evaluation.md +++ b/docs/aws-attested-instance-security-evaluation.md @@ -35,7 +35,7 @@ NitroTPM path meets them today. | P1 | Verifiable platform root of trust | TDX/SNP/Nitro quote verification against vendor root; debug rejected; TCB surfaced | NitroTPM Attestation Documents are verified against the AWS Nitro Attestation PKI, including document timestamp sanity. AWS exposes no TDX/SNP-style TCB advisory field, so a verified attestation is normalized to `tcbStatus = "UpToDate"` and passes the standard authorization gate unchanged. | | P2 | Reproducible or independently computable base image measurement | meta-dstack rebuild plus `dstack-mr` computes `MRTD`/`RTMR0-2` | The unified `os/build.sh` flow emits the AWS image archive with `sha256sum.txt`, `digest.txt`, and `measurement.aws.cbor`; its output directory also contains the reference-PCR side-car `aws-pcrs.json`. `os_image_hash = sha256(sha256sum.txt)` is the same identity used on all platforms; the verifier recomputes it from the downloaded image directory (`dstack/verifier/src/verification.rs`). The hardening audit script `os/yocto/tools/aws/audit-aws-ec2-image-hardening.sh` checks the image for operator mutation channels. | | P3 | Boot command line and root filesystem integrity are measured | `RTMR1/2`, rootfs hash, dm-verity, measured initrd/cmdline | The UKI commits kernel, initrd, and embedded cmdline into `PCR4`; the rootfs is dm-verity-protected. `VmConfig.aws_measurement` is required and must bind `boot_pcr_digest = sha256(PCR4||PCR7||PCR12)` to the attested PCRs, so `PCR12` (external cmdline) is always part of the bound digest — a missing-PCR12 bypass is not expressible. Enforced in guest quote generation (`dstack/dstack-attest/src/attestation.rs`), `verify_os_image_hash_for_aws_nitro_tpm` (`dstack/verifier/src/verification.rs`), and the KMS pipeline via the same verifier check. | -| P4 | Runtime application identity is cryptographically bound | RTMR3 `compose-hash`, `app-id`, `instance-id`, `key-provider`; event log replay | SHA384 `PCR14` event-log replay is the authoritative binding (RTMR3 analogue; non-resettable). Launch events: `system-preparing`, `app-id`, `compose-hash`, zero or more ordered `init-script-hash` events, `instance-id`, `boot-mr-done`, `key-provider`, `storage-fs`, `system-ready` (`dstack/dstack-util/src/system_setup.rs`). GPU launches also include `gpu-policy-hash` and `gpu-attestation` before `instance-id`. `dstack-attest`, `dstack-verifier`, and KMS reject missing/mismatched PCR14 and bad replay. Optionally, the guest extends the raw `MrConfig` V2 `config_id` into `PCR8` once (`PCR8 = sha384(0^48 || config_id)`) so a lightweight third-party verifier can check compose hash + key provider without event-log replay; dstack's own verifier and KMS do not check PCR8. | +| P4 | Runtime application identity is cryptographically bound | RTMR3 `compose-hash`, `app-id`, `instance-id`, `key-provider`; event log replay | SHA384 `PCR14` event-log replay is the authoritative binding (RTMR3 analogue; non-resettable). Launch events: `system-preparing`, `app-id`, `compose-hash`, zero or more ordered `init-script-hash` events, `instance-id`, `boot-mr-done`, `key-provider`, `storage-fs`, `storage-encrypted`, `system-ready` (`dstack/dstack-util/src/system_setup.rs`). GPU launches also include `gpu-policy-hash` and `gpu-attestation` before `instance-id`. `dstack-attest`, `dstack-verifier`, and KMS reject missing/mismatched PCR14 and bad replay. Optionally, the guest extends the raw `MrConfig` V2 `config_id` into `PCR8` once (`PCR8 = sha384(0^48 || config_id)`) so a lightweight third-party verifier can check compose hash + key provider without event-log replay; dstack's own verifier and KMS do not check PCR8. | | P5 | Challenge/liveness and caller key binding | `report_data` challenge or RA-TLS public key hash in quote | RA-TLS binds `report_data` to the TLS certificate public key. KMS key release is bound to the live RA-TLS handshake; external `/verify` callers supply and check their own `report_data` challenge. The low-level NitroTPM document verifier also rejects stale or far-future document timestamps. | | P6 | Secret release only to attested code | dstack KMS verifies attestation, checks auth policy, derives per-app keys | dstack KMS verifies the NitroTPM attestation, runs the same `verify_os_image_hash_for_aws_nitro_tpm` binding check as the verifier, builds `BootInfo` from verified boot PCRs plus PCR14 launch events, and checks auth policy before deriving app keys (`dstack/kms/src/main_service.rs`). AWS NitroTPM key release is gated behind the opt-in `aws_nitro_tpm_key_release` flag (default false in `kms.toml`). | | P7 | Key-release policy is not controlled by the untrusted account admin | KMS runs inside TEE; policy from auth API/contracts; KMS identity measured | Satisfied with dstack KMS or another verifiable secret authority outside the untrusted AWS account admin's control. A NitroTPM-backed dstack KMS keeps root material out of account-admin snapshots and clones. The policy backend (auth-simple in a trusted control plane, or on-chain `DstackKms`/`DstackApp`) must be outside the workload account admin's control. Same-account AWS KMS fails this property if the admin can change key policy, create grants, or call secret-bearing operations through a policy they control. | diff --git a/docs/tutorials/attestation-verification.md b/docs/tutorials/attestation-verification.md index 9808ef94c..cb0342ac0 100644 --- a/docs/tutorials/attestation-verification.md +++ b/docs/tutorials/attestation-verification.md @@ -492,6 +492,7 @@ These are the standard events you'll see in the log: | `os-image-hash` | Guest OS image hash | Should match `tcb_info.os_image_hash` | | `key-provider` | Key provider type | e.g., `kms` | | `storage-fs` | Storage filesystem type | Storage configuration | +| `storage-encrypted` | `1` when the data disk is LUKS-encrypted, `0` when it is not | `0` means the host can read the application's data at rest | | `system-ready` | System ready marker | Always present at end | For a successful GPU launch, the relevant order is `compose-hash`, any diff --git a/dstack/dstack-util/src/system_setup.rs b/dstack/dstack-util/src/system_setup.rs index 0d1d537d7..52bee39e9 100644 --- a/dstack/dstack-util/src/system_setup.rs +++ b/dstack/dstack-util/src/system_setup.rs @@ -3030,6 +3030,11 @@ impl<'a> Stage0<'a> { // Parse kernel command line options let opts = parse_dstack_options(&self.shared).context("Failed to parse kernel cmdline")?; emit_runtime_event("storage-fs", opts.storage_fs.to_string().as_bytes())?; + // The cmdline is not measured on every platform (e.g. GCP TDX). + emit_runtime_event( + "storage-encrypted", + if opts.storage_encrypted { b"1" } else { b"0" }, + )?; info!( "Filesystem options: encryption={}, filesystem={:?}", opts.storage_encrypted, opts.storage_fs From e1b43243dfef69b27904c86515164fd874ebc48d Mon Sep 17 00:00:00 2001 From: Kevin Wang Date: Sat, 19 Sep 2026 23:58:19 -0700 Subject: [PATCH 12/25] fix(dstack-util): check an omitted MrConfig instance_id as empty A relying party cannot tell an omitted instance_id from no_instance_id, so the guest must not skip the check when the host leaves the field out. --- .../src/system_setup/config_id_verifier.rs | 35 ++++++++++++++++--- 1 file changed, 31 insertions(+), 4 deletions(-) diff --git a/dstack/dstack-util/src/system_setup/config_id_verifier.rs b/dstack/dstack-util/src/system_setup/config_id_verifier.rs index fdb00c034..4e4d97142 100644 --- a/dstack/dstack-util/src/system_setup/config_id_verifier.rs +++ b/dstack/dstack-util/src/system_setup/config_id_verifier.rs @@ -184,10 +184,10 @@ fn verify_mr_config_v3_document( bail!("Invalid mr_config app_id"); } } - if let Some(instance_id) = mr_config.instance_id.as_deref() { - if instance_id != local.instance_id { - bail!("Invalid mr_config instance_id"); - } + // Absent means empty: a relying party cannot tell an omitted instance_id + // from `no_instance_id`, so it must not skip the check. + if mr_config.instance_id.as_deref().unwrap_or_default() != local.instance_id { + bail!("Invalid mr_config instance_id"); } if mr_config.key_provider != local.key_provider { bail!("Invalid mr_config key_provider"); @@ -305,6 +305,33 @@ mod tests { Ok(()) } + #[test] + fn mr_config_v3_treats_missing_instance_id_as_empty() { + let compose_hash = [0x22u8; 32]; + let app_id = [0x11u8; 20]; + let document = MrConfigV3::new( + app_id.to_vec(), + compose_hash.to_vec(), + None, + KeyProviderKind::Kms, + Vec::new(), + Vec::new(), + ) + .to_canonical_json(); + let local = |instance_id| LocalMrConfigValues { + compose_hash: &compose_hash, + gpu_policy_hash: &[0x55u8; 32], + init_script_hashes: &[], + app_id: &app_id, + instance_id, + key_provider: KeyProviderKind::Kms, + key_provider_id: &[], + }; + + assert!(verify_mr_config_v3_document(&document, local(&[0x44u8; 20])).is_err()); + assert!(verify_mr_config_v3_document(&document, local(&[])).is_ok()); + } + #[test] fn mr_config_v3_document_must_match_expected_gpu_policy_hash() { let compose_hash = [0x22u8; 32]; From eebbfcafb1386b6f76cfb0d1fbccd8ff006fba03 Mon Sep 17 00:00:00 2001 From: Kevin Wang Date: Sun, 20 Sep 2026 00:12:00 -0700 Subject: [PATCH 13/25] fix(os): drop a sysctl the dstack kernel does not have kernel.unprivileged_userns_clone is a Debian/Ubuntu kernel patch. The image builds mainline from kernel.org -- the source whose sha256 versions.env pins has zero occurrences of the name, while every other key in this file is there -- and none of the ten dstack patches adds it. systemd-sysctl logged "Couldn't write '1' to 'kernel/unprivileged_userns_clone', ignoring" at LOG_WARNING on every boot and exited 0, so the unit succeeded and the setting did nothing. Nothing is lost: mainline has no gate, so with CONFIG_USER_NS=y an unprivileged CLONE_NEWUSER is already allowed, which is all sysbox asks for. --- .../recipes-core/dstack-sysbox/files/99-sysbox-sysctl.conf | 3 --- 1 file changed, 3 deletions(-) diff --git a/os/yocto/layers/meta-dstack/recipes-core/dstack-sysbox/files/99-sysbox-sysctl.conf b/os/yocto/layers/meta-dstack/recipes-core/dstack-sysbox/files/99-sysbox-sysctl.conf index 17cccab61..f2870e8bb 100644 --- a/os/yocto/layers/meta-dstack/recipes-core/dstack-sysbox/files/99-sysbox-sysctl.conf +++ b/os/yocto/layers/meta-dstack/recipes-core/dstack-sysbox/files/99-sysbox-sysctl.conf @@ -1,9 +1,6 @@ # Sysbox recommended sysctl settings # See: https://github.com/nestybox/sysbox/blob/master/docs/user-guide/install-package.md -# Allow unprivileged users to create user namespaces (required for rootless containers) -kernel.unprivileged_userns_clone = 1 - # Raise inotify limits — each sysbox container runs its own init system (e.g. systemd) # which uses inotify heavily; defaults (8192/128) exhaust quickly with multiple containers fs.inotify.max_queued_events = 1048576 From a8f0582386816bb34ed7a1f60979eaa1a2d3a3a8 Mon Sep 17 00:00:00 2001 From: Kevin Wang Date: Sun, 20 Sep 2026 00:06:29 -0700 Subject: [PATCH 14/25] fix(os): order guest agent after dstack-prepare, not tboot tboot.service was replaced by dstack-prepare.service in c901eeda56. 0ff57b26d8 updated app-compose.service but missed this unit, leaving an After= on a unit that no longer exists. The ordering itself still held via Before=dstack-guest-agent.service in dstack-prepare.service; this makes it visible from the guest agent side too. --- os/common/rootfs/dstack-guest-agent.service | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/os/common/rootfs/dstack-guest-agent.service b/os/common/rootfs/dstack-guest-agent.service index a88d395d9..853749662 100644 --- a/os/common/rootfs/dstack-guest-agent.service +++ b/os/common/rootfs/dstack-guest-agent.service @@ -1,7 +1,7 @@ [Unit] Description=dstack Guest Agent Service Requires=dstack-guest-agent.socket -After=network.target tboot.service dstack-guest-agent.socket +After=network.target dstack-prepare.service dstack-guest-agent.socket Before=docker.service [Service] From 4f097440b315a5811428a823a61498c6da338b6f Mon Sep 17 00:00:00 2001 From: Kevin Wang Date: Sun, 20 Sep 2026 00:06:29 -0700 Subject: [PATCH 15/25] fix(os): stop shipping the NVIDIA container runtime at debug level --- .../recipes-graphics/nvidia-container-toolkit/files/config.toml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/os/yocto/layers/meta-nvidia/recipes-graphics/nvidia-container-toolkit/files/config.toml b/os/yocto/layers/meta-nvidia/recipes-graphics/nvidia-container-toolkit/files/config.toml index e09dec2a3..49f2bcb12 100644 --- a/os/yocto/layers/meta-nvidia/recipes-graphics/nvidia-container-toolkit/files/config.toml +++ b/os/yocto/layers/meta-nvidia/recipes-graphics/nvidia-container-toolkit/files/config.toml @@ -12,7 +12,7 @@ ldconfig = "@/sbin/ldconfig" [nvidia-container-runtime] # debug = "/var/log/nvidia-container-runtime.log" -log-level = "debug" +log-level = "info" # Specify the runtimes to consider. This list is processed in order and the PATH # searched for matching executables unless the entry is an absolute path. From 615456412bbaad79df9865e393a6b912442dce03 Mon Sep 17 00:00:00 2001 From: Kevin Wang Date: Sun, 20 Sep 2026 00:07:19 -0700 Subject: [PATCH 16/25] fix(os): warn and skip TSM setup when configfs mount fails `setup_tsm || true` disables `set -e` inside the function, so a failed `mount -t configfs` fell through to `mkdir` under an unmounted directory without any warning. Log the failure and return, matching the existing "configfs not in kernel" case. Boot still continues, since quotes can be obtained through the ioctl fallbacks. --- os/common/rootfs/dstack-prepare.sh | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/os/common/rootfs/dstack-prepare.sh b/os/common/rootfs/dstack-prepare.sh index 40440d3c7..b8d42a890 100755 --- a/os/common/rootfs/dstack-prepare.sh +++ b/os/common/rootfs/dstack-prepare.sh @@ -131,7 +131,10 @@ setup_tsm() { fi if ! mountpoint -q /sys/kernel/config 2>/dev/null; then log "Mounting configfs for TSM..." - mount -t configfs none /sys/kernel/config + mount -t configfs none /sys/kernel/config || { + log "Warning: failed to mount configfs, TSM may not work" + return 1 + } fi if [[ -e /dev/tdx_guest ]] && [[ ! -d /sys/kernel/config/tsm/report/com.intel.dcap ]]; then log "Creating TSM report directory..." From 7d98701df3d8c3747793453f65233bb3f0391eb9 Mon Sep 17 00:00:00 2001 From: Kevin Wang Date: Sun, 20 Sep 2026 00:12:47 -0700 Subject: [PATCH 17/25] fix(kms): make a disabled os image verification visible `image.verify = false` turns off all OS-image binding at `info!` level, after which `BootInfo.os_image_hash` is whatever the caller put in `GetAppKeyRequest.vm_config` - so an `allowedOsImages` allowlist downstream is gating an attacker-supplied string. `enforce_self_authorization = false` already warns at startup for a comparable relaxation. Warn once at startup on the same terms, drop the per-call line to `debug!`, and report the posture as `GetMeta.os_image_verification` so a relying party can tell which kind of allowlist decision it is reading. The proto field is optional and additive: an older KMS leaves it unset, which must not be read as `true`. No SDK consumes `GetMeta`. --- dstack/kms/kms.toml | 2 ++ dstack/kms/rpc/proto/kms_rpc.proto | 3 +++ dstack/kms/src/main_service.rs | 10 ++++++++-- 3 files changed, 13 insertions(+), 2 deletions(-) diff --git a/dstack/kms/kms.toml b/dstack/kms/kms.toml index 0e5751fdd..03adcbe80 100644 --- a/dstack/kms/kms.toml +++ b/dstack/kms/kms.toml @@ -74,6 +74,8 @@ insecure_allow_external_trust_anchors = false # sev_snp_turin = "/etc/dstack/roots/amd-turin-ark.pem" [core.image] +# Verify os_image_hash against the booted image. If false, os_image_hash is +# caller-supplied, so allowedOsImages is not enforced. Never disable in production. verify = true cache_dir = "/usr/share/dstack/images" download_url = "http://localhost:8000/{OS_IMAGE_HASH}.tar.gz" diff --git a/dstack/kms/rpc/proto/kms_rpc.proto b/dstack/kms/rpc/proto/kms_rpc.proto index e491d9335..8c7bbe898 100644 --- a/dstack/kms/rpc/proto/kms_rpc.proto +++ b/dstack/kms/rpc/proto/kms_rpc.proto @@ -58,6 +58,9 @@ message GetMetaResponse { optional string kms_contract_address = 7; optional uint64 chain_id = 8; optional string app_auth_implementation = 9; + // `[core.image] verify`: whether `os_image_hash` is checked against the + // booted image. Unset on older KMS; do not treat unset as true. + optional bool os_image_verification = 10; } message GetKmsKeyRequest { diff --git a/dstack/kms/src/main_service.rs b/dstack/kms/src/main_service.rs index 5e9bc3ffd..e3f35afdf 100644 --- a/dstack/kms/src/main_service.rs +++ b/dstack/kms/src/main_service.rs @@ -30,7 +30,7 @@ use ra_tls::{ }; use scale::Decode; use tokio::sync::OnceCell; -use tracing::{info, warn}; +use tracing::{debug, warn}; use upgrade_authority::{ build_boot_info, ensure_app_id_len, local_kms_boot_info, BootInfo, GetInfoResponse, }; @@ -178,6 +178,11 @@ impl KmsState { "self-authorization is disabled; trusted RPCs will not be gated by KMS self-attestation - do not use in production TEE deployments" ); } + if !config.image.verify { + warn!( + "os image verification is disabled; os_image_hash is caller-supplied and unverified - do not use in production TEE deployments" + ); + } Ok(Self { inner: Arc::new(KmsStateInner { config, @@ -336,7 +341,7 @@ impl RpcHandler { report: &VerifiedAttestation, ) -> Result<()> { if !self.state.config.image.verify { - info!("Image verification is disabled"); + debug!("os image verification is disabled"); return Ok(()); } let mut detail = VerificationDetails::default(); @@ -521,6 +526,7 @@ impl KmsRpc for RpcHandler { chain_id: info.chain_id, gateway_app_id: info.gateway_app_id, app_auth_implementation: info.app_implementation, + os_image_verification: Some(self.state.config.image.verify), }) } From 44ab433dc9d0bbf6d7716008f841ad69b5bca0e4 Mon Sep 17 00:00:00 2001 From: Kevin Wang Date: Sat, 19 Sep 2026 21:49:16 -0700 Subject: [PATCH 18/25] fix(verifier): stop reporting --verify-cert results valid when app info does not decode `is_valid` was the literal `true` for any certificate whose RA-TLS attestation verified, and `decode_app_info(false).ok()` threw the decode error away. A certificate carrying an attestation that verifies but whose app identity does not decode printed a valid result with `app_info: null` -- no app_id, no compose_hash, no os_image_hash -- and exited 0. Report it the way `/verify` does: `is_valid` plus a `reason`, and a non-zero exit. The result file is still written first, so a caller inspecting it sees which check failed. --- dstack/verifier/README.md | 11 +-- dstack/verifier/src/main.rs | 130 ++++++++++++++++++++++++++++++------ 2 files changed, 117 insertions(+), 24 deletions(-) diff --git a/dstack/verifier/README.md b/dstack/verifier/README.md index 8529dc1f6..067e59388 100644 --- a/dstack/verifier/README.md +++ b/dstack/verifier/README.md @@ -141,9 +141,11 @@ certificate alone: cargo run --bin dstack-verifier -- --verify-cert endpoint-cert.pem ``` -The input may be PEM or DER. On success, the verifier prints JSON and writes the -same result next to the input as `endpoint-cert.pem.ratls-verification.json`. -The verification checks that: +The input may be PEM or DER. The verifier prints JSON and writes the same result +next to the input as `endpoint-cert.pem.ratls-verification.json`; `is_valid` +says whether every check below passed, `reason` says which one did not, and the +exit status is non-zero when `is_valid` is `false`. The verification checks +that: 1. the certificate contains a dstack RA-TLS attestation extension; 2. the embedded attestation verifies against the platform root, including AWS @@ -151,7 +153,8 @@ The verification checks that: 3. the attestation `report_data` is `QuoteContentType::RaTlsCert(SubjectPublicKeyInfo)`, so the verified attestation is bound to this exact TLS public key; and -4. the reported `app_info.os_image_hash` is bound to the attested boot +4. the attested app identity decodes; and +5. the reported `app_info.os_image_hash` is bound to the attested boot measurement, surfaced as `app_info.os_image_hash_verified`. This binding is self-contained (no image download) for AWS NitroTPM, SEV-SNP, Nitro Enclave, GCP TDX, and TDX lite. It is reported as `false` for the TDX legacy diff --git a/dstack/verifier/src/main.rs b/dstack/verifier/src/main.rs index 44e29e8ba..b02e04762 100644 --- a/dstack/verifier/src/main.rs +++ b/dstack/verifier/src/main.rs @@ -220,32 +220,20 @@ async fn run_cert_oneshot(file_path: &str, config: &Config) -> anyhow::Result<() .await .map_err(|e| anyhow::anyhow!("failed to verify RA-TLS certificate: {:#}", e))?; - let app_info = verified.decode_app_info(false).ok(); + let app_info = verified.decode_app_info(false); // Bind the reported os_image_hash to the attested boot measurement. For // every platform except TDX legacy this is a self-contained check (no image // download); relying parties should only trust `os_image_hash` when // `os_image_hash_verified` is true. let os_image_hash_verified = verify_cert_os_image_hash(&verified.attestation, config, &attestation_verifier).await; - let output = serde_json::json!({ - "is_valid": true, - "details": { - "tee_variant": verified.attestation.quote.variant(), - "report_data": hex::encode(verified.attestation.report_data), - "public_key_der": hex::encode(&verified.public_key_der), - "app_info": app_info.map(|info| serde_json::json!({ - "app_id": hex::encode(info.app_id), - "compose_hash": hex::encode(info.compose_hash), - "instance_id": hex::encode(info.instance_id), - "device_id": hex::encode(info.device_id), - "mr_system": hex::encode(info.mr_system), - "mr_aggregated": hex::encode(info.mr_aggregated), - "os_image_hash": hex::encode(info.os_image_hash), - "os_image_hash_verified": os_image_hash_verified, - "key_provider_info": hex::encode(info.key_provider_info), - })), - } - }); + let output = cert_oneshot_result( + verified.attestation.quote.variant(), + &verified.attestation.report_data, + &verified.public_key_der, + app_info, + os_image_hash_verified, + ); let output_path = format!("{file_path}.ratls-verification.json"); fs::write(&output_path, serde_json::to_string_pretty(&output)?).map_err(|e| { @@ -258,9 +246,53 @@ async fn run_cert_oneshot(file_path: &str, config: &Config) -> anyhow::Result<() info!("stored certificate verification result at {}", output_path); println!("{}", serde_json::to_string_pretty(&output)?); + if output["is_valid"] != serde_json::json!(true) { + anyhow::bail!("{}", output["reason"].as_str().unwrap_or("invalid")); + } Ok(()) } +/// Render the `--verify-cert` result. +/// +/// `is_valid` is the one field a relying party reads, so it has to cover every +/// step that ran, not just the signature check. App-info decoding is one of +/// those steps: a certificate whose attested app identity does not decode has +/// no `app_id`, `compose_hash` or `os_image_hash` to act on, and reporting it +/// as valid with a null `app_info` puts the burden of noticing on the caller. +/// Same shape as the HTTP `/verify` response: `is_valid` plus a `reason`. +fn cert_oneshot_result( + tee_variant: ra_tls::attestation::TeeVariant, + report_data: &[u8], + public_key_der: &[u8], + app_info: Result, + os_image_hash_verified: bool, +) -> serde_json::Value { + let reason = app_info + .as_ref() + .err() + .map(|err| format!("failed to decode app info from the certificate: {err:#}")); + serde_json::json!({ + "is_valid": reason.is_none(), + "reason": reason, + "details": { + "tee_variant": tee_variant, + "report_data": hex::encode(report_data), + "public_key_der": hex::encode(public_key_der), + "app_info": app_info.ok().map(|info| serde_json::json!({ + "app_id": hex::encode(info.app_id), + "compose_hash": hex::encode(info.compose_hash), + "instance_id": hex::encode(info.instance_id), + "device_id": hex::encode(info.device_id), + "mr_system": hex::encode(info.mr_system), + "mr_aggregated": hex::encode(info.mr_aggregated), + "os_image_hash": hex::encode(info.os_image_hash), + "os_image_hash_verified": os_image_hash_verified, + "key_provider_info": hex::encode(info.key_provider_info), + })), + } + }) +} + /// Verify that an RA-TLS certificate's `os_image_hash` is bound to its attested /// boot measurement. /// @@ -434,6 +466,64 @@ image_download_timeout_secs = 7 } } +#[cfg(test)] +mod cert_oneshot_result_tests { + use super::*; + use ra_tls::attestation::{AppInfo, TeeVariant}; + + fn app_info() -> AppInfo { + serde_json::from_value(serde_json::json!({ + "app_id": "00".repeat(20), + "compose_hash": "11".repeat(32), + "instance_id": "22".repeat(20), + "device_id": "33".repeat(32), + "mr_system": "44".repeat(32), + "mr_aggregated": "55".repeat(32), + "os_image_hash": "66".repeat(32), + "key_provider_info": "", + })) + .unwrap() + } + + /// `is_valid` is the field a relying party branches on, so it must not say + /// "valid" about a certificate whose app identity never decoded -- the + /// result then carries no app_id, compose_hash or os_image_hash at all. + #[test] + fn a_certificate_whose_app_info_does_not_decode_is_not_reported_valid() { + let valid = cert_oneshot_result( + TeeVariant::DstackTdx, + &[0u8; 64], + b"spki", + Ok(app_info()), + true, + ); + assert_eq!(valid["is_valid"], serde_json::json!(true)); + assert_eq!(valid["reason"], serde_json::Value::Null); + assert_eq!( + valid["details"]["app_info"]["compose_hash"], + "11".repeat(32) + ); + + let undecodable = cert_oneshot_result( + TeeVariant::DstackTdx, + &[0u8; 64], + b"spki", + Err(anyhow::anyhow!("no app-id event")), + true, + ); + assert_eq!(undecodable["is_valid"], serde_json::json!(false)); + assert!(undecodable["reason"] + .as_str() + .unwrap() + .contains("no app-id event")); + assert_eq!( + undecodable["details"]["app_info"], + serde_json::Value::Null, + "there is no app info to report" + ); + } +} + #[cfg(test)] mod certificate_profile_tests { use super::*; From 9718d8775a84881d18bc977c117c94a42578cf3d Mon Sep 17 00:00:00 2001 From: Kevin Wang Date: Sat, 19 Sep 2026 21:49:46 -0700 Subject: [PATCH 19/25] docs(verifier): say what event_log_verified actually covers The field documented itself as verifying RTMR 0-2 digests "through replay comparison with the quote". Nothing in dstack-verifier or dstack-attest replays a boot-time event log: the flag is set once the runtime event log replays to its register and the app identity decodes out of its payloads. A TDX quote's RTMR 0-2 entries reach only the `--debug` diff and the three named ACPI digests the lite path cross-checks. Correct the comment rather than the code. Replaying RTMR 0-2 would add nothing: those registers are already checked against measurements recomputed from the OS image, which does not trust the host's event log at all, and is the stronger of the two checks. --- dstack/verifier/README.md | 4 ++-- dstack/verifier/src/types.rs | 20 ++++++++++++++------ 2 files changed, 16 insertions(+), 8 deletions(-) diff --git a/dstack/verifier/README.md b/dstack/verifier/README.md index 8529dc1f6..40db89e34 100644 --- a/dstack/verifier/README.md +++ b/dstack/verifier/README.md @@ -35,7 +35,7 @@ against the returned evidence. "is_valid": true, "details": { "quote_verified": true, - "event_log_verified": true, // See "Verification Process" for semantics + "event_log_verified": true, // runtime event log only; see "Verification Process" "os_image_hash_verified": true, "acpi_tables_verified": true, // true only when TDX ACPI table contents are verified "os_image_is_dev": false, // true=dev image, false=prod, null=unknown/N/A @@ -237,7 +237,7 @@ $ curl -s -d @quote.json localhost:8080/verify | jq The verifier performs the following verification steps: 1. **Quote Verification**: Validates the platform quote using the platform verifier: DCAP for TDX, AMD SNP report verification for SEV-SNP, NSM for Nitro Enclaves, and AWS NitroTPM attestation-document verification for EC2 NitroTPM. -2. **Event Log Verification**: Replays event logs to ensure RTMR/PCR values match and extracts app information. For RTMR3 and AWS NitroTPM PCR14 launch measurements, both the digest and payload integrity are verified. For TDX RTMR 0-2 boot-time measurements, only the digests are verified; the payload content is not validated as dstack does not define semantics for these payloads. +2. **Event Log Verification**: Replays the runtime event log — RTMR3 on TDX, the corresponding launch measurement on SEV-SNP and AWS NitroTPM PCR14 — to ensure it reproduces the quoted register, and extracts app information from its payloads. Both the digests and the payloads are verified there, because `app_id`, `compose_hash` and the rest are read out of them. `event_log_verified` reports that step only. The boot-time event log a TDX quote carries is not replayed: RTMR 0-2 are verified in step 3 by comparing the quoted values against measurements recomputed from the OS image, which does not depend on the host's event log, and dstack defines no semantics for its payloads. 3. **OS Image Hash Verification**: - Treats `vm_config` and any attached measurement material as untrusted inputs until they are bound to the hardware quote - For the full-image TDX path, downloads or loads the image identified by `os_image_hash`, checks the image checksum manifest, uses dstack-mr to compute expected MRTD/RTMR0-2, and compares them against the verified measurements from the quote diff --git a/dstack/verifier/src/types.rs b/dstack/verifier/src/types.rs index d9d71442b..031e692ac 100644 --- a/dstack/verifier/src/types.rs +++ b/dstack/verifier/src/types.rs @@ -79,13 +79,21 @@ impl PolicyBootInfo { #[derive(Debug, Clone, Default, Serialize)] pub struct VerificationDetails { pub quote_verified: bool, - /// Indicates that the event log was verified against the quote. + /// Indicates that the runtime event log was replayed against the quote and + /// the app identity was decoded from it. /// - /// For RTMR3 (runtime measurements), both the digest and payload integrity are verified - /// by replaying the event log and comparing against the quote. For RTMR 0-2 (boot-time - /// measurements), only the digests are verified through replay comparison with the quote; - /// the payload content is not validated. dstack does not define semantics for RTMR 0-2 - /// event log payloads. + /// That is RTMR3 on TDX, and the corresponding launch PCR on SEV-SNP and + /// AWS NitroTPM: both the digests and the payloads are verified, because + /// `app_id`, `compose_hash` and the rest are read out of those payloads. + /// + /// It says nothing about the boot-time event log. Nothing replays the + /// RTMR 0-2 entries a TDX quote carries, and that is deliberate: those + /// registers are verified by comparing the quoted values against + /// measurements recomputed from the OS image (see `os_image_hash_verified` + /// and `acpi_tables_verified`), which does not depend on the host's event + /// log at all. The boot event log is carried for diagnostics -- the + /// `--debug` RTMR diff and the three named ACPI digests the TDX lite path + /// cross-checks -- and dstack defines no semantics for its payloads. pub event_log_verified: bool, pub os_image_hash_verified: bool, /// Indicates that TDX ACPI table contents were verified. From e9e4c3ddd834052282ce7dc749de9f85da9b532a Mon Sep 17 00:00:00 2001 From: Kevin Wang Date: Tue, 22 Sep 2026 22:06:04 -0700 Subject: [PATCH 20/25] verifier: remove the debug verification path The debug request flag only fed a development-time RTMR diff (rtmr_debug) and an ACPI table dump (acpi_tables) on the legacy TDX path. The diff was never wired to the event log, so it always reported the whole expected sequence as missing. Nobody relies on it, so drop the flag, both response fields and the code behind them instead of fixing it. VerificationRequest does not deny unknown fields, so clients still sending debug keep working. --- CHANGELOG.md | 1 + dstack/dstack-mr/cli/src/main.rs | 2 +- dstack/kms/src/main_service.rs | 2 +- dstack/tests/e2e/attestation/run-platform.sh | 2 +- dstack/verifier/src/lib.rs | 5 +- dstack/verifier/src/main.rs | 2 +- dstack/verifier/src/types.rs | 46 --- dstack/verifier/src/verification.rs | 291 ++----------------- 8 files changed, 34 insertions(+), 317 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index e05698858..915483557 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -95,6 +95,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - os: the Yocto guest-OS backend (`os/yocto/`) is deprecated in favor of mkosi (`os/mkosi/`), which is now the default and recommended backend. `os/build.sh` defaults to `--backend mkosi`, and `make os-image` / `make os-repro-check` build with mkosi. The Yocto builds move to `make os-image-yocto` / `make os-repro-check-yocto`; `make os-image-mkosi` / `make os-repro-check-mkosi` remain as aliases. Every Yocto entrypoint prints a deprecation warning, and the backend is kept only to rebuild existing Yocto images. The mkosi build still reads patches, units and scripts from `os/yocto/`, so the directory stays until those files move ### Removed +- verifier: the `debug` request field and the `acpi_tables` / `rtmr_debug` response fields. The per-event RTMR diff never had the events it diffed, so it reported every expected digest as missing. Requests that still send `debug` are accepted and the field is ignored. - sdk: `TlsKeyOptions.path` in the JavaScript SDK. `GetTlsKeyArgs` has no such field and `getTlsKey` never read it, so a caller who set it was silently ignored. Breaking at the type level only, and only for code whose value was already being discarded. `deriveKey`'s `path` is a real, deprecated Tappd-era parameter and stays; the Python, Rust and Go v0 TLS-key options never carried one - guest-agent: the `EmitEvent` RPC no longer records anything -- runtime RTMR3 events are system-owned in 0.6.0, so an app can no longer extend the measurement chain. The method itself stays on the unversioned path and always fails with an error naming the removal, rather than being deleted outright: a deleted method answers HTTP 404 `Service not found: EmitEvent`, which tells a 0.5.x caller nothing about why its events stopped being recorded, while the kept stub fails with a message naming the removal and pointing at `report_data`. **Breaking:** any app extending RTMR3 at runtime must stop; bind app data through `report_data` instead, which is what most callers wanted anyway - gateway: `core.debug.insecure_skip_attestation`. It turned off both checks that make a gateway cluster a trust boundary: the node stopped asking its guest agent for its own app id, and every WaveKV sync and push, plus `ensure_from_gateway`, stopped checking the peer's. With it set, anything that could reach the sync routes could insert entries that replicated to every gateway in the cluster. It existed only because the integration suites could not run without it; they now run against a guest agent simulator and verify quotes through the production path, so nothing sets it. `docs/security/security-model.md` argues dstack's development switches are acceptable because they are visible in attestation measurements or public contract state -- that argument cannot cover the switch deciding whether attestation happens at all, which is why this one is deleted rather than documented. There is no replacement. **Breaking:** the config struct does not use `deny_unknown_fields`, so a leftover line is ignored rather than rejected, and what follows depends on why it was set. On a TDX host with a guest agent the gateway starts normally and peers that cannot present a verifiable app id stop being accepted. On a host with no guest agent -- the usual reason to have set it -- the gateway no longer starts at all, failing with `Failed to get app info`, because `my_app_id` is what every peer check compares against and a node that cannot learn its own identity must not come up without one. Remove the line and make sure every node can attest diff --git a/dstack/dstack-mr/cli/src/main.rs b/dstack/dstack-mr/cli/src/main.rs index 6f52081d3..d8e4f0211 100644 --- a/dstack/dstack-mr/cli/src/main.rs +++ b/dstack/dstack-mr/cli/src/main.rs @@ -325,7 +325,7 @@ fn run_diagnose(config: &DiagnoseConfig) -> Result<()> { let initrd = image_dir.join(&image_info.initrd).display().to_string(); let cmdline = dstack_mr::tdx::measured_kernel_cmdline(&image_info.cmdline); - // Same resolution order as the verifier (see verifier::compute_measurement_details): + // Same resolution order as the verifier (see verifier::compute_measurements): // explicit vm_config.ovmf_variant > image_info.ovmf_variant > legacy default. let ovmf_variant = vm .ovmf_variant diff --git a/dstack/kms/src/main_service.rs b/dstack/kms/src/main_service.rs index e3f35afdf..d4bad106a 100644 --- a/dstack/kms/src/main_service.rs +++ b/dstack/kms/src/main_service.rs @@ -347,7 +347,7 @@ impl RpcHandler { let mut detail = VerificationDetails::default(); self.state .verifier - .verify_os_image_hash(vm_config, report, false, &mut detail) + .verify_os_image_hash(vm_config, report, &mut detail) .await .context("Failed to verify os image hash")?; Ok(()) diff --git a/dstack/tests/e2e/attestation/run-platform.sh b/dstack/tests/e2e/attestation/run-platform.sh index b025b511c..1ce12ac1c 100755 --- a/dstack/tests/e2e/attestation/run-platform.sh +++ b/dstack/tests/e2e/attestation/run-platform.sh @@ -186,7 +186,7 @@ if ! ATTESTATION_HEX=$(dstack-util attest --report-data "$REPORT_DATA" --hex); t exit 1 fi jq -n --arg attestation "$ATTESTATION_HEX" \ - '{attestation: $attestation, debug: true}' > "$WORK/request.json" + '{attestation: $attestation}' > "$WORK/request.json" cat > "$WORK/verifier.toml" <, #[serde(with = "serde_bytes", default)] pub attestation: Option>, - #[serde(default)] - pub debug: Option, } #[derive(Debug, Clone, Serialize)] @@ -115,48 +113,6 @@ pub struct VerificationDetails { /// Canonical auth-policy input matching the KMS bootAuth payload. #[serde(skip_serializing_if = "Option::is_none")] pub boot_info: Option, - #[serde(skip_serializing_if = "Option::is_none")] - pub acpi_tables: Option, - #[serde(skip_serializing_if = "Option::is_none")] - pub rtmr_debug: Option>, -} - -#[derive(Debug, Clone, Serialize)] -pub struct AcpiTables { - pub tables: String, - pub rsdp: String, - pub loader: String, -} - -#[derive(Debug, Clone, Serialize)] -pub struct RtmrMismatch { - pub rtmr: String, - pub expected: String, - pub actual: String, - pub events: Vec, - #[serde(skip_serializing_if = "Vec::is_empty")] - pub missing_expected_digests: Vec, -} - -#[derive(Debug, Clone, Serialize)] -pub struct RtmrEventEntry { - pub index: usize, - pub event_type: u32, - pub event_name: String, - pub actual_digest: String, - #[serde(skip_serializing_if = "Option::is_none")] - pub expected_digest: Option, - pub payload_len: usize, - pub status: RtmrEventStatus, -} - -#[derive(Debug, Clone, Serialize)] -#[serde(rename_all = "snake_case")] -pub enum RtmrEventStatus { - Match, - Mismatch, - Extra, - Missing, } #[cfg(test)] @@ -176,7 +132,6 @@ mod tests { assert_eq!(req.event_log.as_deref(), Some("[]")); assert_eq!(req.vm_config.as_deref(), Some("{}")); assert_eq!(req.attestation, None); - assert_eq!(req.debug, None); } #[test] @@ -194,7 +149,6 @@ mod tests { let req: VerificationRequest = serde_json::from_str("{}").unwrap(); assert_eq!(req.quote, None); assert_eq!(req.attestation, None); - assert_eq!(req.debug, None); } #[test] diff --git a/dstack/verifier/src/verification.rs b/dstack/verifier/src/verification.rs index 0d8f1add9..fc28b8b32 100644 --- a/dstack/verifier/src/verification.rs +++ b/dstack/verifier/src/verification.rs @@ -17,9 +17,7 @@ use cc_eventlog::{ }, TdxEvent, }; -use dstack_mr::{ - tdx::TdxRtmr0AcpiHashes, RtmrLog, RtmrLogs, TdxMeasurementDetails, TdxMeasurements, -}; +use dstack_mr::{tdx::TdxRtmr0AcpiHashes, TdxMeasurements}; use dstack_types::{TdxAttestationVariant, VmConfig}; use hex_literal::hex; use ra_tls::attestation::{ @@ -32,8 +30,7 @@ use tokio::{io::AsyncWriteExt, process::Command}; use tracing::{debug, info, warn}; use crate::types::{ - AcpiTables, PolicyBootInfo, RtmrEventEntry, RtmrEventStatus, RtmrMismatch, VerificationDetails, - VerificationRequest, VerificationResponse, + PolicyBootInfo, VerificationDetails, VerificationRequest, VerificationResponse, }; /// Return the canonical TCB status and advisory list used by auth policy. @@ -88,99 +85,6 @@ fn decode_key_provider_info(bytes: &[u8]) -> Option RtmrMismatch { - let expected_hex = hex::encode(expected); - let actual_hex = hex::encode(actual); - - let mut events = Vec::new(); - - for (&idx, expected_digest) in actual_indices.iter().zip(expected_sequence.iter()) { - match event_log.get(idx) { - Some(event) => { - let event_name = if event.event.is_empty() { - "(unnamed)".to_string() - } else { - event.event.clone() - }; - let status = if event.digest() == expected_digest.as_slice() { - RtmrEventStatus::Match - } else { - RtmrEventStatus::Mismatch - }; - events.push(RtmrEventEntry { - index: idx, - event_type: event.event_type, - event_name, - actual_digest: hex::encode(event.digest()), - expected_digest: Some(hex::encode(expected_digest)), - payload_len: event.event_payload.len(), - status, - }); - } - None => { - events.push(RtmrEventEntry { - index: idx, - event_type: 0, - event_name: "(missing)".to_string(), - actual_digest: String::new(), - expected_digest: Some(hex::encode(expected_digest)), - payload_len: 0, - status: RtmrEventStatus::Missing, - }); - } - } - } - - for &idx in actual_indices.iter().skip(expected_sequence.len()) { - let (event_type, event_name, actual_digest, payload_len) = match event_log.get(idx) { - Some(event) => ( - event.event_type, - if event.event.is_empty() { - "(unnamed)".to_string() - } else { - event.event.clone() - }, - hex::encode(event.digest()), - event.event_payload.len(), - ), - None => (0, "(missing)".to_string(), String::new(), 0), - }; - events.push(RtmrEventEntry { - index: idx, - event_type, - event_name, - actual_digest, - expected_digest: None, - payload_len, - status: RtmrEventStatus::Extra, - }); - } - - let missing_expected_digests = if expected_sequence.len() > actual_indices.len() { - expected_sequence[actual_indices.len()..] - .iter() - .map(hex::encode) - .collect() - } else { - Vec::new() - }; - - RtmrMismatch { - rtmr: rtmr_label.to_string(), - expected: expected_hex.to_string(), - actual: actual_hex.to_string(), - events, - missing_expected_digests, - } -} - // Bump whenever expected RTMR computation changes so stale entries get ignored. // v3: all supported OVMF measurements use the Pre202505 RTMR[0] layout. // @@ -312,7 +216,7 @@ impl CvmVerifier { Ok(()) } - fn compute_measurement_details( + fn compute_measurements( &self, vm_config: &VmConfig, fw_path: &Path, @@ -320,7 +224,7 @@ impl CvmVerifier { initrd_path: &Path, kernel_cmdline: &str, kernel_header_normalized: bool, - ) -> Result { + ) -> Result { let firmware = fw_path.display().to_string(); let kernel = kernel_path.display().to_string(); let initrd = initrd_path.display().to_string(); @@ -329,7 +233,7 @@ impl CvmVerifier { // deployments fall back to the only layout that existed back then. let ovmf_variant = vm_config.ovmf_variant.unwrap_or_default(); - let details = dstack_mr::Machine::builder() + dstack_mr::Machine::builder() .cpu_count(vm_config.cpu_count) .memory_size(vm_config.memory_size) .firmware(&firmware) @@ -356,30 +260,8 @@ impl CvmVerifier { .host_share_mode(vm_config.host_share_mode.clone()) .ovmf_variant(ovmf_variant) .build() - .measure_with_logs() - .context("Failed to compute expected MRs")?; - - Ok(details) - } - - fn compute_measurements( - &self, - vm_config: &VmConfig, - fw_path: &Path, - kernel_path: &Path, - initrd_path: &Path, - kernel_cmdline: &str, - kernel_header_normalized: bool, - ) -> Result { - self.compute_measurement_details( - vm_config, - fw_path, - kernel_path, - initrd_path, - kernel_cmdline, - kernel_header_normalized, - ) - .map(|details| details.measurements) + .measure() + .context("Failed to compute expected MRs") } fn load_or_compute_measurements( @@ -719,7 +601,6 @@ impl CvmVerifier { }; let mut details = VerificationDetails::default(); - let debug = request.debug.unwrap_or(false); let attestation = attestation.into_v1(); let verified = attestation.verify(&self.attestation_verifier).await; let verified_attestation = match verified { @@ -748,7 +629,6 @@ impl CvmVerifier { .verify_os_image_hash( request_vm_config.clone(), &verified_attestation, - debug, &mut details, ) .await; @@ -794,7 +674,6 @@ impl CvmVerifier { &self, vm_config: String, attestation: &VerifiedAttestation, - debug: bool, details: &mut VerificationDetails, ) -> Result { // The raw config string used for platform-specific binding: the explicit @@ -829,22 +708,12 @@ impl CvmVerifier { // rather than degraded to a download. AttestationQuote::DstackTdx(_) => match vm_config.tdx_attestation_variant { TdxAttestationVariant::Legacy => { - self.verify_os_image_hash_for_dstack_tdx( - &vm_config, - attestation, - debug, - details, - ) - .await?; + self.verify_os_image_hash_for_dstack_tdx(&vm_config, attestation, details) + .await?; } TdxAttestationVariant::Lite => { - self.verify_os_image_hash_for_dstack_tdx_lite( - &vm_config, - attestation, - debug, - details, - ) - .await?; + self.verify_os_image_hash_for_dstack_tdx_lite(&vm_config, attestation, details) + .await?; } }, AttestationQuote::DstackNitroEnclave(_) => { @@ -909,16 +778,11 @@ impl CvmVerifier { &self, vm_config: &VmConfig, attestation: &VerifiedAttestation, - debug: bool, details: &mut VerificationDetails, ) -> Result<()> { let Some(report) = &attestation.report.tdx_report() else { bail!("No TDX report"); }; - let Some(tdx_quote) = attestation.tdx_quote() else { - bail!("No TDX quote"); - }; - let event_log = &tdx_quote.event_log; let report = report .report .as_td10() @@ -946,57 +810,24 @@ impl CvmVerifier { details.os_image_version = Some(image_paths.version.clone()); } - let (mrs, expected_logs) = if debug { - let TdxMeasurementDetails { - measurements, - rtmr_logs, - acpi_tables, - } = self - .compute_measurement_details( - vm_config, - &image_paths.fw_path, - &image_paths.kernel_path, - &image_paths.initrd_path, - &image_paths.kernel_cmdline, - image_paths.kernel_header_normalized, - ) - .context("Failed to compute expected measurements")?; - - details.acpi_tables = Some(AcpiTables { - tables: hex::encode(&acpi_tables.tables), - rsdp: hex::encode(&acpi_tables.rsdp), - loader: hex::encode(&acpi_tables.loader), - }); - - (measurements, Some(rtmr_logs)) - } else { - ( - self.load_or_compute_measurements( - vm_config, - &image_paths.fw_path, - &image_paths.kernel_path, - &image_paths.initrd_path, - &image_paths.kernel_cmdline, - image_paths.kernel_header_normalized, - ) - .context("Failed to compute expected measurements")?, - None, + let expected_mrs = self + .load_or_compute_measurements( + vm_config, + &image_paths.fw_path, + &image_paths.kernel_path, + &image_paths.initrd_path, + &image_paths.kernel_cmdline, + image_paths.kernel_header_normalized, ) - }; - - self.compare_tdx_mrs( - Mrs { - mrtd: mrs.mrtd, - rtmr0: mrs.rtmr0, - rtmr1: mrs.rtmr1, - rtmr2: mrs.rtmr2, - }, - verified_mrs, - expected_logs.as_ref(), - event_log, - debug, - details, - )?; + .context("Failed to compute expected measurements")?; + Mrs { + mrtd: expected_mrs.mrtd, + rtmr0: expected_mrs.rtmr0, + rtmr1: expected_mrs.rtmr1, + rtmr2: expected_mrs.rtmr2, + } + .assert_eq(&verified_mrs) + .context("MRs do not match")?; details.acpi_tables_verified = true; Ok(()) } @@ -1005,7 +836,6 @@ impl CvmVerifier { &self, vm_config: &VmConfig, attestation: &VerifiedAttestation, - _debug: bool, details: &mut VerificationDetails, ) -> Result<()> { let Some(report) = &attestation.report.tdx_report() else { @@ -1091,69 +921,6 @@ impl CvmVerifier { .context("MRs do not match") } - fn compare_tdx_mrs( - &self, - expected_mrs: Mrs, - verified_mrs: Mrs, - expected_logs: Option<&RtmrLogs>, - event_log: &[TdxEvent], - debug: bool, - details: &mut VerificationDetails, - ) -> Result<()> { - match expected_mrs.assert_eq(&verified_mrs) { - Ok(()) => Ok(()), - Err(e) => { - let result = Err(e).context("MRs do not match"); - if !debug { - return result; - } - let Some(expected_logs) = expected_logs else { - return result; - }; - let mut rtmr_debug = Vec::new(); - - if expected_mrs.rtmr0 != verified_mrs.rtmr0 { - rtmr_debug.push(collect_rtmr_mismatch( - "RTMR0", - &expected_mrs.rtmr0, - &verified_mrs.rtmr0, - &expected_logs[0], - &[], - event_log, - )); - } - - if expected_mrs.rtmr1 != verified_mrs.rtmr1 { - rtmr_debug.push(collect_rtmr_mismatch( - "RTMR1", - &expected_mrs.rtmr1, - &verified_mrs.rtmr1, - &expected_logs[1], - &[], - event_log, - )); - } - - if expected_mrs.rtmr2 != verified_mrs.rtmr2 { - rtmr_debug.push(collect_rtmr_mismatch( - "RTMR2", - &expected_mrs.rtmr2, - &verified_mrs.rtmr2, - &expected_logs[2], - &[], - event_log, - )); - } - - if !rtmr_debug.is_empty() { - details.rtmr_debug = Some(rtmr_debug); - } - - result - } - } - } - /// Verify Nitro Enclave OS image hash using the signature-verified NSM PCRs. /// /// For Nitro: @@ -2360,7 +2127,6 @@ mod tests { event_log: None, vm_config: None, attestation: Some(attestation), - debug: None, }) .await .expect("verifier runs"); @@ -2585,7 +2351,6 @@ mod tests { event_log: None, vm_config: None, attestation: Some(forged), - debug: None, }) .await .expect("verifier runs"); From 9dcb78a094ccaa6003ceccb5d8e7f8a426e24a68 Mon Sep 17 00:00:00 2001 From: Kevin Wang Date: Tue, 22 Sep 2026 22:56:26 -0700 Subject: [PATCH 21/25] verifier: compare TDX MRs as TdxMeasurements With compare_tdx_mrs gone, the private Mrs struct only mirrored dstack_mr::TdxMeasurements field by field. Compare TdxMeasurements directly and share the TD report extraction between the legacy and lite paths. TdxMeasurementDetails.acpi_tables was only read by the removed debug dump, so drop it and let Tdvf::rtmr0_log return just the log. --- dstack/dstack-mr/src/machine.rs | 5 +- dstack/dstack-mr/src/tdvf.rs | 13 +--- dstack/verifier/src/verification.rs | 115 ++++++++-------------------- 3 files changed, 36 insertions(+), 97 deletions(-) diff --git a/dstack/dstack-mr/src/machine.rs b/dstack/dstack-mr/src/machine.rs index 7925d3df0..c30323030 100644 --- a/dstack/dstack-mr/src/machine.rs +++ b/dstack/dstack-mr/src/machine.rs @@ -2,7 +2,6 @@ // // SPDX-License-Identifier: Apache-2.0 -use crate::acpi::Tables; use crate::tdvf::Tdvf; use crate::util::debug_print_log; use crate::{kernel, OvmfVariant, RtmrLogs, TdxMeasurements}; @@ -115,7 +114,6 @@ pub struct VersionedOptions { pub struct TdxMeasurementDetails { pub measurements: TdxMeasurements, pub rtmr_logs: RtmrLogs, - pub acpi_tables: Tables, } impl Machine<'_> { @@ -132,7 +130,7 @@ impl Machine<'_> { let mrtd = tdvf.mrtd(self).context("Failed to compute MR TD")?; - let (rtmr0_log, acpi_tables) = tdvf + let rtmr0_log = tdvf .rtmr0_log(self) .context("Failed to compute RTMR0 log")?; debug_print_log("RTMR0", &rtmr0_log); @@ -163,7 +161,6 @@ impl Machine<'_> { rtmr2, }, rtmr_logs: [rtmr0_log, rtmr1_log, rtmr2_log], - acpi_tables, }) } } diff --git a/dstack/dstack-mr/src/tdvf.rs b/dstack/dstack-mr/src/tdvf.rs index 3f6393713..23a922151 100644 --- a/dstack/dstack-mr/src/tdvf.rs +++ b/dstack/dstack-mr/src/tdvf.rs @@ -7,7 +7,6 @@ use hex_literal::hex; use scale::Decode; use sha2::{Digest, Sha384}; -use crate::acpi::Tables; use crate::num::read_le; use crate::{measure_log, measure_sha384, utf16_encode, Machine, OvmfVariant, RtmrLog}; @@ -497,23 +496,17 @@ impl<'a> Tdvf<'a> { #[allow(dead_code)] pub fn rtmr0(&self, machine: &Machine) -> Result> { - let (rtmr0_log, _) = self.rtmr0_log(machine)?; - Ok(measure_log(&rtmr0_log)) + Ok(measure_log(&self.rtmr0_log(machine)?)) } - pub fn rtmr0_log(&self, machine: &Machine) -> Result<(RtmrLog, Tables)> { + pub fn rtmr0_log(&self, machine: &Machine) -> Result { let tables = machine.build_tables()?; let acpi_hashes = AcpiTableHashes { tables: measure_sha384(&tables.tables), rsdp: measure_sha384(&tables.rsdp), loader: measure_sha384(&tables.loader), }; - let log = self.rtmr0_log_with_acpi_hashes( - machine.memory_size, - machine.ovmf_variant, - &acpi_hashes, - )?; - Ok((log, tables)) + self.rtmr0_log_with_acpi_hashes(machine.memory_size, machine.ovmf_variant, &acpi_hashes) } pub(crate) fn rtmr0_log_with_acpi_hashes( diff --git a/dstack/verifier/src/verification.rs b/dstack/verifier/src/verification.rs index fc28b8b32..6313177ae 100644 --- a/dstack/verifier/src/verification.rs +++ b/dstack/verifier/src/verification.rs @@ -780,20 +780,7 @@ impl CvmVerifier { attestation: &VerifiedAttestation, details: &mut VerificationDetails, ) -> Result<()> { - let Some(report) = &attestation.report.tdx_report() else { - bail!("No TDX report"); - }; - let report = report - .report - .as_td10() - .context("Failed to decode TD report")?; - - let verified_mrs = Mrs { - mrtd: report.mr_td.to_vec(), - rtmr0: report.rt_mr0.to_vec(), - rtmr1: report.rt_mr1.to_vec(), - rtmr2: report.rt_mr2.to_vec(), - }; + let verified_mrs = verified_tdx_mrs(attestation)?; // Legacy TDX attestation keeps the original KMS verifier semantics: // os_image_hash must be the image digest (digest.txt = @@ -820,14 +807,7 @@ impl CvmVerifier { image_paths.kernel_header_normalized, ) .context("Failed to compute expected measurements")?; - Mrs { - mrtd: expected_mrs.mrtd, - rtmr0: expected_mrs.rtmr0, - rtmr1: expected_mrs.rtmr1, - rtmr2: expected_mrs.rtmr2, - } - .assert_eq(&verified_mrs) - .context("MRs do not match")?; + assert_tdx_mrs_eq(&expected_mrs, &verified_mrs).context("MRs do not match")?; details.acpi_tables_verified = true; Ok(()) } @@ -838,26 +818,11 @@ impl CvmVerifier { attestation: &VerifiedAttestation, details: &mut VerificationDetails, ) -> Result<()> { - let Some(report) = &attestation.report.tdx_report() else { - bail!("No TDX report"); - }; + let verified_mrs = verified_tdx_mrs(attestation)?; let Some(tdx_quote) = attestation.tdx_quote() else { bail!("No TDX quote"); }; let event_log = &tdx_quote.event_log; - // Get boot info from attestation - let report = report - .report - .as_td10() - .context("Failed to decode TD report")?; - - // Extract the verified MRs from the report - let verified_mrs = Mrs { - mrtd: report.mr_td.to_vec(), - rtmr0: report.rt_mr0.to_vec(), - rtmr1: report.rt_mr1.to_vec(), - rtmr2: report.rt_mr2.to_vec(), - }; let document = vm_config .tdx_measurement @@ -903,22 +868,13 @@ impl CvmVerifier { details.acpi_tables_verified = true; // RTMR0 below is rebuilt from the recomputed digests, so the expected // value depends on nothing the host reported about the tables. - let mrs = dstack_mr::tdx::tdx_measurements_from_measurement_document( + let expected_mrs = dstack_mr::tdx::tdx_measurements_from_measurement_document( document, vm_config, &acpi_hashes, ) .context("Failed to compute TDX expected measurements without image download")?; - - let expected_mrs = Mrs { - mrtd: mrs.mrtd.clone(), - rtmr0: mrs.rtmr0.clone(), - rtmr1: mrs.rtmr1.clone(), - rtmr2: mrs.rtmr2.clone(), - }; - expected_mrs - .assert_eq(&verified_mrs) - .context("MRs do not match") + assert_tdx_mrs_eq(&expected_mrs, &verified_mrs).context("MRs do not match") } /// Verify Nitro Enclave OS image hash using the signature-verified NSM PCRs. @@ -1144,46 +1100,39 @@ impl CvmVerifier { } } -#[derive(Debug, Clone)] -struct Mrs { - mrtd: Vec, - rtmr0: Vec, - rtmr1: Vec, - rtmr2: Vec, +/// MRTD and RTMR0-2 from the verified TD report. +fn verified_tdx_mrs(attestation: &VerifiedAttestation) -> Result { + let report = attestation + .report + .tdx_report() + .context("No TDX report")? + .report + .as_td10() + .context("Failed to decode TD report")?; + Ok(TdxMeasurements { + mrtd: report.mr_td.to_vec(), + rtmr0: report.rt_mr0.to_vec(), + rtmr1: report.rt_mr1.to_vec(), + rtmr2: report.rt_mr2.to_vec(), + }) } -impl Mrs { - fn assert_eq(&self, other: &Self) -> Result<()> { - if self.mrtd != other.mrtd { +fn assert_tdx_mrs_eq(expected: &TdxMeasurements, actual: &TdxMeasurements) -> Result<()> { + for (name, expected, actual) in [ + ("MRTD", &expected.mrtd, &actual.mrtd), + ("RTMR0", &expected.rtmr0, &actual.rtmr0), + ("RTMR1", &expected.rtmr1, &actual.rtmr1), + ("RTMR2", &expected.rtmr2, &actual.rtmr2), + ] { + if expected != actual { bail!( - "MRTD mismatch: expected={}, actual={}", - hex::encode(&self.mrtd), - hex::encode(&other.mrtd) + "{name} mismatch: expected={}, actual={}", + hex::encode(expected), + hex::encode(actual) ); } - if self.rtmr0 != other.rtmr0 { - bail!( - "RTMR0 mismatch: expected={}, actual={}", - hex::encode(&self.rtmr0), - hex::encode(&other.rtmr0) - ); - } - if self.rtmr1 != other.rtmr1 { - bail!( - "RTMR1 mismatch: expected={}, actual={}", - hex::encode(&self.rtmr1), - hex::encode(&other.rtmr1) - ); - } - if self.rtmr2 != other.rtmr2 { - bail!( - "RTMR2 mismatch: expected={}, actual={}", - hex::encode(&self.rtmr2), - hex::encode(&other.rtmr2) - ); - } - Ok(()) } + Ok(()) } #[cfg(test)] From b4218e7e9c77ce975908de483224c405e19037ac Mon Sep 17 00:00:00 2001 From: Kevin Wang Date: Sat, 19 Sep 2026 21:51:31 -0700 Subject: [PATCH 22/25] fix(verifier): key the measurement cache on the measured VM shape only vm_config_cache_key hashed the whole VmConfig, including fields the full-image measurement never reads: an arbitrary `image` string, and the `tdx_measurement`/`gcp_measurement`/`aws_measurement` documents, each carrying a caller-sized checksum_file and CBOR blob. A cache miss costs a full firmware and kernel hash plus ACPI generation and leaves a file under /measurements/ that nothing evicts, so one captured quote replayed with a different filler byte per request misses every time and grows the cache without bound. Clear those four before hashing. Clearing rather than listing the fields that matter keeps a VmConfig field added later in the key by default, which is the safe direction: an extra miss, never a stale measurement. --- dstack/verifier/src/verification.rs | 44 ++++++++++++++++++++++++++++- 1 file changed, 43 insertions(+), 1 deletion(-) diff --git a/dstack/verifier/src/verification.rs b/dstack/verifier/src/verification.rs index 6313177ae..e8cb585c8 100644 --- a/dstack/verifier/src/verification.rs +++ b/dstack/verifier/src/verification.rs @@ -142,8 +142,18 @@ impl CvmVerifier { .join(format!("{cache_key}.json")) } + /// Hash only what the measurement reads: `image` and the measurement documents + /// are caller-controlled and unused, so they must not be able to force a miss. + /// Clear them rather than allowlisting, so new `VmConfig` fields stay in the key. fn vm_config_cache_key(vm_config: &VmConfig) -> Result { - let serialized = serde_json::to_vec(vm_config) + let vm_config = VmConfig { + image: None, + tdx_measurement: None, + gcp_measurement: None, + aws_measurement: None, + ..vm_config.clone() + }; + let serialized = serde_json::to_vec(&vm_config) .context("Failed to serialize VM config for cache key computation")?; Ok(hex::encode(Sha256::digest(&serialized))) } @@ -1723,6 +1733,38 @@ mod tests { assert_eq!(entries.len(), 1, "temporary cache files must not survive"); } + #[test] + fn measurement_cache_key_ignores_unmeasured_fields() { + let base: VmConfig = serde_json::from_value(serde_json::json!({ + "os_image_hash": "11".repeat(32), + "cpu_count": 2, + "memory_size": 0x8000_0000u64, + })) + .unwrap(); + let key = |config: &VmConfig| CvmVerifier::vm_config_cache_key(config).unwrap(); + let blob = || vec![0xaa; 4096]; + + let mut padded = base.clone(); + padded.image = Some("x".repeat(4096)); + padded.tdx_measurement = Some(dstack_types::TdxOsImageMeasurementDocument::new( + blob(), + blob(), + )); + padded.gcp_measurement = Some(dstack_types::GcpOsImageMeasurementDocument::new( + blob(), + blob(), + )); + padded.aws_measurement = Some(dstack_types::AwsOsImageMeasurementDocument::new( + blob(), + blob(), + )); + assert_eq!(key(&padded), key(&base)); + + let mut resized = base.clone(); + resized.cpu_count += 1; + assert_ne!(key(&resized), key(&base)); + } + #[test] fn image_cache_pruning_keeps_checksum_identity() { let dir = tempfile::tempdir().expect("temp image directory"); From 9b4170f34ef75d385c04b3229ad778c1960a78b2 Mon Sep 17 00:00:00 2001 From: Kevin Wang Date: Tue, 22 Sep 2026 23:30:13 -0700 Subject: [PATCH 23/25] refactor(verifier): measure the OS image on a blocking thread compute_measurements hashes the firmware and the kernel and generates the ACPI tables -- pure CPU over a multi-hundred-megabyte image -- and was called synchronously from async fns, so it parked a Rocket worker for the whole computation. tpm-qvl already fetches collateral through spawn_blocking for the same reason. Pass ImagePaths instead of its fields one by one, and run the measurement through spawn_blocking. The cache lookup and store stay on the async side: they are small file operations. --- dstack/verifier/src/verification.rs | 61 ++++++++--------------------- 1 file changed, 17 insertions(+), 44 deletions(-) diff --git a/dstack/verifier/src/verification.rs b/dstack/verifier/src/verification.rs index 6313177ae..c4a844bcc 100644 --- a/dstack/verifier/src/verification.rs +++ b/dstack/verifier/src/verification.rs @@ -216,18 +216,10 @@ impl CvmVerifier { Ok(()) } - fn compute_measurements( - &self, - vm_config: &VmConfig, - fw_path: &Path, - kernel_path: &Path, - initrd_path: &Path, - kernel_cmdline: &str, - kernel_header_normalized: bool, - ) -> Result { - let firmware = fw_path.display().to_string(); - let kernel = kernel_path.display().to_string(); - let initrd = initrd_path.display().to_string(); + fn compute_measurements(vm_config: &VmConfig, image: &ImagePaths) -> Result { + let firmware = image.fw_path.display().to_string(); + let kernel = image.kernel_path.display().to_string(); + let initrd = image.initrd_path.display().to_string(); // Prefer the explicit variant the image declared; pre-`ovmf_variant` // deployments fall back to the only layout that existed back then. @@ -239,11 +231,11 @@ impl CvmVerifier { .firmware(&firmware) .kernel(&kernel) .initrd(&initrd) - .kernel_cmdline(kernel_cmdline) + .kernel_cmdline(&image.kernel_cmdline) .root_verity(true) .hotplug_off(vm_config.hotplug_off) .maybe_two_pass_add_pages(vm_config.qemu_single_pass_add_pages) - .normalized_setup_header(kernel_header_normalized) + .normalized_setup_header(image.kernel_header_normalized) .maybe_pic(vm_config.pic) .maybe_qemu_version(vm_config.qemu_version.clone()) .maybe_pci_hole64_size(if vm_config.pci_hole64_size > 0 { @@ -264,14 +256,10 @@ impl CvmVerifier { .context("Failed to compute expected MRs") } - fn load_or_compute_measurements( + async fn load_or_compute_measurements( &self, vm_config: &VmConfig, - fw_path: &Path, - kernel_path: &Path, - initrd_path: &Path, - kernel_cmdline: &str, - kernel_header_normalized: bool, + image: ImagePaths, ) -> Result { let cache_key = Self::vm_config_cache_key(vm_config)?; @@ -279,14 +267,11 @@ impl CvmVerifier { return Ok(measurements); } - let measurements = self.compute_measurements( - vm_config, - fw_path, - kernel_path, - initrd_path, - kernel_cmdline, - kernel_header_normalized, - )?; + let vm_config = vm_config.clone(); + let measurements = + tokio::task::spawn_blocking(move || Self::compute_measurements(&vm_config, &image)) + .await + .context("measurement task failed")??; if let Err(e) = self.store_measurements_in_cache(&cache_key, &measurements) { warn!( @@ -555,14 +540,8 @@ impl CvmVerifier { ) -> Result { let image_paths = self.ensure_image_downloaded(vm_config).await?; - self.load_or_compute_measurements( - vm_config, - &image_paths.fw_path, - &image_paths.kernel_path, - &image_paths.initrd_path, - &image_paths.kernel_cmdline, - image_paths.kernel_header_normalized, - ) + self.load_or_compute_measurements(vm_config, image_paths) + .await } pub async fn verify(&self, request: VerificationRequest) -> Result { @@ -798,14 +777,8 @@ impl CvmVerifier { } let expected_mrs = self - .load_or_compute_measurements( - vm_config, - &image_paths.fw_path, - &image_paths.kernel_path, - &image_paths.initrd_path, - &image_paths.kernel_cmdline, - image_paths.kernel_header_normalized, - ) + .load_or_compute_measurements(vm_config, image_paths) + .await .context("Failed to compute expected measurements")?; assert_tdx_mrs_eq(&expected_mrs, &verified_mrs).context("MRs do not match")?; details.acpi_tables_verified = true; From 33a742d319ec7c5d1fc75bbbfa9dbaeec37fcd0e Mon Sep 17 00:00:00 2001 From: Kevin Wang Date: Tue, 22 Sep 2026 23:38:16 -0700 Subject: [PATCH 24/25] docs(verifier): correct per-platform event_log_verified semantics SEV-SNP has no runtime event log (identity comes from mr_config via HOST_DATA), Nitro Enclave and GCP TDX were missing, and the replay runs during quote verification, so a mismatch fails quote_verified. The --debug diff does not read event log entries, and the lite ACPI digest check is fatal rather than diagnostic. --- dstack/verifier/README.md | 4 ++-- dstack/verifier/src/types.rs | 30 +++++++++++++++++------------- 2 files changed, 19 insertions(+), 15 deletions(-) diff --git a/dstack/verifier/README.md b/dstack/verifier/README.md index 40db89e34..4572e4daf 100644 --- a/dstack/verifier/README.md +++ b/dstack/verifier/README.md @@ -35,7 +35,7 @@ against the returned evidence. "is_valid": true, "details": { "quote_verified": true, - "event_log_verified": true, // runtime event log only; see "Verification Process" + "event_log_verified": true, // app identity decoded; see "Verification Process" "os_image_hash_verified": true, "acpi_tables_verified": true, // true only when TDX ACPI table contents are verified "os_image_is_dev": false, // true=dev image, false=prod, null=unknown/N/A @@ -237,7 +237,7 @@ $ curl -s -d @quote.json localhost:8080/verify | jq The verifier performs the following verification steps: 1. **Quote Verification**: Validates the platform quote using the platform verifier: DCAP for TDX, AMD SNP report verification for SEV-SNP, NSM for Nitro Enclaves, and AWS NitroTPM attestation-document verification for EC2 NitroTPM. -2. **Event Log Verification**: Replays the runtime event log — RTMR3 on TDX, the corresponding launch measurement on SEV-SNP and AWS NitroTPM PCR14 — to ensure it reproduces the quoted register, and extracts app information from its payloads. Both the digests and the payloads are verified there, because `app_id`, `compose_hash` and the rest are read out of them. `event_log_verified` reports that step only. The boot-time event log a TDX quote carries is not replayed: RTMR 0-2 are verified in step 3 by comparing the quoted values against measurements recomputed from the OS image, which does not depend on the host's event log, and dstack defines no semantics for its payloads. +2. **Event Log Verification**: Decodes app information (`app_id`, `compose_hash`, ...) from evidence bound to the quote and reports it as `event_log_verified`. On dstack TDX, GCP TDX and AWS NitroTPM it is read from the runtime event log, whose digests and payloads step 1 already replayed against the quoted register (RTMR3, plus TPM PCR14 on GCP; PCR14 on NitroTPM), so a replay mismatch fails quote verification. SEV-SNP has no runtime event log and takes it from `mr_config`, bound through HOST_DATA; Nitro Enclaves derive it from the PCRs. The RTMR 0-2 entries of a TDX event log are not replayed: on dstack TDX those registers are verified in step 3 against measurements recomputed from the OS image, which does not depend on the host's event log, and dstack defines no semantics for their payloads. 3. **OS Image Hash Verification**: - Treats `vm_config` and any attached measurement material as untrusted inputs until they are bound to the hardware quote - For the full-image TDX path, downloads or loads the image identified by `os_image_hash`, checks the image checksum manifest, uses dstack-mr to compute expected MRTD/RTMR0-2, and compares them against the verified measurements from the quote diff --git a/dstack/verifier/src/types.rs b/dstack/verifier/src/types.rs index 031e692ac..5d37d9e15 100644 --- a/dstack/verifier/src/types.rs +++ b/dstack/verifier/src/types.rs @@ -79,21 +79,25 @@ impl PolicyBootInfo { #[derive(Debug, Clone, Default, Serialize)] pub struct VerificationDetails { pub quote_verified: bool, - /// Indicates that the runtime event log was replayed against the quote and - /// the app identity was decoded from it. + /// Indicates that the app identity was decoded from evidence bound to the + /// quote. /// - /// That is RTMR3 on TDX, and the corresponding launch PCR on SEV-SNP and - /// AWS NitroTPM: both the digests and the payloads are verified, because - /// `app_id`, `compose_hash` and the rest are read out of those payloads. + /// On dstack TDX, GCP TDX and AWS NitroTPM that evidence is the runtime + /// event log, already replayed during quote verification against RTMR3 + /// (plus TPM PCR14 on GCP) or PCR14 on NitroTPM, so a replay mismatch fails + /// `quote_verified` rather than this flag. Digests and payloads are both + /// covered, because `app_id`, `compose_hash` and the rest are read out of + /// the payloads. SEV-SNP has no runtime event log and takes the identity + /// from `mr_config`, bound through HOST_DATA; Nitro Enclave derives it from + /// the PCRs. /// - /// It says nothing about the boot-time event log. Nothing replays the - /// RTMR 0-2 entries a TDX quote carries, and that is deliberate: those - /// registers are verified by comparing the quoted values against - /// measurements recomputed from the OS image (see `os_image_hash_verified` - /// and `acpi_tables_verified`), which does not depend on the host's event - /// log at all. The boot event log is carried for diagnostics -- the - /// `--debug` RTMR diff and the three named ACPI digests the TDX lite path - /// cross-checks -- and dstack defines no semantics for its payloads. + /// It says nothing about the RTMR 0-2 entries a TDX event log carries; + /// nothing replays them. On dstack TDX those registers are verified by + /// comparing the quoted values against measurements recomputed from the OS + /// image (see `os_image_hash_verified`), which does not depend on the + /// host's event log. The TDX lite path only reads the three named ACPI + /// digests from it and requires them to match the recomputed ones; dstack + /// defines no semantics for the other payloads. pub event_log_verified: bool, pub os_image_hash_verified: bool, /// Indicates that TDX ACPI table contents were verified. From 54c78ba628cd464da723639f996a7f8477df22d8 Mon Sep 17 00:00:00 2001 From: Shaw Date: Wed, 23 Sep 2026 01:49:37 -0700 Subject: [PATCH 25/25] fix(os): pin kernel builder base before package snapshot --- os/image/README.md | 5 ++++- os/image/kernel-builder/Dockerfile | 4 +++- os/image/kernel-builder/build.sh | 3 ++- os/mkosi/versions.env | 3 +++ 4 files changed, 12 insertions(+), 3 deletions(-) diff --git a/os/image/README.md b/os/image/README.md index d1f4e1256..b98f516ba 100644 --- a/os/image/README.md +++ b/os/image/README.md @@ -52,7 +52,10 @@ image. The optional `artifacts.kernel_devel` manifest entry is published as `--kernel-devel.tar.gz` beside the image archives. It is not in `sha256sum.txt`, so it does not affect `os_image_hash`. `kernel-builder/` -builds the matching module-builder container image. +builds the matching module-builder container image. Its build script takes the +base image digest and Debian snapshot from `../mkosi/versions.env`; the base must +predate that snapshot so installed runtime libraries do not outrun the pinned +development packages. The Docker build rejects a Debian suite mismatch. ## Kernel setup-header normalization diff --git a/os/image/kernel-builder/Dockerfile b/os/image/kernel-builder/Dockerfile index 31b2d7f23..945fcb5d8 100644 --- a/os/image/kernel-builder/Dockerfile +++ b/os/image/kernel-builder/Dockerfile @@ -6,9 +6,10 @@ # kernel: its exported build tree plus a compiler from the same pinned Debian # snapshot the kernel was built with. Built by build.sh. -ARG BASE_IMAGE=debian:trixie-slim +ARG BASE_IMAGE FROM ${BASE_IMAGE} +ARG DEBIAN_RELEASE ARG DEBIAN_SNAPSHOT ARG KERNEL_RELEASE ARG IMAGE_VERSION=unknown @@ -17,6 +18,7 @@ ARG GIT_REVISION=unknown # Pin apt to the snapshot; trixie also ships deb822 sources that must go. RUN set -eux; \ suite=$(. /etc/os-release && echo "${VERSION_CODENAME}"); \ + test "$suite" = "${DEBIAN_RELEASE:?}"; \ rm -f /etc/apt/sources.list /etc/apt/sources.list.d/*; \ echo "deb [check-valid-until=no] http://snapshot.debian.org/archive/debian/${DEBIAN_SNAPSHOT:?} ${suite} main" \ > /etc/apt/sources.list; \ diff --git a/os/image/kernel-builder/build.sh b/os/image/kernel-builder/build.sh index b179f976d..9b98d894a 100755 --- a/os/image/kernel-builder/build.sh +++ b/os/image/kernel-builder/build.sh @@ -18,7 +18,8 @@ cp "$archive" "$context/kernel-devel.tar.gz" ${DOCKER:-docker} build \ --file "$SELF/Dockerfile" \ - --build-arg "BASE_IMAGE=debian:$DEBIAN_RELEASE-slim" \ + --build-arg "BASE_IMAGE=${KERNEL_BUILDER_BASE_IMAGE:?}" \ + --build-arg "DEBIAN_RELEASE=$DEBIAN_RELEASE" \ --build-arg "DEBIAN_SNAPSHOT=$DEBIAN_SNAPSHOT" \ --build-arg "KERNEL_RELEASE=$KERNEL_VERSION-dstack" \ --build-arg "IMAGE_VERSION=${IMAGE_VERSION:-$DSTACK_VERSION}" \ diff --git a/os/mkosi/versions.env b/os/mkosi/versions.env index 625c382f8..042c8bde4 100644 --- a/os/mkosi/versions.env +++ b/os/mkosi/versions.env @@ -4,6 +4,9 @@ KERNEL_VERSION=6.18.40 KERNEL_SHA256=3712fc1ec839e4daac981176c8518912e8f452650aaedfe4381da4419613a431 DEBIAN_RELEASE=trixie DEBIAN_SNAPSHOT=20260721T000000Z +# Module-builder root predates the snapshot so its installed libraries can be +# upgraded to the pinned development packages without repository downgrades. +KERNEL_BUILDER_BASE_IMAGE=debian:trixie-slim@sha256:020c0d20b9880058cbe785a9db107156c3c75c2ac944a6aa7ab59f2add76a7bd MKOSI_VERSION=26 # mkosi constructs the image, so it is part of the build toolchain and a # reproducibility input in its own right. The major version above is what