diff --git a/CHANGELOG.md b/CHANGELOG.md index a08edfc36..915483557 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -8,6 +8,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## [Unreleased] ### Added +- dstack-util: the measured `gpu-attestation` event now exposes aggregate `dbgstat` and `secboot` values from the NVIDIA-signed claims. `dbgstat` is `enabled` if any attested GPU reports enabled, while `secboot` is true only if every attested GPU reports secure boot; the event schema version is bumped to 3. - sdk: every SDK ships two clients, one per guest-agent surface, and nothing else. `DstackClientV0` speaks the frozen v0.5.11 API on the unversioned paths -- every method a v0.5.x SDK had, including `verify()` (which calls the server `Verify` RPC) and `emit_event()` (which surfaces the agent's removal error verbatim). `DstackClientV1` speaks `dstack.guest.v1` at `/v1` and has exactly its six methods: `issue_cert`, `get_key`, `attest`, `attest_gpu`, `info`, `version`. They are transport mirrors, not a compatibility layer: neither translates a call to the other, and each one's method set is its surface's method set. **v1 derives different key material than v0 for the same inputs** -- see `docs/guest-api-v1.md` for the migration. `TappdClient` is untouched. The blockchain adapters (Ethereum/viem, Solana) remain **v0-only** and stay typed against the v0 `GetKeyResponse`. v1 has no chain-related surface, deliberately: it returns key material, and what an application builds from those bytes is its own business rather than something the SDKs model. @@ -94,6 +95,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - os: the Yocto guest-OS backend (`os/yocto/`) is deprecated in favor of mkosi (`os/mkosi/`), which is now the default and recommended backend. `os/build.sh` defaults to `--backend mkosi`, and `make os-image` / `make os-repro-check` build with mkosi. The Yocto builds move to `make os-image-yocto` / `make os-repro-check-yocto`; `make os-image-mkosi` / `make os-repro-check-mkosi` remain as aliases. Every Yocto entrypoint prints a deprecation warning, and the backend is kept only to rebuild existing Yocto images. The mkosi build still reads patches, units and scripts from `os/yocto/`, so the directory stays until those files move ### Removed +- verifier: the `debug` request field and the `acpi_tables` / `rtmr_debug` response fields. The per-event RTMR diff never had the events it diffed, so it reported every expected digest as missing. Requests that still send `debug` are accepted and the field is ignored. - sdk: `TlsKeyOptions.path` in the JavaScript SDK. `GetTlsKeyArgs` has no such field and `getTlsKey` never read it, so a caller who set it was silently ignored. Breaking at the type level only, and only for code whose value was already being discarded. `deriveKey`'s `path` is a real, deprecated Tappd-era parameter and stays; the Python, Rust and Go v0 TLS-key options never carried one - guest-agent: the `EmitEvent` RPC no longer records anything -- runtime RTMR3 events are system-owned in 0.6.0, so an app can no longer extend the measurement chain. The method itself stays on the unversioned path and always fails with an error naming the removal, rather than being deleted outright: a deleted method answers HTTP 404 `Service not found: EmitEvent`, which tells a 0.5.x caller nothing about why its events stopped being recorded, while the kept stub fails with a message naming the removal and pointing at `report_data`. **Breaking:** any app extending RTMR3 at runtime must stop; bind app data through `report_data` instead, which is what most callers wanted anyway - gateway: `core.debug.insecure_skip_attestation`. It turned off both checks that make a gateway cluster a trust boundary: the node stopped asking its guest agent for its own app id, and every WaveKV sync and push, plus `ensure_from_gateway`, stopped checking the peer's. With it set, anything that could reach the sync routes could insert entries that replicated to every gateway in the cluster. It existed only because the integration suites could not run without it; they now run against a guest agent simulator and verify quotes through the production path, so nothing sets it. `docs/security/security-model.md` argues dstack's development switches are acceptable because they are visible in attestation measurements or public contract state -- that argument cannot cover the switch deciding whether attestation happens at all, which is why this one is deleted rather than documented. There is no replacement. **Breaking:** the config struct does not use `deny_unknown_fields`, so a leftover line is ignored rather than rejected, and what follows depends on why it was set. On a TDX host with a guest agent the gateway starts normally and peers that cannot present a verifiable app id stop being accepted. On a host with no guest agent -- the usual reason to have set it -- the gateway no longer starts at all, failing with `Failed to get app info`, because `my_app_id` is what every peer check compares against and a node that cannot learn its own identity must not come up without one. Remove the line and make sure every node can attest diff --git a/docs/attestation-tdx.md b/docs/attestation-tdx.md index 14c20bcf5..f9744e4ba 100644 --- a/docs/attestation-tdx.md +++ b/docs/attestation-tdx.md @@ -43,7 +43,8 @@ For a GPU launch, any `init-script-hash` events are followed by evaluation, `gpu-attestation`. The `gpu-policy-hash` payload is `SHA-256(JCS(requirements.gpu_policy))`, using `{}` when the field is omitted. The `gpu-attestation` payload is JSON containing the verified device count, -CC/DevTools state, and `evidence_sha256`. +CC/DevTools state, aggregate signed-claim `dbgstat` and `secboot`, and +`evidence_sha256`. The guest-agent returns the complete `nvattest` record captured during boot from `/v1/Attest`, when the request sets `include_boottime_gpu_evidence`, so a verifier can fetch the quote and the GPU evidence in one round trip. `AttestResponse.boottime_gpu_evidence` is a list of `GpuEvidenceBundle` (`{vendor, format, evidence}`); the boot record is the bundle whose `vendor` is `nvidia` and whose `format` is `nvidia-nvattest-boottime-json-v1`, and its `evidence` is hex-encoded bytes that decode to the exact UTF-8 `nvattest` output. It is not trustworthy by itself. (`/v1/AttestGpu` runs a *fresh* attestation against a caller nonce and returns bundles tagged `nvidia-nvattest-collect-evidence-json-v1`, a deliberately distinct format that a boot-record verifier does not appraise; its result is not bound to the TD and must not be used as remote evidence; only the boot-time record below is.) After verifying the TDX quote and replaying the event log to RTMR3, hash the *decoded* bundle bytes — `SHA-256(hex_decode(bundle.evidence))`, never the JSON string as returned nor a re-serialized form — and require the result to equal the `gpu-attestation` event's `evidence_sha256`. See [GPU Security for AI Workloads](./security/security-model.md#gpu-security-for-ai-workloads) for the event schema, ordering, Rego example, and platform differences. diff --git a/docs/auth-simple-operations.md b/docs/auth-simple-operations.md index 416df9e4c..6b8826d8f 100644 --- a/docs/auth-simple-operations.md +++ b/docs/auth-simple-operations.md @@ -227,6 +227,8 @@ The Gateway is a special app that routes traffic to other apps. Once deployed, a The `gatewayAppId` is returned in boot responses and used by KMS for key derivation. +To trust several gateways, join their App IDs with commas (`"0xaaa...,0xbbb..."`). CVMs accept a gateway whose App ID is contained in this value. + --- ## KMS Onboarding (Multi-Node) diff --git a/docs/aws-attested-instance-security-evaluation.md b/docs/aws-attested-instance-security-evaluation.md index 4f3ca9682..3575da342 100644 --- a/docs/aws-attested-instance-security-evaluation.md +++ b/docs/aws-attested-instance-security-evaluation.md @@ -35,7 +35,7 @@ NitroTPM path meets them today. | P1 | Verifiable platform root of trust | TDX/SNP/Nitro quote verification against vendor root; debug rejected; TCB surfaced | NitroTPM Attestation Documents are verified against the AWS Nitro Attestation PKI, including document timestamp sanity. AWS exposes no TDX/SNP-style TCB advisory field, so a verified attestation is normalized to `tcbStatus = "UpToDate"` and passes the standard authorization gate unchanged. | | P2 | Reproducible or independently computable base image measurement | meta-dstack rebuild plus `dstack-mr` computes `MRTD`/`RTMR0-2` | The unified `os/build.sh` flow emits the AWS image archive with `sha256sum.txt`, `digest.txt`, and `measurement.aws.cbor`; its output directory also contains the reference-PCR side-car `aws-pcrs.json`. `os_image_hash = sha256(sha256sum.txt)` is the same identity used on all platforms; the verifier recomputes it from the downloaded image directory (`dstack/verifier/src/verification.rs`). The hardening audit script `os/yocto/tools/aws/audit-aws-ec2-image-hardening.sh` checks the image for operator mutation channels. | | P3 | Boot command line and root filesystem integrity are measured | `RTMR1/2`, rootfs hash, dm-verity, measured initrd/cmdline | The UKI commits kernel, initrd, and embedded cmdline into `PCR4`; the rootfs is dm-verity-protected. `VmConfig.aws_measurement` is required and must bind `boot_pcr_digest = sha256(PCR4||PCR7||PCR12)` to the attested PCRs, so `PCR12` (external cmdline) is always part of the bound digest — a missing-PCR12 bypass is not expressible. Enforced in guest quote generation (`dstack/dstack-attest/src/attestation.rs`), `verify_os_image_hash_for_aws_nitro_tpm` (`dstack/verifier/src/verification.rs`), and the KMS pipeline via the same verifier check. | -| P4 | Runtime application identity is cryptographically bound | RTMR3 `compose-hash`, `app-id`, `instance-id`, `key-provider`; event log replay | SHA384 `PCR14` event-log replay is the authoritative binding (RTMR3 analogue; non-resettable). Launch events: `system-preparing`, `app-id`, `compose-hash`, zero or more ordered `init-script-hash` events, `instance-id`, `boot-mr-done`, `key-provider`, `storage-fs`, `system-ready` (`dstack/dstack-util/src/system_setup.rs`). GPU launches also include `gpu-policy-hash` and `gpu-attestation` before `instance-id`. `dstack-attest`, `dstack-verifier`, and KMS reject missing/mismatched PCR14 and bad replay. Optionally, the guest extends the raw `MrConfig` V2 `config_id` into `PCR8` once (`PCR8 = sha384(0^48 || config_id)`) so a lightweight third-party verifier can check compose hash + key provider without event-log replay; dstack's own verifier and KMS do not check PCR8. | +| P4 | Runtime application identity is cryptographically bound | RTMR3 `compose-hash`, `app-id`, `instance-id`, `key-provider`; event log replay | SHA384 `PCR14` event-log replay is the authoritative binding (RTMR3 analogue; non-resettable). Launch events: `system-preparing`, `app-id`, `compose-hash`, zero or more ordered `init-script-hash` events, `instance-id`, `boot-mr-done`, `key-provider`, `storage-fs`, `storage-encrypted`, `system-ready` (`dstack/dstack-util/src/system_setup.rs`). GPU launches also include `gpu-policy-hash` and `gpu-attestation` before `instance-id`. `dstack-attest`, `dstack-verifier`, and KMS reject missing/mismatched PCR14 and bad replay. Optionally, the guest extends the raw `MrConfig` V2 `config_id` into `PCR8` once (`PCR8 = sha384(0^48 || config_id)`) so a lightweight third-party verifier can check compose hash + key provider without event-log replay; dstack's own verifier and KMS do not check PCR8. | | P5 | Challenge/liveness and caller key binding | `report_data` challenge or RA-TLS public key hash in quote | RA-TLS binds `report_data` to the TLS certificate public key. KMS key release is bound to the live RA-TLS handshake; external `/verify` callers supply and check their own `report_data` challenge. The low-level NitroTPM document verifier also rejects stale or far-future document timestamps. | | P6 | Secret release only to attested code | dstack KMS verifies attestation, checks auth policy, derives per-app keys | dstack KMS verifies the NitroTPM attestation, runs the same `verify_os_image_hash_for_aws_nitro_tpm` binding check as the verifier, builds `BootInfo` from verified boot PCRs plus PCR14 launch events, and checks auth policy before deriving app keys (`dstack/kms/src/main_service.rs`). AWS NitroTPM key release is gated behind the opt-in `aws_nitro_tpm_key_release` flag (default false in `kms.toml`). | | P7 | Key-release policy is not controlled by the untrusted account admin | KMS runs inside TEE; policy from auth API/contracts; KMS identity measured | Satisfied with dstack KMS or another verifiable secret authority outside the untrusted AWS account admin's control. A NitroTPM-backed dstack KMS keeps root material out of account-admin snapshots and clones. The policy backend (auth-simple in a trusted control plane, or on-chain `DstackKms`/`DstackApp`) must be outside the workload account admin's control. Same-account AWS KMS fails this property if the admin can change key policy, create grants, or call secret-bearing operations through a policy they control. | diff --git a/docs/onchain-governance.md b/docs/onchain-governance.md index 42099c069..777675399 100644 --- a/docs/onchain-governance.md +++ b/docs/onchain-governance.md @@ -98,6 +98,8 @@ GATEWAY_APP_ID= \ Output: `Set gateway app ID: ` +To trust more than one gateway (for example during a gateway migration), set the value to several app IDs joined by commas, e.g. `,`. A CVM accepts a gateway whose certificate carries any app ID contained in this string. + Add the gateway's compose hash to the whitelist. To compute the compose hash: ```bash @@ -155,7 +157,7 @@ The central governance contract that manages OS image whitelisting, app registra |----------|-------------| | `addOsImageHash(bytes32)` | Whitelist an OS image hash | | `removeOsImageHash(bytes32)` | Remove an OS image from whitelist | -| `setGatewayAppId(string)` | Set the trusted Gateway app ID | +| `setGatewayAppId(string)` | Set the trusted Gateway app ID(s) | | `registerApp(address)` | Register an app contract | | `deployAndRegisterApp(...)` | Deploy and register app in one transaction | | `isAppAllowed(AppBootInfo)` | Check if an app is allowed to boot | diff --git a/docs/security/security-model.md b/docs/security/security-model.md index 6e7427cea..beed20f01 100644 --- a/docs/security/security-model.md +++ b/docs/security/security-model.md @@ -118,7 +118,7 @@ For every NVML-enumerated GPU, dstack calls `Device::is_cc_enabled()` and `Devic ### Dual Attestation -GPU workloads require verification of both hardware components. The CPU TEE quote verifies the CVM and its measured guest code. NVIDIA-signed evidence, checked against NVIDIA RIMs and certificate status by `nvattest`, verifies the GPU appraisal. After the optional policy and ready-state operations succeed, dstack emits a `gpu-attestation` launch event before `system-ready`. Its versioned payload records the number of appraised devices, asserted CC/DevTools state, and SHA-256 of the complete nvattest JSON output (claims and detached EAT). On TDX, both `gpu-policy-hash` and `gpu-attestation` are append-only RTMR3 events; they are never derived from application-controlled `report_data`. +GPU workloads require verification of both hardware components. The CPU TEE quote verifies the CVM and its measured guest code. NVIDIA-signed evidence, checked against NVIDIA RIMs and certificate status by `nvattest`, verifies the GPU appraisal. After the optional policy and ready-state operations succeed, dstack emits a `gpu-attestation` launch event before `system-ready`. Its versioned payload records the number of appraised devices, asserted CC/DevTools state, aggregate signed-claim debug and secure-boot status, and SHA-256 of the complete nvattest JSON output (claims and detached EAT). On TDX, both `gpu-policy-hash` and `gpu-attestation` are append-only RTMR3 events; they are never derived from application-controlled `report_data`. For a successful TDX GPU launch, the GPU-relevant RTMR3 event order is: @@ -135,15 +135,23 @@ boot-mr-done ```json { - "version": 2, + "version": 3, "provider": "nvidia", "devices": 1, "cc_mode": "on", "devtools": false, + "dbgstat": "disabled", + "secboot": true, "evidence_sha256": "" } ``` +`devtools` is the aggregate NVML state and is true if any device has DevTools +mode enabled. `dbgstat` and `secboot` summarize the NVIDIA-signed claims: +`dbgstat` is `enabled` if any device reports enabled, and `secboot` is true only +if every device reports secure boot. The complete per-device claims remain in +the boot-time evidence committed by `evidence_sha256`. + `/v1/Attest` returns that complete boot-time `nvattest` record when the request sets `include_boottime_gpu_evidence`; it runs no new attestation. `AttestResponse.boottime_gpu_evidence` is a list of `GpuEvidenceBundle`, each carrying `vendor`, `format`, and `evidence`. The boot record is the bundle whose `vendor` is `nvidia` and whose `format` is `nvidia-nvattest-boottime-json-v1`; its `evidence` is hex-encoded bytes that decode to the exact UTF-8 nvattest output as the agent read it from disk. The other format, `nvidia-nvattest-collect-evidence-json-v1`, comes from `/v1/AttestGpu` and is fresh evidence against a caller nonce; the two are deliberately distinct because a verifier for one does not appraise the other. To bind the API result to TDX evidence: verify the quote, replay the event log to the quote's RTMR3, require exactly one pre-`system-ready` `gpu-attestation` event, decode its JSON payload, select the boot-time bundle by `format`, and compare `evidence_sha256` with `SHA-256(hex_decode(bundle.evidence))`. Hash the decoded bytes, not the JSON string as returned and not a re-serialized form: parsing and re-serializing changes the digest and breaks the comparison. Only after this comparison should the verifier inspect the returned claims. This exact-byte comparison includes any whitespace or trailing newline in the decoded record. A verifier must replay the measured event log, require exactly one `gpu-policy-hash` event immediately after `compose-hash`, and compare its 32-byte payload with the expected policy digest (`SHA-256(JCS({}))` for the omitted/default policy). When MrConfigV3 includes `gpu_policy_hash`, it must match the same digest. When GPU protection is required, the verifier must also require exactly one pre-`system-ready` `gpu-attestation` event with `devices > 0` and, when applicable, the expected deployment count. The raw `attestation.out` file is not trusted by itself; if it is supplied for inspection, its digest must match the `gpu-attestation` event. diff --git a/docs/tutorials/attestation-verification.md b/docs/tutorials/attestation-verification.md index 9808ef94c..1cc77c745 100644 --- a/docs/tutorials/attestation-verification.md +++ b/docs/tutorials/attestation-verification.md @@ -486,12 +486,13 @@ These are the standard events you'll see in the log: | `compose-hash` | SHA-256 of docker compose config | Should match `tcb_info.compose_hash` | | `init-script-hash` | SHA-256 of one init script; repeated in configured order (maximum 5) | Should match the independently approved script bytes | | `gpu-policy-hash` | SHA-256 of the JCS-canonicalized GPU policy (default `{}`) | Should match the expected `requirements.gpu_policy` digest | -| `gpu-attestation` | Verified GPU state and digest of the boot-time `nvattest` JSON | Required for an attested GPU launch; verify as described below | +| `gpu-attestation` | Verified GPU state, aggregate `dbgstat`/`secboot`, and digest of the boot-time `nvattest` JSON | Required for an attested GPU launch; verify as described below | | `instance-id` | Unique instance identifier | Should match `instance_id` from response | | `boot-mr-done` | Boot measurements complete | Marker event | | `os-image-hash` | Guest OS image hash | Should match `tcb_info.os_image_hash` | | `key-provider` | Key provider type | e.g., `kms` | | `storage-fs` | Storage filesystem type | Storage configuration | +| `storage-encrypted` | `1` when the data disk is LUKS-encrypted, `0` when it is not | `0` means the host can read the application's data at rest | | `system-ready` | System ready marker | Always present at end | For a successful GPU launch, the relevant order is `compose-hash`, any diff --git a/dstack/dstack-mr/cli/src/main.rs b/dstack/dstack-mr/cli/src/main.rs index 6f52081d3..d8e4f0211 100644 --- a/dstack/dstack-mr/cli/src/main.rs +++ b/dstack/dstack-mr/cli/src/main.rs @@ -325,7 +325,7 @@ fn run_diagnose(config: &DiagnoseConfig) -> Result<()> { let initrd = image_dir.join(&image_info.initrd).display().to_string(); let cmdline = dstack_mr::tdx::measured_kernel_cmdline(&image_info.cmdline); - // Same resolution order as the verifier (see verifier::compute_measurement_details): + // Same resolution order as the verifier (see verifier::compute_measurements): // explicit vm_config.ovmf_variant > image_info.ovmf_variant > legacy default. let ovmf_variant = vm .ovmf_variant diff --git a/dstack/dstack-mr/src/machine.rs b/dstack/dstack-mr/src/machine.rs index 7925d3df0..c30323030 100644 --- a/dstack/dstack-mr/src/machine.rs +++ b/dstack/dstack-mr/src/machine.rs @@ -2,7 +2,6 @@ // // SPDX-License-Identifier: Apache-2.0 -use crate::acpi::Tables; use crate::tdvf::Tdvf; use crate::util::debug_print_log; use crate::{kernel, OvmfVariant, RtmrLogs, TdxMeasurements}; @@ -115,7 +114,6 @@ pub struct VersionedOptions { pub struct TdxMeasurementDetails { pub measurements: TdxMeasurements, pub rtmr_logs: RtmrLogs, - pub acpi_tables: Tables, } impl Machine<'_> { @@ -132,7 +130,7 @@ impl Machine<'_> { let mrtd = tdvf.mrtd(self).context("Failed to compute MR TD")?; - let (rtmr0_log, acpi_tables) = tdvf + let rtmr0_log = tdvf .rtmr0_log(self) .context("Failed to compute RTMR0 log")?; debug_print_log("RTMR0", &rtmr0_log); @@ -163,7 +161,6 @@ impl Machine<'_> { rtmr2, }, rtmr_logs: [rtmr0_log, rtmr1_log, rtmr2_log], - acpi_tables, }) } } diff --git a/dstack/dstack-mr/src/tdvf.rs b/dstack/dstack-mr/src/tdvf.rs index 3f6393713..23a922151 100644 --- a/dstack/dstack-mr/src/tdvf.rs +++ b/dstack/dstack-mr/src/tdvf.rs @@ -7,7 +7,6 @@ use hex_literal::hex; use scale::Decode; use sha2::{Digest, Sha384}; -use crate::acpi::Tables; use crate::num::read_le; use crate::{measure_log, measure_sha384, utf16_encode, Machine, OvmfVariant, RtmrLog}; @@ -497,23 +496,17 @@ impl<'a> Tdvf<'a> { #[allow(dead_code)] pub fn rtmr0(&self, machine: &Machine) -> Result> { - let (rtmr0_log, _) = self.rtmr0_log(machine)?; - Ok(measure_log(&rtmr0_log)) + Ok(measure_log(&self.rtmr0_log(machine)?)) } - pub fn rtmr0_log(&self, machine: &Machine) -> Result<(RtmrLog, Tables)> { + pub fn rtmr0_log(&self, machine: &Machine) -> Result { let tables = machine.build_tables()?; let acpi_hashes = AcpiTableHashes { tables: measure_sha384(&tables.tables), rsdp: measure_sha384(&tables.rsdp), loader: measure_sha384(&tables.loader), }; - let log = self.rtmr0_log_with_acpi_hashes( - machine.memory_size, - machine.ovmf_variant, - &acpi_hashes, - )?; - Ok((log, tables)) + self.rtmr0_log_with_acpi_hashes(machine.memory_size, machine.ovmf_variant, &acpi_hashes) } pub(crate) fn rtmr0_log_with_acpi_hashes( diff --git a/dstack/dstack-util/src/system_setup.rs b/dstack/dstack-util/src/system_setup.rs index 4b37b74cf..d99fa6baa 100644 --- a/dstack/dstack-util/src/system_setup.rs +++ b/dstack/dstack-util/src/system_setup.rs @@ -1362,7 +1362,7 @@ async fn do_sys_setup(stage0: Stage0<'_>) -> Result<()> { mod gpu { use super::*; - const EVENT_VERSION: u32 = 2; + const EVENT_VERSION: u32 = 3; const POLICY_ENTRYPOINT: &str = "data.policy.nv_match"; /// Bound Rego evaluation so a runaway application policy cannot hang boot. const POLICY_TIMEOUT: Duration = Duration::from_secs(10); @@ -1376,6 +1376,8 @@ mod gpu { devices: u32, cc_mode: &'static str, devtools: bool, + dbgstat: NvidiaGpuDebugStatus, + secboot: bool, evidence_sha256: String, } @@ -1392,7 +1394,17 @@ mod gpu { } pub(super) fn event(&self, devtools: bool) -> Result> { - attestation_event(&self.output, self.devices, devtools) + let dbgstat = if self + .parsed_claims + .iter() + .any(|claim| claim.dbgstat == NvidiaGpuDebugStatus::Enabled) + { + NvidiaGpuDebugStatus::Enabled + } else { + NvidiaGpuDebugStatus::Disabled + }; + let secboot = self.parsed_claims.iter().all(|claim| claim.secboot); + attestation_event(&self.output, self.devices, devtools, dbgstat, secboot) } pub(super) fn verify_claim_policy( @@ -1443,7 +1455,7 @@ mod gpu { dbgstat: NvidiaGpuDebugStatus, } - #[derive(Debug, Deserialize, PartialEq, Eq)] + #[derive(Debug, Clone, Copy, Deserialize, Serialize, PartialEq, Eq)] #[serde(rename_all = "lowercase")] enum NvidiaGpuDebugStatus { Disabled, @@ -1596,13 +1608,21 @@ mod gpu { Ok(()) } - fn attestation_event(stdout: &[u8], devices: u32, devtools: bool) -> Result> { + fn attestation_event( + stdout: &[u8], + devices: u32, + devtools: bool, + dbgstat: NvidiaGpuDebugStatus, + secboot: bool, + ) -> Result> { let event = GpuAttestationEvent { version: EVENT_VERSION, provider: "nvidia", devices, cc_mode: "on", devtools, + dbgstat, + secboot, evidence_sha256: hex::encode(sha256(stdout)), }; serde_json::to_vec(&event).context("failed to serialize GPU attestation event") @@ -1959,13 +1979,17 @@ mod gpu { fn event_commits_to_complete_nvattest_output() { let nonce = "22".repeat(32); let output = nvattest_output(&nonce, 1); - let event: Value = - serde_json::from_slice(&attestation_event(&output, 1, true).unwrap()).unwrap(); + let event: Value = serde_json::from_slice( + &attestation_event(&output, 1, true, NvidiaGpuDebugStatus::Enabled, false).unwrap(), + ) + .unwrap(); assert_eq!(event["version"], EVENT_VERSION); assert_eq!(event["devices"], 1); assert!(event.get("policy").is_none()); assert_eq!(event["cc_mode"], "on"); assert_eq!(event["devtools"], true); + assert_eq!(event["dbgstat"], "enabled"); + assert_eq!(event["secboot"], false); assert_eq!(event["evidence_sha256"], hex::encode(sha256(&output))); } @@ -2216,6 +2240,11 @@ struct AppIdValidator { } impl AppIdValidator { + /// `allowed_app_id` may list several gateway app ids (e.g. + /// `","`), so the peer is accepted if its hex app id appears + /// anywhere in it. This relies on the KMS only issuing certificates with + /// 20-byte app ids (`ensure_app_id_len`), so a shorter id cannot match a + /// fragment. fn validate(&self, cert: Option) -> Result<()> { if self.allowed_app_id == "any" { return Ok(()); @@ -2307,7 +2336,7 @@ impl<'a> Stage0<'a> { self.shared.dir.join(APP_KEYS) } - async fn request_app_keys_from_kms_url(&self, kms_url: String) -> Result { + async fn request_app_keys_from_kms_url(&self, kms_url: String) -> Result<(AppKeys, Vec)> { info!("Requesting app keys from KMS: {kms_url}"); let tmp_ca = { info!("Getting temp ca cert"); @@ -2341,9 +2370,6 @@ impl<'a> Stage0<'a> { .await .context("Failed to get app key")?; - emit_runtime_event("os-image-hash", &response.os_image_hash) - .context("failed to extend os-image-hash to the launch measurement")?; - let (_, ca_pem) = x509_parser::pem::parse_x509_pem(tmp_ca.ca_cert.as_bytes()) .context("Failed to parse ca cert")?; let x509 = ca_pem.parse_x509().context("Failed to parse ca cert")?; @@ -2363,34 +2389,31 @@ impl<'a> Stage0<'a> { tmp_ca_cert: tmp_ca.temp_ca_cert, }, }; - Ok(keys) + Ok((keys, response.os_image_hash)) } async fn request_app_keys_from_kms(&self) -> Result { if self.shared.sys_config.kms_urls.is_empty() { bail!("No KMS URLs are set"); } - let keys = 'out: { - let mut error = anyhow!("unknown error"); - for (i, kms_url) in self.shared.sys_config.kms_urls.iter().enumerate() { - let kms_url = kms_rpc_url(kms_url); - let response = self.request_app_keys_from_kms_url(kms_url.clone()).await; - match response { - Ok(response) => { - break 'out response; - } - Err(err) => { - warn!("Failed to get app keys from KMS {kms_url}: {err:?}"); - // Record the first error - if i == 0 { - error = err; - } - } + let mut errors = vec![]; + for kms_url in &self.shared.sys_config.kms_urls { + let kms_url = kms_rpc_url(kms_url); + match self.request_app_keys_from_kms_url(kms_url.clone()).await { + Ok((keys, os_image_hash)) => { + // Measured here, outside the per-URL request, so that a + // failover can never extend os-image-hash twice. + emit_runtime_event("os-image-hash", &os_image_hash) + .context("failed to extend os-image-hash to the launch measurement")?; + return Ok(keys); + } + Err(err) => { + warn!("failed to get app keys from KMS {kms_url}: {err:?}"); + errors.push(format!("{kms_url}: {err:#}")); } } - return Err(error).context("Failed to get app keys from KMS"); - }; - Ok(keys) + } + Err(anyhow!(errors.join("; "))).context("failed to get app keys from KMS") } fn verify_key_provider_id(&self, provider_id: &[u8]) -> Result<()> { @@ -3037,6 +3060,11 @@ impl<'a> Stage0<'a> { // Parse kernel command line options let opts = parse_dstack_options(&self.shared).context("Failed to parse kernel cmdline")?; emit_runtime_event("storage-fs", opts.storage_fs.to_string().as_bytes())?; + // The cmdline is not measured on every platform (e.g. GCP TDX). + emit_runtime_event( + "storage-encrypted", + if opts.storage_encrypted { b"1" } else { b"0" }, + )?; info!( "Filesystem options: encryption={}, filesystem={:?}", opts.storage_encrypted, opts.storage_fs diff --git a/dstack/dstack-util/src/system_setup/config_id_verifier.rs b/dstack/dstack-util/src/system_setup/config_id_verifier.rs index fdb00c034..4e4d97142 100644 --- a/dstack/dstack-util/src/system_setup/config_id_verifier.rs +++ b/dstack/dstack-util/src/system_setup/config_id_verifier.rs @@ -184,10 +184,10 @@ fn verify_mr_config_v3_document( bail!("Invalid mr_config app_id"); } } - if let Some(instance_id) = mr_config.instance_id.as_deref() { - if instance_id != local.instance_id { - bail!("Invalid mr_config instance_id"); - } + // Absent means empty: a relying party cannot tell an omitted instance_id + // from `no_instance_id`, so it must not skip the check. + if mr_config.instance_id.as_deref().unwrap_or_default() != local.instance_id { + bail!("Invalid mr_config instance_id"); } if mr_config.key_provider != local.key_provider { bail!("Invalid mr_config key_provider"); @@ -305,6 +305,33 @@ mod tests { Ok(()) } + #[test] + fn mr_config_v3_treats_missing_instance_id_as_empty() { + let compose_hash = [0x22u8; 32]; + let app_id = [0x11u8; 20]; + let document = MrConfigV3::new( + app_id.to_vec(), + compose_hash.to_vec(), + None, + KeyProviderKind::Kms, + Vec::new(), + Vec::new(), + ) + .to_canonical_json(); + let local = |instance_id| LocalMrConfigValues { + compose_hash: &compose_hash, + gpu_policy_hash: &[0x55u8; 32], + init_script_hashes: &[], + app_id: &app_id, + instance_id, + key_provider: KeyProviderKind::Kms, + key_provider_id: &[], + }; + + assert!(verify_mr_config_v3_document(&document, local(&[0x44u8; 20])).is_err()); + assert!(verify_mr_config_v3_document(&document, local(&[])).is_ok()); + } + #[test] fn mr_config_v3_document_must_match_expected_gpu_policy_hash() { let compose_hash = [0x22u8; 32]; diff --git a/dstack/kms/kms.toml b/dstack/kms/kms.toml index 3a6ad527e..dc6c682b6 100644 --- a/dstack/kms/kms.toml +++ b/dstack/kms/kms.toml @@ -75,6 +75,8 @@ insecure_allow_external_trust_anchors = false # sev_snp_turin = "/etc/dstack/roots/amd-turin-ark.pem" [core.image] +# Verify os_image_hash against the booted image. If false, os_image_hash is +# caller-supplied, so allowedOsImages is not enforced. Never disable in production. verify = true cache_dir = "/usr/share/dstack/images" download_url = "http://localhost:8000/{OS_IMAGE_HASH}.tar.gz" diff --git a/dstack/kms/rpc/proto/kms_rpc.proto b/dstack/kms/rpc/proto/kms_rpc.proto index e491d9335..8c7bbe898 100644 --- a/dstack/kms/rpc/proto/kms_rpc.proto +++ b/dstack/kms/rpc/proto/kms_rpc.proto @@ -58,6 +58,9 @@ message GetMetaResponse { optional string kms_contract_address = 7; optional uint64 chain_id = 8; optional string app_auth_implementation = 9; + // `[core.image] verify`: whether `os_image_hash` is checked against the + // booted image. Unset on older KMS; do not treat unset as true. + optional bool os_image_verification = 10; } message GetKmsKeyRequest { diff --git a/dstack/kms/src/main_service.rs b/dstack/kms/src/main_service.rs index 5e9bc3ffd..d4bad106a 100644 --- a/dstack/kms/src/main_service.rs +++ b/dstack/kms/src/main_service.rs @@ -30,7 +30,7 @@ use ra_tls::{ }; use scale::Decode; use tokio::sync::OnceCell; -use tracing::{info, warn}; +use tracing::{debug, warn}; use upgrade_authority::{ build_boot_info, ensure_app_id_len, local_kms_boot_info, BootInfo, GetInfoResponse, }; @@ -178,6 +178,11 @@ impl KmsState { "self-authorization is disabled; trusted RPCs will not be gated by KMS self-attestation - do not use in production TEE deployments" ); } + if !config.image.verify { + warn!( + "os image verification is disabled; os_image_hash is caller-supplied and unverified - do not use in production TEE deployments" + ); + } Ok(Self { inner: Arc::new(KmsStateInner { config, @@ -336,13 +341,13 @@ impl RpcHandler { report: &VerifiedAttestation, ) -> Result<()> { if !self.state.config.image.verify { - info!("Image verification is disabled"); + debug!("os image verification is disabled"); return Ok(()); } let mut detail = VerificationDetails::default(); self.state .verifier - .verify_os_image_hash(vm_config, report, false, &mut detail) + .verify_os_image_hash(vm_config, report, &mut detail) .await .context("Failed to verify os image hash")?; Ok(()) @@ -521,6 +526,7 @@ impl KmsRpc for RpcHandler { chain_id: info.chain_id, gateway_app_id: info.gateway_app_id, app_auth_implementation: info.app_implementation, + os_image_verification: Some(self.state.config.image.verify), }) } diff --git a/dstack/tests/e2e/attestation/run-platform.sh b/dstack/tests/e2e/attestation/run-platform.sh index b025b511c..1ce12ac1c 100755 --- a/dstack/tests/e2e/attestation/run-platform.sh +++ b/dstack/tests/e2e/attestation/run-platform.sh @@ -186,7 +186,7 @@ if ! ATTESTATION_HEX=$(dstack-util attest --report-data "$REPORT_DATA" --hex); t exit 1 fi jq -n --arg attestation "$ATTESTATION_HEX" \ - '{attestation: $attestation, debug: true}' > "$WORK/request.json" + '{attestation: $attestation}' > "$WORK/request.json" cat > "$WORK/verifier.toml" < anyhow::Result<() .await .map_err(|e| anyhow::anyhow!("failed to verify RA-TLS certificate: {:#}", e))?; - let app_info = verified.decode_app_info(false).ok(); + let app_info = verified.decode_app_info(false); // Bind the reported os_image_hash to the attested boot measurement. For // every platform except TDX legacy this is a self-contained check (no image // download); relying parties should only trust `os_image_hash` when // `os_image_hash_verified` is true. let os_image_hash_verified = verify_cert_os_image_hash(&verified.attestation, config, &attestation_verifier).await; - let output = serde_json::json!({ - "is_valid": true, - "details": { - "tee_variant": verified.attestation.quote.variant(), - "report_data": hex::encode(verified.attestation.report_data), - "public_key_der": hex::encode(&verified.public_key_der), - "app_info": app_info.map(|info| serde_json::json!({ - "app_id": hex::encode(info.app_id), - "compose_hash": hex::encode(info.compose_hash), - "instance_id": hex::encode(info.instance_id), - "device_id": hex::encode(info.device_id), - "mr_system": hex::encode(info.mr_system), - "mr_aggregated": hex::encode(info.mr_aggregated), - "os_image_hash": hex::encode(info.os_image_hash), - "os_image_hash_verified": os_image_hash_verified, - "key_provider_info": hex::encode(info.key_provider_info), - })), - } - }); + let output = cert_oneshot_result( + verified.attestation.quote.variant(), + &verified.attestation.report_data, + &verified.public_key_der, + app_info, + os_image_hash_verified, + ); let output_path = format!("{file_path}.ratls-verification.json"); fs::write(&output_path, serde_json::to_string_pretty(&output)?).map_err(|e| { @@ -258,9 +246,53 @@ async fn run_cert_oneshot(file_path: &str, config: &Config) -> anyhow::Result<() info!("stored certificate verification result at {}", output_path); println!("{}", serde_json::to_string_pretty(&output)?); + if output["is_valid"] != serde_json::json!(true) { + anyhow::bail!("{}", output["reason"].as_str().unwrap_or("invalid")); + } Ok(()) } +/// Render the `--verify-cert` result. +/// +/// `is_valid` is the one field a relying party reads, so it has to cover every +/// step that ran, not just the signature check. App-info decoding is one of +/// those steps: a certificate whose attested app identity does not decode has +/// no `app_id`, `compose_hash` or `os_image_hash` to act on, and reporting it +/// as valid with a null `app_info` puts the burden of noticing on the caller. +/// Same shape as the HTTP `/verify` response: `is_valid` plus a `reason`. +fn cert_oneshot_result( + tee_variant: ra_tls::attestation::TeeVariant, + report_data: &[u8], + public_key_der: &[u8], + app_info: Result, + os_image_hash_verified: bool, +) -> serde_json::Value { + let reason = app_info + .as_ref() + .err() + .map(|err| format!("failed to decode app info from the certificate: {err:#}")); + serde_json::json!({ + "is_valid": reason.is_none(), + "reason": reason, + "details": { + "tee_variant": tee_variant, + "report_data": hex::encode(report_data), + "public_key_der": hex::encode(public_key_der), + "app_info": app_info.ok().map(|info| serde_json::json!({ + "app_id": hex::encode(info.app_id), + "compose_hash": hex::encode(info.compose_hash), + "instance_id": hex::encode(info.instance_id), + "device_id": hex::encode(info.device_id), + "mr_system": hex::encode(info.mr_system), + "mr_aggregated": hex::encode(info.mr_aggregated), + "os_image_hash": hex::encode(info.os_image_hash), + "os_image_hash_verified": os_image_hash_verified, + "key_provider_info": hex::encode(info.key_provider_info), + })), + } + }) +} + /// Verify that an RA-TLS certificate's `os_image_hash` is bound to its attested /// boot measurement. /// @@ -293,7 +325,7 @@ async fn verify_cert_os_image_hash( ); let mut details = VerificationDetails::default(); verifier - .verify_os_image_hash(String::new(), attestation, false, &mut details) + .verify_os_image_hash(String::new(), attestation, &mut details) .await .is_ok() } @@ -434,6 +466,64 @@ image_download_timeout_secs = 7 } } +#[cfg(test)] +mod cert_oneshot_result_tests { + use super::*; + use ra_tls::attestation::{AppInfo, TeeVariant}; + + fn app_info() -> AppInfo { + serde_json::from_value(serde_json::json!({ + "app_id": "00".repeat(20), + "compose_hash": "11".repeat(32), + "instance_id": "22".repeat(20), + "device_id": "33".repeat(32), + "mr_system": "44".repeat(32), + "mr_aggregated": "55".repeat(32), + "os_image_hash": "66".repeat(32), + "key_provider_info": "", + })) + .unwrap() + } + + /// `is_valid` is the field a relying party branches on, so it must not say + /// "valid" about a certificate whose app identity never decoded -- the + /// result then carries no app_id, compose_hash or os_image_hash at all. + #[test] + fn a_certificate_whose_app_info_does_not_decode_is_not_reported_valid() { + let valid = cert_oneshot_result( + TeeVariant::DstackTdx, + &[0u8; 64], + b"spki", + Ok(app_info()), + true, + ); + assert_eq!(valid["is_valid"], serde_json::json!(true)); + assert_eq!(valid["reason"], serde_json::Value::Null); + assert_eq!( + valid["details"]["app_info"]["compose_hash"], + "11".repeat(32) + ); + + let undecodable = cert_oneshot_result( + TeeVariant::DstackTdx, + &[0u8; 64], + b"spki", + Err(anyhow::anyhow!("no app-id event")), + true, + ); + assert_eq!(undecodable["is_valid"], serde_json::json!(false)); + assert!(undecodable["reason"] + .as_str() + .unwrap() + .contains("no app-id event")); + assert_eq!( + undecodable["details"]["app_info"], + serde_json::Value::Null, + "there is no app info to report" + ); + } +} + #[cfg(test)] mod certificate_profile_tests { use super::*; diff --git a/dstack/verifier/src/types.rs b/dstack/verifier/src/types.rs index d9d71442b..53faff88e 100644 --- a/dstack/verifier/src/types.rs +++ b/dstack/verifier/src/types.rs @@ -18,8 +18,6 @@ pub struct VerificationRequest { pub vm_config: Option, #[serde(with = "serde_bytes", default)] pub attestation: Option>, - #[serde(default)] - pub debug: Option, } #[derive(Debug, Clone, Serialize)] @@ -79,13 +77,25 @@ impl PolicyBootInfo { #[derive(Debug, Clone, Default, Serialize)] pub struct VerificationDetails { pub quote_verified: bool, - /// Indicates that the event log was verified against the quote. + /// Indicates that the app identity was decoded from evidence bound to the + /// quote. + /// + /// On dstack TDX, GCP TDX and AWS NitroTPM that evidence is the runtime + /// event log, already replayed during quote verification against RTMR3 + /// (plus TPM PCR14 on GCP) or PCR14 on NitroTPM, so a replay mismatch fails + /// `quote_verified` rather than this flag. Digests and payloads are both + /// covered, because `app_id`, `compose_hash` and the rest are read out of + /// the payloads. SEV-SNP has no runtime event log and takes the identity + /// from `mr_config`, bound through HOST_DATA; Nitro Enclave derives it from + /// the PCRs. /// - /// For RTMR3 (runtime measurements), both the digest and payload integrity are verified - /// by replaying the event log and comparing against the quote. For RTMR 0-2 (boot-time - /// measurements), only the digests are verified through replay comparison with the quote; - /// the payload content is not validated. dstack does not define semantics for RTMR 0-2 - /// event log payloads. + /// It says nothing about the RTMR 0-2 entries a TDX event log carries; + /// nothing replays them. On dstack TDX those registers are verified by + /// comparing the quoted values against measurements recomputed from the OS + /// image (see `os_image_hash_verified`), which does not depend on the + /// host's event log. The TDX lite path only reads the three named ACPI + /// digests from it and requires them to match the recomputed ones; dstack + /// defines no semantics for the other payloads. pub event_log_verified: bool, pub os_image_hash_verified: bool, /// Indicates that TDX ACPI table contents were verified. @@ -115,48 +125,6 @@ pub struct VerificationDetails { /// Canonical auth-policy input matching the KMS bootAuth payload. #[serde(skip_serializing_if = "Option::is_none")] pub boot_info: Option, - #[serde(skip_serializing_if = "Option::is_none")] - pub acpi_tables: Option, - #[serde(skip_serializing_if = "Option::is_none")] - pub rtmr_debug: Option>, -} - -#[derive(Debug, Clone, Serialize)] -pub struct AcpiTables { - pub tables: String, - pub rsdp: String, - pub loader: String, -} - -#[derive(Debug, Clone, Serialize)] -pub struct RtmrMismatch { - pub rtmr: String, - pub expected: String, - pub actual: String, - pub events: Vec, - #[serde(skip_serializing_if = "Vec::is_empty")] - pub missing_expected_digests: Vec, -} - -#[derive(Debug, Clone, Serialize)] -pub struct RtmrEventEntry { - pub index: usize, - pub event_type: u32, - pub event_name: String, - pub actual_digest: String, - #[serde(skip_serializing_if = "Option::is_none")] - pub expected_digest: Option, - pub payload_len: usize, - pub status: RtmrEventStatus, -} - -#[derive(Debug, Clone, Serialize)] -#[serde(rename_all = "snake_case")] -pub enum RtmrEventStatus { - Match, - Mismatch, - Extra, - Missing, } #[cfg(test)] @@ -176,7 +144,6 @@ mod tests { assert_eq!(req.event_log.as_deref(), Some("[]")); assert_eq!(req.vm_config.as_deref(), Some("{}")); assert_eq!(req.attestation, None); - assert_eq!(req.debug, None); } #[test] @@ -194,7 +161,6 @@ mod tests { let req: VerificationRequest = serde_json::from_str("{}").unwrap(); assert_eq!(req.quote, None); assert_eq!(req.attestation, None); - assert_eq!(req.debug, None); } #[test] diff --git a/dstack/verifier/src/verification.rs b/dstack/verifier/src/verification.rs index 0d8f1add9..77752d4fc 100644 --- a/dstack/verifier/src/verification.rs +++ b/dstack/verifier/src/verification.rs @@ -17,9 +17,7 @@ use cc_eventlog::{ }, TdxEvent, }; -use dstack_mr::{ - tdx::TdxRtmr0AcpiHashes, RtmrLog, RtmrLogs, TdxMeasurementDetails, TdxMeasurements, -}; +use dstack_mr::{tdx::TdxRtmr0AcpiHashes, TdxMeasurements}; use dstack_types::{TdxAttestationVariant, VmConfig}; use hex_literal::hex; use ra_tls::attestation::{ @@ -32,8 +30,7 @@ use tokio::{io::AsyncWriteExt, process::Command}; use tracing::{debug, info, warn}; use crate::types::{ - AcpiTables, PolicyBootInfo, RtmrEventEntry, RtmrEventStatus, RtmrMismatch, VerificationDetails, - VerificationRequest, VerificationResponse, + PolicyBootInfo, VerificationDetails, VerificationRequest, VerificationResponse, }; /// Return the canonical TCB status and advisory list used by auth policy. @@ -88,99 +85,6 @@ fn decode_key_provider_info(bytes: &[u8]) -> Option RtmrMismatch { - let expected_hex = hex::encode(expected); - let actual_hex = hex::encode(actual); - - let mut events = Vec::new(); - - for (&idx, expected_digest) in actual_indices.iter().zip(expected_sequence.iter()) { - match event_log.get(idx) { - Some(event) => { - let event_name = if event.event.is_empty() { - "(unnamed)".to_string() - } else { - event.event.clone() - }; - let status = if event.digest() == expected_digest.as_slice() { - RtmrEventStatus::Match - } else { - RtmrEventStatus::Mismatch - }; - events.push(RtmrEventEntry { - index: idx, - event_type: event.event_type, - event_name, - actual_digest: hex::encode(event.digest()), - expected_digest: Some(hex::encode(expected_digest)), - payload_len: event.event_payload.len(), - status, - }); - } - None => { - events.push(RtmrEventEntry { - index: idx, - event_type: 0, - event_name: "(missing)".to_string(), - actual_digest: String::new(), - expected_digest: Some(hex::encode(expected_digest)), - payload_len: 0, - status: RtmrEventStatus::Missing, - }); - } - } - } - - for &idx in actual_indices.iter().skip(expected_sequence.len()) { - let (event_type, event_name, actual_digest, payload_len) = match event_log.get(idx) { - Some(event) => ( - event.event_type, - if event.event.is_empty() { - "(unnamed)".to_string() - } else { - event.event.clone() - }, - hex::encode(event.digest()), - event.event_payload.len(), - ), - None => (0, "(missing)".to_string(), String::new(), 0), - }; - events.push(RtmrEventEntry { - index: idx, - event_type, - event_name, - actual_digest, - expected_digest: None, - payload_len, - status: RtmrEventStatus::Extra, - }); - } - - let missing_expected_digests = if expected_sequence.len() > actual_indices.len() { - expected_sequence[actual_indices.len()..] - .iter() - .map(hex::encode) - .collect() - } else { - Vec::new() - }; - - RtmrMismatch { - rtmr: rtmr_label.to_string(), - expected: expected_hex.to_string(), - actual: actual_hex.to_string(), - events, - missing_expected_digests, - } -} - // Bump whenever expected RTMR computation changes so stale entries get ignored. // v3: all supported OVMF measurements use the Pre202505 RTMR[0] layout. // @@ -238,8 +142,18 @@ impl CvmVerifier { .join(format!("{cache_key}.json")) } + /// Hash only what the measurement reads: `image` and the measurement documents + /// are caller-controlled and unused, so they must not be able to force a miss. + /// Clear them rather than allowlisting, so new `VmConfig` fields stay in the key. fn vm_config_cache_key(vm_config: &VmConfig) -> Result { - let serialized = serde_json::to_vec(vm_config) + let vm_config = VmConfig { + image: None, + tdx_measurement: None, + gcp_measurement: None, + aws_measurement: None, + ..vm_config.clone() + }; + let serialized = serde_json::to_vec(&vm_config) .context("Failed to serialize VM config for cache key computation")?; Ok(hex::encode(Sha256::digest(&serialized))) } @@ -312,34 +226,26 @@ impl CvmVerifier { Ok(()) } - fn compute_measurement_details( - &self, - vm_config: &VmConfig, - fw_path: &Path, - kernel_path: &Path, - initrd_path: &Path, - kernel_cmdline: &str, - kernel_header_normalized: bool, - ) -> Result { - let firmware = fw_path.display().to_string(); - let kernel = kernel_path.display().to_string(); - let initrd = initrd_path.display().to_string(); + fn compute_measurements(vm_config: &VmConfig, image: &ImagePaths) -> Result { + let firmware = image.fw_path.display().to_string(); + let kernel = image.kernel_path.display().to_string(); + let initrd = image.initrd_path.display().to_string(); // Prefer the explicit variant the image declared; pre-`ovmf_variant` // deployments fall back to the only layout that existed back then. let ovmf_variant = vm_config.ovmf_variant.unwrap_or_default(); - let details = dstack_mr::Machine::builder() + dstack_mr::Machine::builder() .cpu_count(vm_config.cpu_count) .memory_size(vm_config.memory_size) .firmware(&firmware) .kernel(&kernel) .initrd(&initrd) - .kernel_cmdline(kernel_cmdline) + .kernel_cmdline(&image.kernel_cmdline) .root_verity(true) .hotplug_off(vm_config.hotplug_off) .maybe_two_pass_add_pages(vm_config.qemu_single_pass_add_pages) - .normalized_setup_header(kernel_header_normalized) + .normalized_setup_header(image.kernel_header_normalized) .maybe_pic(vm_config.pic) .maybe_qemu_version(vm_config.qemu_version.clone()) .maybe_pci_hole64_size(if vm_config.pci_hole64_size > 0 { @@ -356,40 +262,14 @@ impl CvmVerifier { .host_share_mode(vm_config.host_share_mode.clone()) .ovmf_variant(ovmf_variant) .build() - .measure_with_logs() - .context("Failed to compute expected MRs")?; - - Ok(details) - } - - fn compute_measurements( - &self, - vm_config: &VmConfig, - fw_path: &Path, - kernel_path: &Path, - initrd_path: &Path, - kernel_cmdline: &str, - kernel_header_normalized: bool, - ) -> Result { - self.compute_measurement_details( - vm_config, - fw_path, - kernel_path, - initrd_path, - kernel_cmdline, - kernel_header_normalized, - ) - .map(|details| details.measurements) + .measure() + .context("Failed to compute expected MRs") } - fn load_or_compute_measurements( + async fn load_or_compute_measurements( &self, vm_config: &VmConfig, - fw_path: &Path, - kernel_path: &Path, - initrd_path: &Path, - kernel_cmdline: &str, - kernel_header_normalized: bool, + image: ImagePaths, ) -> Result { let cache_key = Self::vm_config_cache_key(vm_config)?; @@ -397,14 +277,11 @@ impl CvmVerifier { return Ok(measurements); } - let measurements = self.compute_measurements( - vm_config, - fw_path, - kernel_path, - initrd_path, - kernel_cmdline, - kernel_header_normalized, - )?; + let vm_config = vm_config.clone(); + let measurements = + tokio::task::spawn_blocking(move || Self::compute_measurements(&vm_config, &image)) + .await + .context("measurement task failed")??; if let Err(e) = self.store_measurements_in_cache(&cache_key, &measurements) { warn!( @@ -673,14 +550,8 @@ impl CvmVerifier { ) -> Result { let image_paths = self.ensure_image_downloaded(vm_config).await?; - self.load_or_compute_measurements( - vm_config, - &image_paths.fw_path, - &image_paths.kernel_path, - &image_paths.initrd_path, - &image_paths.kernel_cmdline, - image_paths.kernel_header_normalized, - ) + self.load_or_compute_measurements(vm_config, image_paths) + .await } pub async fn verify(&self, request: VerificationRequest) -> Result { @@ -719,7 +590,6 @@ impl CvmVerifier { }; let mut details = VerificationDetails::default(); - let debug = request.debug.unwrap_or(false); let attestation = attestation.into_v1(); let verified = attestation.verify(&self.attestation_verifier).await; let verified_attestation = match verified { @@ -748,7 +618,6 @@ impl CvmVerifier { .verify_os_image_hash( request_vm_config.clone(), &verified_attestation, - debug, &mut details, ) .await; @@ -794,7 +663,6 @@ impl CvmVerifier { &self, vm_config: String, attestation: &VerifiedAttestation, - debug: bool, details: &mut VerificationDetails, ) -> Result { // The raw config string used for platform-specific binding: the explicit @@ -829,22 +697,12 @@ impl CvmVerifier { // rather than degraded to a download. AttestationQuote::DstackTdx(_) => match vm_config.tdx_attestation_variant { TdxAttestationVariant::Legacy => { - self.verify_os_image_hash_for_dstack_tdx( - &vm_config, - attestation, - debug, - details, - ) - .await?; + self.verify_os_image_hash_for_dstack_tdx(&vm_config, attestation, details) + .await?; } TdxAttestationVariant::Lite => { - self.verify_os_image_hash_for_dstack_tdx_lite( - &vm_config, - attestation, - debug, - details, - ) - .await?; + self.verify_os_image_hash_for_dstack_tdx_lite(&vm_config, attestation, details) + .await?; } }, AttestationQuote::DstackNitroEnclave(_) => { @@ -909,27 +767,9 @@ impl CvmVerifier { &self, vm_config: &VmConfig, attestation: &VerifiedAttestation, - debug: bool, details: &mut VerificationDetails, ) -> Result<()> { - let Some(report) = &attestation.report.tdx_report() else { - bail!("No TDX report"); - }; - let Some(tdx_quote) = attestation.tdx_quote() else { - bail!("No TDX quote"); - }; - let event_log = &tdx_quote.event_log; - let report = report - .report - .as_td10() - .context("Failed to decode TD report")?; - - let verified_mrs = Mrs { - mrtd: report.mr_td.to_vec(), - rtmr0: report.rt_mr0.to_vec(), - rtmr1: report.rt_mr1.to_vec(), - rtmr2: report.rt_mr2.to_vec(), - }; + let verified_mrs = verified_tdx_mrs(attestation)?; // Legacy TDX attestation keeps the original KMS verifier semantics: // os_image_hash must be the image digest (digest.txt = @@ -946,57 +786,11 @@ impl CvmVerifier { details.os_image_version = Some(image_paths.version.clone()); } - let (mrs, expected_logs) = if debug { - let TdxMeasurementDetails { - measurements, - rtmr_logs, - acpi_tables, - } = self - .compute_measurement_details( - vm_config, - &image_paths.fw_path, - &image_paths.kernel_path, - &image_paths.initrd_path, - &image_paths.kernel_cmdline, - image_paths.kernel_header_normalized, - ) - .context("Failed to compute expected measurements")?; - - details.acpi_tables = Some(AcpiTables { - tables: hex::encode(&acpi_tables.tables), - rsdp: hex::encode(&acpi_tables.rsdp), - loader: hex::encode(&acpi_tables.loader), - }); - - (measurements, Some(rtmr_logs)) - } else { - ( - self.load_or_compute_measurements( - vm_config, - &image_paths.fw_path, - &image_paths.kernel_path, - &image_paths.initrd_path, - &image_paths.kernel_cmdline, - image_paths.kernel_header_normalized, - ) - .context("Failed to compute expected measurements")?, - None, - ) - }; - - self.compare_tdx_mrs( - Mrs { - mrtd: mrs.mrtd, - rtmr0: mrs.rtmr0, - rtmr1: mrs.rtmr1, - rtmr2: mrs.rtmr2, - }, - verified_mrs, - expected_logs.as_ref(), - event_log, - debug, - details, - )?; + let expected_mrs = self + .load_or_compute_measurements(vm_config, image_paths) + .await + .context("Failed to compute expected measurements")?; + assert_tdx_mrs_eq(&expected_mrs, &verified_mrs).context("MRs do not match")?; details.acpi_tables_verified = true; Ok(()) } @@ -1005,29 +799,13 @@ impl CvmVerifier { &self, vm_config: &VmConfig, attestation: &VerifiedAttestation, - _debug: bool, details: &mut VerificationDetails, ) -> Result<()> { - let Some(report) = &attestation.report.tdx_report() else { - bail!("No TDX report"); - }; + let verified_mrs = verified_tdx_mrs(attestation)?; let Some(tdx_quote) = attestation.tdx_quote() else { bail!("No TDX quote"); }; let event_log = &tdx_quote.event_log; - // Get boot info from attestation - let report = report - .report - .as_td10() - .context("Failed to decode TD report")?; - - // Extract the verified MRs from the report - let verified_mrs = Mrs { - mrtd: report.mr_td.to_vec(), - rtmr0: report.rt_mr0.to_vec(), - rtmr1: report.rt_mr1.to_vec(), - rtmr2: report.rt_mr2.to_vec(), - }; let document = vm_config .tdx_measurement @@ -1073,85 +851,13 @@ impl CvmVerifier { details.acpi_tables_verified = true; // RTMR0 below is rebuilt from the recomputed digests, so the expected // value depends on nothing the host reported about the tables. - let mrs = dstack_mr::tdx::tdx_measurements_from_measurement_document( + let expected_mrs = dstack_mr::tdx::tdx_measurements_from_measurement_document( document, vm_config, &acpi_hashes, ) .context("Failed to compute TDX expected measurements without image download")?; - - let expected_mrs = Mrs { - mrtd: mrs.mrtd.clone(), - rtmr0: mrs.rtmr0.clone(), - rtmr1: mrs.rtmr1.clone(), - rtmr2: mrs.rtmr2.clone(), - }; - expected_mrs - .assert_eq(&verified_mrs) - .context("MRs do not match") - } - - fn compare_tdx_mrs( - &self, - expected_mrs: Mrs, - verified_mrs: Mrs, - expected_logs: Option<&RtmrLogs>, - event_log: &[TdxEvent], - debug: bool, - details: &mut VerificationDetails, - ) -> Result<()> { - match expected_mrs.assert_eq(&verified_mrs) { - Ok(()) => Ok(()), - Err(e) => { - let result = Err(e).context("MRs do not match"); - if !debug { - return result; - } - let Some(expected_logs) = expected_logs else { - return result; - }; - let mut rtmr_debug = Vec::new(); - - if expected_mrs.rtmr0 != verified_mrs.rtmr0 { - rtmr_debug.push(collect_rtmr_mismatch( - "RTMR0", - &expected_mrs.rtmr0, - &verified_mrs.rtmr0, - &expected_logs[0], - &[], - event_log, - )); - } - - if expected_mrs.rtmr1 != verified_mrs.rtmr1 { - rtmr_debug.push(collect_rtmr_mismatch( - "RTMR1", - &expected_mrs.rtmr1, - &verified_mrs.rtmr1, - &expected_logs[1], - &[], - event_log, - )); - } - - if expected_mrs.rtmr2 != verified_mrs.rtmr2 { - rtmr_debug.push(collect_rtmr_mismatch( - "RTMR2", - &expected_mrs.rtmr2, - &verified_mrs.rtmr2, - &expected_logs[2], - &[], - event_log, - )); - } - - if !rtmr_debug.is_empty() { - details.rtmr_debug = Some(rtmr_debug); - } - - result - } - } + assert_tdx_mrs_eq(&expected_mrs, &verified_mrs).context("MRs do not match") } /// Verify Nitro Enclave OS image hash using the signature-verified NSM PCRs. @@ -1377,46 +1083,39 @@ impl CvmVerifier { } } -#[derive(Debug, Clone)] -struct Mrs { - mrtd: Vec, - rtmr0: Vec, - rtmr1: Vec, - rtmr2: Vec, +/// MRTD and RTMR0-2 from the verified TD report. +fn verified_tdx_mrs(attestation: &VerifiedAttestation) -> Result { + let report = attestation + .report + .tdx_report() + .context("No TDX report")? + .report + .as_td10() + .context("Failed to decode TD report")?; + Ok(TdxMeasurements { + mrtd: report.mr_td.to_vec(), + rtmr0: report.rt_mr0.to_vec(), + rtmr1: report.rt_mr1.to_vec(), + rtmr2: report.rt_mr2.to_vec(), + }) } -impl Mrs { - fn assert_eq(&self, other: &Self) -> Result<()> { - if self.mrtd != other.mrtd { - bail!( - "MRTD mismatch: expected={}, actual={}", - hex::encode(&self.mrtd), - hex::encode(&other.mrtd) - ); - } - if self.rtmr0 != other.rtmr0 { - bail!( - "RTMR0 mismatch: expected={}, actual={}", - hex::encode(&self.rtmr0), - hex::encode(&other.rtmr0) - ); - } - if self.rtmr1 != other.rtmr1 { +fn assert_tdx_mrs_eq(expected: &TdxMeasurements, actual: &TdxMeasurements) -> Result<()> { + for (name, expected, actual) in [ + ("MRTD", &expected.mrtd, &actual.mrtd), + ("RTMR0", &expected.rtmr0, &actual.rtmr0), + ("RTMR1", &expected.rtmr1, &actual.rtmr1), + ("RTMR2", &expected.rtmr2, &actual.rtmr2), + ] { + if expected != actual { bail!( - "RTMR1 mismatch: expected={}, actual={}", - hex::encode(&self.rtmr1), - hex::encode(&other.rtmr1) + "{name} mismatch: expected={}, actual={}", + hex::encode(expected), + hex::encode(actual) ); } - if self.rtmr2 != other.rtmr2 { - bail!( - "RTMR2 mismatch: expected={}, actual={}", - hex::encode(&self.rtmr2), - hex::encode(&other.rtmr2) - ); - } - Ok(()) } + Ok(()) } #[cfg(test)] @@ -2007,6 +1706,38 @@ mod tests { assert_eq!(entries.len(), 1, "temporary cache files must not survive"); } + #[test] + fn measurement_cache_key_ignores_unmeasured_fields() { + let base: VmConfig = serde_json::from_value(serde_json::json!({ + "os_image_hash": "11".repeat(32), + "cpu_count": 2, + "memory_size": 0x8000_0000u64, + })) + .unwrap(); + let key = |config: &VmConfig| CvmVerifier::vm_config_cache_key(config).unwrap(); + let blob = || vec![0xaa; 4096]; + + let mut padded = base.clone(); + padded.image = Some("x".repeat(4096)); + padded.tdx_measurement = Some(dstack_types::TdxOsImageMeasurementDocument::new( + blob(), + blob(), + )); + padded.gcp_measurement = Some(dstack_types::GcpOsImageMeasurementDocument::new( + blob(), + blob(), + )); + padded.aws_measurement = Some(dstack_types::AwsOsImageMeasurementDocument::new( + blob(), + blob(), + )); + assert_eq!(key(&padded), key(&base)); + + let mut resized = base.clone(); + resized.cpu_count += 1; + assert_ne!(key(&resized), key(&base)); + } + #[test] fn image_cache_pruning_keeps_checksum_identity() { let dir = tempfile::tempdir().expect("temp image directory"); @@ -2360,7 +2091,6 @@ mod tests { event_log: None, vm_config: None, attestation: Some(attestation), - debug: None, }) .await .expect("verifier runs"); @@ -2585,7 +2315,6 @@ mod tests { event_log: None, vm_config: None, attestation: Some(forged), - debug: None, }) .await .expect("verifier runs"); diff --git a/os/common/rootfs/app-compose.service b/os/common/rootfs/app-compose.service index b3979e255..5fa2ae8fd 100644 --- a/os/common/rootfs/app-compose.service +++ b/os/common/rootfs/app-compose.service @@ -1,7 +1,10 @@ [Unit] Description=App Compose Service +# After= alone is satisfied by a failed unit; app must not run without these. +Requires=dstack-prepare.service dstack-guest-agent.service Wants=docker.service containerd.service containerd-stargz-grpc.service After=docker.service containerd.service containerd-stargz-grpc.service dstack-prepare.service dstack-guest-agent.service +OnFailure=dstack-boot-error@%n.service [Service] Type=oneshot diff --git a/os/common/rootfs/app-compose.sh b/os/common/rootfs/app-compose.sh index 4bd2dad79..3bb1f9585 100755 --- a/os/common/rootfs/app-compose.sh +++ b/os/common/rootfs/app-compose.sh @@ -126,10 +126,7 @@ start) docker-compose|nerdctl-compose) echo "Starting containers with runner=$runner snapshotter=$snapshotter" dstack-util notify-host -e "boot.progress" -d "starting containers" || true - if ! compose_start; then - dstack-util notify-host -e "boot.error" -d "failed to start containers" - exit 1 - fi + compose_start if [ "$runner" = docker-compose ]; then echo "Pruning unused Docker images and volumes" docker image prune -af diff --git a/os/common/rootfs/docker.service.d/dstack-guest-agent.conf b/os/common/rootfs/docker.service.d/dstack-guest-agent.conf index 2ca01dce3..e5528c5aa 100644 --- a/os/common/rootfs/docker.service.d/dstack-guest-agent.conf +++ b/os/common/rootfs/docker.service.d/dstack-guest-agent.conf @@ -5,7 +5,11 @@ [Unit] Wants=dstack-guest-agent.service After=dstack-guest-agent.service +OnFailure=dstack-boot-error@%n.service [Service] -TimeoutStartSec=0 -ExecStartPre=/bin/sh -c 'while ! systemctl is-active --quiet dstack-guest-agent.service || [ ! -S /var/run/dstack.sock ] || [ ! -S /var/run/tappd.sock ]; do sleep 1; done' +# The agent restarts itself on crashes; give up once it is parked in `failed` +# or never comes up, so OnFailure= reports the error instead of hanging boot. +# Its sockets come from dstack-guest-agent.socket and exist once it is active. +TimeoutStartSec=300 +ExecStartPre=/bin/sh -c 'until systemctl is-active -q dstack-guest-agent.service; do systemctl is-failed -q dstack-guest-agent.service && exit 1; sleep 1; done' diff --git a/os/common/rootfs/dstack-boot-error@.service b/os/common/rootfs/dstack-boot-error@.service new file mode 100644 index 000000000..3803c22cc --- /dev/null +++ b/os/common/rootfs/dstack-boot-error@.service @@ -0,0 +1,13 @@ +# SPDX-FileCopyrightText: © 2026 Phala Network +# +# SPDX-License-Identifier: Apache-2.0 + +# Pulled in via OnFailure=dstack-boot-error@%n.service to report a failed boot +# unit to the host. +[Unit] +Description=Report %i failure to the host +After=dstack-prepare.service + +[Service] +Type=oneshot +ExecStart=/bin/dstack-util notify-host -e boot.error -d "%i failed" diff --git a/os/common/rootfs/dstack-guest-agent.service b/os/common/rootfs/dstack-guest-agent.service index a88d395d9..ca2ccb65a 100644 --- a/os/common/rootfs/dstack-guest-agent.service +++ b/os/common/rootfs/dstack-guest-agent.service @@ -1,8 +1,9 @@ [Unit] Description=dstack Guest Agent Service Requires=dstack-guest-agent.socket -After=network.target tboot.service dstack-guest-agent.socket +After=network.target dstack-prepare.service dstack-guest-agent.socket Before=docker.service +OnFailure=dstack-boot-error@%n.service [Service] OOMScoreAdjust=-1000 diff --git a/os/common/rootfs/dstack-prepare.sh b/os/common/rootfs/dstack-prepare.sh index 40440d3c7..b8d42a890 100755 --- a/os/common/rootfs/dstack-prepare.sh +++ b/os/common/rootfs/dstack-prepare.sh @@ -131,7 +131,10 @@ setup_tsm() { fi if ! mountpoint -q /sys/kernel/config 2>/dev/null; then log "Mounting configfs for TSM..." - mount -t configfs none /sys/kernel/config + mount -t configfs none /sys/kernel/config || { + log "Warning: failed to mount configfs, TSM may not work" + return 1 + } fi if [[ -e /dev/tdx_guest ]] && [[ ! -d /sys/kernel/config/tsm/report/com.intel.dcap ]]; then log "Creating TSM report directory..." diff --git a/os/common/rootfs/journald.conf b/os/common/rootfs/journald.conf index 90a182d4d..763b6e1f3 100644 --- a/os/common/rootfs/journald.conf +++ b/os/common/rootfs/journald.conf @@ -1,7 +1,9 @@ [Journal] -SystemMaxUse=128M -SystemKeepFree=1G -SystemMaxFileSize=10M -SystemMaxFiles=10 -RuntimeMaxUse=0 +# Nothing creates /var/log/journal, so the journal lives in /run. Keep it there: +# /var/volatile is shared with docker's writable layer. +# RuntimeMaxUse/RuntimeKeepFree stay at their defaults (10%/15% of /run) so the +# budget scales with guest memory. +Storage=volatile +RuntimeMaxFileSize=10M +RuntimeMaxFiles=10 ReadKMsg=no diff --git a/os/image/README.md b/os/image/README.md index d1f4e1256..b98f516ba 100644 --- a/os/image/README.md +++ b/os/image/README.md @@ -52,7 +52,10 @@ image. The optional `artifacts.kernel_devel` manifest entry is published as `--kernel-devel.tar.gz` beside the image archives. It is not in `sha256sum.txt`, so it does not affect `os_image_hash`. `kernel-builder/` -builds the matching module-builder container image. +builds the matching module-builder container image. Its build script takes the +base image digest and Debian snapshot from `../mkosi/versions.env`; the base must +predate that snapshot so installed runtime libraries do not outrun the pinned +development packages. The Docker build rejects a Debian suite mismatch. ## Kernel setup-header normalization diff --git a/os/image/kernel-builder/Dockerfile b/os/image/kernel-builder/Dockerfile index 31b2d7f23..945fcb5d8 100644 --- a/os/image/kernel-builder/Dockerfile +++ b/os/image/kernel-builder/Dockerfile @@ -6,9 +6,10 @@ # kernel: its exported build tree plus a compiler from the same pinned Debian # snapshot the kernel was built with. Built by build.sh. -ARG BASE_IMAGE=debian:trixie-slim +ARG BASE_IMAGE FROM ${BASE_IMAGE} +ARG DEBIAN_RELEASE ARG DEBIAN_SNAPSHOT ARG KERNEL_RELEASE ARG IMAGE_VERSION=unknown @@ -17,6 +18,7 @@ ARG GIT_REVISION=unknown # Pin apt to the snapshot; trixie also ships deb822 sources that must go. RUN set -eux; \ suite=$(. /etc/os-release && echo "${VERSION_CODENAME}"); \ + test "$suite" = "${DEBIAN_RELEASE:?}"; \ rm -f /etc/apt/sources.list /etc/apt/sources.list.d/*; \ echo "deb [check-valid-until=no] http://snapshot.debian.org/archive/debian/${DEBIAN_SNAPSHOT:?} ${suite} main" \ > /etc/apt/sources.list; \ diff --git a/os/image/kernel-builder/build.sh b/os/image/kernel-builder/build.sh index b179f976d..9b98d894a 100755 --- a/os/image/kernel-builder/build.sh +++ b/os/image/kernel-builder/build.sh @@ -18,7 +18,8 @@ cp "$archive" "$context/kernel-devel.tar.gz" ${DOCKER:-docker} build \ --file "$SELF/Dockerfile" \ - --build-arg "BASE_IMAGE=debian:$DEBIAN_RELEASE-slim" \ + --build-arg "BASE_IMAGE=${KERNEL_BUILDER_BASE_IMAGE:?}" \ + --build-arg "DEBIAN_RELEASE=$DEBIAN_RELEASE" \ --build-arg "DEBIAN_SNAPSHOT=$DEBIAN_SNAPSHOT" \ --build-arg "KERNEL_RELEASE=$KERNEL_VERSION-dstack" \ --build-arg "IMAGE_VERSION=${IMAGE_VERSION:-$DSTACK_VERSION}" \ diff --git a/os/mkosi/mkosi.postinst b/os/mkosi/mkosi.postinst index c7622e7e1..d247e2680 100755 --- a/os/mkosi/mkosi.postinst +++ b/os/mkosi/mkosi.postinst @@ -19,3 +19,9 @@ rm -f "$B"/etc/ssh/ssh_host_*_key "$B"/etc/ssh/ssh_host_*_key.pub # Build outputs are merged after package installation, so refresh the final # rootfs cache after manually staged NVIDIA and container-runtime libraries. ldconfig -r "$B" + +# systemd applies the first matching rule across all preset files sorted by +# name, so a distro preset such as Debian's 90-systemd.preset would shadow the +# `disable *` in 99-dstack-default.preset. Keep dstack's policy the only one. +find "$B"/usr/lib/systemd/system-preset "$B"/etc/systemd/system-preset \ + -maxdepth 1 -name '*.preset' ! -name '*dstack*' -print -delete 2>/dev/null || true diff --git a/os/mkosi/mkosi.skeleton/usr/lib/systemd/system-preset/80-dstack.preset b/os/mkosi/mkosi.skeleton/usr/lib/systemd/system-preset/80-dstack.preset index fbc884d3b..b51076f2a 100644 --- a/os/mkosi/mkosi.skeleton/usr/lib/systemd/system-preset/80-dstack.preset +++ b/os/mkosi/mkosi.skeleton/usr/lib/systemd/system-preset/80-dstack.preset @@ -1,5 +1,7 @@ # SPDX-License-Identifier: Apache-2.0 enable systemd-networkd.service +# The only unit WantedBy=network-online.target, which dstack-prepare waits on. +enable systemd-networkd-wait-online.service enable systemd-resolved.service enable chrony.service enable docker.service diff --git a/os/mkosi/tests/acceptance.sh b/os/mkosi/tests/acceptance.sh index 493315a44..e2114dacc 100755 --- a/os/mkosi/tests/acceptance.sh +++ b/os/mkosi/tests/acceptance.sh @@ -124,6 +124,11 @@ test ! -e "$D/../common/rootfs/wg-checker.service" # only meaningful with a terminal disable. Without it, every package pulled in # by Packages= would start at boot with no diff to 80-dstack.preset. grep -q '^disable \*$' "$D/mkosi.skeleton/usr/lib/systemd/system-preset/99-dstack-default.preset" +# ...and it is only terminal if no foreign preset sorts before it. +grep -q "! -name '\*dstack\*'" "$D/mkosi.postinst" || { + echo 'mkosi.postinst must drop foreign preset files' >&2; exit 1; } +grep -qx 'enable systemd-networkd-wait-online.service' \ + "$D/mkosi.skeleton/usr/lib/systemd/system-preset/80-dstack.preset" # The TEE simulator serves synthetic quotes; its preset must not ship in prod. if grep -rq 'dstack-tee-simulator' "$D/mkosi.skeleton/"; then echo 'simulator preset must live in the dev profile skeleton' >&2 diff --git a/os/mkosi/versions.env b/os/mkosi/versions.env index 625c382f8..042c8bde4 100644 --- a/os/mkosi/versions.env +++ b/os/mkosi/versions.env @@ -4,6 +4,9 @@ KERNEL_VERSION=6.18.40 KERNEL_SHA256=3712fc1ec839e4daac981176c8518912e8f452650aaedfe4381da4419613a431 DEBIAN_RELEASE=trixie DEBIAN_SNAPSHOT=20260721T000000Z +# Module-builder root predates the snapshot so its installed libraries can be +# upgraded to the pinned development packages without repository downgrades. +KERNEL_BUILDER_BASE_IMAGE=debian:trixie-slim@sha256:020c0d20b9880058cbe785a9db107156c3c75c2ac944a6aa7ab59f2add76a7bd MKOSI_VERSION=26 # mkosi constructs the image, so it is part of the build toolchain and a # reproducibility input in its own right. The major version above is what diff --git a/os/yocto/layers/meta-dstack/recipes-core/dstack-guest/dstack-guest.bb b/os/yocto/layers/meta-dstack/recipes-core/dstack-guest/dstack-guest.bb index c2beca373..3a7aee1d5 100644 --- a/os/yocto/layers/meta-dstack/recipes-core/dstack-guest/dstack-guest.bb +++ b/os/yocto/layers/meta-dstack/recipes-core/dstack-guest/dstack-guest.bb @@ -93,6 +93,7 @@ do_install() { install -m 0644 ${DSTACK_ROOTFS_FILES}/app-compose.service ${D}${systemd_system_unitdir} install -m 0644 ${DSTACK_ROOTFS_FILES}/dstack-gateway-checker.service ${D}${systemd_system_unitdir} install -m 0644 ${DSTACK_ROOTFS_FILES}/dstack-guest-agent.socket ${D}${systemd_system_unitdir} + install -m 0644 ${DSTACK_ROOTFS_FILES}/dstack-boot-error@.service ${D}${systemd_system_unitdir} install -m 0644 ${DSTACK_ROOTFS_FILES}/llmnr.conf ${D}${sysconfdir}/systemd/resolved.conf.d # Drop-ins the image ships are vendor configuration, so they belong # beside the units in ${systemd_system_unitdir}. /etc is the operator's diff --git a/os/yocto/layers/meta-dstack/recipes-core/dstack-sysbox/files/99-sysbox-sysctl.conf b/os/yocto/layers/meta-dstack/recipes-core/dstack-sysbox/files/99-sysbox-sysctl.conf index 17cccab61..f2870e8bb 100644 --- a/os/yocto/layers/meta-dstack/recipes-core/dstack-sysbox/files/99-sysbox-sysctl.conf +++ b/os/yocto/layers/meta-dstack/recipes-core/dstack-sysbox/files/99-sysbox-sysctl.conf @@ -1,9 +1,6 @@ # Sysbox recommended sysctl settings # See: https://github.com/nestybox/sysbox/blob/master/docs/user-guide/install-package.md -# Allow unprivileged users to create user namespaces (required for rootless containers) -kernel.unprivileged_userns_clone = 1 - # Raise inotify limits — each sysbox container runs its own init system (e.g. systemd) # which uses inotify heavily; defaults (8192/128) exhaust quickly with multiple containers fs.inotify.max_queued_events = 1048576 diff --git a/os/yocto/layers/meta-nvidia/recipes-graphics/nvidia-container-toolkit/files/config.toml b/os/yocto/layers/meta-nvidia/recipes-graphics/nvidia-container-toolkit/files/config.toml index e09dec2a3..49f2bcb12 100644 --- a/os/yocto/layers/meta-nvidia/recipes-graphics/nvidia-container-toolkit/files/config.toml +++ b/os/yocto/layers/meta-nvidia/recipes-graphics/nvidia-container-toolkit/files/config.toml @@ -12,7 +12,7 @@ ldconfig = "@/sbin/ldconfig" [nvidia-container-runtime] # debug = "/var/log/nvidia-container-runtime.log" -log-level = "debug" +log-level = "info" # Specify the runtimes to consider. This list is processed in order and the PATH # searched for matching executables unless the entry is an absolute path.