From 4150c9f89207355a75786fb8d5312613d5a3a297 Mon Sep 17 00:00:00 2001 From: Kevin Wang Date: Mon, 14 Sep 2026 00:47:04 -0700 Subject: [PATCH 01/93] feat(vmm): sweep netd interfaces by VM and keep removals recoverable Split out of the vhost/multiqueue change (#1145); carries the interface lifecycle parts of #1154 and its follow-ups. - netd records each interface's owner in the kernel interface alias - new `RemoveVm`, `ListInterfaces` and `RemoveInterfaceByName` netd RPCs, plus the `dstack-vmm netd list|remove-vm|remove-interface` subcommands - prepare requests carry the VM's workdir, which netd logs - the VMM sweeps a VM's interfaces before every launch and on stop, update, and removal, and tracks outstanding cleanup in `.netd-pending` - removal keeps the VM directory until netd confirms the release - a per-VM launch lock serializes launch, stop, update, and removal - the fake netd used by tests serves the real Netd RPC --- docs/bridge-networking.md | 73 +- docs/libvirt-network-filter.md | 24 +- dstack/vmm/netd-rpc/proto/netd_rpc.proto | 56 ++ dstack/vmm/src/app.rs | 737 +++++++++++++++++--- dstack/vmm/src/app/workdir.rs | 20 + dstack/vmm/src/main.rs | 114 ++++ dstack/vmm/src/main_service.rs | 31 +- dstack/vmm/src/netd.rs | 822 ++++++++++++++++++++++- dstack/vmm/vmm.toml | 6 +- 9 files changed, 1775 insertions(+), 108 deletions(-) diff --git a/docs/bridge-networking.md b/docs/bridge-networking.md index d319a7827..82c11867a 100644 --- a/docs/bridge-networking.md +++ b/docs/bridge-networking.md @@ -171,7 +171,8 @@ needs an `allow` line for the bridge. - `netd` creates a persistent TAP, attaches it to the bridge, binds the nwfilter if the node filters, and the VMM passes `-netdev tap,id=net0,ifname=,...` - Guest MAC address is derived from SHA256 of the VM ID, with an optional configurable prefix (stable across restarts for DHCP IP consistency) - The host DHCP server (dnsmasq) assigns an IP to the VM -- The TAP outlives QEMU and is deleted when the VMM tears the VM's networking down +- The TAP outlives QEMU and is deleted when the VMM tears the VM's networking down, so a VM that crashes does not leave its filter rules attached to a name the next VM could take +- Every interface `netd` creates records which VM of which VMM instance it belongs to, in the kernel's interface alias — see [Who owns an interface](#who-owns-an-interface) - The VMM process needs neither root nor `CAP_NET_ADMIN`; `netd` holds that privilege in a separate service ### MAC address prefix @@ -237,6 +238,76 @@ told. An unpinned mapping goes to the first user-mode NIC; a VM that has none is not refused — it may have been deployed before this — but every mapping it strands is named in the launch log. +## Who owns an interface + +`netd` names an interface `dt<12 hex>`, a digest of (VMM instance, VM, NIC +index). That answers "where is this VM's interface" but not "whose is this +interface" — and the second question is the one a leaked interface poses. So +`netd` also records the identity on the interface itself: + +```console +$ ip -d link show dtc41d9e0b7a52 | grep alias + alias dstack1:0:path-3f9a1c8e7d2b4a60:0a1b2c3d4e5f6071 +``` + +The kernel holds that for exactly the interface's lifetime, so unlike a file on +disk it cannot be written late, lost, or left behind. It is a hint, never an +authority: a record is believed only when re-deriving the interface name from +it reproduces the name it is written on, so a forged, truncated or ambiguous +record reads the same as no record at all. + +Teardown does not need it — a sweep derives the names it deletes. What needs it +is an operator, and a host running several VMM instances, where it is the only +thing that tells one instance's interfaces from another's. + +```bash +# What netd holds on this host +sudo dstack-vmm netd list + +# Everything one VM holds, for a VM whose VMM will never ask again +sudo dstack-vmm netd remove-vm --instance path-3f9a1c8e7d2b4a60 --vm 0a1b2c3d4e5f6071 +``` + +### When a release does not land + +Every stop and every removal asks `netd` to sweep that VM's interfaces, by +deriving each of the 256 names its identity could produce. That needs no +record, and it reaches what a per-NIC teardown cannot: an interface a crash +left behind before anything on disk pointed at it, or one whose NIC the +manifest has since dropped. + +A removal deletes the VM's directory, and that directory — with its `.removing` +marker — is the only thing left that says to try again. So it is deleted only +once the sweep has landed. If `netd` refused, or was not there to ask, the +directory stays and the next VMM start resumes the removal; `RemoveVm` is +idempotent, so the retry costs one round trip. A VM that never asked `netd` for +an interface is unaffected: there is nothing for `netd` to be holding. + +The VMM persists `.netd-pending` before asking netd to prepare an interface and +clears it only after a successful whole-VM sweep. This cleanup marker survives +failed launches and network configuration changes, even if the runtime snapshot +is absent or replaced by a user-mode topology. Older snapshots are promoted to +the marker before cleanup or replacement. A failed cleanup during an update +also leaves the old snapshot intact. + +On an unfiltered node, an unavailable `libvirtd` does not make an otherwise +successful TAP sweep fail. Filtered nodes still require confirmation that their +nwfilter bindings have been released; deleting the TAP alone is not sufficient. + +What no VMM will retry is an interface whose VM directory an operator deleted +by hand, or one recorded under an instance ID no VMM uses any more. `netd list` +shows both, with the instance and VM they are recorded under: + +```bash +sudo dstack-vmm netd list +sudo dstack-vmm netd remove-vm --instance --vm +sudo dstack-vmm netd remove-interface dtc41d9e0b7a52 +``` + +Changing `cvm.instance_id` — or `run_path`, which it is derived from — strands +interfaces the same way. Running VMs keep working until they stop, and +`netd list` still shows the old instance ID, which is what `remove-vm` needs. + ### Mixing networking modes Bridge and user-mode VMs can coexist. Set the global default in `vmm.toml` and override per-VM as needed: diff --git a/docs/libvirt-network-filter.md b/docs/libvirt-network-filter.md index 1e161c183..57ec85bd4 100644 --- a/docs/libvirt-network-filter.md +++ b/docs/libvirt-network-filter.md @@ -20,6 +20,11 @@ The measurable acceptance criteria are: that TAP carries a binding. - `network_filter = "libvirt"` creates the TAP and filter binding before QEMU is submitted to Supervisor, and uses QEMU `-netdev tap`. +- An nwfilter binding outlives the TAP it was bound to, so a teardown clears + the binding at every name that VM could have used, whether or not the + interface is still there. `dstack-vmm netd list` shows a binding whose + interface is already gone as a `binding` row; remove one with + `dstack-vmm netd remove-interface `. - A failed TAP or filter setup prevents QEMU from starting and rolls back all interfaces prepared for that VM. - Normal stop and removal delete the filter binding and TAP. @@ -109,6 +114,19 @@ arguments. It never accepts a command, executable path, TAP name, or raw XML from a client. Filter XML is generated internally with XML escaping and is validated by libvirt. +Teardown by identity only reaches the NIC indices its caller still has a record +of, and that record is written *after* the interface exists — a VMM killed in +between leaves a TAP nothing on disk points at, and a manifest that lost a NIC +leaves the same thing behind. `RemoveVm` names a VM instead of an interface +and derives every name that VM could occupy, so neither has to be recorded for +teardown to work. The VMM sweeps before preparing a launch as well as on stop, +which makes a launch self-healing regardless of what the record says. + +A prepare also carries `workdir`, which `netd` does not need to build the TAP. +It names the VM's directory on the host: untrusted, never read for a decision, +and present only so an operator reading `netd`'s log can get from an opaque TAP +name back to the VM. + ## Deployment modes Production should run one shared service. `netd` reads the `[netd]` section, @@ -199,8 +217,10 @@ sudo dstack-vmm --config ./vmm.toml \ ``` User networking and a caller-supplied netdev never ask `netd` to build an -interface. Bridge and macvtap do ask, and fail closed if `netd` is -unavailable. +interface. The VMM still contacts the socket for such a VM -- every launch and +every stop releases whatever the VM held, before it decides whether it needs +anything built -- but nothing about the VM depends on the answer. Bridge and +macvtap do ask, and fail closed if `netd` is unavailable. Filtered TAP netdevs follow the node's `vhost` and `queues` settings like any other TAP-backed NIC (see [network-data-plane.md](network-data-plane.md)). The diff --git a/dstack/vmm/netd-rpc/proto/netd_rpc.proto b/dstack/vmm/netd-rpc/proto/netd_rpc.proto index 058a83a93..70ec8b991 100644 --- a/dstack/vmm/netd-rpc/proto/netd_rpc.proto +++ b/dstack/vmm/netd-rpc/proto/netd_rpc.proto @@ -26,6 +26,9 @@ message PrepareBridgeRequest { bool filtered = 5; // virtio-net queue pairs. Zero or one creates a single-queue TAP. uint32 queues = 6; + // The VM's working directory on the host. Only logged, so an operator can + // get from a TAP name back to the VM; never used for a decision. + string workdir = 7; } message PrepareMacvtapRequest { @@ -37,6 +40,9 @@ message PrepareMacvtapRequest { string mode = 5; // virtio-net queue pairs. Zero or one creates a single-queue device. uint32 queues = 6; + // The VM's working directory on the host. Only logged; see + // PrepareBridgeRequest.workdir. + string workdir = 7; } // What netd built, so the caller can verify it before handing it to QEMU. @@ -48,6 +54,45 @@ message PreparedInterface { uint32 queues = 3; } +// Names one VM of one VMM instance. +message VmRef { + string instance_id = 1; + string vm_id = 2; +} + +message RemoveVmResponse { + // How many interfaces were deleted. + uint32 removed = 1; +} + +message InterfaceName { + string tap = 1; +} + +message ListInterfacesRequest { + // Only interfaces recorded as this instance's. Empty lists every interface + // netd owns, including ones it cannot attribute. + string instance_id = 1; +} + +// One host resource netd holds. +message InterfaceRecord { + string tap = 1; + // "tap", "macvtap", or "binding" for an nwfilter binding whose interface is + // already gone. + string kind = 2; + // The owner, when the interface carries an ownership record that + // re-derives its name. Absent means the owner is unknown, not that there is + // none. + optional string instance_id = 3; + optional string vm_id = 4; + optional uint32 nic_index = 5; +} + +message InterfaceList { + repeated InterfaceRecord interfaces = 1; +} + message CheckInterfaceRequest { InterfaceIdentity identity = 1; // Also require the nwfilter binding. @@ -64,6 +109,17 @@ service Netd { // Delete an interface and its binding. Deleting one that does not exist // succeeds. rpc RemoveInterface(InterfaceIdentity) returns (google.protobuf.Empty); + // Delete every interface and binding one VM could hold, by deriving each + // name its identity can produce. Reaches interfaces the caller has no record + // of: one a crash left before it was recorded, or one whose NIC the VM no + // longer has. + rpc RemoveVm(VmRef) returns (RemoveVmResponse); + // Delete one interface by name. For interfaces nothing can attribute, such + // as ones built before netd recorded ownership. The name must be one netd + // could have created. + rpc RemoveInterfaceByName(InterfaceName) returns (google.protobuf.Empty); + // Everything netd holds, read off the host. + rpc ListInterfaces(ListInterfacesRequest) returns (InterfaceList); // Verify an interface, and optionally its binding, exists. For operations // and integration diagnostics; launches use the Prepare calls. rpc CheckInterface(CheckInterfaceRequest) returns (google.protobuf.Empty); diff --git a/dstack/vmm/src/app.rs b/dstack/vmm/src/app.rs index 1675aec25..4163876b5 100644 --- a/dstack/vmm/src/app.rs +++ b/dstack/vmm/src/app.rs @@ -307,6 +307,9 @@ pub struct App { state: Arc>, /// Pull status for registry images: tag → status. pub(crate) pull_status: Arc>>, + /// One lock per VM, held across a launch or a teardown. See + /// [`App::launch_lock`]. + launch_locks: Arc>>>>, } const GUEST_AGENT_RPC_TIMEOUT: Duration = Duration::from_secs(30); @@ -334,12 +337,42 @@ impl App { state: Arc::new(Mutex::new(AppState { cid_pool, vms: HashMap::new(), + removing: HashSet::new(), })), config: Arc::new(config), pull_status: Arc::new(Mutex::new(std::collections::HashMap::new())), + launch_locks: Arc::new(Mutex::new(HashMap::new())), } } + /// Serializes everything that creates or deletes one VM's host interfaces. + /// + /// A launch reads whether QEMU is already up and then spends many awaits -- + /// a GPU reset, a whole netd conversation, building the QEMU arguments -- + /// before it launches anything. Nothing used to cover that window. Two + /// `StartVm` calls, or one racing the auto-restart timer, could both read + /// "not running", and the loser's sweep would delete the TAPs the winner's + /// QEMU was already holding open: a live VM silently loses its networking, + /// and the loser's error path then clears the winner's record of it. The + /// authoritative rejection lives in the supervisor, which is reached long + /// after the damage is done. + /// + /// A tokio mutex, because it is held across awaits. Per VM, because a slow + /// start must not stall unrelated ones. Taken by stop and by removal as + /// well as by start: those delete the same interfaces from the other side. + pub(crate) async fn launch_lock(&self, id: &str) -> tokio::sync::OwnedMutexGuard<()> { + self.launch_lock_handle(id).lock_owned().await + } + + fn launch_lock_handle(&self, id: &str) -> Arc> { + let mut locks = self.launch_locks.lock().or_panic("mutex poisoned"); + // A VM that is neither starting nor stopping leaves the map holding + // the only reference, so the map stays the size of what is in flight + // rather than of every VM this process has ever touched. + locks.retain(|_, lock| Arc::strong_count(lock) > 1); + locks.entry(id.to_string()).or_default().clone() + } + pub async fn load_vm( &self, work_dir: impl AsRef, @@ -403,6 +436,19 @@ impl App { Ok(()) } + /// Refuses an operation on a VM that is being removed. + /// + /// Cheap, and taken before the launch lock as well as under it. Removal + /// holds that lock until the VM has exited -- hours, by its own estimate -- + /// so anything that only asked afterwards would wait the removal out in + /// order to be told no. + pub(crate) fn refuse_if_removing(&self, id: &str) -> Result<()> { + if self.lock().is_removing(id) { + bail!("VM is being removed"); + } + Ok(()) + } + pub async fn start_vm(&self, id: &str) -> Result<()> { self.start_vm_with_restart_policy(id, true).await } @@ -417,13 +463,24 @@ impl App { vm.state.auto_restart.reset(); } } - { - let state = self.lock(); - if let Some(vm) = state.get(id) { - if vm.state.removing { - bail!("VM is being removed"); - } - } + // Before the lock as well as after it. Removal holds the lock until the + // VM has exited, so a launch that only asked afterwards would wait that + // out -- hours, by removal's own estimate -- to be told no. Asking + // first is not sufficient on its own, because the marker can be set + // while this waits; asking again under the lock is what makes it + // authoritative. + self.refuse_if_removing(id)?; + // Everything below reads whether this VM is running and acts on the + // answer for as long as the launch takes. See [`App::launch_lock`]. + let _launch = self.launch_lock(id).await; + self.refuse_if_removing(id)?; + // A restart decided before a stop must not outlive it. The decision + // read `started` from disk; `stop_vm` writes it false under this lock, + // so re-reading it here is what makes the stop stick. An explicit start + // sets the flag itself and has nothing to re-read. + if !reset_restart_policy && !self.work_dir(id)?.started().unwrap_or(false) { + debug!(id, "skipping automatic restart: the VM was stopped"); + return Ok(()); } self.sync_dynamic_config(id)?; let is_running = self @@ -470,13 +527,10 @@ impl App { }) .await .context("GPU sanitization task failed")??; - if let Err(error) = self - .prepare_netd_networks(&vm_config, &mut runtime_networks) - .await - { - let _ = work_dir.clear_runtime_networks(); - return Err(error); - } + // Keep any earlier snapshot on failure: preparation may not have + // been able to persist the cleanup marker for legacy NICs yet. + self.prepare_netd_networks(&vm_config, &mut runtime_networks) + .await?; let processes = match vm_config.config_qemu( &work_dir, &self.config.cvm, @@ -485,16 +539,12 @@ impl App { ) { Ok(processes) => processes, Err(error) => { - let _ = self - .remove_netd_networks(&vm_config.manifest.id, &runtime_networks) - .await; + self.release_vm_interfaces(&vm_config.manifest.id).await; return Err(error); } }; if let Err(error) = work_dir.set_runtime_networks(&runtime_networks) { - let _ = self - .remove_netd_networks(&vm_config.manifest.id, &runtime_networks) - .await; + self.release_vm_interfaces(&vm_config.manifest.id).await; return Err(error); } { @@ -504,17 +554,13 @@ impl App { } for process in processes { if let Err(err) = self.supervisor.deploy(&process).await { - if let Err(cleanup_error) = self - .remove_netd_networks(&vm_config.manifest.id, &runtime_networks) - .await - { - warn!(id, %cleanup_error, "failed to roll back netd-managed networking"); - } - if let Err(clear_err) = work_dir.clear_runtime_networks() { - warn!( - id, - "failed to clear runtime networks after start failure: {clear_err}" - ); + if self.release_vm_interfaces(&vm_config.manifest.id).await { + if let Err(clear_err) = work_dir.clear_runtime_networks() { + warn!( + id, + "failed to clear runtime networks after start failure: {clear_err}" + ); + } } if let Some(vm_state) = self.lock().get_mut(id) { vm_state.state.runtime_networks.clear(); @@ -539,13 +585,25 @@ impl App { } pub async fn stop_vm(&self, id: &str) -> Result<()> { + // Removal stops the VM itself and holds the launch lock while it does, + // so this would otherwise wait hours to do again what is already being + // done. + self.refuse_if_removing(id)?; if let Some(vm) = self.lock().get_mut(id) { vm.state.auto_restart.reset(); } + // Teardown deletes the same interfaces a launch creates, and derives + // their names rather than reading a record, so it must not overlap one. + let _launch = self.launch_lock(id).await; self.set_started(id, false)?; self.stop_vm_process(id).await?; - let networks = self.work_dir(id)?.runtime_networks(); - self.remove_netd_networks(id, &networks).await?; + // Not fallible: a VM that has been asked to stop is stopped whether or + // not netd could be reached. What is left behind is reclaimed by this + // VM's next launch, which releases before it prepares, or by its + // removal, which will not finish until the release lands. A VM that is + // never started or removed again keeps its interfaces, and its + // directory is still there to say whose they are. + self.release_vm_interfaces(id).await; Ok(()) } @@ -569,10 +627,41 @@ impl App { mapping.nic_index, ); } + + // Whatever an earlier boot left behind: a crash between creating an + // interface and recording it, a NIC this VM no longer has, or a whole + // backend it no longer uses. Prepare replaces the names it is about to + // use, but only those, so an index nothing will claim again is only + // reachable from here. + // + // Before the early return, and not inside it. A VM that has moved from + // a bridge to user-mode networking needs this release precisely because + // it no longer wants an interface, and gating it on wanting one is how + // the interfaces it left behind would become unreachable to every + // later launch. + let vm_workdir = self.work_dir(&vm.manifest.id)?; + if vm_workdir + .runtime_networks() + .iter() + .any(needs_netd_interface) + { + // Fail before a user-mode boot can overwrite a legacy snapshot + // when its replacement cleanup marker cannot be persisted. + vm_workdir.mark_network_cleanup_pending()?; + } + self.release_vm_interfaces(&vm.manifest.id).await; if !networks.iter().any(needs_netd_interface) { return Ok(()); } + vm_workdir.mark_network_cleanup_pending()?; let qemu_uid = Uid::effective().as_raw(); + // Only ever read back out of a log line: netd is told where the VM + // lives so an operator holding an opaque TAP name can reach the VM + // without going through the VMM first. + let workdir = self + .work_dir(&vm.manifest.id) + .map(|dir| dir.path().display().to_string()) + .unwrap_or_default(); let mut prepared = Vec::new(); for (nic_index, network) in networks.iter_mut().enumerate() { if !needs_netd_interface(network) { @@ -604,6 +693,7 @@ impl App { // may not run libvirt at all. filtered, queues, + workdir: workdir.clone(), }) .await } @@ -615,6 +705,7 @@ impl App { qemu_uid, mode: network.macvtap_mode.clone(), queues, + workdir: workdir.clone(), }) .await } @@ -705,11 +796,12 @@ impl App { } /// A running VM whose snapshot is missing, because a VMM that predates the - /// snapshot started it. + /// snapshot -- or predates it recording what netd built -- started it. /// /// Guessing is all that is left, so guess the way that VMM would have, and - /// then write the guess down, so later teardown does not re-derive it from - /// node configuration that may by then have moved. + /// then write the guess down. Leaving the marker unset would make every + /// later teardown re-derive it from node configuration that may by then + /// have moved, which is the failure this snapshot exists to prevent. /// /// The way *that* VMM would have, not this one: a build old enough to leave /// no snapshot had no vhost and no multiqueue at all, so whatever this @@ -744,31 +836,73 @@ impl App { } } - /// Removes the host interfaces netd built for the NICs in `networks`. - pub(crate) async fn remove_netd_networks( - &self, - vm_id: &str, - networks: &[Networking], - ) -> Result<()> { - let netd = netd::client(&self.config.netd.socket); - let mut first_error = None; - for (nic_index, network) in networks.iter().enumerate().rev() { - if !needs_netd_interface(network) { - continue; + /// Releases every host interface netd holds for this VM. + /// + /// Unconditional and non-fatal, which is one decision made twice. The + /// release path must not be gated on a predicate that can change under it: + /// `needs_netd_interface` reads the VM's *current* backend, and a VM whose + /// NIC was a bridge when its TAP was built and is a user-mode NIC now would + /// skip the release for interfaces that exist. And a VM must be able to + /// stop when the daemon holding its interfaces cannot be reached, or a + /// netd outage becomes a fleet that cannot be stopped. + /// + /// Returns whether netd is known to hold nothing for this VM any more. + /// A removal reads that to decide whether it may delete the workdir: the + /// directory is what says to try again, and deleting it over a failed + /// release is what strands an interface with nothing left to reach it. + pub(crate) async fn release_vm_interfaces(&self, vm_id: &str) -> bool { + let workdir = match self.work_dir(vm_id) { + Ok(workdir) => workdir, + Err(error) => { + warn!(vm_id, %error, "cannot locate network cleanup state"); + return false; } - let identity = InterfaceIdentity { - instance_id: self.config.cvm.instance_id.clone(), - vm_id: vm_id.to_string(), - nic_index: nic_index as u32, - }; - if let Err(error) = netd.remove_interface(identity).await { - first_error.get_or_insert(error); + }; + // Upgrade snapshots written before the marker existed. A failed sweep + // must remain recoverable even if the next boot replaces the snapshot + // with user-mode NICs or an update clears it. + if workdir.runtime_networks().iter().any(needs_netd_interface) { + if let Err(error) = workdir.mark_network_cleanup_pending() { + warn!(vm_id, %error, "cannot persist pending network cleanup"); + return false; } } - if let Some(error) = first_error { - return Err(error).context("failed to remove netd-managed networking"); + // Ask for the release, rather than asking whether it can be asked for. + // A probe first would put a second round trip in front of every stop + // and -- worse -- would make a *busy* netd look like an absent one and + // skip the release entirely. The operation itself cannot be misread + // that way: it succeeds, or it says netd is not there, or netd answers + // with a refusal. + let request = netd::VmRef { + instance_id: self.config.cvm.instance_id.clone(), + vm_id: vm_id.to_string(), + }; + match netd::client(&self.config.netd.socket) + .remove_vm(request) + .await + { + Ok(netd::RemoveVmResponse { removed }) => { + if let Err(error) = workdir.clear_network_cleanup_pending() { + warn!(vm_id, %error, "cannot clear pending network cleanup"); + return false; + } + if removed > 0 { + info!(vm_id, removed, "released netd-managed interfaces"); + } + true + } + Err(error) if netd::is_unreachable(&error) => { + debug!(vm_id, %error, "no netd to release interfaces from"); + false + } + Err(error) => { + warn!( + vm_id, + "failed to release netd-managed interfaces: {error:#}" + ); + false + } } - Ok(()) } pub(crate) async fn stop_vm_process(&self, id: &str) -> Result<()> { @@ -808,23 +942,32 @@ impl App { pub async fn remove_vm(&self, id: &str) -> Result<()> { { let mut state = self.lock(); - let vm = state.get_mut(id).context("VM not found")?; - if vm.state.removing { + state.get(id).context("VM not found")?; + if !state.start_removing(id) { // Already being removed — idempotent return Ok(()); } - vm.state.removing = true; } + // Clear the in-memory mark if anything below fails before ownership of + // the removal is handed to the background task. In particular, a + // removal that cannot persist its crash-recovery marker must not start: + // without that marker a netd outage could leave interfaces no later + // VMM start knows to release. + let mut mark = RemovalMark::new(self.clone(), id); + // Persist the removing marker so crash recovery can resume let work_dir = self.work_dir(id)?; - if let Err(err) = work_dir.set_removing() { - warn!("failed to write .removing marker for {id}: {err:?}"); - } + work_dir + .set_removing() + .with_context(|| format!("failed to write .removing marker for {id}"))?; // User-initiated removal always deletes the workdir let app = self.clone(); let id = id.to_string(); + // `finish_remove_vm` owns the mark from here. Do not clear it in the + // request task while the background removal is still running. + mark.retain(); tokio::spawn(async move { if let Err(err) = app.finish_remove_vm(&id, true).await { error!("Background cleanup failed for {id}: {err:?}"); @@ -839,13 +982,29 @@ impl App { /// /// `delete_workdir`: true for user-initiated removal, false for orphan cleanup. async fn finish_remove_vm(&self, id: &str, delete_workdir: bool) -> Result<()> { + // Ordinary exits clear the mark, including the `?`s below and a panic + // in this task, which `tokio::spawn` would otherwise swallow. The one + // recoverable incomplete outcome explicitly retains it. + let mut mark = RemovalMark::new(self.clone(), id); + // Held across the stop, the wait and the release, not just the release. + // `removing` turns launches away, but a launch that passed that check + // before the marker was set is already inside the lock: it has not + // deployed yet, so the wait below sees nothing running and returns at + // once, and the release then deletes the interfaces of the QEMU that + // launch went on to start. Taking the lock first means the launch + // finishes before removal decides anything, and removal then stops what + // it actually started. + let _launch = self.launch_lock(id).await; // Stop the supervisor process (idempotent if already stopped) if let Err(err) = self.stop_vm_process(id).await { debug!("graceful VM stop during removal failed: {err:?}"); } - // Poll until the process is no longer running, then remove it. - // Some VMs take a long time to stop (e.g. 2+ hours), so we wait indefinitely. + // Poll until the process is no longer running, then remove it. The + // stop above is a SIGKILL, so this is however long the kernel takes to + // tear the VM down -- seconds for a large TD, unbounded for one wedged + // in a device reset. Waiting is still right: what follows deletes the + // interfaces and the workdir it is using. let mut poll_count: u64 = 0; loop { match self.supervisor.info(id).await { @@ -877,25 +1036,42 @@ impl App { } } - let runtime_networks = self.work_dir(id)?.runtime_networks(); - if let Err(error) = self.remove_netd_networks(id, &runtime_networks).await { - warn!(id, %error, "failed to remove netd-managed networking during VM removal"); - } + let vm_path = self.work_dir(id)?; + // Read before the release, because the release is what makes it stale. + // A VM that never asked netd for an interface -- user mode, a custom + // netdev, or one that never launched -- has nothing for netd to be + // holding, so an absent netd is not a reason to keep its directory. + let held_interfaces = vm_path.network_cleanup_pending() + || vm_path.runtime_networks().iter().any(needs_netd_interface); + let released = self.release_vm_interfaces(id).await; // Only delete the workdir for user-initiated removal or if .removing marker exists. // Orphaned supervisor processes without the marker keep their data intact. - let vm_path = self.work_dir(id)?; - if delete_workdir || vm_path.is_removing() { + if !(delete_workdir || vm_path.is_removing()) { if vm_path.path().exists() { - if let Err(err) = fs::remove_dir_all(&vm_path) { - error!("failed to remove VM directory for {id}: {err:?}"); - } + info!( + "VM {id} workdir preserved (orphan cleanup): {}", + vm_path.path().display() + ); } - } else if vm_path.path().exists() { - info!( - "VM {id} workdir preserved (orphan cleanup): {}", - vm_path.path().display() + } else if held_interfaces && !released { + // The `.removing` marker and the directory are what a later boot + // reads to retry this, and `remove_all` is idempotent, so keeping + // them costs one retry and losing them strands every interface + // netd still holds: nothing else on the host can name them. + warn!( + "VM {id} keeps its directory because netd did not release its interfaces; \ + the removal resumes at the next VMM start" ); + // The disk marker says this removal must be retried. Keep the + // matching in-memory state too, so start/update/stop cannot revive + // the VM before that retry happens. + mark.retain(); + return Ok(()); + } else if vm_path.path().exists() { + if let Err(err) = fs::remove_dir_all(&vm_path) { + error!("failed to remove VM directory for {id}: {err:?}"); + } } // Free CID and remove from memory (last step) @@ -915,16 +1091,13 @@ impl App { /// Returns false if a cleanup task is already running for this VM. fn spawn_finish_remove(&self, id: &str) -> bool { { - let mut state = self.lock(); - if let Some(vm) = state.get_mut(id) { - if vm.state.removing { - // Already being cleaned up — skip - return false; - } - vm.state.removing = true; + // Claimed in the set rather than in the entry: an orphaned + // supervisor process has no entry, and that is exactly the case + // where the launch lock is held with nothing turning waiters away. + if !self.lock().start_removing(id) { + // Already being cleaned up — skip + return false; } - // If VM is not in memory (e.g. orphaned supervisor process), no entry to guard - // but we still need to clean up the supervisor process. } let app = self.clone(); let id = id.to_string(); @@ -1995,6 +2168,337 @@ mod tests { use super::mr_config::{mr_config_version, MrConfigVersion}; use super::*; + fn test_app() -> App { + use rocket::figment::providers::Format as _; + let config: Config = rocket::figment::Figment::from( + rocket::figment::providers::Toml::string(crate::config::DEFAULT_CONFIG), + ) + .extract() + .unwrap(); + App::new(config, SupervisorClient::new("http://127.0.0.1:0")) + } + + fn test_config(netd_socket: &Path, run_path: &Path) -> Config { + use rocket::figment::providers::Format as _; + let mut config: Config = rocket::figment::Figment::from( + rocket::figment::providers::Toml::string(crate::config::DEFAULT_CONFIG), + ) + .extract() + .unwrap(); + config.netd.socket = netd_socket.to_path_buf(); + config.cvm.instance_id = "test-instance".to_string(); + config.run_path = run_path.to_path_buf(); + config + } + + fn app_talking_to(netd_socket: &Path) -> App { + let run_path = netd_socket.parent().unwrap_or(Path::new("/nonexistent")); + App::new( + test_config(netd_socket, run_path), + SupervisorClient::new("http://127.0.0.1:0"), + ) + } + + /// A supervisor with no remaining process lets removal exercise the real + /// directory and netd paths without launching a VM. + async fn stopped_supervisor() -> (SupervisorClient, tokio::task::JoinHandle<()>) { + use tokio::io::{AsyncBufReadExt, AsyncWriteExt, BufReader}; + let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap(); + let address = listener.local_addr().unwrap(); + let task = tokio::spawn(async move { + loop { + let (mut stream, _) = listener.accept().await.unwrap(); + let mut reader = BufReader::new(&mut stream); + let mut header = String::new(); + loop { + header.clear(); + assert!(reader.read_line(&mut header).await.unwrap() > 0); + if header == "\r\n" { + break; + } + } + let body = serde_json::to_string(&supervisor_client::supervisor::Response::Data( + Option::::None, + )) + .unwrap(); + stream + .write_all( + format!( + "HTTP/1.1 200 OK\r\nContent-Length: {}\r\nConnection: close\r\n\r\n{}", + body.len(), + body, + ) + .as_bytes(), + ) + .await + .unwrap(); + } + }); + (SupervisorClient::new(&format!("http://{address}")), task) + } + + #[tokio::test] + async fn failed_release_survives_snapshot_replacement_and_removal_retry() { + let (supervisor, server) = stopped_supervisor().await; + let dir = tempfile::tempdir().unwrap(); + let app = App::new( + test_config(&dir.path().join("absent-netd.sock"), dir.path()), + supervisor.clone(), + ); + let workdir = app.work_dir("vm-1").unwrap(); + std::fs::create_dir_all(workdir.path()).unwrap(); + // A legacy launch only left the runtime snapshot, without a marker. + let network = Networking { + nic: NicNetworking { + mode: NetworkingMode::Bridge, + ..Default::default() + }, + ..Default::default() + }; + workdir.set_runtime_networks(&[network]).unwrap(); + assert!(!workdir.network_cleanup_pending()); + assert!(!app.release_vm_interfaces("vm-1").await); + assert!(workdir.network_cleanup_pending()); + // A subsequent user-mode boot can replace the snapshot. Cleanup must + // still remember the bridge interfaces from the earlier boot. + workdir + .set_runtime_networks(&[Networking::default()]) + .unwrap(); + workdir.set_removing().unwrap(); + assert!(app.lock().start_removing("vm-1")); + app.finish_remove_vm("vm-1", true).await.unwrap(); + assert!(workdir.path().exists()); + assert!(workdir.is_removing()); + assert!(workdir.network_cleanup_pending()); + assert!(app.refuse_if_removing("vm-1").is_err()); + + // Simulate restarting the VMM after netd has recovered. + let netd = netd::testing::FakeNetd::spawn(&["RemoveVm"]); + let restarted = App::new(test_config(netd.socket(), dir.path()), supervisor); + restarted.finish_remove_vm("vm-1", true).await.unwrap(); + assert!(!workdir.path().exists()); + assert_eq!(netd.methods(), vec!["RemoveVm"]); + server.abort(); + } + + #[tokio::test] + async fn removal_without_a_snapshot_distinguishes_pending_from_never_prepared() { + let (supervisor, server) = stopped_supervisor().await; + let dir = tempfile::tempdir().unwrap(); + let app = App::new( + test_config(&dir.path().join("absent-netd.sock"), dir.path()), + supervisor, + ); + for (id, pending) in [("interrupted-prepare", true), ("user-only", false)] { + let workdir = app.work_dir(id).unwrap(); + std::fs::create_dir_all(workdir.path()).unwrap(); + if pending { + // Preparation wrote this before contacting netd, then crashed + // without ever writing runtime-networks.json. + workdir.mark_network_cleanup_pending().unwrap(); + } + workdir.set_removing().unwrap(); + app.finish_remove_vm(id, true).await.unwrap(); + assert_eq!(workdir.path().exists(), pending); + assert_eq!(workdir.is_removing(), pending); + } + server.abort(); + } + + #[tokio::test] + async fn successful_sweep_clears_the_pending_marker() { + let netd = netd::testing::FakeNetd::spawn(&["RemoveVm"]); + let dir = tempfile::tempdir().unwrap(); + let app = App::new( + test_config(netd.socket(), dir.path()), + SupervisorClient::new("http://127.0.0.1:0"), + ); + let workdir = app.work_dir("vm-1").unwrap(); + std::fs::create_dir_all(workdir.path()).unwrap(); + workdir.mark_network_cleanup_pending().unwrap(); + assert!(app.release_vm_interfaces("vm-1").await); + assert!(!workdir.network_cleanup_pending()); + assert!(workdir.path().exists()); + } + + /// A netd outage must not become a fleet that cannot be stopped. + #[tokio::test] + async fn a_stop_survives_a_netd_that_is_not_there() { + let app = app_talking_to(Path::new("/nonexistent/dstack-netd.sock")); + // Returns rather than propagating: there is no error type here on + // purpose, because there is no caller that should act on one. + app.release_vm_interfaces("vm-1").await; + } + + /// A removal deletes the workdir, and the workdir is the only thing left + /// that says to retry. So the release has to say whether it landed: an + /// answer means netd holds nothing, and anything else -- a refusal, or a + /// netd that is not there to ask -- means it may still. + #[tokio::test] + async fn a_release_says_whether_netd_still_holds_anything() { + let netd = netd::testing::FakeNetd::spawn(&["RemoveVm"]); + let app = app_talking_to(netd.socket()); + assert!(app.release_vm_interfaces("vm-1").await); + + // Refused: netd is up and still holding whatever it had. + let refusing = netd::testing::FakeNetd::spawn(&[]); + let app = app_talking_to(refusing.socket()); + assert!(!app.release_vm_interfaces("vm-1").await); + + // Not there to ask. A VM that never asked netd for an interface is + // unaffected -- the removal checks that separately -- but one that did + // must keep its directory so a later start can try again. + let app = app_talking_to(Path::new("/nonexistent/dstack-netd.sock")); + assert!(!app.release_vm_interfaces("vm-1").await); + } + + /// A refused sweep is not a reason to fail the stop. What netd holds stays + /// until this VM launches again or until an operator names it, but the + /// stop still succeeds: a netd outage must not become a fleet that cannot + /// be stopped. + #[tokio::test] + async fn a_netd_that_refuses_the_sweep_does_not_fail_the_stop() { + let netd = netd::testing::FakeNetd::spawn(&[]); + let app = app_talking_to(netd.socket()); + app.release_vm_interfaces("vm-1").await; + assert_eq!( + netd.methods(), + vec!["RemoveVm"], + "asked once, and not asked about afterwards" + ); + } + + /// One request, no question in front of it. A probe on the hot path is a + /// second round trip whose failure mode is silence: a netd too slow to + /// answer it reads as an absent one, and an absent netd's release is + /// skipped. + #[tokio::test] + async fn a_stop_asks_netd_to_sweep_without_a_question_first() { + let netd = netd::testing::FakeNetd::spawn(&["RemoveVm"]); + let app = app_talking_to(netd.socket()); + app.release_vm_interfaces("vm-1").await; + + assert_eq!(netd.methods(), vec!["RemoveVm"]); + let sweep = &netd.seen()[0]; + assert_eq!(sweep["vm_id"], "vm-1"); + assert_eq!(sweep["instance_id"], "test-instance"); + // A sweep names no NIC: reaching the indices the caller can no longer + // name is the entire point. + assert!(sweep.get("nic_index").is_none()); + } + + /// The orphan cleanup runs for an ID that never loaded, so a guard living + /// in the VM entry is not there when it holds the launch lock across the + /// whole teardown. Without the mark, `StartVm` on that ID waits the + /// teardown out with no error and no log. + #[tokio::test] + async fn a_removal_with_no_vm_entry_still_turns_operations_away() { + let app = test_app(); + assert!(app.refuse_if_removing("orphan").is_ok()); + assert!(app.lock().start_removing("orphan")); + assert!( + app.refuse_if_removing("orphan").is_err(), + "an orphan being removed is still a VM being removed" + ); + // And the same removal cannot be started twice. + assert!(!app.lock().start_removing("orphan")); + } + + /// An ordinary early error clears the in-memory mark so the caller can + /// retry rather than leaving a VM permanently inaccessible. + #[tokio::test] + async fn a_removal_that_gives_up_early_does_not_leave_the_vm_marked() { + let app = test_app(); + { + let _mark = RemovalMark::new(app.clone(), "vm-1"); + assert!(app.lock().start_removing("vm-1")); + assert!(app.refuse_if_removing("vm-1").is_err()); + } + assert!( + app.refuse_if_removing("vm-1").is_ok(), + "the mark is cleared however the removal ends" + ); + } + + /// Once a removal deliberately keeps its crash-recovery marker, dropping + /// the guard must not make the VM launchable again in the current process. + #[tokio::test] + async fn a_removal_waiting_for_netd_stays_marked() { + let app = test_app(); + assert!(app.lock().start_removing("vm-1")); + { + let mut mark = RemovalMark::new(app.clone(), "vm-1"); + mark.retain(); + } + assert!( + app.refuse_if_removing("vm-1").is_err(), + "the in-memory state must agree with the retained disk marker" + ); + } + + /// A restart decided before a stop must not outlive it. The restart task + /// reads the started flag off disk and only then queues a launch, which + /// waits for the lock the stop is holding; without a re-read under that + /// lock the launch resurrects a VM the operator was told was stopped. + #[tokio::test] + async fn an_automatic_restart_does_not_outlive_the_stop_it_raced() { + let dir = tempfile::tempdir().unwrap(); + let app = App::new( + test_config(Path::new("/nonexistent/netd.sock"), dir.path()), + SupervisorClient::new("http://127.0.0.1:0"), + ); + let work_dir = app.work_dir("vm-1").unwrap(); + std::fs::create_dir_all(work_dir.path()).unwrap(); + + work_dir.set_started(false).unwrap(); + app.start_vm_with_restart_policy("vm-1", false) + .await + .expect("an automatic restart of a stopped VM does nothing"); + + // The flag is the whole difference: with it set, the same call goes on + // to do the work, and fails here for want of a VM to launch. + work_dir.set_started(true).unwrap(); + assert!(app + .start_vm_with_restart_policy("vm-1", false) + .await + .is_err()); + + // An explicit start sets the flag itself and has nothing to re-read, + // so it is never turned away by one. + work_dir.set_started(false).unwrap(); + assert!(app.start_vm("vm-1").await.is_err()); + } + + /// The window a launch spends between reading "not running" and actually + /// starting QEMU is long -- a GPU reset, a netd conversation -- and the + /// sweep inside it deletes interfaces by deriving their names. Two entrants + /// in that window meant the loser deleting the winner's live TAPs. + #[tokio::test] + async fn one_vm_launches_at_a_time_and_the_lock_map_stays_small() { + let app = test_app(); + let held = app.launch_lock("vm-1").await; + + // A different VM is never blocked by it: a slow start must not stall + // every other launch on the node. + let other = tokio::time::timeout(Duration::from_millis(50), app.launch_lock("vm-2")).await; + assert!(other.is_ok(), "an unrelated VM must not wait"); + + // The same VM is. + let same = tokio::time::timeout(Duration::from_millis(50), app.launch_lock("vm-1")).await; + assert!(same.is_err(), "a second entrant must wait for the first"); + + drop(held); + drop(other); + tokio::time::timeout(Duration::from_millis(50), app.launch_lock("vm-1")) + .await + .expect("the lock is released"); + + // Nothing is in flight now, so the map holds nothing either. + assert!(app.launch_locks.lock().unwrap().len() <= 1); + let _ = app.launch_lock("vm-3").await; + assert!(app.launch_locks.lock().unwrap().len() <= 2); + } + #[test] fn accepts_server_generated_ids() { validate_vm_id(&uuid::Uuid::new_v4().to_string()).unwrap(); @@ -3094,6 +3598,14 @@ impl VmState { pub(crate) struct AppState { cid_pool: IdPool, vms: HashMap, + /// The VMs a removal is currently working on. + /// + /// Separate from `VmState::removing` because the set has to outlive the + /// entry. Orphan cleanup runs for IDs that never loaded into `vms`, and + /// `finish_remove_vm` holds the launch lock across the whole teardown, so + /// a guard that lives in the entry cannot turn away the operation that + /// would otherwise wait that teardown out. + removing: HashSet, } impl AppState { @@ -3116,6 +3628,57 @@ impl AppState { pub fn iter_vms(&self) -> impl Iterator { self.vms.values() } + + /// Claims `id` for a removal. False when one already has it. + fn start_removing(&mut self, id: &str) -> bool { + if let Some(vm) = self.vms.get_mut(id) { + vm.state.removing = true; + } + self.removing.insert(id.to_string()) + } + + fn is_removing(&self, id: &str) -> bool { + self.removing.contains(id) + } +} + +/// Clears the in-flight removal mark however the removal ends. +/// +/// `finish_remove_vm` returns early on more than its happy path, and a mark +/// left behind is not a stale flag: every operation on that VM answers "being +/// removed" from then on, including the removal that would retry. +struct RemovalMark { + app: App, + id: String, + clear_on_drop: bool, +} + +impl RemovalMark { + fn new(app: App, id: &str) -> Self { + Self { + app, + id: id.to_string(), + clear_on_drop: true, + } + } + + /// Leave the removal mark in place after this guard goes out of scope. + fn retain(&mut self) { + self.clear_on_drop = false; + } +} + +impl Drop for RemovalMark { + fn drop(&mut self) { + if !self.clear_on_drop { + return; + } + let mut state = self.app.lock(); + state.removing.remove(&self.id); + if let Some(vm) = state.vms.get_mut(&self.id) { + vm.state.removing = false; + } + } } /// Reject VM ids that would escape `run_path` once joined into a filesystem diff --git a/dstack/vmm/src/app/workdir.rs b/dstack/vmm/src/app/workdir.rs index a9822c7ec..5809f73ba 100644 --- a/dstack/vmm/src/app/workdir.rs +++ b/dstack/vmm/src/app/workdir.rs @@ -143,6 +143,26 @@ impl VmWorkDir { self.workdir.join("runtime-networks.json") } + /// Written before asking netd to create anything, and removed only after + /// a successful whole-VM sweep. Unlike the runtime snapshot, this survives + /// failed launches and changes to a topology that no longer uses netd. + pub fn mark_network_cleanup_pending(&self) -> Result<()> { + safe_write::safe_write(self.workdir.join(".netd-pending"), b"") + .context("failed to persist pending network cleanup") + } + + pub fn network_cleanup_pending(&self) -> bool { + self.workdir.join(".netd-pending").exists() + } + + pub fn clear_network_cleanup_pending(&self) -> Result<()> { + match fs::remove_file(self.workdir.join(".netd-pending")) { + Ok(()) => Ok(()), + Err(error) if error.kind() == std::io::ErrorKind::NotFound => Ok(()), + Err(error) => Err(error).context("failed to clear pending network cleanup"), + } + } + pub fn runtime_networks(&self) -> Vec { fs::read_to_string(self.runtime_networks_path()) .ok() diff --git a/dstack/vmm/src/main.rs b/dstack/vmm/src/main.rs index 17b93d194..c2c8e2aaf 100644 --- a/dstack/vmm/src/main.rs +++ b/dstack/vmm/src/main.rs @@ -77,6 +77,48 @@ struct NetdArgs { /// Override the Unix socket configured in [netd]. #[arg(long)] socket: Option, + /// Inspect a running netd instead of starting one. + #[command(subcommand)] + command: Option, +} + +#[derive(Subcommand)] +enum NetdCommand { + /// List every host interface netd holds. + /// + /// Answers the question a leak is made of -- whose is this interface -- + /// which deriving a name from an identity cannot. + List { + /// Only this VMM instance's interfaces. Defaults to every one netd + /// owns, including those it cannot attribute. + #[arg(long)] + instance: Option, + }, + /// Delete one interface by name. + /// + /// For what nothing else can name: an interface built before netd recorded + /// ownership, or by another netd, whose VM is gone. `netd list` shows these + /// with no instance and no VM, so nothing can derive the sweep that would + /// take them; an operator who can tell what they are says so here. + RemoveInterface { + /// The interface name, as `netd list` prints it. + name: String, + }, + /// Delete every interface netd holds for one VM. + /// + /// For a VM whose VMM will never ask again -- one whose directory was + /// deleted by hand, or whose instance is gone. A VMM sweeps its own VMs on + /// every stop and every removal, and keeps a removal pending until the + /// sweep lands; this is for when no VMM will ever run that sweep. + RemoveVm { + /// The `cvm.instance_id` of the VMM that created them. `netd list` + /// shows it. + #[arg(long)] + instance: String, + /// The VM's ID. + #[arg(long)] + vm: String, + }, } #[derive(ClapArgs)] @@ -192,6 +234,74 @@ async fn log_rotation_task(app: App) { } } +/// Client-side netd subcommands. Talks to the socket like the VMM does, so it +/// needs whatever the socket's permissions ask for and not root. +async fn run_netd_command(config: &NetdConfig, command: &NetdCommand) -> Result<()> { + match command { + NetdCommand::List { instance } => { + let interfaces = netd::client(&config.socket) + .list_interfaces(netd::ListInterfacesRequest { + instance_id: instance.clone().unwrap_or_default(), + }) + .await + .context("failed to list netd interfaces")? + .interfaces; + println!( + "{:<16} {:<8} {:<24} {:<38} {:>3}", + "INTERFACE", "KIND", "INSTANCE", "VM", "NIC" + ); + let mut unattributed = 0; + for record in &interfaces { + if record.instance_id.is_none() { + unattributed += 1; + } + println!( + "{:<16} {:<8} {:<24} {:<38} {:>3}", + record.tap, + record.kind, + record.instance_id.as_deref().unwrap_or("-"), + record.vm_id.as_deref().unwrap_or("-"), + record + .nic_index + .map_or_else(|| "-".to_string(), |index| index.to_string()), + ); + } + println!(); + println!("{} interface(s)", interfaces.len()); + if unattributed > 0 { + // Not a fault to fix by hand: an interface built before netd + // recorded ownership, or by another netd, carries no record and + // gets one the next time its VM launches. + println!( + "{unattributed} carry no ownership record; `netd remove-interface` takes \ + one by name" + ); + } + Ok(()) + } + NetdCommand::RemoveInterface { name } => { + netd::client(&config.socket) + .remove_interface_by_name(netd::InterfaceName { tap: name.clone() }) + .await + .with_context(|| format!("failed to remove {name}"))?; + println!("removed {name}"); + Ok(()) + } + NetdCommand::RemoveVm { instance, vm } => { + let removed = netd::client(&config.socket) + .remove_vm(netd::VmRef { + instance_id: instance.clone(), + vm_id: vm.clone(), + }) + .await + .context("failed to remove the VM's interfaces")? + .removed; + println!("removed {removed} interface(s) for {vm}"); + Ok(()) + } + } +} + #[rocket::main] async fn main() -> Result<()> { { @@ -235,6 +345,9 @@ async fn main() -> Result<()> { .context("failed to load [cvm.network_filter] for netd")?, ); } + if let Some(command) = &netd_args.command { + return run_netd_command(&netd_config, command).await; + } return netd::serve(netd_config).await; } @@ -246,6 +359,7 @@ async fn main() -> Result<()> { // Preserve the existing startup validation. The broader static checks are // opt-in through `check-config` until they have seen wider deployment use. + netd::validate_instance_id(&config.cvm.instance_id)?; config .host_api .validate() diff --git a/dstack/vmm/src/main_service.rs b/dstack/vmm/src/main_service.rs index 7d0c7099b..ea4166dcc 100644 --- a/dstack/vmm/src/main_service.rs +++ b/dstack/vmm/src/main_service.rs @@ -928,6 +928,21 @@ impl VmmRpc for RpcHandler { async fn update_vm(self, request: UpdateVmRequest) -> Result { info!(vm_id = %request.id, "update_vm RPC called"); + // A VM being removed is not one to reconfigure. Before the lock, + // because removal holds it across the whole teardown and anything that + // only asked afterwards would wait that out in order to be told no. + self.app.refuse_if_removing(&request.id)?; + // Held from here rather than around the parts that touch the host, + // because everything below writes into the workdir -- the compose + // file first, the manifest last -- and `put_manifest` creates the + // directory it writes into. An update that resumed after a removal + // deleted that directory would recreate it holding nothing but a + // manifest: invisible to `list_vms`, unloadable at every start, and + // claiming the VM's netd interfaces against collection forever. + let _launch = self.app.launch_lock(&request.id).await; + // Again under the lock: removal can have claimed the VM while this + // waited for it. + self.app.refuse_if_removing(&request.id)?; let new_id = if !request.compose_file.is_empty() { // check the compose file is valid let _app_compose: AppCompose = @@ -1008,18 +1023,19 @@ impl VmmRpc for RpcHandler { let networks = networks_from_proto(&request.networks, &cvm)?; resolve_requested_networks(&networks, &cvm, manifest.vcpu)? }; + // Under the launch lock this whole call holds. Reading "not + // running" outside it and acting on the answer inside is the exact + // race the lock exists to close: a launch can start, prepare its + // interfaces and deploy QEMU in between, and the release would + // then delete the interfaces of a VM that is running -- silently, + // since QEMU stays up and the supervisor still reports it healthy. let is_running = self .app .supervisor .info(&request.id) .await? .is_some_and(|info| info.state.status.is_running()); - if !is_running { - let runtime_networks = vm_work_dir.runtime_networks(); - self.app - .remove_netd_networks(&request.id, &runtime_networks) - .await - .context("failed to remove previous netd-managed networking")?; + if !is_running && self.app.release_vm_interfaces(&request.id).await { vm_work_dir.clear_runtime_networks()?; } manifest.networks = networks; @@ -1772,6 +1788,9 @@ mod tests { .expect("a named backend is an override"); } + /// Deliberately restated rather than calling `NetworkingMode::as_str`: the + /// test below checks that what `GetInfo` reports is accepted back, and a + /// helper that shares the production mapping could only ever agree with it. fn networking_mode_name_for_test(mode: NetworkingMode) -> &'static str { match mode { NetworkingMode::Bridge => "bridge", diff --git a/dstack/vmm/src/netd.rs b/dstack/vmm/src/netd.rs index e13facc73..5812edf6c 100644 --- a/dstack/vmm/src/netd.rs +++ b/dstack/vmm/src/netd.rs @@ -9,6 +9,7 @@ //! request and response messages. use std::{ + collections::HashSet, fs::{File, OpenOptions, Permissions}, io::{self, Write as _}, os::{ @@ -31,14 +32,15 @@ use ra_rpc::{CallContext, RpcCall}; use rocket::listener::{unix::UnixStream as RocketUnixStream, Endpoint, Listener}; use sha2::{Digest, Sha256}; use tokio::net::UnixListener; -use tracing::{info, warn}; +use tracing::{debug, info, warn}; use uuid::Uuid; use wait_timeout::ChildExt; use dstack_netd_rpc::netd_server::{NetdRpc, NetdServer}; pub use dstack_netd_rpc::{ - netd_client::NetdClient, CheckInterfaceRequest, InterfaceIdentity, PrepareBridgeRequest, - PrepareMacvtapRequest, PreparedInterface, + netd_client::NetdClient, CheckInterfaceRequest, InterfaceIdentity, InterfaceList, + InterfaceName, InterfaceRecord, ListInterfacesRequest, PrepareBridgeRequest, + PrepareMacvtapRequest, PreparedInterface, RemoveVmResponse, VmRef, }; use crate::config::{NetdConfig, NetworkFilterConfig}; @@ -53,6 +55,20 @@ const LOCK_PATH: &str = "/run/lock/dstack-netd.lock"; /// Upper bound on TAP queue pairs netd will create. Mirrors the VMM's own cap /// so a malformed request cannot ask the kernel for an unbounded device. const MAX_QUEUES: u32 = 64; +/// Highest NIC index an identity may name. Also the width of the space a +/// whole-VM sweep has to enumerate, since it derives names instead of reading a +/// record. +const MAX_NIC_INDEX: u32 = 255; +/// The interface names netd may create. Reserved: anything matching it is +/// netd's to delete, and nothing else on the host may take one. +const TAP_PREFIX: &str = "dt"; +/// Hex characters of digest in a TAP name, after [`TAP_PREFIX`]. +const TAP_DIGEST_CHARS: usize = 12; +/// Version tag on the ownership record. Present so a later format can be told +/// from this one rather than mis-parsed as it. +const ALIAS_PREFIX: &str = "dstack1"; +/// What the kernel stores in an interface alias, minus the terminator. +const MAX_IFALIAS: usize = 255; pub fn tap_name(identity: &InterfaceIdentity) -> String { let input = format!( @@ -60,7 +76,82 @@ pub fn tap_name(identity: &InterfaceIdentity) -> String { identity.instance_id, identity.vm_id, identity.nic_index ); let digest = Sha256::digest(input.as_bytes()); - format!("dt{}", hex::encode(&digest[..6])) + format!( + "{TAP_PREFIX}{}", + hex::encode(&digest[..TAP_DIGEST_CHARS / 2]) + ) +} + +/// The ownership record netd writes onto every interface it creates. +/// +/// The record lives on the resource, so it has exactly the resource's +/// lifetime. A file under `/run` would be a second thing to keep in step with +/// the first, and the failure this whole path exists to fix is precisely a +/// record that got out of step: written after the interface, lost with the +/// directory, and unreadable to anything but the process that wrote it. +/// +/// Never trusted as *authority*. Anything that can reach this socket can also +/// name an identity, and the interface name is a digest of that identity -- +/// so a record is believed only when re-deriving the name from it reproduces +/// the name it is written on. Ambiguity (a separator inside an identity), +/// truncation, and forgery all fail that check and land in the same bucket as +/// no record at all, which is the bucket handled conservatively. +pub fn interface_alias(identity: &InterfaceIdentity) -> String { + format!( + "{ALIAS_PREFIX}:{}:{}:{}", + identity.nic_index, identity.instance_id, identity.vm_id + ) +} + +/// The identity an interface claims, if the claim checks out. +/// +/// `nic_index` first, so the two free-form fields are the last two and a +/// `vm_id` containing the separator still parses. An `instance_id` containing +/// one does not, and is refused at prepare rather than mis-parsed here. +pub fn owner_of(tap: &str, alias: &str) -> Option { + // `trim_end_matches`, not `trim`: sysfs adds a newline, and a `vm_id` + // whose own trailing whitespace were trimmed off here would re-derive a + // name that is not the one it is on, making the interface permanently + // unattributable -- never collected, only removable by hand. + let rest = alias + .trim_end_matches(['\n', '\r']) + .strip_prefix(ALIAS_PREFIX)? + .strip_prefix(':')?; + let (nic_index, rest) = rest.split_once(':')?; + let (instance_id, vm_id) = rest.split_once(':')?; + let identity = InterfaceIdentity { + instance_id: instance_id.to_string(), + vm_id: vm_id.to_string(), + nic_index: nic_index.parse().ok()?, + }; + // The name is the proof. A record that does not reproduce it describes + // some other interface, or nothing. + (tap_name(&identity) == tap).then_some(identity) +} + +/// Whether this name is one netd can have created. See [`TAP_PREFIX`]. +pub fn is_managed_name(interface: &str) -> bool { + let Some(digest) = interface.strip_prefix(TAP_PREFIX) else { + return false; + }; + digest.len() == TAP_DIGEST_CHARS + && digest + .bytes() + .all(|byte| byte.is_ascii_digit() || (b'a'..=b'f').contains(&byte)) +} + +/// Rejects an instance ID no interface could be recorded as belonging to. +/// +/// At startup rather than at the first launch. The VMM derives one that is +/// always valid; an operator who configured their own learns here rather than +/// from the first VM that fails to get a NIC. +pub fn validate_instance_id(instance_id: &str) -> Result<()> { + validate_identity(&InterfaceIdentity { + instance_id: instance_id.to_string(), + vm_id: "0".repeat(64), + nic_index: MAX_NIC_INDEX, + }) + .context("invalid cvm.instance_id") } pub fn instance_id(configured: &str, run_path: &Path) -> String { @@ -278,6 +369,39 @@ impl NetdRpc for NetdHandler { .await } + async fn remove_vm(self, request: VmRef) -> Result { + self.run(move |config| { + let removed = sweep_vm_interfaces( + &config.libvirt_uri, + &request.instance_id, + &request.vm_id, + config.filter_policy().requires_binding(), + )?; + Ok(RemoveVmResponse { removed }) + }) + .await + } + + async fn remove_interface_by_name(self, request: InterfaceName) -> Result<()> { + self.run(move |config| { + let tap = request.tap; + if !is_managed_name(&tap) { + bail!("{tap} is not a name netd could have created"); + } + remove_interface(&config.libvirt_uri, &tap, BindingCleanup::BestEffort) + }) + .await + } + + async fn list_interfaces(self, request: ListInterfacesRequest) -> Result { + self.run(move |config| { + Ok(InterfaceList { + interfaces: list_interfaces(&config.libvirt_uri, &request.instance_id), + }) + }) + .await + } + async fn check_interface(self, request: CheckInterfaceRequest) -> Result<()> { self.run(move |config| { let identity = required_identity(&request.identity)?; @@ -361,6 +485,7 @@ fn prepare_macvtap( add.extend_from_slice(&["type", "macvtap", "mode", mode]); ip(&add)?; let result = (|| { + set_alias(&tap, identity)?; let ifindex = std::fs::read_to_string(Path::new("/sys/class/net").join(&tap).join("ifindex")) .context("failed to read macvtap ifindex")?; @@ -384,7 +509,7 @@ fn prepare_macvtap( })(); match result { Ok(device) => { - info!(%tap, %parent, %mode, %device, %queues, "prepared macvtap"); + info!(%tap, %parent, %mode, %device, %queues, workdir = %request.workdir, "prepared macvtap"); Ok(PreparedInterface { tap, device, @@ -461,6 +586,10 @@ fn prepare_bridge( add.extend_from_slice(&["user", &uid]); ip(&add)?; let result = (|| { + // Before anything else it could fail at. An interface that exists + // without a record is one nothing can attribute, and the window in + // which that is true is the window a crash turns permanent. + set_alias(&tap, identity)?; ip(&["link", "set", "dev", &tap, "master", &request.bridge])?; if filtered { let xml = binding_xml(identity, &request.mac, &tap, filter); @@ -477,7 +606,14 @@ fn prepare_bridge( let _ = remove_interface(libvirt_uri, &tap, BindingCleanup::BestEffort); return Err(error); } - info!(%tap, bridge = %request.bridge, %filtered, %queues, "prepared TAP"); + info!( + %tap, + bridge = %request.bridge, + %filtered, + %queues, + workdir = %request.workdir, + "prepared TAP" + ); Ok(PreparedInterface { tap, device: String::new(), @@ -497,11 +633,177 @@ enum BindingCleanup { /// be running at all, and a stale binding left by an earlier, filtered /// interface at this name is still worth clearing when it is. BestEffort, + /// The caller has already decided about the binding. Used by a pass over + /// many interfaces, which asks libvirt once about all of them rather than + /// once per interface. + Skip, +} + +/// Deletes every interface a VM could hold, by deriving each name rather than +/// consulting a record. +/// +/// `validate_identity` caps the NIC index, so the whole space a VM can occupy +/// is enumerable: 256 names, each a `stat` that usually misses. Cleanup is +/// best-effort about unknown bindings on unfiltered nodes, which need not have +/// libvirtd at all. Filtered nodes must confirm their bindings were released. +/// +/// A name with no interface is not skipped. An nwfilter binding outlives the +/// TAP it was bound to, so the one state teardown must not leave behind is +/// exactly the one a `/sys/class/net` check cannot see: the per-name Remove +/// this replaced deleted the binding unconditionally, and a sweep that reaches +/// less than the thing it replaced is not a sweep. Those names are decided +/// against a single listing, because the whole point of enumerating a bounded +/// space is that deciding one name stays cheap. +fn sweep_vm_interfaces( + libvirt_uri: &str, + instance_id: &str, + vm_id: &str, + requires_binding: bool, +) -> Result { + let identity = InterfaceIdentity { + instance_id: instance_id.to_string(), + vm_id: vm_id.to_string(), + nic_index: 0, + }; + validate_identity(&identity)?; + let bindings = existing_bindings(libvirt_uri); + // Not `bindings.is_some()`. A listing that could not be produced says + // nothing about whether a *deletion* will work, and reading it as "libvirt + // is down, skip the bindings" would mean a node whose listing breaks for + // any reason silently stops cleaning up bindings at all -- which is worse + // than the per-name asking this listing exists to avoid. The pass finds + // out by trying, once. + let mut libvirt = true; + let mut removed = 0; + let mut first_error = (requires_binding && bindings.is_none()) + .then(|| anyhow::anyhow!("cannot confirm nwfilter cleanup: binding listing failed")); + for nic_index in 0..=MAX_NIC_INDEX { + let tap = tap_name(&InterfaceIdentity { + nic_index, + ..identity.clone() + }); + let present = Path::new("/sys/class/net").join(&tap).exists(); + // A pass gets one answer about libvirt, not one per interface. Asking + // again after it has failed is how a hung `libvirtd` turns a bounded + // collection into an unbounded one. + let known_binding = bindings.as_ref().is_some_and(|held| held.contains(&tap)); + let wanted = present || known_binding; + if libvirt && wanted && !is_macvtap(&tap) { + if let Err(error) = delete_binding(libvirt_uri, &tap) { + warn!(%tap, %error, "failed to remove an nwfilter binding"); + libvirt = false; + // Unfiltered nodes do not require a running libvirtd. An + // unknown, possible binding must not make a successful TAP + // deletion fail there. Still retain failures for bindings we + // actually found, including ones left by an older policy. + if requires_binding || known_binding { + first_error.get_or_insert(error); + } + } else if !present { + info!(%tap, %vm_id, "removed orphaned nwfilter binding"); + } + } + if !libvirt && known_binding { + first_error + .get_or_insert_with(|| anyhow::anyhow!("nwfilter binding {tap} was not released")); + } + if !present { + continue; + } + // Keep going after a failure. Stopping at the first one would leave the + // rest of a VM's interfaces behind over one that is stuck. + match remove_interface(libvirt_uri, &tap, BindingCleanup::Skip) { + Err(error) => { + warn!(%tap, %error, "failed to remove interface"); + first_error.get_or_insert(error); + } + Ok(()) => { + info!(%tap, %vm_id, "removed interface"); + removed += 1; + } + } + } + match first_error { + Some(error) => Err(error).context("failed to remove every interface for this VM"), + None => Ok(removed), + } +} + +/// Every resource netd owns, read off the host rather than out of a record. +/// +/// Ownership is the reserved name plus the kernel's own answer about what kind +/// of device it is; attribution is the interface's alias, checked by +/// re-deriving the name from it. A listing never fails for want of libvirt: on +/// a node that does not filter, `libvirtd` need not be running, and an +/// interface inventory that refused to be produced without it would be +/// unavailable exactly where unfiltered TAPs live. +fn list_interfaces(libvirt_uri: &str, instance_id: &str) -> Vec { + let bindings = existing_bindings(libvirt_uri); + let mut records = Vec::new(); + let mut seen = HashSet::new(); + if let Ok(entries) = std::fs::read_dir("/sys/class/net") { + for entry in entries.flatten() { + let Ok(tap) = entry.file_name().into_string() else { + continue; + }; + if !is_managed_name(&tap) { + continue; + } + let kind = if is_macvtap(&tap) { + "macvtap" + } else if is_tuntap(&tap) { + "tap" + } else { + // The name is netd's to use, but this is not a device netd + // creates. Listing it would invite a caller to delete it. + continue; + }; + let alias = + std::fs::read_to_string(Path::new("/sys/class/net").join(&tap).join("ifalias")) + .unwrap_or_default(); + let owner = owner_of(&tap, &alias); + seen.insert(tap.clone()); + records.push(InterfaceRecord { + kind: kind.to_string(), + nic_index: owner.as_ref().map(|identity| identity.nic_index), + instance_id: owner.as_ref().map(|identity| identity.instance_id.clone()), + vm_id: owner.map(|identity| identity.vm_id), + tap, + }); + } + } + // A binding outlives its interface, and an interface is the only thing that + // carries a record, so an orphaned binding can never be attributed. It is + // still netd's: nothing else creates a binding at one of these names. + for name in bindings.into_iter().flatten() { + if !seen.contains(&name) { + records.push(InterfaceRecord { + tap: name, + kind: "binding".to_string(), + instance_id: None, + vm_id: None, + nic_index: None, + }); + } + } + if !instance_id.is_empty() { + records.retain(|record| record.instance_id.as_deref() == Some(instance_id)); + } + records.sort_by(|left, right| left.tap.cmp(&right.tap)); + records } fn remove_interface(libvirt_uri: &str, tap: &str, cleanup: BindingCleanup) -> Result<()> { let macvtap = is_macvtap(tap); if Path::new("/sys/class/net").join(tap).exists() { + // The name is 48 bits of SHA-256, so a collision is not the worry. A + // caller asserting an identity that happens to derive to some + // pre-existing device is: netd runs as root and `ip link delete` does + // not ask what it is deleting. netd creates exactly two kinds of + // device, and the kernel publishes an attribute unique to each. + if !macvtap && !is_tuntap(tap) { + bail!("refusing to delete {tap}: it is neither a tun/tap nor a macvtap device"); + } let _ = ip(&["link", "set", "dev", tap, "down"]); } // A macvtap interface never carries a binding. Anything else might: this @@ -509,6 +811,7 @@ fn remove_interface(libvirt_uri: &str, tap: &str, cleanup: BindingCleanup) -> Re // outlives the interface. if !macvtap { match cleanup { + BindingCleanup::Skip => {} BindingCleanup::Required => delete_binding(libvirt_uri, tap)?, BindingCleanup::BestEffort => { // netd refuses to start without virsh, so the binary is always @@ -527,6 +830,24 @@ fn remove_interface(libvirt_uri: &str, tap: &str, cleanup: BindingCleanup) -> Re Ok(()) } +/// Records who an interface belongs to, on the interface. See +/// [`interface_alias`]. +fn set_alias(tap: &str, identity: &InterfaceIdentity) -> Result<()> { + let alias = interface_alias(identity); + ip(&["link", "set", "dev", tap, "alias", &alias]) + .with_context(|| format!("failed to record ownership on {tap}")) +} + +/// Whether this is a tun/tap device. `tun_flags` is published by the tun +/// driver and by nothing else, so its presence is the kernel's own answer -- +/// as `macvtap/` is for the other kind of device netd creates. +fn is_tuntap(interface: &str) -> bool { + Path::new("/sys/class/net") + .join(interface) + .join("tun_flags") + .exists() +} + fn is_macvtap(interface: &str) -> bool { Path::new("/sys/class/net") .join(interface) @@ -642,9 +963,25 @@ fn validate_identity(identity: &InterfaceIdentity) -> Result<()> { bail!("invalid {label}"); } } - if identity.nic_index > 255 { + if identity.nic_index > MAX_NIC_INDEX { bail!("NIC index is out of range"); } + // An identity that cannot be recorded on the interface is refused rather + // than built unattributed. A host resource nothing can name the owner of + // is the thing this whole path exists to stop producing, and the kernel's + // alias is the only place with the interface's exact lifetime to put it. + let alias = interface_alias(identity); + if alias.len() > MAX_IFALIAS { + bail!( + "identity is too long to record on the interface: {} bytes of {MAX_IFALIAS}", + alias.len() + ); + } + // The record puts the two free-form fields last, so only the first of them + // has to be unambiguous. + if identity.instance_id.contains(':') { + bail!("instance ID must not contain ':'"); + } Ok(()) } @@ -705,13 +1042,47 @@ fn ip(args: &[&str]) -> Result<()> { } fn virsh(uri: &str, args: &[&str], stdin: Option<&[u8]>) -> Result<()> { + virsh_output(uri, args, stdin).map(|_| ()) +} + +fn virsh_output(uri: &str, args: &[&str], stdin: Option<&[u8]>) -> Result { let mut full_args = vec!["--connect", uri]; full_args.extend_from_slice(args); - run_command(VIRSH_PATH, &full_args, stdin) + run_command_with_timeout(VIRSH_PATH, &full_args, stdin, COMMAND_TIMEOUT) +} + +/// Every nwfilter binding libvirt holds at a name netd could have created. +/// +/// One call, so that a sweep can decide 256 names against a set instead of +/// asking libvirt 256 times. `None` means libvirt could not be asked at all, +/// which on a node running unfiltered TAPs is the normal state -- `virsh` must +/// be installed for netd to start, but `libvirtd` need not be running. +/// +/// The command has no machine-readable mode: it prints a two-line header and +/// then one binding per line, interface name first, and it accepts no options +/// at all -- `--name` is not one of them, and asking for it fails the whole +/// call. Narrowing to netd's own name space is what makes parsing a human +/// table safe: a header, a rule line, or a column that moves cannot produce a +/// `dt` name, and a binding at any other name is not netd's to reason about. +fn existing_bindings(uri: &str) -> Option> { + match virsh_output(uri, &["nwfilter-binding-list"], None) { + Ok(output) => Some( + output + .lines() + .filter_map(|line| line.split_whitespace().next()) + .filter(|name| is_managed_name(name)) + .map(str::to_string) + .collect(), + ), + Err(error) => { + debug!("could not list nwfilter bindings: {error:#}"); + None + } + } } fn run_command(program: &str, args: &[&str], stdin: Option<&[u8]>) -> Result<()> { - run_command_with_timeout(program, args, stdin, COMMAND_TIMEOUT) + run_command_with_timeout(program, args, stdin, COMMAND_TIMEOUT).map(|_| ()) } fn run_command_with_timeout( @@ -719,7 +1090,7 @@ fn run_command_with_timeout( args: &[&str], stdin: Option<&[u8]>, command_timeout: Duration, -) -> Result<()> { +) -> Result { let mut child = Command::new(program) .args(args) .stdin(if stdin.is_some() { @@ -751,7 +1122,7 @@ fn run_command_with_timeout( let error = String::from_utf8_lossy(&output.stderr); bail!("{} failed: {}", Path::new(program).display(), error.trim()); } - Ok(()) + Ok(String::from_utf8_lossy(&output.stdout).into_owned()) } fn require_executable(path: &str) -> Result<()> { @@ -775,6 +1146,168 @@ fn prepare_socket_path(socket: &Path) -> Result<()> { Ok(()) } +/// A netd that exists only to be talked to. +/// +/// The VMM's side of the netd conversation -- what it refuses, what it does +/// when netd refuses -- needs no privileged daemon, only something that serves +/// the same RPC on a socket. This is that, scripted per method, recording what +/// it was asked so a test can assert on the conversation rather than on its +/// effects. +#[cfg(test)] +pub(crate) mod testing { + use std::{ + path::{Path, PathBuf}, + sync::{Arc, Mutex}, + }; + + use anyhow::{bail, Result}; + use ra_rpc::{CallContext, RpcCall}; + use serde_json::Value; + use tokio::net::UnixListener; + + use super::{ + CheckInterfaceRequest, InterfaceIdentity, InterfaceList, InterfaceName, + ListInterfacesRequest, NetdListener, NetdRpc, NetdServer, PrepareBridgeRequest, + PrepareMacvtapRequest, PreparedInterface, RemoveVmResponse, VmRef, + }; + + #[derive(Clone)] + pub(crate) struct FakeState { + /// Methods answered with success; every other one is refused. + handles: Arc>, + seen: Arc>>, + } + + pub(crate) struct FakeHandler(FakeState); + + impl RpcCall for FakeHandler { + type PrpcService = NetdServer; + + fn construct(context: CallContext<'_, FakeState>) -> Result { + Ok(Self(context.state.clone())) + } + } + + impl FakeHandler { + fn record(&self, method: &str, request: impl serde::Serialize) -> Result<()> { + self.0 + .seen + .lock() + .expect("poisoned") + .push((method.to_string(), serde_json::to_value(request)?)); + if !self.0.handles.iter().any(|name| name == method) { + bail!("fake netd refuses {method}"); + } + Ok(()) + } + } + + impl NetdRpc for FakeHandler { + async fn prepare_bridge(self, request: PrepareBridgeRequest) -> Result { + let queues = request.queues.max(1); + self.record("PrepareBridge", request)?; + Ok(PreparedInterface { + tap: "dtdeadbeef00".into(), + device: String::new(), + queues, + }) + } + + async fn prepare_macvtap( + self, + request: PrepareMacvtapRequest, + ) -> Result { + let queues = request.queues.max(1); + self.record("PrepareMacvtap", request)?; + Ok(PreparedInterface { + tap: "dtdeadbeef00".into(), + device: "/dev/tap1".into(), + queues, + }) + } + + async fn remove_interface(self, request: InterfaceIdentity) -> Result<()> { + self.record("RemoveInterface", request) + } + + async fn remove_vm(self, request: VmRef) -> Result { + self.record("RemoveVm", request)?; + Ok(RemoveVmResponse { removed: 0 }) + } + + async fn remove_interface_by_name(self, request: InterfaceName) -> Result<()> { + self.record("RemoveInterfaceByName", request) + } + + async fn list_interfaces(self, request: ListInterfacesRequest) -> Result { + self.record("ListInterfaces", request)?; + Ok(InterfaceList::default()) + } + + async fn check_interface(self, request: CheckInterfaceRequest) -> Result<()> { + self.record("CheckInterface", request) + } + } + + pub(crate) struct FakeNetd { + _dir: tempfile::TempDir, + socket: PathBuf, + seen: Arc>>, + } + + impl FakeNetd { + /// Serves on a fresh socket, answering `handles` and refusing the + /// rest. The socket is bound before this returns, so a caller can + /// connect at once. + pub(crate) fn spawn(handles: &[&str]) -> Self { + let dir = tempfile::tempdir().expect("tempdir"); + let socket = dir.path().join("netd.sock"); + let listener = UnixListener::bind(&socket).expect("bind"); + let state = FakeState { + handles: Arc::new(handles.iter().map(|name| name.to_string()).collect()), + seen: Arc::new(Mutex::new(Vec::new())), + }; + let seen = state.seen.clone(); + tokio::spawn(async move { + let figment = rocket::Config::figment().merge(("log_level", "off")); + let ignite = rocket::custom(figment) + .manage(state) + .mount("/prpc", ra_rpc::prpc_routes!(FakeState, FakeHandler)) + .ignite() + .await + .expect("ignite fake netd"); + let _ = ignite.launch_on(NetdListener(listener)).await; + }); + Self { + _dir: dir, + socket, + seen, + } + } + + pub(crate) fn socket(&self) -> &Path { + &self.socket + } + + /// Every request it was sent, in order. + pub(crate) fn seen(&self) -> Vec { + self.calls() + .into_iter() + .map(|(_, request)| request) + .collect() + } + + /// The method of every request it was sent, in order. + pub(crate) fn methods(&self) -> Vec { + self.calls().into_iter().map(|(method, _)| method).collect() + } + + fn calls(&self) -> Vec<(String, Value)> { + self.seen.lock().expect("poisoned").clone() + } + } +} + #[cfg(test)] mod tests { use super::*; @@ -806,6 +1339,7 @@ mod tests { qemu_uid: 1000, filtered: true, queues: 0, + workdir: String::new(), }; let filter = NetworkFilterConfig { mode: crate::config::NetworkFilterMode::Libvirt, @@ -857,6 +1391,7 @@ mod tests { qemu_uid: 1000, filtered: false, queues: 4, + workdir: String::new(), }; let filtering = NetworkFilterConfig { mode: crate::config::NetworkFilterMode::Libvirt, @@ -907,6 +1442,7 @@ mod tests { qemu_uid: 1000, mode: "bridge".into(), queues: 4, + workdir: String::new(), }; let error = match prepare_macvtap("test:///default", &request, &filtering) { Err(error) => error, @@ -928,6 +1464,7 @@ mod tests { qemu_uid: 1000, filtered: true, queues: 1, + workdir: String::new(), }; // Nothing on the wire can name a filter: the field does not exist. let wire = serde_json::to_value(&request).unwrap(); @@ -963,6 +1500,269 @@ mod tests { assert!(started.elapsed() < Duration::from_secs(2)); } + /// A whole-VM sweep names a VM and no NIC: reaching the indices the caller + /// can no longer name is the point, so netd derives the whole space. That + /// space is bounded by what an identity may say, which is what makes + /// deriving it cheap enough to do on every launch. + #[test] + fn a_whole_vm_sweep_covers_a_bounded_space() { + let value = serde_json::to_value(VmRef { + instance_id: "instance".into(), + vm_id: "vm".into(), + }) + .unwrap(); + assert!(value.get("nic_index").is_none()); + + let mut identity = identity("instance", "vm", MAX_NIC_INDEX); + assert!(validate_identity(&identity).is_ok()); + identity.nic_index = MAX_NIC_INDEX + 1; + assert!(validate_identity(&identity).is_err()); + } + + /// The record is a hint; the name is the proof. Everything that can go + /// wrong with reading a string off an interface -- forged, truncated, + /// ambiguous, absent -- has to land in the same bucket, and it has to be + /// the bucket a collection treats conservatively. + #[test] + fn an_interface_says_whose_it_is_and_the_name_is_what_proves_it() { + let nic = identity("path-abc", "vm-1", 3); + let tap = tap_name(&nic); + let alias = interface_alias(&nic); + assert_eq!(alias, "dstack1:3:path-abc:vm-1"); + + let owner = owner_of(&tap, &alias).expect("its own record checks out"); + assert_eq!(owner.instance_id, "path-abc"); + assert_eq!(owner.vm_id, "vm-1"); + assert_eq!(owner.nic_index, 3); + + // A record naming some other interface proves nothing about this one. + // This is what makes the record unforgeable without making it + // authoritative: anything that can reach the socket can write a + // string, but only the true identity re-derives the name. + let forged = interface_alias(&identity("path-abc", "someone-elses-vm", 3)); + assert!(owner_of(&tap, &forged).is_none()); + assert!(owner_of(&tap, "").is_none()); + assert!(owner_of(&tap, "dstack1:3:path-abc").is_none()); + assert!(owner_of(&tap, &alias[..alias.len() - 2]).is_none()); + // A format this build does not know is not this format. + assert!(owner_of(&tap, &alias.replace("dstack1", "dstack2")).is_none()); + + // The two free-form fields are last and only the first of them has to + // be unambiguous, so a VM ID carrying the separator still reads back. + let odd = identity("path-abc", "vm:with:colons", 0); + assert_eq!( + owner_of(&tap_name(&odd), &interface_alias(&odd)).map(|owner| owner.vm_id), + Some("vm:with:colons".to_string()) + ); + // An instance ID carrying it is refused instead of mis-parsed. + assert!(validate_identity(&identity("path:abc", "vm-1", 0)).is_err()); + } + + /// An identity that cannot be recorded would produce an interface nothing + /// can attribute, which is the state this whole path exists to stop + /// creating. Refusing it is the only answer that keeps the invariant. + #[test] + fn an_identity_too_long_to_record_is_refused() { + let long = "v".repeat(128); + assert!(validate_identity(&identity("instance", &long, 0)).is_ok()); + let identity_too_long = identity(&"i".repeat(128), &long, 255); + assert!(interface_alias(&identity_too_long).len() > MAX_IFALIAS); + let error = validate_identity(&identity_too_long) + .unwrap_err() + .to_string(); + assert!(error.contains("too long to record"), "{error}"); + } + + /// The name space netd claims. A collection deletes what matches, so what + /// matches has to be exactly what netd can produce. + #[test] + fn the_managed_name_space_is_exactly_what_netd_produces() { + assert!(is_managed_name(&tap_name(&identity("instance", "vm", 0)))); + assert!(is_managed_name("dt0123456789ab")); + assert!(!is_managed_name("dt0123456789AB"), "digests are lower case"); + assert!(!is_managed_name("dt0123456789a"), "one short"); + assert!(!is_managed_name("dt0123456789abc"), "one long"); + assert!(!is_managed_name("dtzzzzzzzzzzzz")); + assert!(!is_managed_name("virbr0")); + assert!(!is_managed_name("eth0")); + // The whole space fits in IFNAMSIZ, or the kernel would refuse the + // names this reserves. + assert!(tap_name(&identity("instance", "vm", 255)).len() < 16); + } + + /// The command prints a table for a human and accepts no options to make it + /// print anything else, so this parses one. Narrowing to netd's own name + /// space is what makes that safe. + #[test] + fn the_binding_listing_reads_a_table_meant_for_a_person() { + let output = "\ + Port Dev Filter +--------------------------------- + dt1e053266e9f7 clean-traffic + dt28b105b3031a clean-traffic + vnet3 some-other-filter +"; + let names: HashSet = output + .lines() + .filter_map(|line| line.split_whitespace().next()) + .filter(|name| is_managed_name(name)) + .map(str::to_string) + .collect(); + assert_eq!(names.len(), 2); + assert!(names.contains("dt1e053266e9f7")); + // The header, the rule, and a binding that is not netd's all fall out. + assert!(!names.contains("Port")); + assert!(!names.contains("vnet3")); + } + + /// Everything else here reasons about strings. This puts the reasoning + /// next to the kernel: that an alias survives on a device netd actually + /// creates, that enumeration finds it, that the guards refuse what they + /// are meant to, and that removal leaves nothing. + /// + /// Refuses to run in the host's network namespace, so it cannot touch a + /// real node's interfaces even when it fails. Unsharing one from inside + /// the test is not enough: `/sys/class/net` keeps showing the old + /// namespace until sysfs is remounted, which is most of what `ip netns + /// exec` does. So it asks to be put in one: + /// + /// ```text + /// cargo test -p dstack-vmm --bins --no-run + /// sudo ip netns add dstack-netd-test + /// sudo ip netns exec dstack-netd-test \ + /// target/debug/deps/dstack_vmm- --ignored --test-threads=1 + /// sudo ip netns del dstack-netd-test + /// ``` + #[test] + #[ignore = "needs root and its own network namespace; see the doc comment"] + fn a_real_interface_carries_its_record_and_removal_leaves_nothing() { + assert!( + nix::unistd::Uid::effective().is_root(), + "this test needs root" + ); + let (mine, init) = ( + std::fs::read_link("/proc/self/ns/net").unwrap(), + std::fs::read_link("/proc/1/ns/net").unwrap(), + ); + assert_ne!( + mine, init, + "run this inside its own network namespace; it creates and deletes interfaces" + ); + // Nothing in this namespace to talk to, which is also the state of a + // node that does not filter: the listing has to work without libvirt. + let uri = "qemu:///nonexistent-for-this-test"; + + let nic = identity("test-instance", "vm-1", 2); + let tap = tap_name(&nic); + ip(&["tuntap", "add", "dev", &tap, "mode", "tap"]).unwrap(); + set_alias(&tap, &nic).unwrap(); + assert!(is_tuntap(&tap), "the kernel publishes tun_flags for a TAP"); + + let records = list_interfaces(uri, ""); + let record = records + .iter() + .find(|record| record.tap == tap) + .expect("an interface netd created is one netd can find"); + assert_eq!(record.instance_id.as_deref(), Some("test-instance")); + assert_eq!(record.vm_id.as_deref(), Some("vm-1")); + assert_eq!(record.nic_index, Some(2)); + assert_eq!(record.kind, "tap"); + // Narrowing by instance is what keeps one VMM's collection off + // another's interfaces. + assert_eq!(list_interfaces(uri, "test-instance").len(), 1); + assert!(list_interfaces(uri, "someone-else").is_empty()); + + // A device with one of netd's names that netd did not create. The name + // is 48 bits of digest, so this is not about collisions -- it is that + // `ip link delete` does not ask what it is deleting, and netd runs as + // root. + let impostor = tap_name(&identity("test-instance", "not-a-tap", 0)); + ip(&["link", "add", &impostor, "type", "dummy"]).unwrap(); + assert!(is_managed_name(&impostor)); + assert!( + !list_interfaces(uri, "") + .iter() + .any(|record| record.tap == impostor), + "a device netd did not create is not offered up for collection" + ); + let refused = remove_interface(uri, &impostor, BindingCleanup::Skip).unwrap_err(); + assert!(refused.to_string().contains("refusing to delete")); + + // An interface whose record does not re-derive its own name proves + // nothing, and lands in the same bucket as no record at all. + ip(&[ + "link", + "set", + "dev", + &tap, + "alias", + "dstack1:2:test-instance:some-other-vm", + ]) + .unwrap(); + let records = list_interfaces(uri, ""); + let record = records.iter().find(|record| record.tap == tap).unwrap(); + assert!(record.instance_id.is_none(), "a forged record is no record"); + + remove_interface(uri, &tap, BindingCleanup::Skip).unwrap(); + assert!(!Path::new("/sys/class/net").join(&tap).exists()); + assert!(!list_interfaces(uri, "") + .iter() + .any(|record| record.tap == tap)); + // Removing what is not there is not an error: a sweep derives names + // and most of them miss. + remove_interface(uri, &tap, BindingCleanup::Skip).unwrap(); + } + + /// Run with the same isolated-network-namespace setup as the test above. + #[test] + #[ignore = "needs root and its own network namespace"] + fn sweeps_without_libvirt_follow_the_nodes_filter_policy() { + assert!(nix::unistd::Uid::effective().is_root()); + assert_ne!( + std::fs::read_link("/proc/self/ns/net").unwrap(), + std::fs::read_link("/proc/1/ns/net").unwrap(), + "run this inside its own network namespace", + ); + let uri = "qemu:///nonexistent-for-this-test"; + for requires_binding in [false, true] { + let nic = identity("sweep-test", "vm-1", 0); + let tap = tap_name(&nic); + ip(&["tuntap", "add", "dev", &tap, "mode", "tap"]).unwrap(); + let mut config = NetdConfig { + libvirt_uri: uri.into(), + ..Default::default() + }; + config.network_filter = Some(NetworkFilterConfig { + mode: if requires_binding { + crate::config::NetworkFilterMode::Libvirt + } else { + crate::config::NetworkFilterMode::None + }, + ..Default::default() + }); + let sweep = || { + sweep_vm_interfaces( + &config.libvirt_uri, + &nic.instance_id, + &nic.vm_id, + config.filter_policy().requires_binding(), + ) + }; + let result = sweep(); + // Even a binding failure must not leave the TAP on the bridge. + assert!(!Path::new("/sys/class/net").join(&tap).exists()); + if requires_binding { + assert!(result.is_err()); + // A retry cannot claim success just because the TAP is gone: + // its binding may still exist in the unreachable libvirt. + assert!(sweep().is_err()); + } else { + assert_eq!(result.unwrap(), 1); + assert_eq!(sweep().unwrap(), 0); + } + } + } + /// The VMM tells "netd is not running" apart from "netd refused". Both /// ways a Unix socket can have nobody behind it must read as the former, /// through the context the pRPC client stacks on the transport error. diff --git a/dstack/vmm/vmm.toml b/dstack/vmm/vmm.toml index 068cece16..019a5ea83 100644 --- a/dstack/vmm/vmm.toml +++ b/dstack/vmm/vmm.toml @@ -46,7 +46,11 @@ max_allocable_memory_in_mb = 100_000 # MB qmp_socket = false # The user to run the VM as. If empty, the VM will be run as the current user. # Unique namespace when multiple dstack-vmm instances share one host. When -# empty, a stable value is derived from run_path. +# empty, a stable value is derived from run_path. It is what netd records on +# every host interface this VMM asks for, and the name space those interfaces' +# names are derived in. Two VMMs on one host must not share one: each would +# build interfaces at names the other can also produce. Left empty it is +# derived from run_path, which cannot collide. May not contain ":". instance_id = "" # Network choices that deployment RPC callers may make. Macvtap is excluded # by default because libvirt nwfilter applies only to bridge interfaces. From 0606026c27746c2590de1f592db33691460cc036 Mon Sep 17 00:00:00 2001 From: Kevin Wang Date: Mon, 14 Sep 2026 08:33:43 -0700 Subject: [PATCH 02/93] fix(vmm): make netd interface cleanup converge without a restart - `.netd-pending` is the only record that netd may hold interfaces for a VM. The runtime snapshot no longer takes part in cleanup decisions, and a VM without the marker never contacts netd to release anything. - A removal retries the release with backoff inside its background task until netd answers, instead of returning and waiting for the next VMM start. The VM stays marked as being removed meanwhile, so the removal mark is handed to the task and no longer needs to be retained by hand. - The VM directory is deleted with `.removing` last, and `.removing` is written durably, so a removal interrupted by a crash is resumed. - A periodic reconciliation releases what stopped VMs still hold, which covers a stop that could not reach netd and a VM that exited on its own. - `create_vm` removes a VM whose first start failed through the normal removal path, so interfaces netd could not release keep their marker. --- docs/bridge-networking.md | 38 ++-- docs/libvirt-network-filter.md | 14 +- dstack/vmm/src/app.rs | 401 +++++++++++++++++---------------- dstack/vmm/src/app/workdir.rs | 59 ++++- dstack/vmm/src/main.rs | 14 +- dstack/vmm/src/main_service.rs | 28 +-- dstack/vmm/src/netd.rs | 14 +- 7 files changed, 327 insertions(+), 241 deletions(-) diff --git a/docs/bridge-networking.md b/docs/bridge-networking.md index 82c11867a..7007d4c86 100644 --- a/docs/bridge-networking.md +++ b/docs/bridge-networking.md @@ -270,25 +270,25 @@ sudo dstack-vmm netd remove-vm --instance path-3f9a1c8e7d2b4a60 --vm 0a1b2c3d4e5 ### When a release does not land -Every stop and every removal asks `netd` to sweep that VM's interfaces, by -deriving each of the 256 names its identity could produce. That needs no -record, and it reaches what a per-NIC teardown cannot: an interface a crash -left behind before anything on disk pointed at it, or one whose NIC the -manifest has since dropped. - -A removal deletes the VM's directory, and that directory — with its `.removing` -marker — is the only thing left that says to try again. So it is deleted only -once the sweep has landed. If `netd` refused, or was not there to ask, the -directory stays and the next VMM start resumes the removal; `RemoveVm` is -idempotent, so the retry costs one round trip. A VM that never asked `netd` for -an interface is unaffected: there is nothing for `netd` to be holding. - -The VMM persists `.netd-pending` before asking netd to prepare an interface and -clears it only after a successful whole-VM sweep. This cleanup marker survives -failed launches and network configuration changes, even if the runtime snapshot -is absent or replaced by a user-mode topology. Older snapshots are promoted to -the marker before cleanup or replacement. A failed cleanup during an update -also leaves the old snapshot intact. +The VMM writes `.netd-pending` into the VM's directory before it asks `netd` +to build anything, and deletes it only after `netd` has swept every interface +that VM could hold. The marker is the only record the VMM keeps of what `netd` +may hold, so it survives failed launches and changes to a topology that no +longer uses `netd`. A VM without it has nothing to release, and its stop, +removal and launch never contact `netd` for one. + +A sweep derives each of the 256 names the VM's identity could produce, so it +needs no record of which NICs existed: it also reaches an interface a crash left +behind, or one whose NIC the manifest has since dropped. It is idempotent, and +every path that releases a VM retries it until it lands: + +- a launch sweeps before it prepares; +- a stop that could not reach `netd` leaves the marker, and the VMM checks + stopped VMs with a marker every minute and sweeps them; +- a removal keeps the VM's directory, and the VM marked as being removed, and + retries the sweep with backoff until `netd` answers. The directory is deleted + after that, with `.removing` last, so a removal interrupted by a crash is + resumed by the next VMM start. On an unfiltered node, an unavailable `libvirtd` does not make an otherwise successful TAP sweep fail. Filtered nodes still require confirmation that their diff --git a/docs/libvirt-network-filter.md b/docs/libvirt-network-filter.md index 57ec85bd4..125f4b761 100644 --- a/docs/libvirt-network-filter.md +++ b/docs/libvirt-network-filter.md @@ -119,8 +119,10 @@ of, and that record is written *after* the interface exists — a VMM killed in between leaves a TAP nothing on disk points at, and a manifest that lost a NIC leaves the same thing behind. `RemoveVm` names a VM instead of an interface and derives every name that VM could occupy, so neither has to be recorded for -teardown to work. The VMM sweeps before preparing a launch as well as on stop, -which makes a launch self-healing regardless of what the record says. +teardown to work. The VMM keeps a single `.netd-pending` marker per VM from +before its first prepare until a sweep succeeds, and sweeps whenever that marker +is present: before a launch, on stop, on removal, and periodically for stopped +VMs. See [bridge-networking.md](bridge-networking.md#when-a-release-does-not-land). A prepare also carries `workdir`, which `netd` does not need to build the TAP. It names the VM's directory on the host: untrusted, never read for a decision, @@ -217,10 +219,10 @@ sudo dstack-vmm --config ./vmm.toml \ ``` User networking and a caller-supplied netdev never ask `netd` to build an -interface. The VMM still contacts the socket for such a VM -- every launch and -every stop releases whatever the VM held, before it decides whether it needs -anything built -- but nothing about the VM depends on the answer. Bridge and -macvtap do ask, and fail closed if `netd` is unavailable. +interface, and a VM that has never used `netd` never contacts it. A VM that +moved from bridge or macvtap to user networking still releases what it held, +but nothing about its launch depends on the answer. Bridge and macvtap do ask, +and fail closed if `netd` is unavailable. Filtered TAP netdevs follow the node's `vhost` and `queues` settings like any other TAP-backed NIC (see [network-data-plane.md](network-data-plane.md)). The diff --git a/dstack/vmm/src/app.rs b/dstack/vmm/src/app.rs index 4163876b5..fd63a3083 100644 --- a/dstack/vmm/src/app.rs +++ b/dstack/vmm/src/app.rs @@ -300,6 +300,14 @@ pub(crate) enum PullStatus { Failed(String), } +/// First delay before a removal asks netd again to release a VM's interfaces. +#[cfg(not(test))] +const RELEASE_RETRY_INITIAL: Duration = Duration::from_secs(2); +#[cfg(test)] +const RELEASE_RETRY_INITIAL: Duration = Duration::from_millis(20); +/// Longest delay between those attempts. +const RELEASE_RETRY_MAX: Duration = Duration::from_secs(60); + #[derive(Clone)] pub struct App { pub config: Arc, @@ -598,11 +606,9 @@ impl App { self.set_started(id, false)?; self.stop_vm_process(id).await?; // Not fallible: a VM that has been asked to stop is stopped whether or - // not netd could be reached. What is left behind is reclaimed by this - // VM's next launch, which releases before it prepares, or by its - // removal, which will not finish until the release lands. A VM that is - // never started or removed again keeps its interfaces, and its - // directory is still there to say whose they are. + // not netd could be reached. What is left behind keeps its marker and + // is released by `reconcile_network_cleanup`, the next launch, or the + // removal, whichever comes first. self.release_vm_interfaces(id).await; Ok(()) } @@ -628,27 +634,15 @@ impl App { ); } - // Whatever an earlier boot left behind: a crash between creating an - // interface and recording it, a NIC this VM no longer has, or a whole - // backend it no longer uses. Prepare replaces the names it is about to - // use, but only those, so an index nothing will claim again is only + // Whatever an earlier boot left behind: a failed stop, a crash between + // creating an interface and using it, a NIC this VM no longer has, or + // a backend it no longer uses. Prepare replaces the names it is about + // to use, but only those, so an index nothing will claim again is only // reachable from here. // - // Before the early return, and not inside it. A VM that has moved from - // a bridge to user-mode networking needs this release precisely because - // it no longer wants an interface, and gating it on wanting one is how - // the interfaces it left behind would become unreachable to every - // later launch. + // Before the early return: a VM that moved from a bridge to user-mode + // networking still has to give back what it held. let vm_workdir = self.work_dir(&vm.manifest.id)?; - if vm_workdir - .runtime_networks() - .iter() - .any(needs_netd_interface) - { - // Fail before a user-mode boot can overwrite a legacy snapshot - // when its replacement cleanup marker cannot be persisted. - vm_workdir.mark_network_cleanup_pending()?; - } self.release_vm_interfaces(&vm.manifest.id).await; if !networks.iter().any(needs_netd_interface) { return Ok(()); @@ -838,13 +832,12 @@ impl App { /// Releases every host interface netd holds for this VM. /// - /// Unconditional and non-fatal, which is one decision made twice. The - /// release path must not be gated on a predicate that can change under it: - /// `needs_netd_interface` reads the VM's *current* backend, and a VM whose - /// NIC was a bridge when its TAP was built and is a user-mode NIC now would - /// skip the release for interfaces that exist. And a VM must be able to - /// stop when the daemon holding its interfaces cannot be reached, or a - /// netd outage becomes a fleet that cannot be stopped. + /// Gated on `.netd-pending` alone, never on the VM's current networking: + /// a VM whose NIC was a bridge when its TAP was built and is a user-mode + /// NIC now still holds that TAP. Non-fatal, because a VM must be able to + /// stop when netd cannot be reached; what a failed release leaves behind + /// keeps its marker, and [`App::reconcile_network_cleanup`], the next + /// launch or the removal retries it. /// /// Returns whether netd is known to hold nothing for this VM any more. /// A removal reads that to decide whether it may delete the workdir: the @@ -858,14 +851,8 @@ impl App { return false; } }; - // Upgrade snapshots written before the marker existed. A failed sweep - // must remain recoverable even if the next boot replaces the snapshot - // with user-mode NICs or an update clears it. - if workdir.runtime_networks().iter().any(needs_netd_interface) { - if let Err(error) = workdir.mark_network_cleanup_pending() { - warn!(vm_id, %error, "cannot persist pending network cleanup"); - return false; - } + if !workdir.network_cleanup_pending() { + return true; } // Ask for the release, rather than asking whether it can be asked for. // A probe first would put a second round trip in front of every stop @@ -949,27 +936,19 @@ impl App { } } - // Clear the in-memory mark if anything below fails before ownership of - // the removal is handed to the background task. In particular, a - // removal that cannot persist its crash-recovery marker must not start: - // without that marker a netd outage could leave interfaces no later - // VMM start knows to release. - let mut mark = RemovalMark::new(self.clone(), id); - - // Persist the removing marker so crash recovery can resume - let work_dir = self.work_dir(id)?; - work_dir + // Clears the in-memory mark if the removal cannot start. In particular, + // a removal that cannot persist its crash-recovery marker must not + // start: a crash would then forget it. + let mark = RemovalMark::new(self.clone(), id); + self.work_dir(id)? .set_removing() .with_context(|| format!("failed to write .removing marker for {id}"))?; // User-initiated removal always deletes the workdir let app = self.clone(); - let id = id.to_string(); - // `finish_remove_vm` owns the mark from here. Do not clear it in the - // request task while the background removal is still running. - mark.retain(); tokio::spawn(async move { - if let Err(err) = app.finish_remove_vm(&id, true).await { + let id = mark.id.clone(); + if let Err(err) = app.finish_remove_vm(mark, true).await { error!("Background cleanup failed for {id}: {err:?}"); } }); @@ -981,11 +960,11 @@ impl App { /// remove from supervisor, optionally delete workdir, and free CID. /// /// `delete_workdir`: true for user-initiated removal, false for orphan cleanup. - async fn finish_remove_vm(&self, id: &str, delete_workdir: bool) -> Result<()> { - // Ordinary exits clear the mark, including the `?`s below and a panic - // in this task, which `tokio::spawn` would otherwise swallow. The one - // recoverable incomplete outcome explicitly retains it. - let mut mark = RemovalMark::new(self.clone(), id); + /// + /// Owns the removal mark, which is cleared however this ends, including the + /// `?`s below and a panic in the task. + async fn finish_remove_vm(&self, mark: RemovalMark, delete_workdir: bool) -> Result<()> { + let id = mark.id.as_str(); // Held across the stop, the wait and the release, not just the release. // `removing` turns launches away, but a launch that passed that check // before the marker was set is already inside the lock: it has not @@ -1037,39 +1016,33 @@ impl App { } let vm_path = self.work_dir(id)?; - // Read before the release, because the release is what makes it stale. - // A VM that never asked netd for an interface -- user mode, a custom - // netdev, or one that never launched -- has nothing for netd to be - // holding, so an absent netd is not a reason to keep its directory. - let held_interfaces = vm_path.network_cleanup_pending() - || vm_path.runtime_networks().iter().any(needs_netd_interface); - let released = self.release_vm_interfaces(id).await; - // Only delete the workdir for user-initiated removal or if .removing marker exists. // Orphaned supervisor processes without the marker keep their data intact. if !(delete_workdir || vm_path.is_removing()) { + self.release_vm_interfaces(id).await; if vm_path.path().exists() { info!( "VM {id} workdir preserved (orphan cleanup): {}", vm_path.path().display() ); } - } else if held_interfaces && !released { - // The `.removing` marker and the directory are what a later boot - // reads to retry this, and `remove_all` is idempotent, so keeping - // them costs one retry and losing them strands every interface - // netd still holds: nothing else on the host can name them. - warn!( - "VM {id} keeps its directory because netd did not release its interfaces; \ - the removal resumes at the next VMM start" - ); - // The disk marker says this removal must be retried. Keep the - // matching in-memory state too, so start/update/stop cannot revive - // the VM before that retry happens. - mark.retain(); - return Ok(()); - } else if vm_path.path().exists() { - if let Err(err) = fs::remove_dir_all(&vm_path) { + } else { + // The directory holds `.netd-pending`, the only record of what netd + // may still hold for this VM, so it goes only once the release has + // landed. The release is idempotent: retry until it does. The VM + // stays marked as being removed meanwhile, so nothing can start it, + // and a repeated removal request is already in progress. + let mut delay = RELEASE_RETRY_INITIAL; + while !self.release_vm_interfaces(id).await { + warn!( + "VM {id} is waiting for netd to release its interfaces; retrying in {delay:?}" + ); + tokio::time::sleep(delay).await; + delay = (delay * 2).min(RELEASE_RETRY_MAX); + } + if let Err(err) = vm_path.remove_all() { + // `.removing` is deleted last, so what is left is resumed by + // the next reload. error!("failed to remove VM directory for {id}: {err:?}"); } } @@ -1099,11 +1072,12 @@ impl App { return false; } } + let mark = RemovalMark::new(self.clone(), id); let app = self.clone(); - let id = id.to_string(); tokio::spawn(async move { + let id = mark.id.clone(); // Don't pass delete_workdir=true; rely on .removing marker check inside - if let Err(err) = app.finish_remove_vm(&id, false).await { + if let Err(err) = app.finish_remove_vm(mark, false).await { error!("Background cleanup failed for {id}: {err:?}"); } }); @@ -1676,6 +1650,45 @@ impl App { Ok(()) } + /// Releases what netd still holds for VMs that are not running. + /// + /// A stop whose release failed, or a VM that exited on its own, leaves + /// `.netd-pending` behind with nothing scheduled to act on it. Checking + /// here makes that state converge without waiting for the VM to be + /// started or removed. Costs a file check per VM when nothing is pending. + pub(crate) async fn reconcile_network_cleanup(&self) { + let ids: Vec = self.lock().vms.keys().cloned().collect(); + for id in ids { + self.reconcile_vm_network_cleanup(&id).await; + } + } + + async fn reconcile_vm_network_cleanup(&self, id: &str) { + let pending = self + .work_dir(id) + .is_ok_and(|workdir| workdir.network_cleanup_pending()); + // A removal releases the VM itself, and holds the launch lock for as + // long as that takes. + if !pending || self.refuse_if_removing(id).is_err() { + return; + } + // A launch holds this lock until QEMU is deployed, so the running + // check below cannot see a launch half done. + let _launch = self.launch_lock(id).await; + if self.refuse_if_removing(id).is_err() { + return; + } + match self.supervisor.info(id).await { + Ok(Some(info)) if info.state.status.is_running() => return, + Ok(_) => {} + Err(error) => { + debug!(id, "skipping network cleanup: {error:#}"); + return; + } + } + self.release_vm_interfaces(id).await; + } + pub(crate) async fn try_restart_exited_vms(&self) -> Result<()> { let running_vms = self .supervisor @@ -2237,71 +2250,76 @@ mod tests { (SupervisorClient::new(&format!("http://{address}")), task) } + /// A VM directory holding `.netd-pending`, as a launch leaves it. + fn pending_workdir(app: &App, id: &str) -> VmWorkDir { + let workdir = app.work_dir(id).unwrap(); + std::fs::create_dir_all(workdir.path()).unwrap(); + workdir.mark_network_cleanup_pending().unwrap(); + workdir + } + + /// A removal keeps retrying while netd is away, keeps the VM marked as + /// being removed so nothing can start it, and finishes on its own once netd + /// comes back -- without a VMM restart or another request. #[tokio::test] - async fn failed_release_survives_snapshot_replacement_and_removal_retry() { + async fn a_removal_retries_until_netd_releases_and_then_finishes() { let (supervisor, server) = stopped_supervisor().await; let dir = tempfile::tempdir().unwrap(); - let app = App::new( - test_config(&dir.path().join("absent-netd.sock"), dir.path()), - supervisor.clone(), - ); - let workdir = app.work_dir("vm-1").unwrap(); - std::fs::create_dir_all(workdir.path()).unwrap(); - // A legacy launch only left the runtime snapshot, without a marker. - let network = Networking { - nic: NicNetworking { - mode: NetworkingMode::Bridge, - ..Default::default() - }, - ..Default::default() - }; - workdir.set_runtime_networks(&[network]).unwrap(); - assert!(!workdir.network_cleanup_pending()); - assert!(!app.release_vm_interfaces("vm-1").await); - assert!(workdir.network_cleanup_pending()); - // A subsequent user-mode boot can replace the snapshot. Cleanup must - // still remember the bridge interfaces from the earlier boot. - workdir - .set_runtime_networks(&[Networking::default()]) - .unwrap(); + let socket = dir.path().join("netd.sock"); + let app = App::new(test_config(&socket, dir.path()), supervisor); + let workdir = pending_workdir(&app, "vm-1"); workdir.set_removing().unwrap(); + assert!(app.lock().start_removing("vm-1")); - app.finish_remove_vm("vm-1", true).await.unwrap(); + let mark = RemovalMark::new(app.clone(), "vm-1"); + let removal = tokio::spawn({ + let app = app.clone(); + async move { app.finish_remove_vm(mark, true).await } + }); + + tokio::time::sleep(Duration::from_millis(200)).await; + assert!( + !removal.is_finished(), + "nothing released, so nothing removed" + ); assert!(workdir.path().exists()); - assert!(workdir.is_removing()); assert!(workdir.network_cleanup_pending()); assert!(app.refuse_if_removing("vm-1").is_err()); + // Asking again while it retries is the same removal. + assert!(!app.lock().start_removing("vm-1")); - // Simulate restarting the VMM after netd has recovered. - let netd = netd::testing::FakeNetd::spawn(&["RemoveVm"]); - let restarted = App::new(test_config(netd.socket(), dir.path()), supervisor); - restarted.finish_remove_vm("vm-1", true).await.unwrap(); + let netd = netd::testing::FakeNetd::spawn_at(&socket, &["RemoveVm"]); + tokio::time::timeout(Duration::from_secs(10), removal) + .await + .expect("the removal converges once netd answers") + .unwrap() + .unwrap(); assert!(!workdir.path().exists()); - assert_eq!(netd.methods(), vec!["RemoveVm"]); + assert!(app.refuse_if_removing("vm-1").is_ok()); + assert!(netd.methods().iter().all(|method| method == "RemoveVm")); server.abort(); } + /// A VM that never asked netd for an interface has nothing to wait for: + /// its removal completes with netd absent. #[tokio::test] - async fn removal_without_a_snapshot_distinguishes_pending_from_never_prepared() { + async fn a_removal_with_nothing_pending_does_not_wait_for_netd() { let (supervisor, server) = stopped_supervisor().await; let dir = tempfile::tempdir().unwrap(); let app = App::new( test_config(&dir.path().join("absent-netd.sock"), dir.path()), supervisor, ); - for (id, pending) in [("interrupted-prepare", true), ("user-only", false)] { - let workdir = app.work_dir(id).unwrap(); - std::fs::create_dir_all(workdir.path()).unwrap(); - if pending { - // Preparation wrote this before contacting netd, then crashed - // without ever writing runtime-networks.json. - workdir.mark_network_cleanup_pending().unwrap(); - } - workdir.set_removing().unwrap(); - app.finish_remove_vm(id, true).await.unwrap(); - assert_eq!(workdir.path().exists(), pending); - assert_eq!(workdir.is_removing(), pending); - } + let workdir = app.work_dir("user-only").unwrap(); + std::fs::create_dir_all(workdir.path()).unwrap(); + workdir.set_removing().unwrap(); + assert!(app.lock().start_removing("user-only")); + let mark = RemovalMark::new(app.clone(), "user-only"); + tokio::time::timeout(Duration::from_secs(10), app.finish_remove_vm(mark, true)) + .await + .unwrap() + .unwrap(); + assert!(!workdir.path().exists()); server.abort(); } @@ -2313,21 +2331,35 @@ mod tests { test_config(netd.socket(), dir.path()), SupervisorClient::new("http://127.0.0.1:0"), ); - let workdir = app.work_dir("vm-1").unwrap(); - std::fs::create_dir_all(workdir.path()).unwrap(); - workdir.mark_network_cleanup_pending().unwrap(); + let workdir = pending_workdir(&app, "vm-1"); assert!(app.release_vm_interfaces("vm-1").await); assert!(!workdir.network_cleanup_pending()); assert!(workdir.path().exists()); } + /// The marker is the only thing that says netd may hold something. Without + /// it there is nothing to release and nobody to ask, so a user-mode VM on a + /// node with no netd never waits on one. + #[tokio::test] + async fn a_vm_with_nothing_pending_never_asks_netd() { + let netd = netd::testing::FakeNetd::spawn(&["RemoveVm"]); + let app = app_talking_to(netd.socket()); + assert!(app.release_vm_interfaces("vm-1").await); + assert!(netd.methods().is_empty()); + } + /// A netd outage must not become a fleet that cannot be stopped. #[tokio::test] async fn a_stop_survives_a_netd_that_is_not_there() { - let app = app_talking_to(Path::new("/nonexistent/dstack-netd.sock")); - // Returns rather than propagating: there is no error type here on - // purpose, because there is no caller that should act on one. - app.release_vm_interfaces("vm-1").await; + let dir = tempfile::tempdir().unwrap(); + let app = App::new( + test_config(Path::new("/nonexistent/dstack-netd.sock"), dir.path()), + SupervisorClient::new("http://127.0.0.1:0"), + ); + let workdir = pending_workdir(&app, "vm-1"); + // Returns rather than propagating, and keeps the marker for a retry. + assert!(!app.release_vm_interfaces("vm-1").await); + assert!(workdir.network_cleanup_pending()); } /// A removal deletes the workdir, and the workdir is the only thing left @@ -2336,36 +2368,13 @@ mod tests { /// netd that is not there to ask -- means it may still. #[tokio::test] async fn a_release_says_whether_netd_still_holds_anything() { - let netd = netd::testing::FakeNetd::spawn(&["RemoveVm"]); - let app = app_talking_to(netd.socket()); - assert!(app.release_vm_interfaces("vm-1").await); - - // Refused: netd is up and still holding whatever it had. - let refusing = netd::testing::FakeNetd::spawn(&[]); - let app = app_talking_to(refusing.socket()); - assert!(!app.release_vm_interfaces("vm-1").await); - - // Not there to ask. A VM that never asked netd for an interface is - // unaffected -- the removal checks that separately -- but one that did - // must keep its directory so a later start can try again. - let app = app_talking_to(Path::new("/nonexistent/dstack-netd.sock")); - assert!(!app.release_vm_interfaces("vm-1").await); - } - - /// A refused sweep is not a reason to fail the stop. What netd holds stays - /// until this VM launches again or until an operator names it, but the - /// stop still succeeds: a netd outage must not become a fleet that cannot - /// be stopped. - #[tokio::test] - async fn a_netd_that_refuses_the_sweep_does_not_fail_the_stop() { - let netd = netd::testing::FakeNetd::spawn(&[]); - let app = app_talking_to(netd.socket()); - app.release_vm_interfaces("vm-1").await; - assert_eq!( - netd.methods(), - vec!["RemoveVm"], - "asked once, and not asked about afterwards" - ); + for (handles, released) in [(&["RemoveVm"][..], true), (&[][..], false)] { + let netd = netd::testing::FakeNetd::spawn(handles); + let app = app_talking_to(netd.socket()); + let workdir = pending_workdir(&app, "vm-1"); + assert_eq!(app.release_vm_interfaces("vm-1").await, released); + assert_eq!(workdir.network_cleanup_pending(), !released); + } } /// One request, no question in front of it. A probe on the hot path is a @@ -2373,9 +2382,10 @@ mod tests { /// answer it reads as an absent one, and an absent netd's release is /// skipped. #[tokio::test] - async fn a_stop_asks_netd_to_sweep_without_a_question_first() { + async fn a_release_asks_netd_to_sweep_without_a_question_first() { let netd = netd::testing::FakeNetd::spawn(&["RemoveVm"]); let app = app_talking_to(netd.socket()); + pending_workdir(&app, "vm-1"); app.release_vm_interfaces("vm-1").await; assert_eq!(netd.methods(), vec!["RemoveVm"]); @@ -2387,6 +2397,37 @@ mod tests { assert!(sweep.get("nic_index").is_none()); } + /// A stop whose release failed leaves the marker with nothing scheduled to + /// act on it. Reconciliation releases it once netd answers, and leaves a + /// VM that is being removed to its removal. + #[tokio::test] + async fn reconciliation_releases_what_a_stopped_vm_still_holds() { + let (supervisor, server) = stopped_supervisor().await; + let netd = netd::testing::FakeNetd::spawn(&["RemoveVm"]); + let app = App::new( + test_config(netd.socket(), netd.socket().parent().unwrap()), + supervisor, + ); + let workdir = pending_workdir(&app, "vm-1"); + + assert!(app.lock().start_removing("vm-1")); + app.reconcile_vm_network_cleanup("vm-1").await; + assert!( + netd.methods().is_empty(), + "a removal owns this VM's release" + ); + drop(RemovalMark::new(app.clone(), "vm-1")); + + app.reconcile_vm_network_cleanup("vm-1").await; + assert_eq!(netd.methods(), vec!["RemoveVm"]); + assert!(!workdir.network_cleanup_pending()); + + // Nothing pending: nothing to ask. + app.reconcile_vm_network_cleanup("vm-1").await; + assert_eq!(netd.methods(), vec!["RemoveVm"]); + server.abort(); + } + /// The orphan cleanup runs for an ID that never loaded, so a guard living /// in the VM entry is not there when it holds the launch lock across the /// whole teardown. Without the mark, `StartVm` on that ID waits the @@ -2420,22 +2461,6 @@ mod tests { ); } - /// Once a removal deliberately keeps its crash-recovery marker, dropping - /// the guard must not make the VM launchable again in the current process. - #[tokio::test] - async fn a_removal_waiting_for_netd_stays_marked() { - let app = test_app(); - assert!(app.lock().start_removing("vm-1")); - { - let mut mark = RemovalMark::new(app.clone(), "vm-1"); - mark.retain(); - } - assert!( - app.refuse_if_removing("vm-1").is_err(), - "the in-memory state must agree with the retained disk marker" - ); - } - /// A restart decided before a stop must not outlive it. The restart task /// reads the started flag off disk and only then queues a launch, which /// waits for the lock the stop is holding; without a re-read under that @@ -3650,7 +3675,6 @@ impl AppState { struct RemovalMark { app: App, id: String, - clear_on_drop: bool, } impl RemovalMark { @@ -3658,21 +3682,12 @@ impl RemovalMark { Self { app, id: id.to_string(), - clear_on_drop: true, } } - - /// Leave the removal mark in place after this guard goes out of scope. - fn retain(&mut self) { - self.clear_on_drop = false; - } } impl Drop for RemovalMark { fn drop(&mut self) { - if !self.clear_on_drop { - return; - } let mut state = self.app.lock(); state.removing.remove(&self.id); if let Some(vm) = state.vms.get_mut(&self.id) { diff --git a/dstack/vmm/src/app/workdir.rs b/dstack/vmm/src/app/workdir.rs index 5809f73ba..55c070873 100644 --- a/dstack/vmm/src/app/workdir.rs +++ b/dstack/vmm/src/app/workdir.rs @@ -143,9 +143,13 @@ impl VmWorkDir { self.workdir.join("runtime-networks.json") } - /// Written before asking netd to create anything, and removed only after - /// a successful whole-VM sweep. Unlike the runtime snapshot, this survives - /// failed launches and changes to a topology that no longer uses netd. + /// The only record that netd may hold host interfaces for this VM. + /// + /// Written durably before asking netd to create anything, and removed only + /// after a successful whole-VM sweep, so it is present whenever netd holds + /// something. The reverse need not hold: host interfaces do not survive a + /// reboot while this file does, and a marker with nothing behind it costs + /// one sweep. pub fn mark_network_cleanup_pending(&self) -> Result<()> { safe_write::safe_write(self.workdir.join(".netd-pending"), b"") .context("failed to persist pending network cleanup") @@ -232,7 +236,40 @@ impl VmWorkDir { } pub fn set_removing(&self) -> Result<()> { - fs::write(self.removing_marker(), "").context("failed to write .removing marker") + // Durable: this is what makes a removal resume after a crash. + safe_write::safe_write(self.removing_marker(), b"") + .context("failed to write .removing marker") + } + + /// Deletes the directory, `.removing` last. + /// + /// A deletion interrupted part-way leaves the marker behind, so the next + /// VMM start finishes the removal instead of loading what is left as a VM. + pub fn remove_all(&self) -> Result<()> { + let marker = self.removing_marker(); + let entries = match fs::read_dir(&self.workdir) { + Ok(entries) => entries, + Err(error) if error.kind() == std::io::ErrorKind::NotFound => return Ok(()), + Err(error) => return Err(error.into()), + }; + for entry in entries { + let path = entry?.path(); + if path == marker { + continue; + } + if path.symlink_metadata()?.is_dir() { + fs::remove_dir_all(&path)?; + } else { + fs::remove_file(&path)?; + } + } + match fs::remove_file(&marker) { + Ok(()) => {} + Err(error) if error.kind() == std::io::ErrorKind::NotFound => {} + Err(error) => return Err(error.into()), + } + fs::remove_dir(&self.workdir)?; + Ok(()) } pub fn path(&self) -> &Path { @@ -329,4 +366,18 @@ mod tests { fs::remove_dir_all(temp)?; Ok(()) } + + /// `.removing` goes last, so an interrupted deletion is still a removal. + #[test] + fn remove_all_deletes_everything_and_tolerates_a_missing_directory() { + let dir = tempfile::tempdir().unwrap(); + let workdir = VmWorkDir::new(dir.path().join("vm")); + std::fs::create_dir_all(workdir.path().join("shared")).unwrap(); + std::fs::write(workdir.path().join("shared/file"), b"x").unwrap(); + workdir.mark_network_cleanup_pending().unwrap(); + workdir.set_removing().unwrap(); + workdir.remove_all().unwrap(); + assert!(!workdir.path().exists()); + workdir.remove_all().unwrap(); + } } diff --git a/dstack/vmm/src/main.rs b/dstack/vmm/src/main.rs index c2c8e2aaf..e09c1e42d 100644 --- a/dstack/vmm/src/main.rs +++ b/dstack/vmm/src/main.rs @@ -107,9 +107,8 @@ enum NetdCommand { /// Delete every interface netd holds for one VM. /// /// For a VM whose VMM will never ask again -- one whose directory was - /// deleted by hand, or whose instance is gone. A VMM sweeps its own VMs on - /// every stop and every removal, and keeps a removal pending until the - /// sweep lands; this is for when no VMM will ever run that sweep. + /// deleted by hand, or whose instance is gone. A VMM retries the sweep for + /// its own VMs until it lands; this is for when no VMM will ever run it. RemoveVm { /// The `cvm.instance_id` of the VMM that created them. `netd list` /// shows it. @@ -218,6 +217,14 @@ async fn auto_restart_task(app: App) { } } +async fn network_cleanup_task(app: App) { + let mut interval = tokio::time::interval(Duration::from_secs(60)); + loop { + interval.tick().await; + app.reconcile_network_cleanup().await; + } +} + async fn log_rotation_task(app: App) { if app.config.cvm.log.max_bytes == 0 { info!("Log rotation is disabled"); @@ -469,6 +476,7 @@ async fn main() -> Result<()> { state.reload_vms().await.context("Failed to reload VMs")?; tokio::spawn(auto_restart_task(state.clone())); tokio::spawn(log_rotation_task(state.clone())); + tokio::spawn(network_cleanup_task(state.clone())); tokio::select! { result = run_external_api(state.clone(), figment.clone(), api_auth) => { diff --git a/dstack/vmm/src/main_service.rs b/dstack/vmm/src/main_service.rs index ea4166dcc..9082aaa1b 100644 --- a/dstack/vmm/src/main_service.rs +++ b/dstack/vmm/src/main_service.rs @@ -849,27 +849,29 @@ impl VmmRpc for RpcHandler { warn!("Failed to set started: {}", err); } - let result = self + if let Err(err) = self .app .load_vm(&work_dir, &Default::default(), false) .await - .context("Failed to load VM"); - let result = match result { - Ok(()) => { - if !request.stopped { - self.app.start_vm(&id).await - } else { - Ok(()) - } - } - Err(err) => Err(err), - }; - if let Err(err) = result { + .context("Failed to load VM") + { + // Never started, so netd holds nothing for it. if let Err(err) = fs::remove_dir_all(&work_dir) { warn!("Failed to remove work dir: {}", err); } return Err(err); } + if !request.stopped { + if let Err(err) = self.app.start_vm(&id).await { + // A failed start may leave interfaces netd could not release. + // The normal removal keeps the directory, and with it + // `.netd-pending`, until they are gone. + if let Err(remove_err) = self.app.remove_vm(&id).await { + warn!(vm_id = %id, "failed to remove VM after start failure: {remove_err:#}"); + } + return Err(err); + } + } Ok(Id { id }) } diff --git a/dstack/vmm/src/netd.rs b/dstack/vmm/src/netd.rs index 5812edf6c..39c03d350 100644 --- a/dstack/vmm/src/netd.rs +++ b/dstack/vmm/src/netd.rs @@ -1250,7 +1250,7 @@ pub(crate) mod testing { } pub(crate) struct FakeNetd { - _dir: tempfile::TempDir, + _dir: Option, socket: PathBuf, seen: Arc>>, } @@ -1261,7 +1261,15 @@ pub(crate) mod testing { /// connect at once. pub(crate) fn spawn(handles: &[&str]) -> Self { let dir = tempfile::tempdir().expect("tempdir"); - let socket = dir.path().join("netd.sock"); + let mut netd = Self::spawn_at(&dir.path().join("netd.sock"), handles); + netd._dir = Some(dir); + netd + } + + /// As [`FakeNetd::spawn`], on a socket path the caller chose -- for a + /// netd that comes up after the VMM already started asking for it. + pub(crate) fn spawn_at(socket: &Path, handles: &[&str]) -> Self { + let socket = socket.to_path_buf(); let listener = UnixListener::bind(&socket).expect("bind"); let state = FakeState { handles: Arc::new(handles.iter().map(|name| name.to_string()).collect()), @@ -1279,7 +1287,7 @@ pub(crate) mod testing { let _ = ignite.launch_on(NetdListener(listener)).await; }); Self { - _dir: dir, + _dir: None, socket, seen, } From 2f4414df448d539f6f53cb26fd97faa47660b857 Mon Sep 17 00:00:00 2001 From: Kevin Wang Date: Mon, 14 Sep 2026 18:34:39 -0700 Subject: [PATCH 03/93] fix(vmm): don't strand VMs on netd-less nodes and confirm manual binding removal - A launch whose first netd request never reached a netd clears the `.netd-pending` marker it just wrote, unless an earlier boot still left one. Without this, a bridge deployment on a node without netd left the VM stuck in removal, holding its CID, until netd was installed. - `RemoveInterfaceByName` reports failure when a binding that exists, or is all the name could still hold, was not deleted, following the node's filter policy like the sweep. It previously printed "removed" regardless. - Listing nwfilter bindings uses a 10s timeout, and a sweep whose listing timed out skips binding deletion instead of waiting on the same hung libvirtd again, so a sweep fits inside the VMM's request timeout. - Reconciliation skips running VMs from one supervisor listing instead of taking every running VM's launch lock each pass, and does not burst after a slow pass. - Drop comments that described the removed legacy marker backfill. --- dstack/vmm/src/app.rs | 138 +++++++++++++++++++++++++++---- dstack/vmm/src/main.rs | 2 + dstack/vmm/src/netd.rs | 183 ++++++++++++++++++++++++++++++++++------- 3 files changed, 278 insertions(+), 45 deletions(-) diff --git a/dstack/vmm/src/app.rs b/dstack/vmm/src/app.rs index fd63a3083..6736a363c 100644 --- a/dstack/vmm/src/app.rs +++ b/dstack/vmm/src/app.rs @@ -535,8 +535,6 @@ impl App { }) .await .context("GPU sanitization task failed")??; - // Keep any earlier snapshot on failure: preparation may not have - // been able to persist the cleanup marker for legacy NICs yet. self.prepare_netd_networks(&vm_config, &mut runtime_networks) .await?; let processes = match vm_config.config_qemu( @@ -647,6 +645,9 @@ impl App { if !networks.iter().any(needs_netd_interface) { return Ok(()); } + // Still set only if the release above did not land, in which case netd + // may hold something from before this launch no matter how it ends. + let held_before_launch = vm_workdir.network_cleanup_pending(); vm_workdir.mark_network_cleanup_pending()?; let qemu_uid = Uid::effective().as_raw(); // Only ever read back out of a log line: netd is told where the VM @@ -708,18 +709,28 @@ impl App { let response = match result { Ok(response) => response, Err(error) => { - // The client may have timed out while netd was still - // finishing this Prepare. Remove the in-flight identity - // first; netd runs operations in arrival order, so the - // removal runs after the Prepare completes. - if let Err(cleanup_error) = netd.remove_interface(identity.clone()).await { - warn!(%cleanup_error, "failed to roll back in-flight netd network"); + let unreachable = netd::is_unreachable(&error); + if unreachable && prepared.is_empty() && !held_before_launch { + // The first request of this launch never reached a + // netd, so none holds anything for this VM. Keeping + // the marker would make every later stop and removal + // wait for a netd this node may never run. + if let Err(clear_error) = vm_workdir.clear_network_cleanup_pending() { + warn!(%clear_error, "failed to clear pending network cleanup"); + } + } else { + // The client may have timed out while netd was still + // finishing this Prepare. Remove the in-flight identity + // first; netd runs operations in arrival order, so the + // removal runs after the Prepare completes. + if let Err(cleanup_error) = netd.remove_interface(identity.clone()).await { + warn!(%cleanup_error, "failed to roll back in-flight netd network"); + } + self.roll_back_prepared_networks(prepared).await; } - self.roll_back_prepared_networks(prepared).await; // netd's own message is about a socket or a TAP, so // neither the NIC that asked nor what a node has to install // to satisfy it appears anywhere in the failure. - let unreachable = netd::is_unreachable(&error); let mode = network.nic.mode.as_str(); let error = Err(error).context("failed to prepare netd-managed networking"); return if unreachable { @@ -790,12 +801,11 @@ impl App { } /// A running VM whose snapshot is missing, because a VMM that predates the - /// snapshot -- or predates it recording what netd built -- started it. + /// snapshot started it. /// /// Guessing is all that is left, so guess the way that VMM would have, and - /// then write the guess down. Leaving the marker unset would make every - /// later teardown re-derive it from node configuration that may by then - /// have moved, which is the failure this snapshot exists to prevent. + /// then write the guess down, so what is reported for this boot does not + /// drift with node configuration that may by then have moved. /// /// The way *that* VMM would have, not this one: a build old enough to leave /// no snapshot had no vhost and no multiqueue at all, so whatever this @@ -1658,8 +1668,35 @@ impl App { /// started or removed. Costs a file check per VM when nothing is pending. pub(crate) async fn reconcile_network_cleanup(&self) { let ids: Vec = self.lock().vms.keys().cloned().collect(); - for id in ids { - self.reconcile_vm_network_cleanup(&id).await; + let pending: Vec = ids + .into_iter() + .filter(|id| { + self.work_dir(id) + .is_ok_and(|workdir| workdir.network_cleanup_pending()) + }) + .collect(); + if pending.is_empty() { + return; + } + // A running VM keeps its marker for as long as it runs. One listing + // skips those up front, so a pass does not queue behind every running + // VM's launch lock only to find it running. Each remaining VM is still + // checked again under its lock. + let running: HashSet = match self.supervisor.list().await { + Ok(processes) => processes + .into_iter() + .filter(|process| process.state.status.is_running()) + .map(|process| process.config.id) + .collect(), + Err(error) => { + debug!("skipping network cleanup: {error:#}"); + return; + } + }; + for id in pending { + if !running.contains(&id) { + self.reconcile_vm_network_cleanup(&id).await; + } } } @@ -2348,6 +2385,75 @@ mod tests { assert!(netd.methods().is_empty()); } + fn bridge_vm(app: &App, id: &str) -> (VmConfig, Vec) { + let mut manifest = test_manifest(2048); + manifest.id = id.to_string(); + manifest.networks = vec![NicNetworking { + mode: NetworkingMode::Bridge, + bridge: "dstack-br0".to_string(), + ..NicNetworking::default() + }]; + let networks = app.runtime_networks(&manifest); + let vm = VmConfig { + workdir: app.work_dir(id).unwrap().path().to_path_buf(), + manifest, + image: test_tdx_image(true), + cid: 3, + gateway_enabled: false, + }; + (vm, networks) + } + + /// A launch that never reached a netd built nothing, so it must not leave + /// a marker that makes every later stop and removal wait for one. What + /// an earlier boot may still hold keeps its marker. + #[tokio::test] + async fn a_launch_that_never_reached_netd_leaves_nothing_pending() { + let dir = tempfile::tempdir().unwrap(); + let app = App::new( + test_config(&dir.path().join("absent-netd.sock"), dir.path()), + SupervisorClient::new("http://127.0.0.1:0"), + ); + let (vm, mut networks) = bridge_vm(&app, "vm-1"); + let workdir = app.work_dir("vm-1").unwrap(); + std::fs::create_dir_all(workdir.path()).unwrap(); + + let error = app + .prepare_netd_networks(&vm, &mut networks) + .await + .unwrap_err(); + assert!( + format!("{error:#}").contains("run dstack-vmm netd"), + "{error:#}" + ); + assert!(!workdir.network_cleanup_pending()); + + workdir.mark_network_cleanup_pending().unwrap(); + app.prepare_netd_networks(&vm, &mut networks) + .await + .unwrap_err(); + assert!( + workdir.network_cleanup_pending(), + "an unreleased earlier boot is still pending" + ); + } + + /// A netd that answered, even with a refusal, may have built something: + /// the marker stays. + #[tokio::test] + async fn a_launch_netd_refused_stays_pending() { + let netd = netd::testing::FakeNetd::spawn(&[]); + let app = app_talking_to(netd.socket()); + let (vm, mut networks) = bridge_vm(&app, "vm-1"); + let workdir = app.work_dir("vm-1").unwrap(); + std::fs::create_dir_all(workdir.path()).unwrap(); + app.prepare_netd_networks(&vm, &mut networks) + .await + .unwrap_err(); + assert!(workdir.network_cleanup_pending()); + assert_eq!(netd.methods(), vec!["PrepareBridge", "RemoveInterface"]); + } + /// A netd outage must not become a fleet that cannot be stopped. #[tokio::test] async fn a_stop_survives_a_netd_that_is_not_there() { diff --git a/dstack/vmm/src/main.rs b/dstack/vmm/src/main.rs index e09c1e42d..0ca409af4 100644 --- a/dstack/vmm/src/main.rs +++ b/dstack/vmm/src/main.rs @@ -219,6 +219,8 @@ async fn auto_restart_task(app: App) { async fn network_cleanup_task(app: App) { let mut interval = tokio::time::interval(Duration::from_secs(60)); + // A pass can outlast the period while netd is slow; do not burst after it. + interval.set_missed_tick_behavior(tokio::time::MissedTickBehavior::Delay); loop { interval.tick().await; app.reconcile_network_cleanup().await; diff --git a/dstack/vmm/src/netd.rs b/dstack/vmm/src/netd.rs index 39c03d350..b07899f9c 100644 --- a/dstack/vmm/src/netd.rs +++ b/dstack/vmm/src/netd.rs @@ -49,6 +49,8 @@ use crate::config::{NetdConfig, NetworkFilterConfig}; /// operations queued ahead of it. const REQUEST_TIMEOUT: Duration = Duration::from_secs(30); const COMMAND_TIMEOUT: Duration = Duration::from_secs(30); +/// Bound on listing nwfilter bindings. See [`existing_bindings`]. +const LISTING_TIMEOUT: Duration = Duration::from_secs(10); const IP_PATH: &str = "/usr/sbin/ip"; const VIRSH_PATH: &str = "/usr/bin/virsh"; const LOCK_PATH: &str = "/run/lock/dstack-netd.lock"; @@ -384,11 +386,11 @@ impl NetdRpc for NetdHandler { async fn remove_interface_by_name(self, request: InterfaceName) -> Result<()> { self.run(move |config| { - let tap = request.tap; - if !is_managed_name(&tap) { - bail!("{tap} is not a name netd could have created"); - } - remove_interface(&config.libvirt_uri, &tap, BindingCleanup::BestEffort) + remove_interface_by_name( + &config.libvirt_uri, + &request.tap, + config.filter_policy().requires_binding(), + ) }) .await } @@ -666,14 +668,17 @@ fn sweep_vm_interfaces( nic_index: 0, }; validate_identity(&identity)?; - let bindings = existing_bindings(libvirt_uri); - // Not `bindings.is_some()`. A listing that could not be produced says + let listing = existing_bindings(libvirt_uri); + // Not `listing.is_ok()`. A listing that could not be produced says // nothing about whether a *deletion* will work, and reading it as "libvirt // is down, skip the bindings" would mean a node whose listing breaks for // any reason silently stops cleaning up bindings at all -- which is worse // than the per-name asking this listing exists to avoid. The pass finds - // out by trying, once. - let mut libvirt = true; + // out by trying, once. The exception is a listing that timed out: a + // `libvirtd` that did not answer it will not answer a deletion either, and + // waiting for that one too would outlast the caller's request. + let mut libvirt = !listing.as_ref().is_err_and(timed_out); + let bindings = listing.ok(); let mut removed = 0; let mut first_error = (requires_binding && bindings.is_none()) .then(|| anyhow::anyhow!("cannot confirm nwfilter cleanup: binding listing failed")); @@ -729,6 +734,59 @@ fn sweep_vm_interfaces( } } +/// Deletes one interface, and any binding at its name, for an operator. +/// +/// `netd list` shows a binding whose interface is gone as its own row, and this +/// is how an operator removes it, so a binding that is known to exist, or that +/// is all the name could still hold, has to be confirmed gone before this +/// reports success. Only a binding that merely might sit next to a live +/// interface on a node that does not filter is best effort: `libvirtd` need +/// not be running there, and the interface still has to go. The same rules as +/// [`sweep_vm_interfaces`]. +fn remove_interface_by_name(libvirt_uri: &str, tap: &str, requires_binding: bool) -> Result<()> { + if !is_managed_name(tap) { + bail!("{tap} is not a name netd could have created"); + } + let present = Path::new("/sys/class/net").join(tap).exists(); + if present && !is_macvtap(tap) && !is_tuntap(tap) { + bail!("refusing to delete {tap}: it is neither a tun/tap nor a macvtap device"); + } + let mut binding_error = None; + if !(present && is_macvtap(tap)) { + let known_binding = existing_bindings(libvirt_uri).map(|held| held.contains(tap)); + match known_binding { + Ok(false) if !present => {} + Ok(known) => { + if let Err(error) = delete_binding(libvirt_uri, tap) { + if known || requires_binding { + binding_error = Some(error); + } else { + warn!(%tap, "could not clear a possible nwfilter binding: {error:#}"); + } + } + } + Err(listing_error) => { + if let Err(error) = delete_binding(libvirt_uri, tap) { + if present && !requires_binding { + warn!(%tap, "could not clear a possible nwfilter binding: {error:#}"); + } else { + binding_error = Some(error.context(format!( + "cannot confirm the binding is gone: {listing_error:#}" + ))); + } + } + } + } + } + // The interface goes even when its binding could not: a TAP left on the + // bridge is worse than a binding with nothing to filter. + remove_interface(libvirt_uri, tap, BindingCleanup::Skip)?; + match binding_error { + Some(error) => Err(error), + None => Ok(()), + } +} + /// Every resource netd owns, read off the host rather than out of a record. /// /// Ownership is the reserved name plus the kernel's own answer about what kind @@ -738,7 +796,7 @@ fn sweep_vm_interfaces( /// interface inventory that refused to be produced without it would be /// unavailable exactly where unfiltered TAPs live. fn list_interfaces(libvirt_uri: &str, instance_id: &str) -> Vec { - let bindings = existing_bindings(libvirt_uri); + let bindings = existing_bindings(libvirt_uri).ok(); let mut records = Vec::new(); let mut seen = HashSet::new(); if let Ok(entries) = std::fs::read_dir("/sys/class/net") { @@ -1064,27 +1122,51 @@ fn virsh_output(uri: &str, args: &[&str], stdin: Option<&[u8]>) -> Result Option> { - match virsh_output(uri, &["nwfilter-binding-list"], None) { - Ok(output) => Some( - output - .lines() - .filter_map(|line| line.split_whitespace().next()) - .filter(|name| is_managed_name(name)) - .map(str::to_string) - .collect(), - ), - Err(error) => { - debug!("could not list nwfilter bindings: {error:#}"); - None - } - } +fn existing_bindings(uri: &str) -> Result> { + // Shorter than a mutation's timeout. A sweep or a manual removal asks this + // first, and the whole request has to fit in the VMM's own request + // timeout, which a hung `libvirtd` would otherwise use up on its own. + let output = run_command_with_timeout( + VIRSH_PATH, + &["--connect", uri, "nwfilter-binding-list"], + None, + LISTING_TIMEOUT, + ) + .inspect_err(|error| debug!("could not list nwfilter bindings: {error:#}"))?; + Ok(output + .lines() + .filter_map(|line| line.split_whitespace().next()) + .filter(|name| is_managed_name(name)) + .map(str::to_string) + .collect()) } fn run_command(program: &str, args: &[&str], stdin: Option<&[u8]>) -> Result<()> { run_command_with_timeout(program, args, stdin, COMMAND_TIMEOUT).map(|_| ()) } +/// A helper that did not finish in time. Typed so a caller can tell a hung +/// daemon, which will not answer the next request either, from a refusal. +#[derive(Debug)] +struct CommandTimedOut { + program: String, + timeout: Duration, +} + +impl std::fmt::Display for CommandTimedOut { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + write!(f, "{} timed out after {:?}", self.program, self.timeout) + } +} + +impl std::error::Error for CommandTimedOut {} + +fn timed_out(error: &anyhow::Error) -> bool { + error + .chain() + .any(|cause| cause.downcast_ref::().is_some()) +} + fn run_command_with_timeout( program: &str, args: &[&str], @@ -1112,10 +1194,11 @@ fn run_command_with_timeout( if child.wait_timeout(command_timeout)?.is_none() { let _ = child.kill(); let _ = child.wait(); - bail!( - "{} timed out after {command_timeout:?}", - Path::new(program).display() - ); + return Err(CommandTimedOut { + program: Path::new(program).display().to_string(), + timeout: command_timeout, + } + .into()); } let output = child.wait_with_output()?; if !output.status.success() { @@ -1506,6 +1589,11 @@ mod tests { .unwrap_err(); assert!(error.to_string().contains("timed out")); assert!(started.elapsed() < Duration::from_secs(2)); + // Recognised through whatever context a caller adds. + assert!(timed_out(&error.context("while listing"))); + let refused = run_command_with_timeout("/bin/sh", &["-c", "exit 1"], None, COMMAND_TIMEOUT) + .unwrap_err(); + assert!(!timed_out(&refused)); } /// A whole-VM sweep names a VM and no NIC: reaching the indices the caller @@ -1771,6 +1859,43 @@ mod tests { } } + /// Run with the same isolated-network-namespace setup as the tests above. + /// A manual removal reports success only once nothing at the name is left + /// that the node's policy needs confirmed. + #[test] + #[ignore = "needs root and its own network namespace"] + fn a_manual_removal_does_not_claim_a_binding_it_could_not_confirm() { + assert!(nix::unistd::Uid::effective().is_root()); + assert_ne!( + std::fs::read_link("/proc/self/ns/net").unwrap(), + std::fs::read_link("/proc/1/ns/net").unwrap(), + "run this inside its own network namespace", + ); + let uri = "qemu:///nonexistent-for-this-test"; + let tap = tap_name(&identity("manual-test", "vm-1", 0)); + + // Nothing but a possible binding at the name, and no libvirt to ask. + let error = remove_interface_by_name(uri, &tap, false).unwrap_err(); + assert!(format!("{error:#}").contains("cannot confirm"), "{error:#}"); + + for requires_binding in [false, true] { + ip(&["tuntap", "add", "dev", &tap, "mode", "tap"]).unwrap(); + let result = remove_interface_by_name(uri, &tap, requires_binding); + assert!( + !Path::new("/sys/class/net").join(&tap).exists(), + "the interface goes whatever happened to its binding" + ); + assert_eq!(result.is_err(), requires_binding, "{result:?}"); + } + + let impostor = tap_name(&identity("manual-test", "not-a-tap", 0)); + ip(&["link", "add", &impostor, "type", "dummy"]).unwrap(); + let refused = remove_interface_by_name(uri, &impostor, false).unwrap_err(); + assert!(refused.to_string().contains("refusing to delete")); + ip(&["link", "delete", "dev", &impostor]).unwrap(); + assert!(remove_interface_by_name(uri, "eth0", false).is_err()); + } + /// The VMM tells "netd is not running" apart from "netd refused". Both /// ways a Unix socket can have nobody behind it must read as the former, /// through the context the pRPC client stacks on the transport error. From f0e3f7892a369a180fa1e06f939100aa97b7b58e Mon Sep 17 00:00:00 2001 From: Kevin Wang Date: Mon, 14 Sep 2026 18:52:37 -0700 Subject: [PATCH 04/93] fix(netd): read helper output while waiting and tighten binding teardown - `run_command_with_timeout` drains stdout and stderr (and feeds stdin) on their own threads while it waits. Waiting first turned any output larger than a pipe buffer into a timeout, so on a host with a couple of thousand nwfilter bindings every sweep read as a hung libvirtd. - A sweep and a manual removal set a present interface down before deleting its binding, so a QEMU a stop only signalled does not reach the bridge unfiltered while the binding goes. - A listed binding at a name that is now a macvtap is deleted too, instead of being skipped while the sweep reports success. - A manual removal whose binding listing timed out does not also wait for a deletion from the same libvirtd. - `resize_vm` refuses a VM being removed and holds its launch lock, like `update_vm`, so it cannot recreate a directory a removal just deleted. `remove_all` puts `.removing` back if the directory could not be removed. - Correct the netd trust-boundary doc now that `RemoveInterfaceByName` accepts a netd-shaped TAP name. --- docs/libvirt-network-filter.md | 8 +- dstack/vmm/src/app/workdir.rs | 10 ++- dstack/vmm/src/main_service.rs | 6 ++ dstack/vmm/src/netd.rs | 138 +++++++++++++++++++++++---------- 4 files changed, 119 insertions(+), 43 deletions(-) diff --git a/docs/libvirt-network-filter.md b/docs/libvirt-network-filter.md index 125f4b761..97e659c0c 100644 --- a/docs/libvirt-network-filter.md +++ b/docs/libvirt-network-filter.md @@ -110,9 +110,11 @@ Deploy VMM instances that do not share this trust boundary with dedicated netd sockets and distinct filesystem permissions. `netd` invokes fixed absolute `ip` and `virsh` executables with separate -arguments. It never accepts a command, executable path, TAP name, or raw XML -from a client. Filter XML is generated internally with XML escaping and is -validated by libvirt. +arguments. It never accepts a command, executable path, or raw XML from a +client. The only TAP name it accepts is the one `RemoveInterfaceByName` takes, +and only a name of the form netd itself creates (`dt` and 12 lowercase hex +digits), on a tun/tap or macvtap device. Filter XML is generated internally with +XML escaping and is validated by libvirt. Teardown by identity only reaches the NIC indices its caller still has a record of, and that record is written *after* the interface exists — a VMM killed in diff --git a/dstack/vmm/src/app/workdir.rs b/dstack/vmm/src/app/workdir.rs index 55c070873..fd67072f3 100644 --- a/dstack/vmm/src/app/workdir.rs +++ b/dstack/vmm/src/app/workdir.rs @@ -268,7 +268,15 @@ impl VmWorkDir { Err(error) if error.kind() == std::io::ErrorKind::NotFound => {} Err(error) => return Err(error.into()), } - fs::remove_dir(&self.workdir)?; + if let Err(error) = fs::remove_dir(&self.workdir) { + // Something wrote into the directory while it was being emptied. + // Put the marker back, so what is left is still a removal to the + // next reload rather than a VM that fails to load. + if self.workdir.exists() { + let _ = self.set_removing(); + } + return Err(error.into()); + } Ok(()) } diff --git a/dstack/vmm/src/main_service.rs b/dstack/vmm/src/main_service.rs index 9082aaa1b..5d72e4f20 100644 --- a/dstack/vmm/src/main_service.rs +++ b/dstack/vmm/src/main_service.rs @@ -1112,6 +1112,12 @@ impl VmmRpc for RpcHandler { "resize_vm RPC called" ); validate_resize_request(&request)?; + // The same guard as `update_vm`, for the same reason: this writes the + // manifest, and `put_manifest` recreates a directory a removal has + // just deleted. Before the lock, and again under it. + self.app.refuse_if_removing(&request.id)?; + let _launch = self.app.launch_lock(&request.id).await; + self.app.refuse_if_removing(&request.id)?; let vm_work_dir = self.app.work_dir(&request.id)?; let mut manifest = vm_work_dir.manifest().context("failed to read manifest")?; self.apply_resource_updates( diff --git a/dstack/vmm/src/netd.rs b/dstack/vmm/src/netd.rs index b07899f9c..5ff6d524a 100644 --- a/dstack/vmm/src/netd.rs +++ b/dstack/vmm/src/netd.rs @@ -692,8 +692,13 @@ fn sweep_vm_interfaces( // again after it has failed is how a hung `libvirtd` turns a bounded // collection into an unbounded one. let known_binding = bindings.as_ref().is_some_and(|held| held.contains(&tap)); - let wanted = present || known_binding; - if libvirt && wanted && !is_macvtap(&tap) { + if present { + take_down(&tap); + } + // A macvtap never carries a binding of its own, but a name that was a + // filtered TAP before can still hold one that is listed. + let wanted = known_binding || (present && !is_macvtap(&tap)); + if libvirt && wanted { if let Err(error) = delete_binding(libvirt_uri, &tap) { warn!(%tap, %error, "failed to remove an nwfilter binding"); libvirt = false; @@ -748,36 +753,47 @@ fn remove_interface_by_name(libvirt_uri: &str, tap: &str, requires_binding: bool bail!("{tap} is not a name netd could have created"); } let present = Path::new("/sys/class/net").join(tap).exists(); - if present && !is_macvtap(tap) && !is_tuntap(tap) { + let macvtap = present && is_macvtap(tap); + if present && !macvtap && !is_tuntap(tap) { bail!("refusing to delete {tap}: it is neither a tun/tap nor a macvtap device"); } - let mut binding_error = None; - if !(present && is_macvtap(tap)) { - let known_binding = existing_bindings(libvirt_uri).map(|held| held.contains(tap)); - match known_binding { - Ok(false) if !present => {} - Ok(known) => { - if let Err(error) = delete_binding(libvirt_uri, tap) { - if known || requires_binding { - binding_error = Some(error); - } else { - warn!(%tap, "could not clear a possible nwfilter binding: {error:#}"); - } - } - } - Err(listing_error) => { - if let Err(error) = delete_binding(libvirt_uri, tap) { - if present && !requires_binding { - warn!(%tap, "could not clear a possible nwfilter binding: {error:#}"); - } else { - binding_error = Some(error.context(format!( - "cannot confirm the binding is gone: {listing_error:#}" - ))); - } - } + if present { + take_down(tap); + } + let possible = |error: anyhow::Error| { + warn!(%tap, "could not clear a possible nwfilter binding: {error:#}"); + None + }; + let binding_error = match existing_bindings(libvirt_uri).map(|held| held.contains(tap)) { + // Nothing listed at the name, and nothing there that could carry one. + Ok(false) if !present || macvtap => None, + Ok(known) => match delete_binding(libvirt_uri, tap) { + Ok(()) => None, + Err(error) if known || requires_binding => Some(error), + Err(error) => possible(error), + }, + // A macvtap carries no binding of its own, and there is no listing + // saying an older one is left. + Err(_) if macvtap => None, + Err(listing_error) => { + // A `libvirtd` that did not answer the listing will not answer a + // deletion either, and waiting for it would outlast the request. + let attempt = if timed_out(&listing_error) { + Err(listing_error.context("cannot confirm the binding is gone")) + } else { + delete_binding(libvirt_uri, tap).map_err(|error| { + error.context(format!( + "cannot confirm the binding is gone: {listing_error:#}" + )) + }) + }; + match attempt { + Ok(()) => None, + Err(error) if present && !requires_binding => possible(error), + Err(error) => Some(error), } } - } + }; // The interface goes even when its binding could not: a TAP left on the // bridge is worse than a binding with nothing to filter. remove_interface(libvirt_uri, tap, BindingCleanup::Skip)?; @@ -888,6 +904,17 @@ fn remove_interface(libvirt_uri: &str, tap: &str, cleanup: BindingCleanup) -> Re Ok(()) } +/// Sets a device netd created down before anything else about it is torn down. +/// +/// A stop only signals QEMU, so it may still be sending when its release runs. +/// Down first, so its traffic does not reach the bridge unfiltered for as long +/// as deleting the binding takes. Never touches a device netd did not create. +fn take_down(interface: &str) { + if is_macvtap(interface) || is_tuntap(interface) { + let _ = ip(&["link", "set", "dev", interface, "down"]); + } +} + /// Records who an interface belongs to, on the interface. See /// [`interface_alias`]. fn set_alias(tap: &str, identity: &InterfaceIdentity) -> Result<()> { @@ -1184,14 +1211,23 @@ fn run_command_with_timeout( .stderr(Stdio::piped()) .spawn() .with_context(|| format!("failed to execute {program}"))?; + // Drain both pipes while waiting. A child whose output fills a pipe + // blocks until someone reads it, so waiting first and reading afterwards + // turns any output larger than the pipe buffer -- a binding listing on a + // busy host -- into a timeout. + let stdout = drain(child.stdout.take().context("missing command stdout")?); + let stderr = drain(child.stderr.take().context("missing command stderr")?); if let Some(input) = stdin { - child - .stdin - .take() - .context("missing command stdin")? - .write_all(input)?; + let mut pipe = child.stdin.take().context("missing command stdin")?; + let input = input.to_vec(); + // On its own thread for the same reason: a child that writes before + // it has read all of its input must not stall the write past the + // timeout. Dropping the pipe afterwards is the child's EOF. + std::thread::spawn(move || { + let _ = pipe.write_all(&input); + }); } - if child.wait_timeout(command_timeout)?.is_none() { + let Some(status) = child.wait_timeout(command_timeout)? else { let _ = child.kill(); let _ = child.wait(); return Err(CommandTimedOut { @@ -1199,13 +1235,23 @@ fn run_command_with_timeout( timeout: command_timeout, } .into()); - } - let output = child.wait_with_output()?; - if !output.status.success() { - let error = String::from_utf8_lossy(&output.stderr); + }; + let stdout = stdout.join().unwrap_or_default(); + let stderr = stderr.join().unwrap_or_default(); + if !status.success() { + let error = String::from_utf8_lossy(&stderr); bail!("{} failed: {}", Path::new(program).display(), error.trim()); } - Ok(String::from_utf8_lossy(&output.stdout).into_owned()) + Ok(String::from_utf8_lossy(&stdout).into_owned()) +} + +/// Reads a pipe to its end on its own thread. +fn drain(mut pipe: impl io::Read + Send + 'static) -> std::thread::JoinHandle> { + std::thread::spawn(move || { + let mut buffer = Vec::new(); + let _ = pipe.read_to_end(&mut buffer); + buffer + }) } fn require_executable(path: &str) -> Result<()> { @@ -1589,6 +1635,20 @@ mod tests { .unwrap_err(); assert!(error.to_string().contains("timed out")); assert!(started.elapsed() < Duration::from_secs(2)); + // Output larger than a pipe buffer is read, not mistaken for a hang. + let large = run_command_with_timeout( + "/bin/sh", + &["-c", "head -c 1000000 /dev/zero | tr '\\0' x"], + None, + Duration::from_secs(10), + ) + .unwrap(); + assert_eq!(large.len(), 1_000_000); + // Input is still delivered, and its EOF still arrives. + let echoed = + run_command_with_timeout("/bin/cat", &[], Some(b"binding xml"), COMMAND_TIMEOUT) + .unwrap(); + assert_eq!(echoed, "binding xml"); // Recognised through whatever context a caller adds. assert!(timed_out(&error.context("while listing"))); let refused = run_command_with_timeout("/bin/sh", &["-c", "exit 1"], None, COMMAND_TIMEOUT) From b4cad122a677f52ddbd18123c89d5ad2441e4609 Mon Sep 17 00:00:00 2001 From: Kevin Wang Date: Mon, 14 Sep 2026 19:04:43 -0700 Subject: [PATCH 05/93] fix(netd): keep sweep and manual removal rules consistent - A sweep leaves the binding alone at a name held by a device netd did not create, and lets `remove_interface` refuse it, as a manual removal does. - A manual removal of a macvtap on a filtering node does not report success when the binding listing failed; like a sweep, it requires confirmation. --- dstack/vmm/src/netd.rs | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-) diff --git a/dstack/vmm/src/netd.rs b/dstack/vmm/src/netd.rs index 5ff6d524a..f132f5bda 100644 --- a/dstack/vmm/src/netd.rs +++ b/dstack/vmm/src/netd.rs @@ -695,9 +695,13 @@ fn sweep_vm_interfaces( if present { take_down(&tap); } + let macvtap = present && is_macvtap(&tap); + // A device netd did not create is refused below, before anything at + // its name is touched, as a manual removal refuses it. + let foreign = present && !macvtap && !is_tuntap(&tap); // A macvtap never carries a binding of its own, but a name that was a // filtered TAP before can still hold one that is listed. - let wanted = known_binding || (present && !is_macvtap(&tap)); + let wanted = !foreign && (known_binding || (present && !macvtap)); if libvirt && wanted { if let Err(error) = delete_binding(libvirt_uri, &tap) { warn!(%tap, %error, "failed to remove an nwfilter binding"); @@ -773,8 +777,9 @@ fn remove_interface_by_name(libvirt_uri: &str, tap: &str, requires_binding: bool Err(error) => possible(error), }, // A macvtap carries no binding of its own, and there is no listing - // saying an older one is left. - Err(_) if macvtap => None, + // saying an older one is left. A filtering node still needs that + // confirmed, as a sweep does. + Err(_) if macvtap && !requires_binding => None, Err(listing_error) => { // A `libvirtd` that did not answer the listing will not answer a // deletion either, and waiting for it would outlast the request. From a78b08249d9d5db799a1425bb33af39f1e65c8ca Mon Sep 17 00:00:00 2001 From: Kevin Wang Date: Mon, 14 Sep 2026 21:32:42 -0700 Subject: [PATCH 06/93] os/kernel: add TDX guest scaling knobs Add two guest kernel patches for many-vCPU TDX guests: wake_up_q() notification IPI batching (tdx_wake_q_batch) and PV single-target function IPIs (tdx_pv_single_ipi). Both are runtime switches, on by default in TDX guests and unavailable elsewhere; boot parameters or sysfs writes can turn them off. Build the haltpoll cpuidle driver as a module so guest halt polling can be enabled and disabled at runtime. It stays opt-in. On a 32-vCPU guest the new defaults were not slower than the previous behavior on any tested workload and raised the synchronization benchmark geometric mean from 13.6% to 38.5% of an ordinary VM. --- os/mkosi/components/kernel/kernel-build.sh | 4 +- os/mkosi/components/kernel/kernel.config | 5 + os/mkosi/components/kernel/kernel.sh | 4 +- ...-smp-batch-wake-q-ipis-in-tdx-guests.patch | 387 ++++++++++++++++++ ...e-pv-single-target-ipi-in-tdx-guests.patch | 110 +++++ 5 files changed, 508 insertions(+), 2 deletions(-) create mode 100644 os/yocto/layers/meta-dstack/recipes-kernel/linux/files/0007-sched-smp-batch-wake-q-ipis-in-tdx-guests.patch create mode 100644 os/yocto/layers/meta-dstack/recipes-kernel/linux/files/0008-x86-kvm-runtime-pv-single-target-ipi-in-tdx-guests.patch diff --git a/os/mkosi/components/kernel/kernel-build.sh b/os/mkosi/components/kernel/kernel-build.sh index f6ea91c31..dfad7ab58 100755 --- a/os/mkosi/components/kernel/kernel-build.sh +++ b/os/mkosi/components/kernel/kernel-build.sh @@ -56,7 +56,9 @@ for patch in \ "$ROOT/os/yocto/layers/meta-dstack/recipes-kernel/linux/files/0003-dma-direct-return-struct-page-from-alloc-from-pool.patch" \ "$ROOT/os/yocto/layers/meta-dstack/recipes-kernel/linux/files/0004-dma-pool-free-atomic-pool-pages-by-physical-address.patch" \ "$ROOT/os/yocto/layers/meta-dstack/recipes-kernel/linux/files/0005-swiotlb-preserve-allocation-virtual-address.patch" \ - "$ROOT/os/yocto/layers/meta-dstack/recipes-kernel/linux/files/0006-swiotlb-free-dynamic-pools-from-process-context.patch"; do + "$ROOT/os/yocto/layers/meta-dstack/recipes-kernel/linux/files/0006-swiotlb-free-dynamic-pools-from-process-context.patch" \ + "$ROOT/os/yocto/layers/meta-dstack/recipes-kernel/linux/files/0007-sched-smp-batch-wake-q-ipis-in-tdx-guests.patch" \ + "$ROOT/os/yocto/layers/meta-dstack/recipes-kernel/linux/files/0008-x86-kvm-runtime-pv-single-target-ipi-in-tdx-guests.patch"; do patch -d "$src" -p1 --fuzz=0 < "$patch" done diff --git a/os/mkosi/components/kernel/kernel.config b/os/mkosi/components/kernel/kernel.config index 822e539b9..f21a9ea2e 100644 --- a/os/mkosi/components/kernel/kernel.config +++ b/os/mkosi/components/kernel/kernel.config @@ -15,6 +15,11 @@ CONFIG_ACPI_TABLE_UPGRADE=y CONFIG_HYPERVISOR_GUEST=y CONFIG_PARAVIRT=y CONFIG_KVM_GUEST=y +# Guest halt polling is built as a module so it can be enabled and disabled at +# runtime (modprobe/rmmod cpuidle_haltpoll). Unloading restores the default +# idle path and host halt polling; a built-in driver can only bind at boot. +CONFIG_CPU_IDLE_GOV_HALTPOLL=y +CONFIG_HALTPOLL_CPUIDLE=m CONFIG_INTEL_TDX_GUEST=y CONFIG_TDX_GUEST_DRIVER=y CONFIG_TSM_REPORTS=y diff --git a/os/mkosi/components/kernel/kernel.sh b/os/mkosi/components/kernel/kernel.sh index 62817415b..e02749e0d 100644 --- a/os/mkosi/components/kernel/kernel.sh +++ b/os/mkosi/components/kernel/kernel.sh @@ -13,7 +13,9 @@ component_cache_key() { "$ROOT/os/yocto/layers/meta-dstack/recipes-kernel/linux/files/0003-dma-direct-return-struct-page-from-alloc-from-pool.patch" \ "$ROOT/os/yocto/layers/meta-dstack/recipes-kernel/linux/files/0004-dma-pool-free-atomic-pool-pages-by-physical-address.patch" \ "$ROOT/os/yocto/layers/meta-dstack/recipes-kernel/linux/files/0005-swiotlb-preserve-allocation-virtual-address.patch" \ - "$ROOT/os/yocto/layers/meta-dstack/recipes-kernel/linux/files/0006-swiotlb-free-dynamic-pools-from-process-context.patch" + "$ROOT/os/yocto/layers/meta-dstack/recipes-kernel/linux/files/0006-swiotlb-free-dynamic-pools-from-process-context.patch" \ + "$ROOT/os/yocto/layers/meta-dstack/recipes-kernel/linux/files/0007-sched-smp-batch-wake-q-ipis-in-tdx-guests.patch" \ + "$ROOT/os/yocto/layers/meta-dstack/recipes-kernel/linux/files/0008-x86-kvm-runtime-pv-single-target-ipi-in-tdx-guests.patch" key_tools gcc ld make pahole key_packages binutils dwarves bc bison flex libssl-dev libelf-dev } diff --git a/os/yocto/layers/meta-dstack/recipes-kernel/linux/files/0007-sched-smp-batch-wake-q-ipis-in-tdx-guests.patch b/os/yocto/layers/meta-dstack/recipes-kernel/linux/files/0007-sched-smp-batch-wake-q-ipis-in-tdx-guests.patch new file mode 100644 index 000000000..27d48c6cd --- /dev/null +++ b/os/yocto/layers/meta-dstack/recipes-kernel/linux/files/0007-sched-smp-batch-wake-q-ipis-in-tdx-guests.patch @@ -0,0 +1,387 @@ +From 0000000000000000000000000000000000000000 Mon Sep 17 00:00:00 2001 +From: Kevin Wang +Date: Mon, 14 Sep 2026 10:11:06 -0700 +Subject: [PATCH 7/8] sched/smp: batch wake_up_q() notification IPIs in TDX + guests + +Every IPI sent by a TDX guest costs a TD exit and a TD entry. When a +task wakes many sleeping tasks through wake_up_q() (futex wake, +condition variable broadcast, OpenMP thread pools), each wakee on a +halted CPU costs one separate IPI hypercall, and the waker spends most +of its time in TD transitions instead of issuing wakeups. + +Add a runtime switch, tdx_wake_q_batch +(/sys/module/kernel/parameters/tdx_wake_q_batch), that keeps publishing +each wakeup immediately but defers the notification: target CPUs are +collected in a stack-local mask and notified with one mask IPI. A group +ends after 8 wakeups, after about 50 microseconds of sender time or when +the sender needs to reschedule, so the first wakee of a group waits at most +for the rest of that small group to be queued. Wakees that are polling in +idle still need no IPI at all. + +Batched TTWU entries use a separate per-CPU llist so that a synchronous +call-single callback is never hidden behind a deferred notification. The +IPI handler and the explicit idle, migration and CPU-down flush paths drain +both lists. Only task-context wake_up_q() and fair-class wakeups take the +batched path; RT, deadline, stop and ext tasks, interrupt-context wakeups +and direct try_to_wake_up() callers keep immediate notification. The switch +can only be enabled in a TDX guest without PREEMPT_RT. It is on by default +in TDX guests from a late initcall, so early boot and SMP bring-up keep +immediate notification; an explicit tdx_wake_q_batch= boot parameter is +preserved. + +With guest halt polling enabled in a 32-vCPU TDX guest, enabling batching +raised a PyTorch CPU operator with 32 passive OpenMP threads from 257 to +4440 MiB/s and doubled two many-thread pool synchronization benchmarks. + +Upstream-Status: Inappropriate [dstack TDX guest tuning, not submitted] +--- + kernel/sched/core.c | 131 +++++++++++++++++++++++++++++++++--- + kernel/sched/wake_q_batch.h | 16 +++++ + kernel/smp.c | 49 +++++++++++++- + 3 files changed, 184 insertions(+), 12 deletions(-) + create mode 100644 kernel/sched/wake_q_batch.h + +diff --git a/kernel/sched/core.c b/kernel/sched/core.c +index b5b5897b5..ece2b671b 100644 +--- a/kernel/sched/core.c ++++ b/kernel/sched/core.c +@@ -62,6 +62,7 @@ + #include + #include + #include ++#include "wake_q_batch.h" + #include + #include + #include +@@ -1070,10 +1071,109 @@ void wake_q_add_safe(struct wake_q_head *head, struct task_struct *task) + put_task_struct(task); + } + ++/* ++ * TDX guests pay a full TD exit for every wakeup IPI. When enabled, wake_up_q() ++ * publishes each queued wakeup immediately but coalesces the notification IPIs ++ * of a small group into one mask IPI. A group ends after ++ * WAKE_Q_BATCH_TASKS wakeups, WAKE_Q_BATCH_NS of sender time or a pending ++ * reschedule, so the extra delay for the first wakee stays small. ++ */ ++#define WAKE_Q_BATCH_TASKS 8 ++#define WAKE_Q_BATCH_NS 50000 ++static bool tdx_wake_q_batch; ++/* Set once the boot command line or sysfs chose a value explicitly. */ ++static bool tdx_wake_q_batch_set; ++ ++static bool wake_q_batch_available(void) ++{ ++#if defined(CONFIG_X86_64) && defined(CONFIG_SMP) && !defined(CONFIG_PREEMPT_RT) ++ return cpu_feature_enabled(X86_FEATURE_TDX_GUEST); ++#else ++ return false; ++#endif ++} ++ ++static int param_set_tdx_wake_q_batch(const char *val, ++ const struct kernel_param *kp) ++{ ++ bool enabled; ++ int ret; ++ ++ ret = kstrtobool(val ? val : "1", &enabled); ++ if (ret) ++ return ret; ++ if (enabled && !wake_q_batch_available()) ++ return -EOPNOTSUPP; ++ WRITE_ONCE(*(bool *)kp->arg, enabled); ++ tdx_wake_q_batch_set = true; ++ return 0; ++} ++ ++static const struct kernel_param_ops tdx_wake_q_batch_ops = { ++ .flags = KERNEL_PARAM_OPS_FL_NOARG, ++ .set = param_set_tdx_wake_q_batch, ++ .get = param_get_bool, ++}; ++__core_param_cb(tdx_wake_q_batch, &tdx_wake_q_batch_ops, &tdx_wake_q_batch, 0644); ++ ++/* ++ * Enabled by default in TDX guests, but only once boot is complete, so early ++ * boot and SMP bring-up keep immediate notification. ++ */ ++static int __init tdx_wake_q_batch_default(void) ++{ ++ if (!tdx_wake_q_batch_set && wake_q_batch_available()) ++ WRITE_ONCE(tdx_wake_q_batch, true); ++ return 0; ++} ++late_initcall(tdx_wake_q_batch_default); ++ ++static int try_to_wake_up_mask(struct task_struct *p, unsigned int state, ++ int wake_flags, struct cpumask *mask); ++ ++static void wake_up_q_batch(struct wake_q_head *head) ++{ ++ struct wake_q_node *node = head->first; ++ struct task_struct *refs[WAKE_Q_BATCH_TASKS]; ++ cpumask_t mask; ++ unsigned int nr, i; ++ u64 start; ++ ++ while (node != WAKE_Q_TAIL) { ++ /* Keep CPU-down exclusion until all published wakeups are kicked. */ ++ preempt_disable(); ++ cpumask_clear(&mask); ++ nr = 0; ++ start = local_clock(); ++ do { ++ struct task_struct *task; ++ ++ task = container_of(node, struct task_struct, wake_q); ++ node = node->next; ++ /* Same publication and state barriers as wake_up_q(). */ ++ WRITE_ONCE(task->wake_q.next, NULL); ++ try_to_wake_up_mask(task, TASK_NORMAL, 0, &mask); ++ refs[nr++] = task; ++ } while (node != WAKE_Q_TAIL && nr < WAKE_Q_BATCH_TASKS && ++ !need_resched() && local_clock() - start < WAKE_Q_BATCH_NS); ++ ++ /* No batching state survives preempt_enable() or task release. */ ++ sched_ttwu_flush_batch(&mask); ++ preempt_enable(); ++ for (i = 0; i < nr; i++) ++ put_task_struct(refs[i]); ++ } ++} ++ + void wake_up_q(struct wake_q_head *head) + { + struct wake_q_node *node = head->first; + ++ if (READ_ONCE(tdx_wake_q_batch) && in_task()) { ++ wake_up_q_batch(head); ++ return; ++ } ++ + while (node != WAKE_Q_TAIL) { + struct task_struct *task; + +@@ -3846,7 +3946,8 @@ bool call_function_single_prep_ipi(int cpu) + * via sched_ttwu_wakeup() for activation so the wakee incurs the cost + * of the wakeup instead of the waker. + */ +-static void __ttwu_queue_wakelist(struct task_struct *p, int cpu, int wake_flags) ++static void __ttwu_queue_wakelist(struct task_struct *p, int cpu, int wake_flags, ++ struct cpumask *mask) + { + struct rq *rq = cpu_rq(cpu); + +@@ -3854,7 +3955,11 @@ static void __ttwu_queue_wakelist(struct task_struct *p, int cpu, int wake_flags + + WRITE_ONCE(rq->ttwu_pending, 1); + #ifdef CONFIG_SMP +- __smp_call_single_queue(cpu, &p->wake_entry.llist); ++ /* Publish under pi_lock: migration_cpu_stop() must already see this. */ ++ if (mask && p->sched_class == &fair_sched_class) ++ sched_ttwu_queue_batch(cpu, &p->wake_entry.llist, mask); ++ else ++ __smp_call_single_queue(cpu, &p->wake_entry.llist); + #endif + } + +@@ -3950,23 +4055,25 @@ static inline bool ttwu_queue_cond(struct task_struct *p, int cpu) + return false; + } + +-static bool ttwu_queue_wakelist(struct task_struct *p, int cpu, int wake_flags) ++static bool ttwu_queue_wakelist(struct task_struct *p, int cpu, int wake_flags, ++ struct cpumask *mask) + { + if (sched_feat(TTWU_QUEUE) && ttwu_queue_cond(p, cpu)) { + sched_clock_cpu(cpu); /* Sync clocks across CPUs */ +- __ttwu_queue_wakelist(p, cpu, wake_flags); ++ __ttwu_queue_wakelist(p, cpu, wake_flags, mask); + return true; + } + + return false; + } + +-static void ttwu_queue(struct task_struct *p, int cpu, int wake_flags) ++static void ttwu_queue(struct task_struct *p, int cpu, int wake_flags, ++ struct cpumask *mask) + { + struct rq *rq = cpu_rq(cpu); + struct rq_flags rf; + +- if (ttwu_queue_wakelist(p, cpu, wake_flags)) ++ if (ttwu_queue_wakelist(p, cpu, wake_flags, mask)) + return; + + rq_lock(rq, &rf); +@@ -4145,7 +4252,8 @@ bool ttwu_state_match(struct task_struct *p, unsigned int state, int *success) + * Return: %true if @p->state changes (an actual wakeup was done), + * %false otherwise. + */ +-int try_to_wake_up(struct task_struct *p, unsigned int state, int wake_flags) ++static int try_to_wake_up_mask(struct task_struct *p, unsigned int state, ++ int wake_flags, struct cpumask *mask) + { + guard(preempt)(); + int cpu, success = 0; +@@ -4269,7 +4377,7 @@ int try_to_wake_up(struct task_struct *p, unsigned int state, int wake_flags) + * scheduling. + */ + if (smp_load_acquire(&p->on_cpu) && +- ttwu_queue_wakelist(p, task_cpu(p), wake_flags)) ++ ttwu_queue_wakelist(p, task_cpu(p), wake_flags, mask)) + break; + + /* +@@ -4295,7 +4403,7 @@ int try_to_wake_up(struct task_struct *p, unsigned int state, int wake_flags) + set_task_cpu(p, cpu); + } + +- ttwu_queue(p, cpu, wake_flags); ++ ttwu_queue(p, cpu, wake_flags, mask); + } + out: + if (success) +@@ -4304,6 +4412,11 @@ int try_to_wake_up(struct task_struct *p, unsigned int state, int wake_flags) + return success; + } + ++int try_to_wake_up(struct task_struct *p, unsigned int state, int wake_flags) ++{ ++ return try_to_wake_up_mask(p, state, wake_flags, NULL); ++} ++ + static bool __task_needs_rq_lock(struct task_struct *p) + { + unsigned int state = READ_ONCE(p->__state); +diff --git a/kernel/sched/wake_q_batch.h b/kernel/sched/wake_q_batch.h +new file mode 100644 +index 000000000..73212d061 +--- /dev/null ++++ b/kernel/sched/wake_q_batch.h +@@ -0,0 +1,16 @@ ++/* SPDX-License-Identifier: GPL-2.0-only */ ++#ifndef _KERNEL_SCHED_WAKE_Q_BATCH_H ++#define _KERNEL_SCHED_WAKE_Q_BATCH_H ++ ++struct cpumask; ++struct llist_node; ++ ++#ifdef CONFIG_SMP ++void sched_ttwu_queue_batch(int cpu, struct llist_node *node, ++ struct cpumask *mask); ++void sched_ttwu_flush_batch(struct cpumask *mask); ++#else ++static inline void sched_ttwu_flush_batch(struct cpumask *mask) { } ++#endif ++ ++#endif /* _KERNEL_SCHED_WAKE_Q_BATCH_H */ +diff --git a/kernel/smp.c b/kernel/smp.c +index 02f52291f..50593533e 100644 +--- a/kernel/smp.c ++++ b/kernel/smp.c +@@ -34,6 +34,7 @@ + + #include "smpboot.h" + #include "sched/smp.h" ++#include "sched/wake_q_batch.h" + + #define CSD_TYPE(_csd) ((_csd)->node.u_flags & CSD_FLAG_TYPE_MASK) + +@@ -46,6 +47,8 @@ struct call_function_data { + static DEFINE_PER_CPU_ALIGNED(struct call_function_data, cfd_data); + + static DEFINE_PER_CPU_SHARED_ALIGNED(struct llist_head, call_single_queue); ++/* Batched TTWU wakeups; kept apart so no synchronous CSD waits on a deferred kick. */ ++static DEFINE_PER_CPU_SHARED_ALIGNED(struct llist_head, wake_q_batch_queue); + + static DEFINE_PER_CPU(atomic_t, trigger_backtrace) = ATOMIC_INIT(1); + +@@ -105,8 +108,10 @@ void __init call_function_init(void) + { + int i; + +- for_each_possible_cpu(i) ++ for_each_possible_cpu(i) { + init_llist_head(&per_cpu(call_single_queue, i)); ++ init_llist_head(&per_cpu(wake_q_batch_queue, i)); ++ } + + smpcfd_prepare_cpu(smp_processor_id()); + } +@@ -413,6 +418,33 @@ void __smp_call_single_queue(int cpu, struct llist_node *node) + send_call_function_single_ipi(cpu); + } + ++/* ++ * Queue a TTWU entry for a batched wakeup: the entry is published right away, ++ * only the notification is deferred by adding @cpu to @mask. ++ */ ++void sched_ttwu_queue_batch(int cpu, struct llist_node *node, ++ struct cpumask *mask) ++{ ++ lockdep_assert_preemption_disabled(); ++ if (trace_csd_queue_cpu_enabled()) { ++ call_single_data_t *csd; ++ ++ csd = container_of(node, call_single_data_t, node.llist); ++ trace_csd_queue_cpu(cpu, _RET_IP_, sched_ttwu_pending, csd); ++ } ++ if (llist_add(node, &per_cpu(wake_q_batch_queue, cpu)) && ++ call_function_single_prep_ipi(cpu)) ++ cpumask_set_cpu(cpu, mask); ++} ++ ++void sched_ttwu_flush_batch(struct cpumask *mask) ++{ ++ lockdep_assert_preemption_disabled(); ++ if (!cpumask_empty(mask)) ++ send_call_function_ipi_mask(mask); ++ cpumask_clear(mask); ++} ++ + /* + * Insert a previously allocated call_single_data_t element + * for execution on the given CPU. data must already have +@@ -552,7 +584,7 @@ static void __flush_smp_call_function_queue(bool warn_cpu_offline) + } + + if (!entry) +- return; ++ goto batch_wakes; + + /* + * Second; run all !SYNC callbacks. +@@ -592,6 +624,16 @@ static void __flush_smp_call_function_queue(bool warn_cpu_offline) + csd = llist_entry(entry, typeof(*csd), node.llist); + csd_do_func(sched_ttwu_pending, entry, csd); + } ++ ++batch_wakes: ++ /* Batched wakeups; also reached from idle, migration and CPU-down flushes. */ ++ entry = llist_del_all(this_cpu_ptr(&wake_q_batch_queue)); ++ entry = llist_reverse_order(entry); ++ if (entry) { ++ WARN_ON_ONCE(warn_cpu_offline && !cpu_online(smp_processor_id())); ++ csd = llist_entry(entry, typeof(*csd), node.llist); ++ csd_do_func(sched_ttwu_pending, entry, csd); ++ } + } + + +@@ -612,7 +654,8 @@ void flush_smp_call_function_queue(void) + unsigned int was_pending; + unsigned long flags; + +- if (llist_empty(this_cpu_ptr(&call_single_queue))) ++ if (llist_empty(this_cpu_ptr(&call_single_queue)) && ++ llist_empty(this_cpu_ptr(&wake_q_batch_queue))) + return; + + local_irq_save(flags); +-- +2.43.0 + diff --git a/os/yocto/layers/meta-dstack/recipes-kernel/linux/files/0008-x86-kvm-runtime-pv-single-target-ipi-in-tdx-guests.patch b/os/yocto/layers/meta-dstack/recipes-kernel/linux/files/0008-x86-kvm-runtime-pv-single-target-ipi-in-tdx-guests.patch new file mode 100644 index 000000000..d02fc3763 --- /dev/null +++ b/os/yocto/layers/meta-dstack/recipes-kernel/linux/files/0008-x86-kvm-runtime-pv-single-target-ipi-in-tdx-guests.patch @@ -0,0 +1,110 @@ +From 0000000000000000000000000000000000000000 Mon Sep 17 00:00:00 2001 +From: Kevin Wang +Date: Mon, 14 Sep 2026 10:11:07 -0700 +Subject: [PATCH 8/8] x86/kvm: allow PV single-target function IPIs in TDX + guests at runtime + +In a TDX guest a single-target IPI written to the x2APIC ICR is emulated +through a #VE and a TDVMCALL. The KVM PV IPI hypercall avoids the #VE step, +but only the mask IPI callbacks use it; CALL_FUNCTION_SINGLE IPIs, which +carry cross-CPU wakeups, still go through the emulated ICR write. + +Add tdx_pv_single_ipi (boot parameter and +/sys/module/kernel/parameters/tdx_pv_single_ipi) that routes +smp_ops.send_call_func_single_ipi through the PV mask IPI path. Both +callbacks are built in and valid at any time, so the pointer is simply +switched. Reschedule, NMI and shorthand IPIs are unchanged. The option is +only accepted in a TDX guest and takes effect once PV IPIs are set up. It +is on by default in TDX guests unless the boot command line sets it. + +The benefit depends on how many wakeup IPIs the workload still sends: in +default 32-vCPU and 248-vCPU TDX guests it improved synchronization-heavy +benchmarks by about 32% and 10% (geometric mean), but with guest halt +polling and wake-queue batching it was neutral to slightly negative, so it +stays a runtime switch that guests which enable halt polling can turn off. + +Upstream-Status: Inappropriate [dstack TDX guest tuning, not submitted] +--- + arch/x86/kernel/kvm.c | 59 +++++++++++++++++++++++++++++++++++++++++++ + 1 file changed, 59 insertions(+) + +diff --git a/arch/x86/kernel/kvm.c b/arch/x86/kernel/kvm.c +index b67d7c59d..e3372ee3e 100644 +--- a/arch/x86/kernel/kvm.c ++++ b/arch/x86/kernel/kvm.c +@@ -561,6 +561,57 @@ static void kvm_send_ipi_mask(const struct cpumask *mask, int vector) + __send_ipi_mask(mask, vector); + } + ++/* ++ * A single-target x2APIC ICR write in a TDX guest is emulated through #VE and ++ * a TDVMCALL. Optionally send CALL_FUNCTION_SINGLE IPIs, which carry ++ * cross-CPU wakeups, through the PV IPI hypercall instead. Reschedule, NMI ++ * and shorthand IPIs are unchanged. On by default in TDX guests; it can be ++ * toggled at runtime through /sys/module/kernel/parameters/tdx_pv_single_ipi. ++ */ ++static bool tdx_pv_single_ipi; ++/* Set once the boot command line or sysfs chose a value explicitly. */ ++static bool tdx_pv_single_ipi_set; ++static bool kvm_pv_ipi_ready; ++static void (*kvm_native_single_ipi)(int cpu); ++ ++static void kvm_send_call_func_single_ipi(int cpu) ++{ ++ __send_ipi_mask(cpumask_of(cpu), CALL_FUNCTION_SINGLE_VECTOR); ++} ++ ++static void kvm_apply_pv_single_ipi(void) ++{ ++ /* Both callbacks are built in and valid at all times. */ ++ WRITE_ONCE(smp_ops.send_call_func_single_ipi, ++ READ_ONCE(tdx_pv_single_ipi) ? kvm_send_call_func_single_ipi : ++ kvm_native_single_ipi); ++} ++ ++static int param_set_tdx_pv_single_ipi(const char *val, ++ const struct kernel_param *kp) ++{ ++ bool enabled; ++ int ret; ++ ++ ret = kstrtobool(val ? val : "1", &enabled); ++ if (ret) ++ return ret; ++ if (enabled && !cpu_feature_enabled(X86_FEATURE_TDX_GUEST)) ++ return -EOPNOTSUPP; ++ WRITE_ONCE(tdx_pv_single_ipi, enabled); ++ tdx_pv_single_ipi_set = true; ++ if (READ_ONCE(kvm_pv_ipi_ready)) ++ kvm_apply_pv_single_ipi(); ++ return 0; ++} ++ ++static const struct kernel_param_ops tdx_pv_single_ipi_ops = { ++ .flags = KERNEL_PARAM_OPS_FL_NOARG, ++ .set = param_set_tdx_pv_single_ipi, ++ .get = param_get_bool, ++}; ++__core_param_cb(tdx_pv_single_ipi, &tdx_pv_single_ipi_ops, &tdx_pv_single_ipi, 0644); ++ + static void kvm_send_ipi_mask_allbutself(const struct cpumask *mask, int vector) + { + unsigned int this_cpu = smp_processor_id(); +@@ -629,6 +680,14 @@ static __init void kvm_setup_pv_ipi(void) + { + apic_update_callback(send_IPI_mask, kvm_send_ipi_mask); + apic_update_callback(send_IPI_mask_allbutself, kvm_send_ipi_mask_allbutself); ++ if (cpu_feature_enabled(X86_FEATURE_TDX_GUEST)) { ++ kvm_native_single_ipi = smp_ops.send_call_func_single_ipi; ++ /* On by default in TDX guests unless the command line chose. */ ++ if (!tdx_pv_single_ipi_set) ++ WRITE_ONCE(tdx_pv_single_ipi, true); ++ WRITE_ONCE(kvm_pv_ipi_ready, true); ++ kvm_apply_pv_single_ipi(); ++ } + pr_info("setup PV IPIs\n"); + } + +-- +2.43.0 + From f950ff901ed0b36d08590a77d1f09aa073fe8669 Mon Sep 17 00:00:00 2001 From: Kevin Wang Date: Mon, 14 Sep 2026 21:32:42 -0700 Subject: [PATCH 07/93] os: ship experimental tdx-guest-tune.sh helper Install a TDX guest scaling profile helper as /usr/lib/dstack/tdx-guest-tune.sh with throughput, default and off profiles. It is kept out of PATH because it may be removed later; the kernel switches remain the stable interface. --- os/common/rootfs/tdx-guest-tune.sh | 157 ++++++++++++++++++ .../dstack-rust/dstack-rust-build.sh | 4 + 2 files changed, 161 insertions(+) create mode 100755 os/common/rootfs/tdx-guest-tune.sh diff --git a/os/common/rootfs/tdx-guest-tune.sh b/os/common/rootfs/tdx-guest-tune.sh new file mode 100755 index 000000000..1ade51b7b --- /dev/null +++ b/os/common/rootfs/tdx-guest-tune.sh @@ -0,0 +1,157 @@ +#!/bin/sh +# SPDX-FileCopyrightText: © 2026 Phala Network +# +# SPDX-License-Identifier: Apache-2.0 + +# Runtime scaling policy for many-vCPU TDX guests (dstack kernel with +# tdx_wake_q_batch, tdx_pv_single_ipi and CONFIG_HALTPOLL_CPUIDLE=m). +# +# Experimental helper that may be removed from future images; the stable +# interface is the kernel switches it sets. See docs/tdx-guest-scaling.md. +# +# Usage: /usr/lib/dstack/tdx-guest-tune.sh [throughput|default|off|status] [options] +# throughput guest halt polling + wake-queue IPI batching (highest throughput, +# extra CPU while idle-polling; for vCPUs that are not overcommitted) +# default kernel defaults: no guest polling, wake-queue batching and PV +# single-target IPIs on ("nopoll" is accepted as an alias) +# off all switches off and polling module unloaded (pre-patch behavior) +# status print the current state +# Options: +# --poll-us N guest halt-poll window in microseconds (default: by vCPU count) +# --pv-single Y|N override tdx_pv_single_ipi for the selected profile +# --force apply even if this is not a TDX guest +# Environment: TDX_TUNE_PROFILE, TDX_TUNE_POLL_US, TDX_TUNE_PV_SINGLE. +set -eu + +KP=/sys/module/kernel/parameters +HP=/sys/module/haltpoll/parameters +IDLE=/sys/devices/system/cpu/cpuidle + +profile=${TDX_TUNE_PROFILE:-throughput} +[ "$profile" = nopoll ] && profile=default +poll_us=${TDX_TUNE_POLL_US:-} +pv_single=${TDX_TUNE_PV_SINGLE:-} +force=0 + +log() { echo "tdx-guest-tune: $*"; } +die() { echo "tdx-guest-tune: $*" >&2; exit 1; } + +while [ $# -gt 0 ]; do + case $1 in + throughput|default|off|status) profile=$1 ;; + nopoll) profile=default ;; + --poll-us) [ $# -ge 2 ] || die "--poll-us needs a value"; poll_us=$2; shift ;; + --pv-single) [ $# -ge 2 ] || die "--pv-single needs Y or N"; pv_single=$2; shift ;; + --force) force=1 ;; + -h|--help) sed -n '12,23p' "$0"; exit 0 ;; + *) die "unknown argument: $1" ;; + esac + shift +done + +readf() { cat "$1" 2>/dev/null || echo "-"; } + +status() { + driver=$(readf "$IDLE/current_driver") + governor=$(readf "$IDLE/current_governor") + loaded=no + grep -q '^cpuidle_haltpoll ' /proc/modules 2>/dev/null && loaded=yes + log "vcpus=$(nproc) tdx_guest=$(is_tdx && echo yes || echo no) idle_driver=$driver governor=$governor haltpoll_module=$loaded" + log "guest_halt_poll_ns=$(readf "$HP/guest_halt_poll_ns") allow_shrink=$(readf "$HP/guest_halt_poll_allow_shrink") grow_start=$(readf "$HP/guest_halt_poll_grow_start")" + log "tdx_wake_q_batch=$(readf "$KP/tdx_wake_q_batch") tdx_pv_single_ipi=$(readf "$KP/tdx_pv_single_ipi")" +} + +is_tdx() { + [ -e /dev/tdx_guest ] || grep -qw tdx_guest /proc/cpuinfo 2>/dev/null +} + +write() { # write FILE VALUE + [ -w "$1" ] || die "missing or read-only: $1 (kernel without the dstack TDX knobs?)" + printf '%s\n' "$2" > "$1" +} + +# Window derived from measurements: 32 vCPUs -> 200 us, 248 vCPUs -> 4 ms. +# Both are ~20-25x the TD exit round trip seen under that vCPU concurrency +# (about 9 us and 170 us); intermediate steps are interpolated, not measured. +default_poll_us() { + n=$(nproc) + if [ "$n" -le 32 ]; then echo 200 + elif [ "$n" -le 64 ]; then echo 500 + elif [ "$n" -le 128 ]; then echo 1000 + else echo 4000 + fi +} + +# Enable guest halt polling with a fixed, non-shrinking window. +enable_polling() { + ns=$(( $1 * 1000 )) + # Clamp any previously grown per-CPU limit: allow shrinking to zero on the + # next long idle, then set the new window before disabling shrink again. + write "$HP/guest_halt_poll_allow_shrink" Y + write "$HP/guest_halt_poll_shrink" 0 + write "$HP/guest_halt_poll_ns" 1000 + sleep 1 + write "$HP/guest_halt_poll_shrink" 2 + write "$HP/guest_halt_poll_grow_start" "$ns" + write "$HP/guest_halt_poll_ns" "$ns" + write "$HP/guest_halt_poll_allow_shrink" N + if [ "$(readf "$IDLE/current_driver")" != haltpoll ]; then + modprobe cpuidle_haltpoll force=1 || + die "cannot load cpuidle_haltpoll (image without CONFIG_HALTPOLL_CPUIDLE=m?)" + fi +} + +# Restore the haltpoll governor parameters to their kernel defaults. +reset_polling() { + write "$HP/guest_halt_poll_allow_shrink" Y + write "$HP/guest_halt_poll_grow_start" 50000 + write "$HP/guest_halt_poll_ns" 200000 +} + +disable_polling() { + if grep -q '^cpuidle_haltpoll ' /proc/modules 2>/dev/null; then + rmmod cpuidle_haltpoll + elif [ "$(readf "$IDLE/current_driver")" = haltpoll ]; then + log "warning: haltpoll is built in or bound at boot; only shortening its window" + write "$HP/guest_halt_poll_allow_shrink" Y + write "$HP/guest_halt_poll_ns" 50000 + fi +} + +case $profile in + status) status; exit 0 ;; + throughput|default|off) ;; + *) die "unknown profile: $profile" ;; +esac + +[ "$(id -u)" -eq 0 ] || die "must run as root" +if ! is_tdx && [ "$force" -ne 1 ]; then + die "not a TDX guest; use --force to apply anyway" +fi +case ${pv_single:-} in ""|Y|N) ;; *) die "--pv-single must be Y or N" ;; esac +case ${poll_us:-} in ""|*[!0-9]*) [ -z "${poll_us:-}" ] || die "--poll-us must be an integer" ;; esac + +case $profile in + throughput) + [ -n "${poll_us:-}" ] || poll_us=$(default_poll_us) + enable_polling "$poll_us" + write "$KP/tdx_wake_q_batch" Y + # With polling and batching, PV single-target IPIs measured -5%..+5%. + write "$KP/tdx_pv_single_ipi" "${pv_single:-N}" + ;; + default) + disable_polling + reset_polling + write "$KP/tdx_wake_q_batch" Y + # Without polling most wakeups need an IPI; PV single IPIs measured +10%..+32%. + write "$KP/tdx_pv_single_ipi" "${pv_single:-Y}" + ;; + off) + disable_polling + reset_polling + write "$KP/tdx_wake_q_batch" N + write "$KP/tdx_pv_single_ipi" "${pv_single:-N}" + ;; +esac +log "applied profile: $profile" +status diff --git a/os/mkosi/components/dstack-rust/dstack-rust-build.sh b/os/mkosi/components/dstack-rust/dstack-rust-build.sh index 32b167edc..4d3a66e74 100755 --- a/os/mkosi/components/dstack-rust/dstack-rust-build.sh +++ b/os/mkosi/components/dstack-rust/dstack-rust-build.sh @@ -12,6 +12,10 @@ install -d "$DEST/usr/bin" "$DEST/usr/lib/systemd/system" \ for s in dstack-prepare ephemeral-docker app-compose; do install -m0755 "$ROOT/os/common/rootfs/$s.sh" "$DEST/usr/bin/$s.sh" done +# Experimental helper, kept out of PATH so it is not mistaken for a stable +# command; see docs/tdx-guest-scaling.md. +install -D -m0755 "$ROOT/os/common/rootfs/tdx-guest-tune.sh" \ + "$DEST/usr/lib/dstack/tdx-guest-tune.sh" install -m0644 "$ROOT/os/common/rootfs/"*.service \ "$ROOT/os/common/rootfs/dstack-guest-agent.socket" "$DEST/usr/lib/systemd/system/" install -m0644 "$ROOT/os/common/rootfs/journald.conf" "$DEST/etc/systemd/journald.conf.d/dstack.conf" From e4ff854291f284cb388ff94dfdd9a7b4e30b883a Mon Sep 17 00:00:00 2001 From: Kevin Wang Date: Mon, 14 Sep 2026 21:32:42 -0700 Subject: [PATCH 08/93] docs: describe TDX guest scaling knobs and profiles --- docs/tdx-guest-scaling.md | 91 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 91 insertions(+) create mode 100644 docs/tdx-guest-scaling.md diff --git a/docs/tdx-guest-scaling.md b/docs/tdx-guest-scaling.md new file mode 100644 index 000000000..93f37982b --- /dev/null +++ b/docs/tdx-guest-scaling.md @@ -0,0 +1,91 @@ +# TDX guest scaling knobs + +Many-vCPU TDX guests can run synchronization-heavy workloads (thread pools, +PyTorch CPU operators, `docker pull` extraction) several times slower than the +same guest without TDX. Most of the gap is wakeup overhead: every IPI and every +idle HLT costs a TD exit and a TD entry, which is several times more expensive +than an ordinary VM exit, and on large guests the cost grows further when many +vCPUs exit at the same time. + +The dstack guest kernel has three runtime switches for this. + +| Switch | Default | Where | Effect | +| --- | --- | --- | --- | +| `tdx_wake_q_batch` | on in TDX guests | `/sys/module/kernel/parameters/tdx_wake_q_batch` (also a boot parameter) | `wake_up_q()` (futex wake, condition variable broadcast, OpenMP pools) publishes wakeups immediately but sends the notification IPIs for up to 8 wakees as one mask IPI. Enabled at the end of boot. | +| `tdx_pv_single_ipi` | on in TDX guests | `/sys/module/kernel/parameters/tdx_pv_single_ipi` (also a boot parameter) | Single-target function IPIs use the KVM PV IPI hypercall instead of an emulated x2APIC ICR write. | +| `cpuidle_haltpoll` module | not loaded | `modprobe cpuidle_haltpoll force=1` / `rmmod cpuidle_haltpoll` | Idle vCPUs poll in the guest before halting. Wakeups that arrive during the window need neither an IPI nor a TD exit, at the cost of CPU spent polling. | + +The two kernel switches cannot be enabled outside a TDX guest, so ordinary VMs +are unaffected. Boot with `tdx_wake_q_batch=0 tdx_pv_single_ipi=0` +(or write `N` to the sysfs files) to get the previous behavior. + +The haltpoll governor parameters stay available under +`/sys/module/haltpoll/parameters/`. For TDX, a fixed window that does not +shrink works best (`guest_halt_poll_allow_shrink=N`, +`guest_halt_poll_grow_start` equal to `guest_halt_poll_ns`); the default +adaptive window tends to collapse to zero under slow wakeups and stay there. + +## Profiles + +| Profile | Settings | When | +| --- | --- | --- | +| default | kernel defaults: no guest polling, `tdx_wake_q_batch=Y`, `tdx_pv_single_ipi=Y` | all guests; no extra idle CPU | +| throughput | haltpoll module with a non-shrinking window, `tdx_wake_q_batch=Y`, `tdx_pv_single_ipi=N` | vCPUs are not overcommitted and throughput matters more than idle CPU | +| off | all switches off | reference behavior before these patches | + +Measured poll windows for `throughput`: 200 µs for a 32-vCPU guest and 4 ms +for a 248-vCPU guest. Both are roughly 20-25 times the TD exit round trip +observed at that vCPU concurrency; values in between are interpolations. + +Polling costs CPU while vCPUs idle-poll. Compared with an ordinary VM running +the same image, a `throughput` TD used about 0.15-0.2 extra host cores at idle +or at 200 requests per second, independent of vCPU count (4-32 vCPUs), and +about 0.9 cores more than `default` at 32 vCPUs with every vCPU 5% busy. The +`default` profile stayed within 0.02 cores of `off` in the same tests. + +## Helper script + +The image ships `/usr/lib/dstack/tdx-guest-tune.sh`, which applies one of the +profiles above, picks the poll window from the vCPU count, and prints the +resulting state: + +```bash +/usr/lib/dstack/tdx-guest-tune.sh throughput # or: default, off, status +/usr/lib/dstack/tdx-guest-tune.sh throughput --poll-us 1000 --pv-single Y +``` + +It refuses to run outside a TDX guest unless `--force` is given, and fails with +a message if the kernel lacks one of the switches. + +The script is experimental and may be removed from future images; the kernel +switches above are the stable interface. It is outside `PATH` for that reason. +When calling it from an app's `pre_launch_script`, guard the call so the app +still starts on images without the script (the pre-launch script runs under +`set -e`): + +```bash +t=/usr/lib/dstack/tdx-guest-tune.sh +if [ -x "$t" ]; then "$t" throughput || echo "tdx-guest-tune failed, continuing"; fi +``` + +## Observed results + +Emerald Rapids host, 32-vCPU guest, the same image as TD and as an ordinary VM, +switches changed at runtime on one TD boot, two alternating rounds (MiB/s; docker +in seconds): + +| Workload | Ordinary VM | TD `off` | TD `default` | +| --- | ---: | ---: | ---: | +| pool 1x32 | 2063 | 78 | 644 | +| PyTorch PASSIVE 1x32 | 2976 | 61 | 460 | +| pool 8x32 | 13137 | 2071 | 3852 | +| pool 4x8 | 37989 | 3612 | 12746 | +| PyTorch PASSIVE 8x32, 2 ms sleep | 784 | 478 | 656 | +| pool 1x4 | 11056 | 2129 | 5082 | +| `docker pull python:3.12`, local registry (s) | 29.6 | 48.0 | 43.5 | +| Geometric mean vs ordinary VM | 100% | 13.6% | 38.5% | + +`default` was not slower than `off` on any workload, and timer and request +wakeup latency stayed comparable. The `throughput` profile closes more of the +remaining gap on wakeup-heavy workloads in exchange for polling CPU. Results +depend on workload shape. From 38cb94bc1637e3d7f7873c02ab0f672ec971d1d3 Mon Sep 17 00:00:00 2001 From: Kevin Wang Date: Mon, 14 Sep 2026 20:31:54 -0700 Subject: [PATCH 09/93] os/kernel: stop accepting 4 MiB per allocation before watermarks are set --- os/mkosi/components/kernel/kernel-build.sh | 3 +- os/mkosi/components/kernel/kernel.sh | 3 +- ...p-accepted-reserve-before-watermarks.patch | 135 ++++++++++++++++++ 3 files changed, 139 insertions(+), 2 deletions(-) create mode 100644 os/yocto/layers/meta-dstack/recipes-kernel/linux/files/0009-mm-page-alloc-keep-accepted-reserve-before-watermarks.patch diff --git a/os/mkosi/components/kernel/kernel-build.sh b/os/mkosi/components/kernel/kernel-build.sh index dfad7ab58..4917a529c 100755 --- a/os/mkosi/components/kernel/kernel-build.sh +++ b/os/mkosi/components/kernel/kernel-build.sh @@ -58,7 +58,8 @@ for patch in \ "$ROOT/os/yocto/layers/meta-dstack/recipes-kernel/linux/files/0005-swiotlb-preserve-allocation-virtual-address.patch" \ "$ROOT/os/yocto/layers/meta-dstack/recipes-kernel/linux/files/0006-swiotlb-free-dynamic-pools-from-process-context.patch" \ "$ROOT/os/yocto/layers/meta-dstack/recipes-kernel/linux/files/0007-sched-smp-batch-wake-q-ipis-in-tdx-guests.patch" \ - "$ROOT/os/yocto/layers/meta-dstack/recipes-kernel/linux/files/0008-x86-kvm-runtime-pv-single-target-ipi-in-tdx-guests.patch"; do + "$ROOT/os/yocto/layers/meta-dstack/recipes-kernel/linux/files/0008-x86-kvm-runtime-pv-single-target-ipi-in-tdx-guests.patch" \ + "$ROOT/os/yocto/layers/meta-dstack/recipes-kernel/linux/files/0009-mm-page-alloc-keep-accepted-reserve-before-watermarks.patch"; do patch -d "$src" -p1 --fuzz=0 < "$patch" done diff --git a/os/mkosi/components/kernel/kernel.sh b/os/mkosi/components/kernel/kernel.sh index e02749e0d..1ad75df73 100644 --- a/os/mkosi/components/kernel/kernel.sh +++ b/os/mkosi/components/kernel/kernel.sh @@ -15,7 +15,8 @@ component_cache_key() { "$ROOT/os/yocto/layers/meta-dstack/recipes-kernel/linux/files/0005-swiotlb-preserve-allocation-virtual-address.patch" \ "$ROOT/os/yocto/layers/meta-dstack/recipes-kernel/linux/files/0006-swiotlb-free-dynamic-pools-from-process-context.patch" \ "$ROOT/os/yocto/layers/meta-dstack/recipes-kernel/linux/files/0007-sched-smp-batch-wake-q-ipis-in-tdx-guests.patch" \ - "$ROOT/os/yocto/layers/meta-dstack/recipes-kernel/linux/files/0008-x86-kvm-runtime-pv-single-target-ipi-in-tdx-guests.patch" + "$ROOT/os/yocto/layers/meta-dstack/recipes-kernel/linux/files/0008-x86-kvm-runtime-pv-single-target-ipi-in-tdx-guests.patch" \ + "$ROOT/os/yocto/layers/meta-dstack/recipes-kernel/linux/files/0009-mm-page-alloc-keep-accepted-reserve-before-watermarks.patch" key_tools gcc ld make pahole key_packages binutils dwarves bc bison flex libssl-dev libelf-dev } diff --git a/os/yocto/layers/meta-dstack/recipes-kernel/linux/files/0009-mm-page-alloc-keep-accepted-reserve-before-watermarks.patch b/os/yocto/layers/meta-dstack/recipes-kernel/linux/files/0009-mm-page-alloc-keep-accepted-reserve-before-watermarks.patch new file mode 100644 index 000000000..ae02c62f3 --- /dev/null +++ b/os/yocto/layers/meta-dstack/recipes-kernel/linux/files/0009-mm-page-alloc-keep-accepted-reserve-before-watermarks.patch @@ -0,0 +1,135 @@ +From 0000000000000000000000000000000000000000 Mon Sep 17 00:00:00 2001 +From: Kevin Wang +Date: Mon, 14 Sep 2026 20:30:52 -0700 +Subject: [PATCH 09/10] mm/page_alloc: keep a small accepted reserve before + watermarks are initialized + +Until init_per_zone_wmark_min() runs as a postcore initcall, the zone +watermarks are zero and cond_accept_memory() accepts one MAX_ORDER page +whenever it is called. get_page_from_freelist() and alloc_pages_bulk() +call it before every allocation, so each page allocation made during +early boot, including all of smp_init(), accepts another 4 MiB even +when the zone has plenty of accepted free memory. + +This is expensive for TDX guests on hosts that map private memory at +4K only. Accepting 4 MiB then takes 1024 TDG.MEM.PAGE.ACCEPT calls, each +one a TD exit and a TDH.MEM.PAGE.AUG. The number of allocations grows +with the number of CPUs brought up, and each exit gets slower as more +idle vCPUs take the periodic tick. On a Granite Rapids host running a +7.0 kernel, a 248-vCPU, 32 GiB TD accepts about 13 GiB during SMP +bring-up alone and needs 575-680 seconds to reach init. + +Before the watermarks are initialized, skip the proactive accept while +the zone still has at least two MAX_ORDER pages of accepted free memory. +Callers that retry after a failed watermark check or a failed allocation +pass force=true and still accept one MAX_ORDER page, so an allocation +that cannot be served from accepted memory makes progress as before. +Nothing changes once the watermarks are initialized. + +With this change the same TD brings up all 248 CPUs in about 2 seconds +and reaches init in 114 seconds. +--- + mm/page_alloc.c | 30 +++++++++++++++++++----------- + 1 file changed, 19 insertions(+), 11 deletions(-) + +diff --git a/mm/page_alloc.c b/mm/page_alloc.c +index 775e02b79..08d411add 100644 +--- a/mm/page_alloc.c ++++ b/mm/page_alloc.c +@@ -318,7 +318,7 @@ EXPORT_SYMBOL(nr_online_nodes); + + static bool page_contains_unaccepted(struct page *page, unsigned int order); + static bool cond_accept_memory(struct zone *zone, unsigned int order, +- int alloc_flags); ++ int alloc_flags, bool force); + static bool __free_unaccepted(struct page *page); + + int page_group_by_mobility_disabled __read_mostly; +@@ -3854,7 +3854,7 @@ get_page_from_freelist(gfp_t gfp_mask, unsigned int order, int alloc_flags, + continue; + } + +- cond_accept_memory(zone, order, alloc_flags); ++ cond_accept_memory(zone, order, alloc_flags, false); + + /* + * Detect whether the number of free pages is below high +@@ -3881,7 +3881,7 @@ get_page_from_freelist(gfp_t gfp_mask, unsigned int order, int alloc_flags, + gfp_mask)) { + int ret; + +- if (cond_accept_memory(zone, order, alloc_flags)) ++ if (cond_accept_memory(zone, order, alloc_flags, true)) + goto try_this_zone; + + /* +@@ -3934,7 +3934,7 @@ get_page_from_freelist(gfp_t gfp_mask, unsigned int order, int alloc_flags, + + return page; + } else { +- if (cond_accept_memory(zone, order, alloc_flags)) ++ if (cond_accept_memory(zone, order, alloc_flags, true)) + goto try_this_zone; + + /* Try again if zone has deferred pages */ +@@ -5118,7 +5118,7 @@ unsigned long alloc_pages_bulk_noprof(gfp_t gfp, int preferred_nid, + goto failed; + } + +- cond_accept_memory(zone, 0, alloc_flags); ++ cond_accept_memory(zone, 0, alloc_flags, false); + retry_this_zone: + mark = wmark_pages(zone, alloc_flags & ALLOC_WMARK_MASK) + nr_pages; + if (zone_watermark_fast(zone, 0, mark, +@@ -5127,7 +5127,7 @@ unsigned long alloc_pages_bulk_noprof(gfp_t gfp, int preferred_nid, + break; + } + +- if (cond_accept_memory(zone, 0, alloc_flags)) ++ if (cond_accept_memory(zone, 0, alloc_flags, true)) + goto retry_this_zone; + + /* Try again if zone has deferred pages */ +@@ -7522,7 +7522,7 @@ static bool try_to_accept_memory_one(struct zone *zone) + } + + static bool cond_accept_memory(struct zone *zone, unsigned int order, +- int alloc_flags) ++ int alloc_flags, bool force) + { + long to_accept, wmark; + bool ret = false; +@@ -7539,10 +7539,18 @@ static bool cond_accept_memory(struct zone *zone, unsigned int order, + /* + * Watermarks have not been initialized yet. + * +- * Accepting one MAX_ORDER page to ensure progress. +- */ +- if (!wmark) ++ * Keep a small reserve of accepted free memory instead of accepting ++ * one MAX_ORDER page on every allocation. Accept one MAX_ORDER page ++ * when the allocation could not be satisfied, to ensure progress. ++ */ ++ if (!wmark) { ++ if (!force && ++ zone_page_state(zone, NR_FREE_PAGES) - ++ zone_page_state(zone, NR_UNACCEPTED) >= ++ 2 * MAX_ORDER_NR_PAGES) ++ return false; + return try_to_accept_memory_one(zone); ++ } + + /* How much to accept to get to promo watermark? */ + to_accept = wmark - +@@ -7586,7 +7594,7 @@ static bool page_contains_unaccepted(struct page *page, unsigned int order) + } + + static bool cond_accept_memory(struct zone *zone, unsigned int order, +- int alloc_flags) ++ int alloc_flags, bool force) + { + return false; + } +-- +2.43.0 + From 2a3cc497e1c637acce2e121f11a9938554bb9473 Mon Sep 17 00:00:00 2001 From: Kevin Wang Date: Mon, 14 Sep 2026 20:31:54 -0700 Subject: [PATCH 10/93] os/kernel: convert the TDX SWIOTLB buffer to shared after SMP bring-up --- os/mkosi/components/kernel/kernel-build.sh | 3 +- os/mkosi/components/kernel/kernel.sh | 3 +- ...-swiotlb-to-shared-after-smp-bringup.patch | 78 +++++++++++++++++++ 3 files changed, 82 insertions(+), 2 deletions(-) create mode 100644 os/yocto/layers/meta-dstack/recipes-kernel/linux/files/0010-x86-tdx-convert-swiotlb-to-shared-after-smp-bringup.patch diff --git a/os/mkosi/components/kernel/kernel-build.sh b/os/mkosi/components/kernel/kernel-build.sh index 4917a529c..c9031e71f 100755 --- a/os/mkosi/components/kernel/kernel-build.sh +++ b/os/mkosi/components/kernel/kernel-build.sh @@ -59,7 +59,8 @@ for patch in \ "$ROOT/os/yocto/layers/meta-dstack/recipes-kernel/linux/files/0006-swiotlb-free-dynamic-pools-from-process-context.patch" \ "$ROOT/os/yocto/layers/meta-dstack/recipes-kernel/linux/files/0007-sched-smp-batch-wake-q-ipis-in-tdx-guests.patch" \ "$ROOT/os/yocto/layers/meta-dstack/recipes-kernel/linux/files/0008-x86-kvm-runtime-pv-single-target-ipi-in-tdx-guests.patch" \ - "$ROOT/os/yocto/layers/meta-dstack/recipes-kernel/linux/files/0009-mm-page-alloc-keep-accepted-reserve-before-watermarks.patch"; do + "$ROOT/os/yocto/layers/meta-dstack/recipes-kernel/linux/files/0009-mm-page-alloc-keep-accepted-reserve-before-watermarks.patch" \ + "$ROOT/os/yocto/layers/meta-dstack/recipes-kernel/linux/files/0010-x86-tdx-convert-swiotlb-to-shared-after-smp-bringup.patch"; do patch -d "$src" -p1 --fuzz=0 < "$patch" done diff --git a/os/mkosi/components/kernel/kernel.sh b/os/mkosi/components/kernel/kernel.sh index 1ad75df73..40dc02848 100644 --- a/os/mkosi/components/kernel/kernel.sh +++ b/os/mkosi/components/kernel/kernel.sh @@ -16,7 +16,8 @@ component_cache_key() { "$ROOT/os/yocto/layers/meta-dstack/recipes-kernel/linux/files/0006-swiotlb-free-dynamic-pools-from-process-context.patch" \ "$ROOT/os/yocto/layers/meta-dstack/recipes-kernel/linux/files/0007-sched-smp-batch-wake-q-ipis-in-tdx-guests.patch" \ "$ROOT/os/yocto/layers/meta-dstack/recipes-kernel/linux/files/0008-x86-kvm-runtime-pv-single-target-ipi-in-tdx-guests.patch" \ - "$ROOT/os/yocto/layers/meta-dstack/recipes-kernel/linux/files/0009-mm-page-alloc-keep-accepted-reserve-before-watermarks.patch" + "$ROOT/os/yocto/layers/meta-dstack/recipes-kernel/linux/files/0009-mm-page-alloc-keep-accepted-reserve-before-watermarks.patch" \ + "$ROOT/os/yocto/layers/meta-dstack/recipes-kernel/linux/files/0010-x86-tdx-convert-swiotlb-to-shared-after-smp-bringup.patch" key_tools gcc ld make pahole key_packages binutils dwarves bc bison flex libssl-dev libelf-dev } diff --git a/os/yocto/layers/meta-dstack/recipes-kernel/linux/files/0010-x86-tdx-convert-swiotlb-to-shared-after-smp-bringup.patch b/os/yocto/layers/meta-dstack/recipes-kernel/linux/files/0010-x86-tdx-convert-swiotlb-to-shared-after-smp-bringup.patch new file mode 100644 index 000000000..396cdea60 --- /dev/null +++ b/os/yocto/layers/meta-dstack/recipes-kernel/linux/files/0010-x86-tdx-convert-swiotlb-to-shared-after-smp-bringup.patch @@ -0,0 +1,78 @@ +From 0000000000000000000000000000000000000000 Mon Sep 17 00:00:00 2001 +From: Kevin Wang +Date: Mon, 14 Sep 2026 20:30:52 -0700 +Subject: [PATCH 10/10] x86/tdx: convert the SWIOTLB buffer to shared after SMP + bring-up + +mem_encrypt_init() converts the default SWIOTLB buffer from private to +shared before SMP bring-up. The buffer lives in memory the firmware has +already accepted, so the VMM has to remove every private page backing +it. For each removed page the VMM forces all vCPUs out of the TD, and at +this point the not-yet-woken APs busy-wait inside the TD in firmware, +so every removal waits for a TD exit from every AP. With 248 vCPUs the +default 1 GiB buffer takes about 68 seconds to convert. + +Defer the conversion for TDX guests to fs_initcall_sync. By then the +APs are online, idle in HLT outside the TD, and have stopped the +periodic tick because clocksource_done_booting() enabled high +resolution timers. Converting right after smp_init() is not enough: the +idle APs still take the jiffies tick and re-enter the TD about 1000 +times per second each, and the conversion took over 200 seconds. No +device maps the bounce buffer before device probing starts at +device_initcall. Other memory encryption platforms are unchanged. + +With this change the conversion takes about 6 seconds and the 248-vCPU +TD reaches init in 39-51 seconds. +--- + arch/x86/mm/mem_encrypt.c | 29 +++++++++++++++++++++++++++-- + 1 file changed, 27 insertions(+), 2 deletions(-) + +diff --git a/arch/x86/mm/mem_encrypt.c b/arch/x86/mm/mem_encrypt.c +index 95bae74fd..f405c4123 100644 +--- a/arch/x86/mm/mem_encrypt.c ++++ b/arch/x86/mm/mem_encrypt.c +@@ -85,14 +85,39 @@ static void print_mem_encrypt_feature_info(void) + } + } + ++/* ++ * Converting the SWIOTLB buffer to shared makes the VMM remove every private ++ * page it backs. For each page the VMM has to force all vCPUs out of the TD, ++ * and before SMP bring-up the not-yet-woken APs are busy-waiting inside the ++ * TD in firmware, so every removal waits for an exit from every AP. With ++ * hundreds of vCPUs this takes minutes for the default 1 GiB buffer. Once ++ * the APs are online they idle in HLT outside the TD and the removal is ++ * cheap only after they have also stopped the periodic tick, which happens ++ * once clocksource_done_booting() (fs_initcall) enables high resolution ++ * timers and NOHZ. No device uses the bounce buffer before device probing ++ * starts, so do the conversion from fs_initcall_sync. ++ */ ++static int __init tdx_swiotlb_update_mem_attributes(void) ++{ ++ if (cpu_feature_enabled(X86_FEATURE_TDX_GUEST)) ++ swiotlb_update_mem_attributes(); ++ return 0; ++} ++fs_initcall_sync(tdx_swiotlb_update_mem_attributes); ++ + /* Architecture __weak replacement functions */ + void __init mem_encrypt_init(void) + { + if (!cc_platform_has(CC_ATTR_MEM_ENCRYPT)) + return; + +- /* Call into SWIOTLB to update the SWIOTLB DMA buffers */ +- swiotlb_update_mem_attributes(); ++ /* ++ * Call into SWIOTLB to update the SWIOTLB DMA buffers. TDX guests ++ * defer this until secondary CPUs are online, see ++ * tdx_swiotlb_update_mem_attributes(). ++ */ ++ if (!cpu_feature_enabled(X86_FEATURE_TDX_GUEST)) ++ swiotlb_update_mem_attributes(); + + snp_secure_tsc_prepare(); + +-- +2.43.0 + From 50d4f066715de9b8e085e38192658e8dd4924f84 Mon Sep 17 00:00:00 2001 From: Kevin Wang Date: Mon, 14 Sep 2026 20:32:14 -0700 Subject: [PATCH 11/93] docs: describe the large TDX guest boot-time fixes --- docs/tdx-guest-scaling.md | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) diff --git a/docs/tdx-guest-scaling.md b/docs/tdx-guest-scaling.md index 93f37982b..879c7bd92 100644 --- a/docs/tdx-guest-scaling.md +++ b/docs/tdx-guest-scaling.md @@ -68,6 +68,22 @@ t=/usr/lib/dstack/tdx-guest-tune.sh if [ -x "$t" ]; then "$t" throughput || echo "tdx-guest-tune failed, continuing"; fi ``` +## Boot time of large guests + +Two always-on guest kernel patches keep the boot time of many-vCPU TDs from +growing with the square of the vCPU count on hosts that map TD private memory +at 4K only (upstream TDX KVM, 7.0): + +- `0009` stops the page allocator from accepting 4 MiB of unaccepted memory on + every allocation before the zone watermarks are initialized. Before, SMP + bring-up of a 248-vCPU TD accepted about 13 GiB one 4K page at a time. +- `0010` converts the default SWIOTLB buffer to shared at `fs_initcall_sync` + instead of before SMP bring-up. Before, the host had to force every + still-spinning AP out of the TD for each of the buffer's 262144 pages. + +A 248-vCPU, 32 GiB TD on a Granite Rapids host with a 7.0 kernel reaches init in +about 40-50 seconds instead of 575-680 seconds. The SWIOTLB size is unchanged. + ## Observed results Emerald Rapids host, 32-vCPU guest, the same image as TD and as an ordinary VM, From 70fdc28b54b905afd4fb11412e65e7c6ebd9da49 Mon Sep 17 00:00:00 2001 From: Kevin Wang Date: Mon, 14 Sep 2026 20:55:48 -0700 Subject: [PATCH 12/93] os/kernel: warn on SWIOTLB bounce before the default pool is shared Patch 0010 defers the private-to-shared conversion of the default SWIOTLB pool to fs_initcall_sync on TDX guests. A bounce through the pool before that would fail silently: the device reads a buffer it cannot decrypt. Record in SWIOTLB when swiotlb_update_mem_attributes() has run and WARN_ON_ONCE() in swiotlb_tbl_map_single() if the default pool is used before that on a platform with memory encryption, so a future early DMA user is caught instead of corrupting data. The flag is __ro_after_init; it is written from __init code before mark_rodata_ro(). Patches 0001-0010 still apply to pristine 6.18.40 with --fuzz=0, and swiotlb.o and mem_encrypt.o build with W=1 without new warnings. --- ...-swiotlb-to-shared-after-smp-bringup.patch | 69 +++++++++++++++++-- 1 file changed, 64 insertions(+), 5 deletions(-) diff --git a/os/yocto/layers/meta-dstack/recipes-kernel/linux/files/0010-x86-tdx-convert-swiotlb-to-shared-after-smp-bringup.patch b/os/yocto/layers/meta-dstack/recipes-kernel/linux/files/0010-x86-tdx-convert-swiotlb-to-shared-after-smp-bringup.patch index 396cdea60..896275a2d 100644 --- a/os/yocto/layers/meta-dstack/recipes-kernel/linux/files/0010-x86-tdx-convert-swiotlb-to-shared-after-smp-bringup.patch +++ b/os/yocto/layers/meta-dstack/recipes-kernel/linux/files/0010-x86-tdx-convert-swiotlb-to-shared-after-smp-bringup.patch @@ -21,17 +21,25 @@ times per second each, and the conversion took over 200 seconds. No device maps the bounce buffer before device probing starts at device_initcall. Other memory encryption platforms are unchanged. +A bounce through the still-private default pool would fail silently: +the device reads a buffer it cannot decrypt. Record in SWIOTLB when +swiotlb_update_mem_attributes() has run and WARN_ON_ONCE() in +swiotlb_tbl_map_single() if the default pool is used before that on a +platform with memory encryption, so a future early DMA user is caught +instead of corrupting data. + With this change the conversion takes about 6 seconds and the 248-vCPU TD reaches init in 39-51 seconds. --- - arch/x86/mm/mem_encrypt.c | 29 +++++++++++++++++++++++++++-- - 1 file changed, 27 insertions(+), 2 deletions(-) + arch/x86/mm/mem_encrypt.c | 30 ++++++++++++++++++++++++++++-- + kernel/dma/swiotlb.c | 25 +++++++++++++++++++++---- + 2 files changed, 49 insertions(+), 6 deletions(-) diff --git a/arch/x86/mm/mem_encrypt.c b/arch/x86/mm/mem_encrypt.c -index 95bae74fd..f405c4123 100644 +index 95bae74fd..48618d2dd 100644 --- a/arch/x86/mm/mem_encrypt.c +++ b/arch/x86/mm/mem_encrypt.c -@@ -85,14 +85,39 @@ static void print_mem_encrypt_feature_info(void) +@@ -85,14 +85,40 @@ static void print_mem_encrypt_feature_info(void) } } @@ -45,7 +53,8 @@ index 95bae74fd..f405c4123 100644 + * cheap only after they have also stopped the periodic tick, which happens + * once clocksource_done_booting() (fs_initcall) enables high resolution + * timers and NOHZ. No device uses the bounce buffer before device probing -+ * starts, so do the conversion from fs_initcall_sync. ++ * starts, so do the conversion from fs_initcall_sync. swiotlb_tbl_map_single() ++ * warns if a bounce through the default pool ever happens before that. + */ +static int __init tdx_swiotlb_update_mem_attributes(void) +{ @@ -73,6 +82,56 @@ index 95bae74fd..f405c4123 100644 snp_secure_tsc_prepare(); +diff --git a/kernel/dma/swiotlb.c b/kernel/dma/swiotlb.c +index fa7377b01..bc0900f1d 100644 +--- a/kernel/dma/swiotlb.c ++++ b/kernel/dma/swiotlb.c +@@ -250,6 +250,13 @@ static inline unsigned long nr_slots(u64 val) + return DIV_ROUND_UP(val, IO_TLB_SIZE); + } + ++/* ++ * Set once swiotlb_update_mem_attributes() has run. On platforms with memory ++ * encryption the default pool is private until then, so a bounce through it ++ * would hand the device a buffer it cannot read and fail silently. ++ */ ++static bool swiotlb_default_pool_decrypted __ro_after_init; ++ + /* + * Early SWIOTLB allocation may be too early to allow an architecture to + * perform the desired operations. This function allows the architecture to +@@ -261,10 +268,12 @@ void __init swiotlb_update_mem_attributes(void) + struct io_tlb_pool *mem = &io_tlb_default_mem.defpool; + unsigned long bytes; + +- if (!mem->nslabs || mem->late_alloc) +- return; +- bytes = PAGE_ALIGN(mem->nslabs << IO_TLB_SHIFT); +- set_memory_decrypted((unsigned long)mem->vaddr, bytes >> PAGE_SHIFT); ++ if (mem->nslabs && !mem->late_alloc) { ++ bytes = PAGE_ALIGN(mem->nslabs << IO_TLB_SHIFT); ++ set_memory_decrypted((unsigned long)mem->vaddr, ++ bytes >> PAGE_SHIFT); ++ } ++ swiotlb_default_pool_decrypted = true; + } + + static void swiotlb_init_io_tlb_pool(struct io_tlb_pool *mem, phys_addr_t start, +@@ -1435,6 +1444,14 @@ phys_addr_t swiotlb_tbl_map_single(struct device *dev, phys_addr_t orig_addr, + return (phys_addr_t)DMA_MAPPING_ERROR; + } + ++ /* ++ * Catch a bounce through the default pool before it has been made ++ * shared with the device, see swiotlb_update_mem_attributes(). ++ */ ++ WARN_ON_ONCE(pool == &io_tlb_default_mem.defpool && ++ !swiotlb_default_pool_decrypted && ++ cc_platform_has(CC_ATTR_MEM_ENCRYPT)); ++ + /* + * If dma_skip_sync was set, reset it on first SWIOTLB buffer + * mapping to always sync SWIOTLB buffers. -- 2.43.0 From 2eb9f7f32e7fc6ebdea4414ff4b686c5747700dc Mon Sep 17 00:00:00 2001 From: Leechael Yim Date: Fri, 14 Aug 2026 19:49:34 +0800 Subject: [PATCH 13/93] fix(vmm): sanitize GPUs via VFIO hot reset instead of sysfs SBR The sanitize-on-attach path issued the Secondary Bus Reset by writing Bridge Control in the upstream bridge sysfs config space and re-probed devices through /sys/bus/pci/drivers_probe. Both files are writable by root only, so the feature could not be enabled in production where dstack-vmm runs as an unprivileged user with no sudo. Switch to the VFIO_DEVICE_PCI_HOT_RESET ioctl, which makes the kernel perform the same bus reset. The ioctl is authorized by device ownership rather than privilege: the caller presents fds for every VFIO group affected by the reset, and the /dev/vfio group nodes are the same ones QEMU opens to attach the GPU, so the VMM user already has access. A single group fd suffices because every sanitized GPU sits alone behind a dedicated PCIe bridge and alone in its IOMMU group. The bridge topology check is kept as defense, and the kernel-reported set of affected devices must all belong to the GPU own group or the launch is aborted. Devices stay bound to vfio-pci across the reset, so the drivers_probe re-probe logic is no longer needed and is removed. Not yet validated on GPU hardware; see plans/2026-08-14-vfio-gpu-hot-reset.md for the pending experiment. --- dstack/vmm/src/gpu_reset.rs | 352 ++++++++++++++++++++++++++++++------ dstack/vmm/vmm.toml | 5 +- 2 files changed, 295 insertions(+), 62 deletions(-) diff --git a/dstack/vmm/src/gpu_reset.rs b/dstack/vmm/src/gpu_reset.rs index e58876b9a..f35afb238 100644 --- a/dstack/vmm/src/gpu_reset.rs +++ b/dstack/vmm/src/gpu_reset.rs @@ -2,9 +2,36 @@ // // SPDX-License-Identifier: Apache-2.0 +//! GPU sanitization before QEMU attach. +//! +//! Terminating QEMU while GSP/SPDM initialization is in progress can leave +//! stale FSP/SPDM session state in a GPU. VFIO's attach-time FLR does not +//! clear that state; a PCIe Secondary Bus Reset does. The next guest +//! otherwise fails with an SPDM timeout followed by GSP/RmInitAdapter errors. +//! +//! The VMM runs as an unprivileged user, so it cannot issue the SBR by +//! writing Bridge Control in the upstream bridge's sysfs config space: that +//! file is writable by root only, as is /sys/bus/pci/drivers_probe. Instead +//! the VFIO_DEVICE_PCI_HOT_RESET ioctl asks the kernel to perform the same +//! bus reset. The ioctl is authorized by device ownership rather than +//! privilege: the caller must present an fd for every VFIO group affected by +//! the reset. The group nodes under /dev/vfio are the same ones QEMU opens +//! to attach the GPU, so the VMM user already has access to them. +//! +//! A single group fd suffices only because every sanitized GPU sits alone +//! behind a dedicated PCIe bridge and alone in its IOMMU group. The bridge +//! topology is validated first, and the kernel-reported set of devices +//! affected by the reset must all belong to the GPU's own group; anything +//! else aborts the launch rather than risking disruption to other devices. +//! +//! A VFIO group can be opened by only one process at a time, so every fd is +//! closed again before QEMU is spawned. + use std::{ collections::BTreeSet, fs::{File, OpenOptions}, + mem::size_of, + os::fd::{AsRawFd, FromRawFd, OwnedFd}, os::unix::fs::FileExt, path::{Path, PathBuf}, thread, @@ -18,12 +45,44 @@ use crate::{app::GpuConfig, config::GpuConfig as HostGpuConfig}; const PCI_SYSFS_DEVICES: &str = "/sys/bus/pci/devices"; const PCI_BRIDGE_CLASS: u32 = 0x0604; -const PCI_BRIDGE_CONTROL: u64 = 0x3e; -const PCI_BRIDGE_CTL_BUS_RESET: u16 = 1 << 6; -const SBR_ASSERT_TIME: Duration = Duration::from_millis(100); const SBR_POLL_INTERVAL: Duration = Duration::from_millis(100); const SBR_STABLE_TIME: Duration = Duration::from_millis(500); +const VFIO_CONTAINER: &str = "/dev/vfio/vfio"; +const VFIO_GROUP_DIR: &str = "/dev/vfio"; +const VFIO_API_VERSION: i32 = 0; +const VFIO_TYPE1_IOMMU: usize = 1; +const VFIO_TYPE1V2_IOMMU: usize = 3; +const VFIO_GROUP_FLAGS_VIABLE: u32 = 1; + +// _IO(';', 100 + n) from : no argument size is encoded. +const fn vfio_io(nr: u64) -> libc::c_ulong { + ((b';' as u64) << 8 | (100 + nr)) as libc::c_ulong +} +const VFIO_GET_API_VERSION: libc::c_ulong = vfio_io(0); +const VFIO_CHECK_EXTENSION: libc::c_ulong = vfio_io(1); +const VFIO_SET_IOMMU: libc::c_ulong = vfio_io(2); +const VFIO_GROUP_GET_STATUS: libc::c_ulong = vfio_io(3); +const VFIO_GROUP_SET_CONTAINER: libc::c_ulong = vfio_io(4); +const VFIO_GROUP_GET_DEVICE_FD: libc::c_ulong = vfio_io(6); +const VFIO_DEVICE_PCI_GET_HOT_RESET_INFO: libc::c_ulong = vfio_io(12); +const VFIO_DEVICE_PCI_HOT_RESET: libc::c_ulong = vfio_io(13); + +#[repr(C)] +struct VfioGroupStatus { + argsz: u32, + flags: u32, +} + +#[repr(C)] +#[derive(Clone, Copy)] +struct VfioPciDependentDevice { + group_id: u32, + segment: u16, + bus: u8, + devfn: u8, +} + /// Clears device-internal state that can survive VFIO's attach-time FLR. /// /// Each selected GPU must sit alone behind a dedicated PCIe bridge. Resetting @@ -33,63 +92,252 @@ pub fn sanitize_on_attach(host: &HostGpuConfig, devices: &GpuConfig) -> Result<( if !host.enabled || !host.sanitize_on_attach || devices.gpus.is_empty() { return Ok(()); } - sanitize_at( - Path::new(PCI_SYSFS_DEVICES), - devices, - Duration::from_millis(host.sbr_timeout_ms), - ) + let slots = devices + .gpus + .iter() + .map(|gpu| gpu.slot.clone()) + .collect::>(); + sanitize_slots(&slots, Duration::from_millis(host.sbr_timeout_ms)) } -fn sanitize_at(sysfs_devices: &Path, devices: &GpuConfig, timeout: Duration) -> Result<()> { - let selected = devices - .gpus +/// Sanitizes the given GPU slots. Entry point for the `sanitize-gpu` +/// subcommand; `sanitize_on_attach` funnels here as well. +pub fn sanitize_slots(slots: &[String], timeout: Duration) -> Result<()> { + if slots.is_empty() { + bail!("no GPU slots specified"); + } + let selected = slots .iter() - .map(|gpu| normalize_slot(&gpu.slot)) + .map(|slot| normalize_slot(slot)) .collect::>(); - let mut bridges = BTreeSet::new(); + sanitize_at(Path::new(PCI_SYSFS_DEVICES), &selected, timeout) +} - for gpu in &selected { +fn sanitize_at(sysfs_devices: &Path, selected: &BTreeSet, timeout: Duration) -> Result<()> { + for gpu in selected { let bridge = upstream_bridge(sysfs_devices, gpu)?; ensure_dedicated_bridge(&bridge, gpu)?; - bridges.insert(bridge); } - for bridge in bridges { - info!(bridge = %bridge.display(), "sanitizing GPU with PCIe Secondary Bus Reset"); - secondary_bus_reset(&bridge) - .with_context(|| format!("failed to sanitize GPU using bridge {}", bridge.display()))?; + for gpu in selected { + info!(gpu = %gpu, "sanitizing GPU with VFIO PCI hot reset"); + hot_reset(sysfs_devices, gpu) + .with_context(|| format!("failed to sanitize GPU {gpu} with VFIO hot reset"))?; + } + wait_for_vfio_ready(sysfs_devices, selected, timeout)?; + Ok(()) +} + +/// Issues a Secondary Bus Reset on the GPU's upstream bridge through VFIO. +/// +/// All fds are dropped on return so that QEMU can open the group afterwards. +fn hot_reset(sysfs_devices: &Path, slot: &str) -> Result<()> { + let group_id = iommu_group_id(sysfs_devices, slot)?; + let container = open_container()?; + let group = open_group(&container, group_id)?; + let device = open_device(&group, slot)?; + + let dependents = hot_reset_dependents(&device)?; + let foreign = dependents + .iter() + .filter(|dep| dep.group_id != group_id) + .map(format_dependent) + .collect::>(); + if !foreign.is_empty() { + bail!( + "refusing hot reset of GPU {slot}: it would also reset devices outside \ + IOMMU group {group_id}: {}", + foreign.join(", ") + ); } - wait_for_vfio_ready(sysfs_devices, &selected, timeout)?; + info!( + gpu = %slot, + affected = %dependents.iter().map(format_dependent).collect::>().join(", "), + "issuing VFIO PCI hot reset" + ); + + // struct vfio_pci_hot_reset { argsz, flags, count, group_fds[] } + let mut request = [0_u32; 4]; + request[0] = (size_of::<[u32; 4]>()) as u32; + request[2] = 1; + request[3] = group.as_raw_fd() as u32; + vfio_ioctl( + &device, + VFIO_DEVICE_PCI_HOT_RESET, + request.as_mut_ptr() as usize, + "VFIO_DEVICE_PCI_HOT_RESET", + )?; Ok(()) } +fn open_container() -> Result { + let container = OpenOptions::new() + .read(true) + .write(true) + .open(VFIO_CONTAINER) + .with_context(|| format!("failed to open {VFIO_CONTAINER}"))?; + let version = vfio_ioctl(&container, VFIO_GET_API_VERSION, 0, "VFIO_GET_API_VERSION")?; + if version != VFIO_API_VERSION { + bail!("unsupported VFIO API version {version}"); + } + Ok(container) +} + +fn open_group(container: &File, group_id: u32) -> Result { + let path = format!("{VFIO_GROUP_DIR}/{group_id}"); + let group = OpenOptions::new() + .read(true) + .write(true) + .open(&path) + .with_context(|| format!("failed to open {path} (is the VMM user in its group?)"))?; + + let mut status = VfioGroupStatus { + argsz: size_of::() as u32, + flags: 0, + }; + vfio_ioctl( + &group, + VFIO_GROUP_GET_STATUS, + &mut status as *mut _ as usize, + "VFIO_GROUP_GET_STATUS", + )?; + if status.flags & VFIO_GROUP_FLAGS_VIABLE == 0 { + bail!("IOMMU group {group_id} is not viable; are all of its devices bound to vfio-pci?"); + } + + let container_fd = container.as_raw_fd(); + vfio_ioctl( + &group, + VFIO_GROUP_SET_CONTAINER, + &container_fd as *const _ as usize, + "VFIO_GROUP_SET_CONTAINER", + )?; + // An IOMMU backend must be set before device fds can be handed out. No + // DMA mappings are created; the container exists only for this reset. + let type1v2 = vfio_ioctl( + container, + VFIO_CHECK_EXTENSION, + VFIO_TYPE1V2_IOMMU, + "VFIO_CHECK_EXTENSION", + )?; + let iommu = if type1v2 > 0 { + VFIO_TYPE1V2_IOMMU + } else { + VFIO_TYPE1_IOMMU + }; + vfio_ioctl(container, VFIO_SET_IOMMU, iommu, "VFIO_SET_IOMMU")?; + Ok(group) +} + +fn open_device(group: &File, slot: &str) -> Result { + let name = std::ffi::CString::new(slot).context("invalid PCI slot name")?; + let fd = vfio_ioctl( + group, + VFIO_GROUP_GET_DEVICE_FD, + name.as_ptr() as usize, + "VFIO_GROUP_GET_DEVICE_FD", + )?; + Ok(unsafe { OwnedFd::from_raw_fd(fd) }) +} + +/// Returns the devices the kernel reports as affected by a hot reset. +fn hot_reset_dependents(device: &OwnedFd) -> Result> { + // struct vfio_pci_hot_reset_info { argsz, flags, count, devices[] } + const HEADER: usize = size_of::<[u32; 3]>(); + const ENTRY: usize = size_of::(); + + let mut probe = [HEADER as u32, 0, 0]; + let ret = unsafe { + libc::ioctl( + device.as_raw_fd(), + VFIO_DEVICE_PCI_GET_HOT_RESET_INFO, + probe.as_mut_ptr(), + ) + }; + if ret < 0 { + let err = std::io::Error::last_os_error(); + // ENOSPC is the expected way to learn the entry count. + if err.raw_os_error() != Some(libc::ENOSPC) { + return Err(err).context("VFIO_DEVICE_PCI_GET_HOT_RESET_INFO failed"); + } + } + let count = probe[2] as usize; + + // u64 storage keeps the buffer aligned for the header and entries. + let mut buffer = vec![0_u64; (HEADER + count * ENTRY).div_ceil(size_of::())]; + let header = buffer.as_mut_ptr() as *mut u32; + unsafe { + *header = (HEADER + count * ENTRY) as u32; + } + vfio_ioctl( + device, + VFIO_DEVICE_PCI_GET_HOT_RESET_INFO, + buffer.as_mut_ptr() as usize, + "VFIO_DEVICE_PCI_GET_HOT_RESET_INFO", + )?; + let filled = unsafe { *header.add(2) } as usize; + let entries = unsafe { + std::slice::from_raw_parts( + (buffer.as_ptr() as *const u8).add(HEADER) as *const VfioPciDependentDevice, + filled.min(count), + ) + }; + Ok(entries.to_vec()) +} + +fn format_dependent(dep: &VfioPciDependentDevice) -> String { + format!( + "{:04x}:{:02x}:{:02x}.{:x} (group {})", + dep.segment, + dep.bus, + dep.devfn >> 3, + dep.devfn & 0x7, + dep.group_id + ) +} + +fn vfio_ioctl(fd: &impl AsRawFd, request: libc::c_ulong, arg: usize, what: &str) -> Result { + let ret = unsafe { libc::ioctl(fd.as_raw_fd(), request, arg) }; + if ret < 0 { + return Err(std::io::Error::last_os_error()).with_context(|| format!("{what} failed")); + } + Ok(ret) +} + +fn iommu_group_id(sysfs_devices: &Path, slot: &str) -> Result { + let link = sysfs_devices.join(slot).join("iommu_group"); + let group = link + .canonicalize() + .with_context(|| format!("failed to resolve IOMMU group of {slot}"))?; + group + .file_name() + .and_then(|name| name.to_str()) + .and_then(|name| name.parse().ok()) + .with_context(|| format!("invalid IOMMU group path {}", group.display())) +} + fn wait_for_vfio_ready( sysfs_devices: &Path, devices: &BTreeSet, timeout: Duration, ) -> Result<()> { - let drivers_probe = sysfs_devices - .parent() - .context("PCI sysfs devices directory has no parent")? - .join("drivers_probe"); - let deadline = Instant::now() + timeout; + let started = Instant::now(); + let deadline = started + timeout; let mut stable_since = None; loop { let not_ready = devices .iter() - .filter(|slot| { - let device = sysfs_devices.join(slot); - if !driver_is_vfio(&device) { - let _ = fs_err::write(&drivers_probe, format!("{slot}\n")); - } - !is_vfio_ready(&device) - }) + .filter(|slot| !is_vfio_ready(&sysfs_devices.join(slot))) .cloned() .collect::>(); if not_ready.is_empty() { let since = *stable_since.get_or_insert_with(Instant::now); if since.elapsed() >= SBR_STABLE_TIME { - info!(count = devices.len(), "all sanitized GPUs are VFIO-ready"); + info!( + count = devices.len(), + elapsed_ms = started.elapsed().as_millis() as u64, + "all sanitized GPUs are VFIO-ready" + ); return Ok(()); } } else { @@ -194,39 +442,12 @@ fn is_pci_slot(name: &str) -> bool { .all(|(i, c)| matches!(i, 4 | 7 | 10) || c.is_ascii_hexdigit()) } -fn secondary_bus_reset(bridge: &Path) -> Result<()> { - let config_path = bridge.join("config"); - let config = OpenOptions::new() - .read(true) - .write(true) - .open(&config_path) - .with_context(|| format!("failed to open {}", config_path.display()))?; - let original = read_u16(&config, PCI_BRIDGE_CONTROL)?; - write_u16( - &config, - PCI_BRIDGE_CONTROL, - original | PCI_BRIDGE_CTL_BUS_RESET, - )?; - thread::sleep(SBR_ASSERT_TIME); - write_u16( - &config, - PCI_BRIDGE_CONTROL, - original & !PCI_BRIDGE_CTL_BUS_RESET, - )?; - Ok(()) -} - fn read_u16(file: &File, offset: u64) -> Result { let mut value = [0_u8; 2]; file.read_exact_at(&mut value, offset)?; Ok(u16::from_le_bytes(value)) } -fn write_u16(file: &File, offset: u64, value: u16) -> Result<()> { - file.write_all_at(&value.to_le_bytes(), offset)?; - Ok(()) -} - #[cfg(test)] mod tests { use super::*; @@ -247,6 +468,17 @@ mod tests { assert!(!is_pci_slot("0000:0g:00.0")); } + #[test] + fn formats_dependent_devices_with_pci_slot_and_function() { + let dep = VfioPciDependentDevice { + group_id: 46, + segment: 0, + bus: 0x0f, + devfn: (0x00 << 3) | 0x1, + }; + assert_eq!(format_dependent(&dep), "0000:0f:00.1 (group 46)"); + } + #[test] fn skips_sanitization_when_gpu_passthrough_is_disabled() { let host = HostGpuConfig { diff --git a/dstack/vmm/vmm.toml b/dstack/vmm/vmm.toml index 019a5ea83..61107342e 100644 --- a/dstack/vmm/vmm.toml +++ b/dstack/vmm/vmm.toml @@ -254,8 +254,9 @@ include = [] # Allow attach all GPUs allow_attach_all = true # Issue a Secondary Bus Reset on each GPU's dedicated upstream PCIe bridge -# immediately before QEMU attaches the device. This clears stale GPU FSP/SPDM -# state left by a VM that terminated during GPU firmware initialization. +# immediately before QEMU attaches the device, using a VFIO PCI hot reset so +# that no root privileges are needed. This clears stale GPU FSP/SPDM state +# left by a VM that terminated during GPU firmware initialization. sanitize_on_attach = true # Maximum time for all reset GPUs to become continuously VFIO-ready. sbr_timeout_ms = 10000 From cfbee8573e85ce1578c9747add7c99f2b08e1554 Mon Sep 17 00:00:00 2001 From: Leechael Yim Date: Fri, 14 Aug 2026 19:49:41 +0800 Subject: [PATCH 14/93] feat(vmm): add sanitize-gpu subcommand for manual GPU reset Expose the sanitize path as "dstack-vmm sanitize-gpu ..." so operators can reset GPUs by hand and the pending hardware experiment can exercise exactly the code path used at VM launch, running as the unprivileged VMM user. The subcommand needs no server configuration, only /dev/vfio access, and is handled before config loading like the other special modes. --- dstack/vmm/src/main.rs | 21 +++++++++++++++++++++ 1 file changed, 21 insertions(+) diff --git a/dstack/vmm/src/main.rs b/dstack/vmm/src/main.rs index 0ca409af4..70dc8dc38 100644 --- a/dstack/vmm/src/main.rs +++ b/dstack/vmm/src/main.rs @@ -65,6 +65,8 @@ enum Command { CheckConfig, /// One-shot VM execution mode for debugging Run(RunArgs), + /// Sanitize GPUs with a VFIO PCI hot reset (debugging/ops) + SanitizeGpu(SanitizeGpuArgs), /// Run the privileged TAP and libvirt nwfilter broker. Netd(NetdArgs), /// Internal per-VM QEMU/swtpm launcher. @@ -132,6 +134,16 @@ struct RunArgs { dry_run: bool, } +#[derive(ClapArgs)] +struct SanitizeGpuArgs { + /// PCI slots of the GPUs to reset, e.g. 0000:0f:00.0 + #[arg(required = true)] + slots: Vec, + /// Maximum time in milliseconds for the GPUs to become VFIO-ready again + #[arg(long, default_value_t = 10_000)] + timeout_ms: u64, +} + #[derive(ClapArgs)] struct VmLauncherArgs { /// Path to the generated VM launch specification. @@ -327,6 +339,14 @@ async fn main() -> Result<()> { return vm_launcher::run(Path::new(&launcher_args.spec)).await; } + // Needs no server configuration; only /dev/vfio access. + if let Some(Command::SanitizeGpu(sanitize_args)) = &args.command { + return gpu_reset::sanitize_slots( + &sanitize_args.slots, + Duration::from_millis(sanitize_args.timeout_ms), + ); + } + let figment = config::load_config_figment(args.config.as_deref()); if let Some(Command::Netd(netd_args)) = &args.command { let mut netd_config: NetdConfig = figment @@ -394,6 +414,7 @@ async fn main() -> Result<()> { return Ok(()); } Command::Netd(_) => unreachable!("netd mode handled before server startup"), + Command::SanitizeGpu(_) => unreachable!("sanitize-gpu handled before config loading"), Command::Run(run_args) => { // One-shot VM execution mode return one_shot::run_one_shot( From 57de57cf8efc2d9daa9a718b2032577eeaa5606f Mon Sep 17 00:00:00 2001 From: Kevin Wang Date: Mon, 17 Aug 2026 06:09:14 -0700 Subject: [PATCH 15/93] refactor(vmm): use vfio-ioctls for GPU hot reset --- dstack/Cargo.lock | 41 ++++++++ dstack/Cargo.toml | 1 + dstack/vmm/Cargo.toml | 1 + dstack/vmm/src/gpu_reset.rs | 199 ++++-------------------------------- 4 files changed, 62 insertions(+), 180 deletions(-) diff --git a/dstack/Cargo.lock b/dstack/Cargo.lock index 227ef2fae..db1f4312c 100644 --- a/dstack/Cargo.lock +++ b/dstack/Cargo.lock @@ -2442,6 +2442,7 @@ dependencies = [ "tracing-subscriber", "url", "uuid", + "vfio-ioctls", "wait-timeout", "which 7.0.3", ] @@ -8329,12 +8330,52 @@ version = "0.9.5" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "0b928f33d975fc6ad9f86c8f283853ad26bdd5b10b7f1542aa2fa15e2289105a" +[[package]] +name = "vfio-bindings" +version = "0.6.2" +source = "git+https://github.com/kvinwang/vfio.git?rev=d509db2#d509db2b403d258659c8dcf597a7b189d9512940" + +[[package]] +name = "vfio-ioctls" +version = "0.8.0" +source = "git+https://github.com/kvinwang/vfio.git?rev=d509db2#d509db2b403d258659c8dcf597a7b189d9512940" +dependencies = [ + "byteorder", + "libc", + "log", + "thiserror 2.0.18", + "vfio-bindings", + "vm-memory", + "vmm-sys-util", +] + [[package]] name = "virtue" version = "0.0.18" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "051eb1abcf10076295e815102942cc58f9d5e3b4560e46e53c21e8ff6f3af7b1" +[[package]] +name = "vm-memory" +version = "0.18.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9b55e753c7725603745cb32b2287ef7ef3da05c03c7702cda3fa8abe25ae0465" +dependencies = [ + "libc", + "thiserror 2.0.18", + "winapi", +] + +[[package]] +name = "vmm-sys-util" +version = "0.15.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "506c62fdf617a5176827c2f9afbcf1be155b03a9b4bf9617a60dbc07e3a1642f" +dependencies = [ + "bitflags 1.3.2", + "libc", +] + [[package]] name = "void" version = "1.0.2" diff --git a/dstack/Cargo.toml b/dstack/Cargo.toml index 6223e773e..d5cec92b0 100644 --- a/dstack/Cargo.toml +++ b/dstack/Cargo.toml @@ -126,6 +126,7 @@ dstack-verifier = { path = "verifier", default-features = false } size-parser = { path = "size-parser" } nvattest = { path = "nvattest" } wavekv = "2.1" +vfio-ioctls = { git = "https://github.com/kvinwang/vfio.git", rev = "d509db2", default-features = false } # Core dependencies anyhow = { version = "1.0.97", default-features = false } diff --git a/dstack/vmm/Cargo.toml b/dstack/vmm/Cargo.toml index 22387fcef..0850c1697 100644 --- a/dstack/vmm/Cargo.toml +++ b/dstack/vmm/Cargo.toml @@ -68,6 +68,7 @@ tar.workspace = true tempfile.workspace = true wait-timeout.workspace = true listenfd.workspace = true +vfio-ioctls.workspace = true [dev-dependencies] insta.workspace = true diff --git a/dstack/vmm/src/gpu_reset.rs b/dstack/vmm/src/gpu_reset.rs index f35afb238..0ce2efa18 100644 --- a/dstack/vmm/src/gpu_reset.rs +++ b/dstack/vmm/src/gpu_reset.rs @@ -29,17 +29,17 @@ use std::{ collections::BTreeSet, - fs::{File, OpenOptions}, - mem::size_of, - os::fd::{AsRawFd, FromRawFd, OwnedFd}, + fs::File, os::unix::fs::FileExt, path::{Path, PathBuf}, + sync::Arc, thread, time::{Duration, Instant}, }; use anyhow::{bail, Context, Result}; use tracing::info; +use vfio_ioctls::{PciHotResetDevice, VfioContainer, VfioDevice}; use crate::{app::GpuConfig, config::GpuConfig as HostGpuConfig}; @@ -48,41 +48,6 @@ const PCI_BRIDGE_CLASS: u32 = 0x0604; const SBR_POLL_INTERVAL: Duration = Duration::from_millis(100); const SBR_STABLE_TIME: Duration = Duration::from_millis(500); -const VFIO_CONTAINER: &str = "/dev/vfio/vfio"; -const VFIO_GROUP_DIR: &str = "/dev/vfio"; -const VFIO_API_VERSION: i32 = 0; -const VFIO_TYPE1_IOMMU: usize = 1; -const VFIO_TYPE1V2_IOMMU: usize = 3; -const VFIO_GROUP_FLAGS_VIABLE: u32 = 1; - -// _IO(';', 100 + n) from : no argument size is encoded. -const fn vfio_io(nr: u64) -> libc::c_ulong { - ((b';' as u64) << 8 | (100 + nr)) as libc::c_ulong -} -const VFIO_GET_API_VERSION: libc::c_ulong = vfio_io(0); -const VFIO_CHECK_EXTENSION: libc::c_ulong = vfio_io(1); -const VFIO_SET_IOMMU: libc::c_ulong = vfio_io(2); -const VFIO_GROUP_GET_STATUS: libc::c_ulong = vfio_io(3); -const VFIO_GROUP_SET_CONTAINER: libc::c_ulong = vfio_io(4); -const VFIO_GROUP_GET_DEVICE_FD: libc::c_ulong = vfio_io(6); -const VFIO_DEVICE_PCI_GET_HOT_RESET_INFO: libc::c_ulong = vfio_io(12); -const VFIO_DEVICE_PCI_HOT_RESET: libc::c_ulong = vfio_io(13); - -#[repr(C)] -struct VfioGroupStatus { - argsz: u32, - flags: u32, -} - -#[repr(C)] -#[derive(Clone, Copy)] -struct VfioPciDependentDevice { - group_id: u32, - segment: u16, - bus: u8, - devfn: u8, -} - /// Clears device-internal state that can survive VFIO's attach-time FLR. /// /// Each selected GPU must sit alone behind a dedicated PCIe bridge. Resetting @@ -133,11 +98,16 @@ fn sanitize_at(sysfs_devices: &Path, selected: &BTreeSet, timeout: Durat /// All fds are dropped on return so that QEMU can open the group afterwards. fn hot_reset(sysfs_devices: &Path, slot: &str) -> Result<()> { let group_id = iommu_group_id(sysfs_devices, slot)?; - let container = open_container()?; - let group = open_group(&container, group_id)?; - let device = open_device(&group, slot)?; - - let dependents = hot_reset_dependents(&device)?; + let container = Arc::new(VfioContainer::new(None).context("failed to open VFIO container")?); + let group = container + .get_group(group_id) + .with_context(|| format!("failed to open VFIO group {group_id}"))?; + let device = VfioDevice::new(&sysfs_devices.join(slot), container) + .with_context(|| format!("failed to open VFIO device {slot}"))?; + + let dependents = device + .pci_hot_reset_info() + .context("VFIO_DEVICE_GET_PCI_HOT_RESET_INFO failed")?; let foreign = dependents .iter() .filter(|dep| dep.group_id != group_id) @@ -156,136 +126,13 @@ fn hot_reset(sysfs_devices: &Path, slot: &str) -> Result<()> { "issuing VFIO PCI hot reset" ); - // struct vfio_pci_hot_reset { argsz, flags, count, group_fds[] } - let mut request = [0_u32; 4]; - request[0] = (size_of::<[u32; 4]>()) as u32; - request[2] = 1; - request[3] = group.as_raw_fd() as u32; - vfio_ioctl( - &device, - VFIO_DEVICE_PCI_HOT_RESET, - request.as_mut_ptr() as usize, - "VFIO_DEVICE_PCI_HOT_RESET", - )?; + device + .pci_hot_reset(&[&group]) + .context("VFIO_DEVICE_PCI_HOT_RESET failed")?; Ok(()) } -fn open_container() -> Result { - let container = OpenOptions::new() - .read(true) - .write(true) - .open(VFIO_CONTAINER) - .with_context(|| format!("failed to open {VFIO_CONTAINER}"))?; - let version = vfio_ioctl(&container, VFIO_GET_API_VERSION, 0, "VFIO_GET_API_VERSION")?; - if version != VFIO_API_VERSION { - bail!("unsupported VFIO API version {version}"); - } - Ok(container) -} - -fn open_group(container: &File, group_id: u32) -> Result { - let path = format!("{VFIO_GROUP_DIR}/{group_id}"); - let group = OpenOptions::new() - .read(true) - .write(true) - .open(&path) - .with_context(|| format!("failed to open {path} (is the VMM user in its group?)"))?; - - let mut status = VfioGroupStatus { - argsz: size_of::() as u32, - flags: 0, - }; - vfio_ioctl( - &group, - VFIO_GROUP_GET_STATUS, - &mut status as *mut _ as usize, - "VFIO_GROUP_GET_STATUS", - )?; - if status.flags & VFIO_GROUP_FLAGS_VIABLE == 0 { - bail!("IOMMU group {group_id} is not viable; are all of its devices bound to vfio-pci?"); - } - - let container_fd = container.as_raw_fd(); - vfio_ioctl( - &group, - VFIO_GROUP_SET_CONTAINER, - &container_fd as *const _ as usize, - "VFIO_GROUP_SET_CONTAINER", - )?; - // An IOMMU backend must be set before device fds can be handed out. No - // DMA mappings are created; the container exists only for this reset. - let type1v2 = vfio_ioctl( - container, - VFIO_CHECK_EXTENSION, - VFIO_TYPE1V2_IOMMU, - "VFIO_CHECK_EXTENSION", - )?; - let iommu = if type1v2 > 0 { - VFIO_TYPE1V2_IOMMU - } else { - VFIO_TYPE1_IOMMU - }; - vfio_ioctl(container, VFIO_SET_IOMMU, iommu, "VFIO_SET_IOMMU")?; - Ok(group) -} - -fn open_device(group: &File, slot: &str) -> Result { - let name = std::ffi::CString::new(slot).context("invalid PCI slot name")?; - let fd = vfio_ioctl( - group, - VFIO_GROUP_GET_DEVICE_FD, - name.as_ptr() as usize, - "VFIO_GROUP_GET_DEVICE_FD", - )?; - Ok(unsafe { OwnedFd::from_raw_fd(fd) }) -} - -/// Returns the devices the kernel reports as affected by a hot reset. -fn hot_reset_dependents(device: &OwnedFd) -> Result> { - // struct vfio_pci_hot_reset_info { argsz, flags, count, devices[] } - const HEADER: usize = size_of::<[u32; 3]>(); - const ENTRY: usize = size_of::(); - - let mut probe = [HEADER as u32, 0, 0]; - let ret = unsafe { - libc::ioctl( - device.as_raw_fd(), - VFIO_DEVICE_PCI_GET_HOT_RESET_INFO, - probe.as_mut_ptr(), - ) - }; - if ret < 0 { - let err = std::io::Error::last_os_error(); - // ENOSPC is the expected way to learn the entry count. - if err.raw_os_error() != Some(libc::ENOSPC) { - return Err(err).context("VFIO_DEVICE_PCI_GET_HOT_RESET_INFO failed"); - } - } - let count = probe[2] as usize; - - // u64 storage keeps the buffer aligned for the header and entries. - let mut buffer = vec![0_u64; (HEADER + count * ENTRY).div_ceil(size_of::())]; - let header = buffer.as_mut_ptr() as *mut u32; - unsafe { - *header = (HEADER + count * ENTRY) as u32; - } - vfio_ioctl( - device, - VFIO_DEVICE_PCI_GET_HOT_RESET_INFO, - buffer.as_mut_ptr() as usize, - "VFIO_DEVICE_PCI_GET_HOT_RESET_INFO", - )?; - let filled = unsafe { *header.add(2) } as usize; - let entries = unsafe { - std::slice::from_raw_parts( - (buffer.as_ptr() as *const u8).add(HEADER) as *const VfioPciDependentDevice, - filled.min(count), - ) - }; - Ok(entries.to_vec()) -} - -fn format_dependent(dep: &VfioPciDependentDevice) -> String { +fn format_dependent(dep: &PciHotResetDevice) -> String { format!( "{:04x}:{:02x}:{:02x}.{:x} (group {})", dep.segment, @@ -296,14 +143,6 @@ fn format_dependent(dep: &VfioPciDependentDevice) -> String { ) } -fn vfio_ioctl(fd: &impl AsRawFd, request: libc::c_ulong, arg: usize, what: &str) -> Result { - let ret = unsafe { libc::ioctl(fd.as_raw_fd(), request, arg) }; - if ret < 0 { - return Err(std::io::Error::last_os_error()).with_context(|| format!("{what} failed")); - } - Ok(ret) -} - fn iommu_group_id(sysfs_devices: &Path, slot: &str) -> Result { let link = sysfs_devices.join(slot).join("iommu_group"); let group = link @@ -470,11 +309,11 @@ mod tests { #[test] fn formats_dependent_devices_with_pci_slot_and_function() { - let dep = VfioPciDependentDevice { + let dep = PciHotResetDevice { group_id: 46, segment: 0, bus: 0x0f, - devfn: (0x00 << 3) | 0x1, + devfn: 0x1, }; assert_eq!(format_dependent(&dep), "0000:0f:00.1 (group 46)"); } From c975611e6742350eefdbcd313c65060dad0b85a1 Mon Sep 17 00:00:00 2001 From: Kevin Wang Date: Mon, 17 Aug 2026 06:42:58 -0700 Subject: [PATCH 16/93] refactor(vmm): keep VFIO group handling encapsulated --- dstack/Cargo.lock | 4 ++-- dstack/Cargo.toml | 2 +- dstack/vmm/src/gpu_reset.rs | 9 +++------ 3 files changed, 6 insertions(+), 9 deletions(-) diff --git a/dstack/Cargo.lock b/dstack/Cargo.lock index db1f4312c..d9a136405 100644 --- a/dstack/Cargo.lock +++ b/dstack/Cargo.lock @@ -8333,12 +8333,12 @@ checksum = "0b928f33d975fc6ad9f86c8f283853ad26bdd5b10b7f1542aa2fa15e2289105a" [[package]] name = "vfio-bindings" version = "0.6.2" -source = "git+https://github.com/kvinwang/vfio.git?rev=d509db2#d509db2b403d258659c8dcf597a7b189d9512940" +source = "git+https://github.com/kvinwang/vfio.git?rev=1b560c2#1b560c20c8c294db0df5c6d8e0346ee7f9d89b62" [[package]] name = "vfio-ioctls" version = "0.8.0" -source = "git+https://github.com/kvinwang/vfio.git?rev=d509db2#d509db2b403d258659c8dcf597a7b189d9512940" +source = "git+https://github.com/kvinwang/vfio.git?rev=1b560c2#1b560c20c8c294db0df5c6d8e0346ee7f9d89b62" dependencies = [ "byteorder", "libc", diff --git a/dstack/Cargo.toml b/dstack/Cargo.toml index d5cec92b0..61cc8e56f 100644 --- a/dstack/Cargo.toml +++ b/dstack/Cargo.toml @@ -126,7 +126,7 @@ dstack-verifier = { path = "verifier", default-features = false } size-parser = { path = "size-parser" } nvattest = { path = "nvattest" } wavekv = "2.1" -vfio-ioctls = { git = "https://github.com/kvinwang/vfio.git", rev = "d509db2", default-features = false } +vfio-ioctls = { git = "https://github.com/kvinwang/vfio.git", rev = "1b560c2", default-features = false } # Core dependencies anyhow = { version = "1.0.97", default-features = false } diff --git a/dstack/vmm/src/gpu_reset.rs b/dstack/vmm/src/gpu_reset.rs index 0ce2efa18..5e76404d0 100644 --- a/dstack/vmm/src/gpu_reset.rs +++ b/dstack/vmm/src/gpu_reset.rs @@ -99,15 +99,12 @@ fn sanitize_at(sysfs_devices: &Path, selected: &BTreeSet, timeout: Durat fn hot_reset(sysfs_devices: &Path, slot: &str) -> Result<()> { let group_id = iommu_group_id(sysfs_devices, slot)?; let container = Arc::new(VfioContainer::new(None).context("failed to open VFIO container")?); - let group = container - .get_group(group_id) - .with_context(|| format!("failed to open VFIO group {group_id}"))?; let device = VfioDevice::new(&sysfs_devices.join(slot), container) .with_context(|| format!("failed to open VFIO device {slot}"))?; let dependents = device .pci_hot_reset_info() - .context("VFIO_DEVICE_GET_PCI_HOT_RESET_INFO failed")?; + .context("failed to query PCI hot reset scope")?; let foreign = dependents .iter() .filter(|dep| dep.group_id != group_id) @@ -127,8 +124,8 @@ fn hot_reset(sysfs_devices: &Path, slot: &str) -> Result<()> { ); device - .pci_hot_reset(&[&group]) - .context("VFIO_DEVICE_PCI_HOT_RESET failed")?; + .pci_hot_reset_own_group() + .context("failed to perform PCI hot reset")?; Ok(()) } From 42844e0d29721b826ada626a51203d538b744278 Mon Sep 17 00:00:00 2001 From: Kevin Wang Date: Mon, 14 Sep 2026 23:17:54 -0700 Subject: [PATCH 17/93] ci: publish release images to GHCR --- .github/workflows/gateway-release.yml | 13 +++++++------ .github/workflows/kms-release.yml | 13 +++++++------ .github/workflows/local-key-provider-release.yml | 15 ++++++++------- .github/workflows/verifier-release.yml | 15 ++++++++------- 4 files changed, 30 insertions(+), 26 deletions(-) diff --git a/.github/workflows/gateway-release.yml b/.github/workflows/gateway-release.yml index 998209bc5..62310a9b0 100644 --- a/.github/workflows/gateway-release.yml +++ b/.github/workflows/gateway-release.yml @@ -49,11 +49,12 @@ jobs: with: ref: ${{ github.event_name == 'workflow_dispatch' && env.RELEASE_TAG || github.sha }} - - name: Log in to Docker Hub + - name: Log in to GitHub Container Registry uses: docker/login-action@v3 with: - username: ${{ vars.DOCKERHUB_USERNAME }} - password: ${{ secrets.DOCKERHUB_TOKEN }} + registry: ghcr.io + username: ${{ github.actor }} + password: ${{ secrets.GITHUB_TOKEN }} - name: Set up Docker Buildx uses: docker/setup-buildx-action@v3 @@ -71,7 +72,7 @@ jobs: METADATA_FILE: ${{ runner.temp }}/build-metadata.json GIT_REV: ${{ env.GIT_REV }} IMAGE_VERSION: ${{ env.VERSION }} - IMAGE_NAME: ${{ vars.DOCKERHUB_ORG }}/dstack-gateway:${{ env.VERSION }} + IMAGE_NAME: ghcr.io/dstack-tee/dstack-gateway:${{ env.VERSION }} IMAGE_SOURCE_URL: ${{ github.server_url }}/${{ github.repository }} DSTACK_SRC_URL: ${{ github.server_url }}/${{ github.repository }}.git run: | @@ -87,7 +88,7 @@ jobs: - name: Generate artifact attestation uses: actions/attest-build-provenance@v1 with: - subject-name: "docker.io/${{ vars.DOCKERHUB_ORG }}/dstack-gateway" + subject-name: "ghcr.io/dstack-tee/dstack-gateway" subject-digest: ${{ steps.build-and-push.outputs.digest }} push-to-registry: true @@ -100,7 +101,7 @@ jobs: body: | ## Docker Image Information - **Image**: `docker.io/${{ vars.DOCKERHUB_ORG }}/dstack-gateway:${{ env.VERSION }}` + **Image**: `ghcr.io/dstack-tee/dstack-gateway:${{ env.VERSION }}` **Digest (SHA256)**: `${{ steps.build-and-push.outputs.digest }}` diff --git a/.github/workflows/kms-release.yml b/.github/workflows/kms-release.yml index 122d886b6..64903e2d8 100644 --- a/.github/workflows/kms-release.yml +++ b/.github/workflows/kms-release.yml @@ -56,11 +56,12 @@ jobs: dstack/kms/auth-eth/lib/openzeppelin-contracts-upgradeable \ dstack/kms/auth-eth/lib/openzeppelin-foundry-upgrades - - name: Log in to Docker Hub + - name: Log in to GitHub Container Registry uses: docker/login-action@v3 with: - username: ${{ vars.DOCKERHUB_USERNAME }} - password: ${{ secrets.DOCKERHUB_TOKEN }} + registry: ghcr.io + username: ${{ github.actor }} + password: ${{ secrets.GITHUB_TOKEN }} - name: Set up Docker Buildx uses: docker/setup-buildx-action@v3 @@ -78,7 +79,7 @@ jobs: METADATA_FILE: ${{ runner.temp }}/build-metadata.json GIT_REV: ${{ env.GIT_REV }} IMAGE_VERSION: ${{ env.VERSION }} - IMAGE_NAME: ${{ vars.DOCKERHUB_ORG }}/dstack-kms:${{ env.VERSION }} + IMAGE_NAME: ghcr.io/dstack-tee/dstack-kms:${{ env.VERSION }} IMAGE_SOURCE_URL: ${{ github.server_url }}/${{ github.repository }} DSTACK_SRC_URL: ${{ github.server_url }}/${{ github.repository }}.git run: | @@ -94,7 +95,7 @@ jobs: - name: Generate artifact attestation uses: actions/attest-build-provenance@v1 with: - subject-name: "docker.io/${{ vars.DOCKERHUB_ORG }}/dstack-kms" + subject-name: "ghcr.io/dstack-tee/dstack-kms" subject-digest: ${{ steps.build-and-push.outputs.digest }} push-to-registry: true @@ -119,7 +120,7 @@ jobs: body: | ## Docker Image Information - **Image**: `docker.io/${{ vars.DOCKERHUB_ORG }}/dstack-kms:${{ env.VERSION }}` + **Image**: `ghcr.io/dstack-tee/dstack-kms:${{ env.VERSION }}` **Digest (SHA256)**: `${{ steps.build-and-push.outputs.digest }}` diff --git a/.github/workflows/local-key-provider-release.yml b/.github/workflows/local-key-provider-release.yml index f8c028189..f0700217a 100644 --- a/.github/workflows/local-key-provider-release.yml +++ b/.github/workflows/local-key-provider-release.yml @@ -40,11 +40,12 @@ jobs: echo "VERSION=$VERSION" >> "$GITHUB_ENV" echo "Parsed version: $VERSION" - - name: Log in to Docker Hub + - name: Log in to GitHub Container Registry uses: docker/login-action@v4 with: - username: ${{ vars.DOCKERHUB_USERNAME }} - password: ${{ secrets.DOCKERHUB_TOKEN }} + registry: ghcr.io + username: ${{ github.actor }} + password: ${{ secrets.GITHUB_TOKEN }} - name: Set up Docker Buildx uses: docker/setup-buildx-action@v4 @@ -56,21 +57,21 @@ jobs: context: dstack file: dstack/local-key-provider/build/Dockerfile.key-provider push: true - tags: ${{ vars.DOCKERHUB_ORG }}/local-key-provider:${{ env.VERSION }} + tags: ghcr.io/dstack-tee/local-key-provider:${{ env.VERSION }} platforms: linux/amd64 provenance: false - name: Generate artifact attestation uses: actions/attest-build-provenance@v4 with: - subject-name: "docker.io/${{ vars.DOCKERHUB_ORG }}/local-key-provider" + subject-name: "ghcr.io/dstack-tee/local-key-provider" subject-digest: ${{ steps.build-and-push.outputs.digest }} push-to-registry: true - name: Export enclave measurements env: IMAGE_DIGEST: ${{ steps.build-and-push.outputs.digest }} - IMAGE_NAME: ${{ vars.DOCKERHUB_ORG }}/local-key-provider + IMAGE_NAME: ghcr.io/dstack-tee/local-key-provider run: | docker pull "${IMAGE_NAME}@${IMAGE_DIGEST}" docker run --rm \ @@ -94,7 +95,7 @@ jobs: body: | ## Docker Image Information - **Image**: `docker.io/${{ vars.DOCKERHUB_ORG }}/local-key-provider:${{ env.VERSION }}` + **Image**: `ghcr.io/dstack-tee/local-key-provider:${{ env.VERSION }}` **Digest (SHA256)**: `${{ steps.build-and-push.outputs.digest }}` diff --git a/.github/workflows/verifier-release.yml b/.github/workflows/verifier-release.yml index 857a6382c..e837cf318 100644 --- a/.github/workflows/verifier-release.yml +++ b/.github/workflows/verifier-release.yml @@ -49,11 +49,12 @@ jobs: with: ref: ${{ github.event_name == 'workflow_dispatch' && env.RELEASE_TAG || github.sha }} - - name: Log in to Docker Hub + - name: Log in to GitHub Container Registry uses: docker/login-action@v3 with: - username: ${{ vars.DOCKERHUB_USERNAME }} - password: ${{ secrets.DOCKERHUB_TOKEN }} + registry: ghcr.io + username: ${{ github.actor }} + password: ${{ secrets.GITHUB_TOKEN }} - name: Set up Docker Buildx uses: docker/setup-buildx-action@v3 @@ -71,8 +72,8 @@ jobs: METADATA_FILE: ${{ runner.temp }}/build-metadata.json GIT_REV: ${{ env.GIT_REV }} IMAGE_VERSION: ${{ env.VERSION }} - IMAGE_NAME: ${{ vars.DOCKERHUB_ORG }}/dstack-verifier:${{ env.VERSION }} - LATEST_IMAGE_NAME: ${{ vars.DOCKERHUB_ORG }}/dstack-verifier:latest + IMAGE_NAME: ghcr.io/dstack-tee/dstack-verifier:${{ env.VERSION }} + LATEST_IMAGE_NAME: ghcr.io/dstack-tee/dstack-verifier:latest IMAGE_SOURCE_URL: ${{ github.server_url }}/${{ github.repository }} DSTACK_SRC_URL: ${{ github.server_url }}/${{ github.repository }}.git run: | @@ -89,7 +90,7 @@ jobs: - name: Generate artifact attestation uses: actions/attest-build-provenance@v1 with: - subject-name: "docker.io/${{ vars.DOCKERHUB_ORG }}/dstack-verifier" + subject-name: "ghcr.io/dstack-tee/dstack-verifier" subject-digest: ${{ steps.build-and-push.outputs.digest }} push-to-registry: true @@ -102,7 +103,7 @@ jobs: body: | ## Docker Image Information - **Image**: `docker.io/${{ vars.DOCKERHUB_ORG }}/dstack-verifier:${{ env.VERSION }}` + **Image**: `ghcr.io/dstack-tee/dstack-verifier:${{ env.VERSION }}` **Digest (SHA256)**: `${{ steps.build-and-push.outputs.digest }}` From d697fbcb835d104b79e7572c51b1afb21d3b14f6 Mon Sep 17 00:00:00 2001 From: Kevin Wang Date: Mon, 14 Sep 2026 23:28:04 -0700 Subject: [PATCH 18/93] ci: harden GHCR release publishing --- .github/workflows/gateway-release.yml | 19 +++++-- .github/workflows/kms-release.yml | 21 +++++-- .../workflows/local-key-provider-release.yml | 55 ++++++++++++++----- .github/workflows/verifier-release.yml | 19 +++++-- docs/rc-testing-runbook.md | 4 +- dstack/gateway/dstack-app/builder/README.md | 6 +- dstack/kms/dstack-app/builder/README.md | 6 +- dstack/local-key-provider/README.md | 2 +- dstack/verifier/README.md | 2 +- dstack/verifier/builder/README.md | 6 +- 10 files changed, 97 insertions(+), 43 deletions(-) diff --git a/.github/workflows/gateway-release.yml b/.github/workflows/gateway-release.yml index 62310a9b0..116e3739c 100644 --- a/.github/workflows/gateway-release.yml +++ b/.github/workflows/gateway-release.yml @@ -45,19 +45,19 @@ jobs: # Manual runs rebuild an existing release tag, not the selected branch. - name: Checkout repository - uses: actions/checkout@v5 + uses: actions/checkout@fbc6f3992d24b796d5a048ff273f7fcc4a7b6c09 # v5 with: ref: ${{ github.event_name == 'workflow_dispatch' && env.RELEASE_TAG || github.sha }} - name: Log in to GitHub Container Registry - uses: docker/login-action@v3 + uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9 # v3 with: registry: ghcr.io username: ${{ github.actor }} password: ${{ secrets.GITHUB_TOKEN }} - name: Set up Docker Buildx - uses: docker/setup-buildx-action@v3 + uses: docker/setup-buildx-action@8d2750c68a42422c14e847fe6c8ac0403b4cbd6f # v3 - name: Resolve the revision to build run: echo "GIT_REV=$(git rev-parse HEAD)" >> "$GITHUB_ENV" @@ -85,15 +85,24 @@ jobs: fi echo "digest=$digest" >> "$GITHUB_OUTPUT" + - name: Make package public + env: + GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + gh api --method PATCH \ + "/orgs/${GITHUB_REPOSITORY_OWNER}/packages/container/dstack-gateway" \ + -f visibility=public \ + --jq 'select(.visibility == "public") | .html_url' + - name: Generate artifact attestation - uses: actions/attest-build-provenance@v1 + uses: actions/attest-build-provenance@92c65d2898f1f53cfdc910b962cecff86e7f8fcc # v1 with: subject-name: "ghcr.io/dstack-tee/dstack-gateway" subject-digest: ${{ steps.build-and-push.outputs.digest }} push-to-registry: true - name: GitHub Release - uses: softprops/action-gh-release@v2 + uses: softprops/action-gh-release@3bb12739c298aeb8a4eeaf626c5b8d85266b0e65 # v2 with: tag_name: ${{ env.RELEASE_TAG }} target_commitish: ${{ env.GIT_REV }} diff --git a/.github/workflows/kms-release.yml b/.github/workflows/kms-release.yml index 64903e2d8..cccc6e7f4 100644 --- a/.github/workflows/kms-release.yml +++ b/.github/workflows/kms-release.yml @@ -45,7 +45,7 @@ jobs: # Manual runs rebuild an existing release tag, not the selected branch. - name: Checkout repository - uses: actions/checkout@v5 + uses: actions/checkout@fbc6f3992d24b796d5a048ff273f7fcc4a7b6c09 # v5 with: ref: ${{ github.event_name == 'workflow_dispatch' && env.RELEASE_TAG || github.sha }} @@ -57,14 +57,14 @@ jobs: dstack/kms/auth-eth/lib/openzeppelin-foundry-upgrades - name: Log in to GitHub Container Registry - uses: docker/login-action@v3 + uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9 # v3 with: registry: ghcr.io username: ${{ github.actor }} password: ${{ secrets.GITHUB_TOKEN }} - name: Set up Docker Buildx - uses: docker/setup-buildx-action@v3 + uses: docker/setup-buildx-action@8d2750c68a42422c14e847fe6c8ac0403b4cbd6f # v3 - name: Resolve the revision to build run: echo "GIT_REV=$(git rev-parse HEAD)" >> "$GITHUB_ENV" @@ -92,15 +92,24 @@ jobs: fi echo "digest=$digest" >> "$GITHUB_OUTPUT" + - name: Make package public + env: + GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + gh api --method PATCH \ + "/orgs/${GITHUB_REPOSITORY_OWNER}/packages/container/dstack-kms" \ + -f visibility=public \ + --jq 'select(.visibility == "public") | .html_url' + - name: Generate artifact attestation - uses: actions/attest-build-provenance@v1 + uses: actions/attest-build-provenance@92c65d2898f1f53cfdc910b962cecff86e7f8fcc # v1 with: subject-name: "ghcr.io/dstack-tee/dstack-kms" subject-digest: ${{ steps.build-and-push.outputs.digest }} push-to-registry: true - name: Install Foundry - uses: foundry-rs/foundry-toolchain@v1 + uses: foundry-rs/foundry-toolchain@908c540300062bd5a7e473851cdb4282204cee09 # v1 - name: Compile contracts with Foundry run: | @@ -109,7 +118,7 @@ jobs: forge build - name: GitHub Release - uses: softprops/action-gh-release@v2 + uses: softprops/action-gh-release@3bb12739c298aeb8a4eeaf626c5b8d85266b0e65 # v2 with: tag_name: ${{ env.RELEASE_TAG }} target_commitish: ${{ env.GIT_REV }} diff --git a/.github/workflows/local-key-provider-release.yml b/.github/workflows/local-key-provider-release.yml index f0700217a..984184c06 100644 --- a/.github/workflows/local-key-provider-release.yml +++ b/.github/workflows/local-key-provider-release.yml @@ -25,44 +25,70 @@ jobs: build-and-release: runs-on: ubuntu-latest steps: - - name: Checkout repository - uses: actions/checkout@v7 - - - name: Parse version from tag + - name: Parse and check version env: DISPATCH_VERSION: ${{ inputs.version }} run: | - if [[ "$GITHUB_REF" == refs/tags/local-key-provider-v* ]]; then - VERSION=${GITHUB_REF#refs/tags/local-key-provider-v} - else + if [ "$GITHUB_EVENT_NAME" = workflow_dispatch ]; then VERSION=$DISPATCH_VERSION + else + case "$GITHUB_REF" in + refs/tags/local-key-provider-v*) VERSION=${GITHUB_REF#refs/tags/local-key-provider-v} ;; + *) echo "Expected a local-key-provider-v* tag" >&2; exit 1 ;; + esac + fi + if [[ ! "$VERSION" =~ ^[A-Za-z0-9_][A-Za-z0-9_.-]{0,127}$ ]]; then + echo "Version must be a valid Docker tag" >&2 + exit 1 fi echo "VERSION=$VERSION" >> "$GITHUB_ENV" - echo "Parsed version: $VERSION" + echo "RELEASE_TAG=local-key-provider-v$VERSION" >> "$GITHUB_ENV" + + # Manual runs rebuild an existing release tag, not the selected branch. + - name: Checkout repository + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7 + with: + ref: ${{ github.event_name == 'workflow_dispatch' && env.RELEASE_TAG || github.sha }} - name: Log in to GitHub Container Registry - uses: docker/login-action@v4 + uses: docker/login-action@dbcb813823bdd20940b903addbd779551569679f # v4 with: registry: ghcr.io username: ${{ github.actor }} password: ${{ secrets.GITHUB_TOKEN }} - name: Set up Docker Buildx - uses: docker/setup-buildx-action@v4 + uses: docker/setup-buildx-action@37fe631027851001ddb9b187196cc803df7f5f0e # v4 + + - name: Resolve the revision to build + run: echo "GIT_REV=$(git rev-parse HEAD)" >> "$GITHUB_ENV" - name: Build and push Docker image id: build-and-push - uses: docker/build-push-action@v7 + uses: docker/build-push-action@53b7df96c91f9c12dcc8a07bcb9ccacbed38856a # v7 with: context: dstack file: dstack/local-key-provider/build/Dockerfile.key-provider push: true tags: ghcr.io/dstack-tee/local-key-provider:${{ env.VERSION }} + labels: | + org.opencontainers.image.source=${{ github.server_url }}/${{ github.repository }} + org.opencontainers.image.revision=${{ env.GIT_REV }} + org.opencontainers.image.version=${{ env.VERSION }} platforms: linux/amd64 provenance: false + - name: Make package public + env: + GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + gh api --method PATCH \ + "/orgs/${GITHUB_REPOSITORY_OWNER}/packages/container/local-key-provider" \ + -f visibility=public \ + --jq 'select(.visibility == "public") | .html_url' + - name: Generate artifact attestation - uses: actions/attest-build-provenance@v4 + uses: actions/attest-build-provenance@8beda2b7ed98355c0e97c0a63bec38ae472e66c4 # v4 with: subject-name: "ghcr.io/dstack-tee/local-key-provider" subject-digest: ${{ steps.build-and-push.outputs.digest }} @@ -87,10 +113,11 @@ jobs: echo "MRSIGNER=$MRSIGNER" >> "$GITHUB_ENV" - name: GitHub Release - uses: softprops/action-gh-release@v3 + uses: softprops/action-gh-release@5113cdc90fd4d541c801c55356214017bf5ae34b # v3 with: name: "Local Key Provider Release v${{ env.VERSION }}" - tag_name: local-key-provider-v${{ env.VERSION }} + tag_name: ${{ env.RELEASE_TAG }} + target_commitish: ${{ env.GIT_REV }} files: local-key-provider.sig.json body: | ## Docker Image Information diff --git a/.github/workflows/verifier-release.yml b/.github/workflows/verifier-release.yml index e837cf318..a599d12cf 100644 --- a/.github/workflows/verifier-release.yml +++ b/.github/workflows/verifier-release.yml @@ -45,19 +45,19 @@ jobs: # Manual runs rebuild an existing release tag, not the selected branch. - name: Checkout repository - uses: actions/checkout@v5 + uses: actions/checkout@fbc6f3992d24b796d5a048ff273f7fcc4a7b6c09 # v5 with: ref: ${{ github.event_name == 'workflow_dispatch' && env.RELEASE_TAG || github.sha }} - name: Log in to GitHub Container Registry - uses: docker/login-action@v3 + uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9 # v3 with: registry: ghcr.io username: ${{ github.actor }} password: ${{ secrets.GITHUB_TOKEN }} - name: Set up Docker Buildx - uses: docker/setup-buildx-action@v3 + uses: docker/setup-buildx-action@8d2750c68a42422c14e847fe6c8ac0403b4cbd6f # v3 - name: Resolve the revision to build run: echo "GIT_REV=$(git rev-parse HEAD)" >> "$GITHUB_ENV" @@ -87,15 +87,24 @@ jobs: fi echo "digest=$digest" >> "$GITHUB_OUTPUT" + - name: Make package public + env: + GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + gh api --method PATCH \ + "/orgs/${GITHUB_REPOSITORY_OWNER}/packages/container/dstack-verifier" \ + -f visibility=public \ + --jq 'select(.visibility == "public") | .html_url' + - name: Generate artifact attestation - uses: actions/attest-build-provenance@v1 + uses: actions/attest-build-provenance@92c65d2898f1f53cfdc910b962cecff86e7f8fcc # v1 with: subject-name: "ghcr.io/dstack-tee/dstack-verifier" subject-digest: ${{ steps.build-and-push.outputs.digest }} push-to-registry: true - name: GitHub Release - uses: softprops/action-gh-release@v2 + uses: softprops/action-gh-release@3bb12739c298aeb8a4eeaf626c5b8d85266b0e65 # v2 with: tag_name: ${{ env.RELEASE_TAG }} target_commitish: ${{ env.GIT_REV }} diff --git a/docs/rc-testing-runbook.md b/docs/rc-testing-runbook.md index 1e4478679..ba08a70ea 100644 --- a/docs/rc-testing-runbook.md +++ b/docs/rc-testing-runbook.md @@ -99,8 +99,8 @@ the CLI's default `--url`. On both machines used last time it pointed at a live deployment with credentials, so any `deploy` that omitted `--url` would have landed there. **Pass `--url http://127.0.0.1:` on every single invocation.** -**Component tags publish to Docker Hub.** `kms-v*`, `gateway-v*`, `verifier-v*` -and friends trigger release workflows that push to `${DOCKERHUB_ORG}` and cut +**Component tags publish to GHCR.** `kms-v*`, `gateway-v*`, `verifier-v*` +and friends trigger release workflows that push to `ghcr.io/dstack-tee` and cut GitHub releases. If you only want images in a private registry, build locally and push by hand — do not push those tags. diff --git a/dstack/gateway/dstack-app/builder/README.md b/dstack/gateway/dstack-app/builder/README.md index 18e074452..b8aedad53 100644 --- a/dstack/gateway/dstack-app/builder/README.md +++ b/dstack/gateway/dstack-app/builder/README.md @@ -59,17 +59,17 @@ GIT_REV=HEAD \ IMAGE_VERSION=0.6.0 \ IMAGE_SOURCE_URL=https://github.com/Dstack-TEE/dstack \ OCI_TAR=/tmp/gateway.oci.tar \ - ./build-image.sh dstacktee/dstack-gateway:0.6.0 + ./build-image.sh ghcr.io/dstack-tee/dstack-gateway:0.6.0 python3 -c 'import json,tarfile;t=tarfile.open("/tmp/gateway.oci.tar");print(json.load(t.extractfile("index.json"))["manifests"][0]["digest"])' ``` `IMAGE_VERSION` is part of the image metadata, so it must match the release for the digests to match. The printed digest is what the registry reports for -`dstacktee/dstack-gateway:0.6.0`; compare it with: +`ghcr.io/dstack-tee/dstack-gateway:0.6.0`; compare it with: ```bash -docker buildx imagetools inspect dstacktee/dstack-gateway:0.6.0 --format '{{.Manifest.Digest}}' +docker buildx imagetools inspect ghcr.io/dstack-tee/dstack-gateway:0.6.0 --format '{{.Manifest.Digest}}' ``` ## Image metadata diff --git a/dstack/kms/dstack-app/builder/README.md b/dstack/kms/dstack-app/builder/README.md index 923b34ebc..382197e5d 100644 --- a/dstack/kms/dstack-app/builder/README.md +++ b/dstack/kms/dstack-app/builder/README.md @@ -58,17 +58,17 @@ GIT_REV=HEAD \ IMAGE_VERSION=0.6.0 \ IMAGE_SOURCE_URL=https://github.com/Dstack-TEE/dstack \ OCI_TAR=/tmp/kms.oci.tar \ - ./build-image.sh dstacktee/dstack-kms:0.6.0 + ./build-image.sh ghcr.io/dstack-tee/dstack-kms:0.6.0 python3 -c 'import json,tarfile;t=tarfile.open("/tmp/kms.oci.tar");print(json.load(t.extractfile("index.json"))["manifests"][0]["digest"])' ``` `IMAGE_VERSION` is part of the image metadata, so it must match the release for the digests to match. The printed digest is what the registry reports for -`dstacktee/dstack-kms:0.6.0`; compare it with: +`ghcr.io/dstack-tee/dstack-kms:0.6.0`; compare it with: ```bash -docker buildx imagetools inspect dstacktee/dstack-kms:0.6.0 --format '{{.Manifest.Digest}}' +docker buildx imagetools inspect ghcr.io/dstack-tee/dstack-kms:0.6.0 --format '{{.Manifest.Digest}}' ``` This is the digest that `deploy-to-vmm.sh` pins in `KMS_IMAGE`, and that in turn diff --git a/dstack/local-key-provider/README.md b/dstack/local-key-provider/README.md index cf3cc09ae..ddd28e7bc 100644 --- a/dstack/local-key-provider/README.md +++ b/dstack/local-key-provider/README.md @@ -24,6 +24,6 @@ provider under Gramine. The container build uses the repository workspace's lockfile. Tagged releases (`local-key-provider-v*`) publish the image as -`docker.io/dstacktee/local-key-provider:`. The corresponding GitHub +`ghcr.io/dstack-tee/local-key-provider:`. The corresponding GitHub release records the image digest and enclave measurements, attaches the full SIGSTRUCT metadata, and links to its Sigstore build-provenance attestation. diff --git a/dstack/verifier/README.md b/dstack/verifier/README.md index 232ede636..8529dc1f6 100644 --- a/dstack/verifier/README.md +++ b/dstack/verifier/README.md @@ -169,7 +169,7 @@ check with the policy fields emitted by `/verify`: accepted OS image, ```yaml services: dstack-verifier: - image: dstacktee/dstack-verifier:latest + image: ghcr.io/dstack-tee/dstack-verifier:latest ports: - "8080:8080" restart: unless-stopped diff --git a/dstack/verifier/builder/README.md b/dstack/verifier/builder/README.md index 9e409e925..728028156 100644 --- a/dstack/verifier/builder/README.md +++ b/dstack/verifier/builder/README.md @@ -59,17 +59,17 @@ GIT_REV=HEAD \ IMAGE_VERSION=0.6.0 \ IMAGE_SOURCE_URL=https://github.com/Dstack-TEE/dstack \ OCI_TAR=/tmp/verifier.oci.tar \ - ./build-image.sh dstacktee/dstack-verifier:0.6.0 + ./build-image.sh ghcr.io/dstack-tee/dstack-verifier:0.6.0 python3 -c 'import json,tarfile;t=tarfile.open("/tmp/verifier.oci.tar");print(json.load(t.extractfile("index.json"))["manifests"][0]["digest"])' ``` `IMAGE_VERSION` is part of the image metadata, so it must match the release for the digests to match. The printed digest is what the registry reports for -`dstacktee/dstack-verifier:0.6.0`; compare it with: +`ghcr.io/dstack-tee/dstack-verifier:0.6.0`; compare it with: ```bash -docker buildx imagetools inspect dstacktee/dstack-verifier:0.6.0 --format '{{.Manifest.Digest}}' +docker buildx imagetools inspect ghcr.io/dstack-tee/dstack-verifier:0.6.0 --format '{{.Manifest.Digest}}' ``` Note that the release also publishes a `:latest` tag pointing at the same From 0c6b751fa46b369418e9125ce14e7622f35e105f Mon Sep 17 00:00:00 2001 From: Kevin Wang Date: Mon, 14 Sep 2026 23:55:56 -0700 Subject: [PATCH 19/93] ci: verify GHCR images are public --- .github/workflows/gateway-release.yml | 15 +++++++++------ .github/workflows/kms-release.yml | 15 +++++++++------ .github/workflows/local-key-provider-release.yml | 15 +++++++++------ .github/workflows/verifier-release.yml | 15 +++++++++------ 4 files changed, 36 insertions(+), 24 deletions(-) diff --git a/.github/workflows/gateway-release.yml b/.github/workflows/gateway-release.yml index 116e3739c..b3017ded0 100644 --- a/.github/workflows/gateway-release.yml +++ b/.github/workflows/gateway-release.yml @@ -85,14 +85,17 @@ jobs: fi echo "digest=$digest" >> "$GITHUB_OUTPUT" - - name: Make package public + - name: Verify anonymous pull access env: - GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} + IMAGE_DIGEST: ${{ steps.build-and-push.outputs.digest }} run: | - gh api --method PATCH \ - "/orgs/${GITHUB_REPOSITORY_OWNER}/packages/container/dstack-gateway" \ - -f visibility=public \ - --jq 'select(.visibility == "public") | .html_url' + token=$(curl -fsSL \ + "https://ghcr.io/token?scope=repository:dstack-tee/dstack-gateway:pull" \ + | jq -er .token) + curl -fsSL -o /dev/null \ + -H "Authorization: Bearer $token" \ + -H "Accept: application/vnd.oci.image.manifest.v1+json" \ + "https://ghcr.io/v2/dstack-tee/dstack-gateway/manifests/$IMAGE_DIGEST" - name: Generate artifact attestation uses: actions/attest-build-provenance@92c65d2898f1f53cfdc910b962cecff86e7f8fcc # v1 diff --git a/.github/workflows/kms-release.yml b/.github/workflows/kms-release.yml index cccc6e7f4..9b5cda2ec 100644 --- a/.github/workflows/kms-release.yml +++ b/.github/workflows/kms-release.yml @@ -92,14 +92,17 @@ jobs: fi echo "digest=$digest" >> "$GITHUB_OUTPUT" - - name: Make package public + - name: Verify anonymous pull access env: - GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} + IMAGE_DIGEST: ${{ steps.build-and-push.outputs.digest }} run: | - gh api --method PATCH \ - "/orgs/${GITHUB_REPOSITORY_OWNER}/packages/container/dstack-kms" \ - -f visibility=public \ - --jq 'select(.visibility == "public") | .html_url' + token=$(curl -fsSL \ + "https://ghcr.io/token?scope=repository:dstack-tee/dstack-kms:pull" \ + | jq -er .token) + curl -fsSL -o /dev/null \ + -H "Authorization: Bearer $token" \ + -H "Accept: application/vnd.oci.image.manifest.v1+json" \ + "https://ghcr.io/v2/dstack-tee/dstack-kms/manifests/$IMAGE_DIGEST" - name: Generate artifact attestation uses: actions/attest-build-provenance@92c65d2898f1f53cfdc910b962cecff86e7f8fcc # v1 diff --git a/.github/workflows/local-key-provider-release.yml b/.github/workflows/local-key-provider-release.yml index 984184c06..49d39d410 100644 --- a/.github/workflows/local-key-provider-release.yml +++ b/.github/workflows/local-key-provider-release.yml @@ -78,14 +78,17 @@ jobs: platforms: linux/amd64 provenance: false - - name: Make package public + - name: Verify anonymous pull access env: - GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} + IMAGE_DIGEST: ${{ steps.build-and-push.outputs.digest }} run: | - gh api --method PATCH \ - "/orgs/${GITHUB_REPOSITORY_OWNER}/packages/container/local-key-provider" \ - -f visibility=public \ - --jq 'select(.visibility == "public") | .html_url' + token=$(curl -fsSL \ + "https://ghcr.io/token?scope=repository:dstack-tee/local-key-provider:pull" \ + | jq -er .token) + curl -fsSL -o /dev/null \ + -H "Authorization: Bearer $token" \ + -H "Accept: application/vnd.oci.image.manifest.v1+json" \ + "https://ghcr.io/v2/dstack-tee/local-key-provider/manifests/$IMAGE_DIGEST" - name: Generate artifact attestation uses: actions/attest-build-provenance@8beda2b7ed98355c0e97c0a63bec38ae472e66c4 # v4 diff --git a/.github/workflows/verifier-release.yml b/.github/workflows/verifier-release.yml index a599d12cf..f3ca48f79 100644 --- a/.github/workflows/verifier-release.yml +++ b/.github/workflows/verifier-release.yml @@ -87,14 +87,17 @@ jobs: fi echo "digest=$digest" >> "$GITHUB_OUTPUT" - - name: Make package public + - name: Verify anonymous pull access env: - GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} + IMAGE_DIGEST: ${{ steps.build-and-push.outputs.digest }} run: | - gh api --method PATCH \ - "/orgs/${GITHUB_REPOSITORY_OWNER}/packages/container/dstack-verifier" \ - -f visibility=public \ - --jq 'select(.visibility == "public") | .html_url' + token=$(curl -fsSL \ + "https://ghcr.io/token?scope=repository:dstack-tee/dstack-verifier:pull" \ + | jq -er .token) + curl -fsSL -o /dev/null \ + -H "Authorization: Bearer $token" \ + -H "Accept: application/vnd.oci.image.manifest.v1+json" \ + "https://ghcr.io/v2/dstack-tee/dstack-verifier/manifests/$IMAGE_DIGEST" - name: Generate artifact attestation uses: actions/attest-build-provenance@92c65d2898f1f53cfdc910b962cecff86e7f8fcc # v1 From bc760b29a5effe3a323bbbaf389120939bf227a3 Mon Sep 17 00:00:00 2001 From: Kevin Wang Date: Tue, 15 Sep 2026 00:08:07 -0700 Subject: [PATCH 20/93] ci: build KMS contracts from checked-out dependencies --- .github/workflows/kms-release.yml | 1 - 1 file changed, 1 deletion(-) diff --git a/.github/workflows/kms-release.yml b/.github/workflows/kms-release.yml index 9b5cda2ec..eedb5d66a 100644 --- a/.github/workflows/kms-release.yml +++ b/.github/workflows/kms-release.yml @@ -117,7 +117,6 @@ jobs: - name: Compile contracts with Foundry run: | cd dstack/kms/auth-eth - forge install forge build - name: GitHub Release From 3b75a4c09ed907830bc6ec0dd151189a5520cce9 Mon Sep 17 00:00:00 2001 From: Leechael Yim Date: Wed, 9 Sep 2026 14:03:12 +0800 Subject: [PATCH 21/93] feat(vmm): add a status filter to the console instance list The console could search by keyword but not by lifecycle status. The list is paginated server-side, so a client-only filter would only cover the current page and would report the wrong total. Add StatusRequest.status and apply it in list_vms before pagination, using the same running/stopped/exited/stopping/removing values already shown in the Status column. Empty status keeps the previous all-status behavior. The toolbar select resets to page 1, matching page-size changes. Status computation is shared through vm_runtime_status so the filter and the list agree. Verified with `npm run build` in dstack/vmm/ui, `cargo check -p dstack-vmm-rpc`, and prek on the changed files. --- dstack/vmm/rpc/proto/vmm_rpc.proto | 5 +- dstack/vmm/src/app.rs | 8 +++ dstack/vmm/src/app/vm_info.rs | 51 ++++++++++++++----- dstack/vmm/ui/src/composables/useVmManager.ts | 10 ++++ dstack/vmm/ui/src/styles/main.css | 18 +++++++ dstack/vmm/ui/src/templates/app.html | 13 +++++ 6 files changed, 92 insertions(+), 13 deletions(-) diff --git a/dstack/vmm/rpc/proto/vmm_rpc.proto b/dstack/vmm/rpc/proto/vmm_rpc.proto index 62bf381be..209c350a5 100644 --- a/dstack/vmm/rpc/proto/vmm_rpc.proto +++ b/dstack/vmm/rpc/proto/vmm_rpc.proto @@ -229,7 +229,7 @@ message UpdateVmRequest { repeated NetworkingConfig networks = 20; } -// Parameters for listing CVMs with pagination and keyword filtering. +// Parameters for listing CVMs with pagination and filtering. message StatusRequest { // List of VM IDs repeated string ids = 1; @@ -241,6 +241,9 @@ message StatusRequest { uint32 page = 4; // Page size uint32 page_size = 5; + // Filter by VM status: running, stopped, exited, stopping, or removing. + // Empty means all statuses. + string status = 6; } // Summary of the current fleet status. diff --git a/dstack/vmm/src/app.rs b/dstack/vmm/src/app.rs index 6736a363c..1920a0a14 100644 --- a/dstack/vmm/src/app.rs +++ b/dstack/vmm/src/app.rs @@ -1409,6 +1409,14 @@ impl App { }) .cloned() .collect::>(); + if !request.status.is_empty() { + infos.retain(|vm| match self.work_dir(&vm.config.manifest.id) { + Ok(work_dir) => { + vm.runtime_status(vms.get(&vm.config.manifest.id), &work_dir) == request.status + } + Err(_) => false, + }); + } infos.sort_by(|a, b| { a.config .manifest diff --git a/dstack/vmm/src/app/vm_info.rs b/dstack/vmm/src/app/vm_info.rs index 48af6f864..1e60c72d7 100644 --- a/dstack/vmm/src/app/vm_info.rs +++ b/dstack/vmm/src/app/vm_info.rs @@ -126,6 +126,19 @@ fn sanitize_optional>(value: Option) -> Option { value.filter(|value| !value.as_ref().trim().is_empty()) } +pub(crate) fn vm_runtime_status(removing: bool, started: bool, is_running: bool) -> &'static str { + if removing { + "removing" + } else { + match (started, is_running) { + (true, true) => "running", + (true, false) => "exited", + (false, true) => "stopping", + (false, false) => "stopped", + } + } +} + impl VmInfo { /// Takes no `CvmConfig` on purpose. Everything it reports about a VM's /// data plane was decided when that VM launched and written into @@ -271,6 +284,18 @@ fn app_url(id: &str, custom_gateway_urls: &[String], gateway: &GatewayConfig) -> } impl VmState { + pub(crate) fn runtime_status( + &self, + process: Option<&ProcessInfo>, + workdir: &VmWorkDir, + ) -> &'static str { + vm_runtime_status( + self.state.removing, + workdir.started().unwrap_or(false), + process.is_some_and(|info| info.state.status.is_running()), + ) + } + pub fn merged_info(&self, process: Option<&ProcessInfo>, workdir: &VmWorkDir) -> VmInfo { fn truncate(duration: Duration) -> Duration { Duration::from_secs(duration.as_secs()) @@ -287,17 +312,7 @@ impl VmState { } let is_running = process.is_some_and(|info| info.state.status.is_running()); - let started = workdir.started().unwrap_or(false); - let status = if self.state.removing { - "removing" - } else { - match (started, is_running) { - (true, true) => "running", - (true, false) => "exited", - (false, true) => "stopping", - (false, false) => "stopped", - } - }; + let status = self.runtime_status(process, workdir); let uptime = display_timestamp(process.and_then(|info| info.state.started_at.as_ref())); let exited_at = display_timestamp(process.and_then(|info| info.state.stopped_at.as_ref())); let instance_id = sanitize_optional( @@ -328,9 +343,21 @@ impl VmState { #[cfg(test)] mod tests { - use super::{interfaces_to_proto, networking_to_proto, sanitize_optional}; + use super::{ + interfaces_to_proto, networking_to_proto, sanitize_optional, vm_runtime_status, + }; use crate::config::{NetworkingMode, NicNetworking}; + #[test] + fn runtime_status_covers_lifecycle() { + assert_eq!(vm_runtime_status(true, true, true), "removing"); + assert_eq!(vm_runtime_status(true, false, false), "removing"); + assert_eq!(vm_runtime_status(false, true, true), "running"); + assert_eq!(vm_runtime_status(false, true, false), "exited"); + assert_eq!(vm_runtime_status(false, false, true), "stopping"); + assert_eq!(vm_runtime_status(false, false, false), "stopped"); + } + /// Custom mode hands the operator the whole netdev string and the VMM never /// parses it, so it has no data-plane state to report. Reporting the /// resolved defaults instead asserted "vhost off, one queue" over a netdev diff --git a/dstack/vmm/ui/src/composables/useVmManager.ts b/dstack/vmm/ui/src/composables/useVmManager.ts index 976a75c7c..0060bd59d 100644 --- a/dstack/vmm/ui/src/composables/useVmManager.ts +++ b/dstack/vmm/ui/src/composables/useVmManager.ts @@ -323,6 +323,7 @@ function useVmManager() { const expandedVMs = ref(new Set() as Set); const networkInfo = ref({} as Record); const searchQuery = ref(''); + const statusFilter = ref(''); const currentPage = ref(1); const pageInput = ref(1); const pageSize = ref(Number.parseInt(localStorage.getItem('pageSize') || '50', 10)); @@ -708,6 +709,7 @@ type CreateVmPayloadSource = { const request: VmmTypes.IStatusRequest = { brief: true, keyword: searchQuery.value || undefined, + status: statusFilter.value || undefined, page: currentPage.value, page_size: pageSize.value, }; @@ -1609,6 +1611,12 @@ type CreateVmPayloadSource = { loadVMList(); } + function onStatusFilterChange() { + currentPage.value = 1; + pageInput.value = 1; + loadVMList(); + } + async function startVm(id: string) { try { await vmmRpc.startVm({ id }); @@ -1911,6 +1919,7 @@ type CreateVmPayloadSource = { expandedVMs, networkInfo, searchQuery, + statusFilter, currentPage, pageInput, pageSize, @@ -1966,6 +1975,7 @@ type CreateVmPayloadSource = { nextPage, prevPage, onPageSizeChange, + onStatusFilterChange, copyToClipboard, downloadAppCompose, downloadUserConfig, diff --git a/dstack/vmm/ui/src/styles/main.css b/dstack/vmm/ui/src/styles/main.css index 113109a42..3f4f51be2 100644 --- a/dstack/vmm/ui/src/styles/main.css +++ b/dstack/vmm/ui/src/styles/main.css @@ -216,6 +216,24 @@ h1, h2, h3, h4, h5, h6 { background: var(--color-primary-hover); } +.status-filter-select { + border: 1px solid var(--color-border); + border-radius: var(--radius-md); + padding: 10px 12px; + font-size: 13px; + outline: none; + cursor: pointer; + background: var(--color-bg-primary); + color: var(--color-text-secondary); + font-weight: 500; + box-shadow: var(--shadow-sm); +} + +.status-filter-select:hover, +.status-filter-select:focus { + border-color: var(--color-text-tertiary); +} + .vm-count { display: flex; align-items: center; diff --git a/dstack/vmm/ui/src/templates/app.html b/dstack/vmm/ui/src/templates/app.html index 965f7bcc2..bb24dabe0 100644 --- a/dstack/vmm/ui/src/templates/app.html +++ b/dstack/vmm/ui/src/templates/app.html @@ -127,6 +127,19 @@

dstack-vmm

+
Total Instances: {{ totalVMs }} From 0f8d6b995c91c78577b61b811cfde9ec5e8e7a99 Mon Sep 17 00:00:00 2001 From: Leechael Yim Date: Tue, 15 Sep 2026 15:26:54 +0800 Subject: [PATCH 22/93] style(vmm): rustfmt the status-filter test imports prek and cargo fmt --check failed on the four-item use super line introduced while resolving the rebase conflict. --- dstack/vmm/src/app/vm_info.rs | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/dstack/vmm/src/app/vm_info.rs b/dstack/vmm/src/app/vm_info.rs index 1e60c72d7..8b5b524d4 100644 --- a/dstack/vmm/src/app/vm_info.rs +++ b/dstack/vmm/src/app/vm_info.rs @@ -343,9 +343,7 @@ impl VmState { #[cfg(test)] mod tests { - use super::{ - interfaces_to_proto, networking_to_proto, sanitize_optional, vm_runtime_status, - }; + use super::{interfaces_to_proto, networking_to_proto, sanitize_optional, vm_runtime_status}; use crate::config::{NetworkingMode, NicNetworking}; #[test] From c11c410e30f3448481b85ed7ee55522c7c6afd96 Mon Sep 17 00:00:00 2001 From: Wim Yedema Date: Tue, 15 Sep 2026 14:16:41 +0000 Subject: [PATCH 23/93] fix(yocto): avoid preserving source groups during unpack The dstack recipes copy source trees from outside BitBake's work directory. Archive-mode rsync attempts to preserve their group ownership, which can fail when the build user cannot assign the source group.\n\nDisable group preservation while retaining the other archive-mode attributes. --- .../meta-dstack/recipes-core/dstack-guest/dstack-guest.bb | 4 ++-- .../recipes-core/dstack-tee-simulator/dstack-tee-simulator.bb | 2 +- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/os/yocto/layers/meta-dstack/recipes-core/dstack-guest/dstack-guest.bb b/os/yocto/layers/meta-dstack/recipes-core/dstack-guest/dstack-guest.bb index 63757007e..caed68b31 100644 --- a/os/yocto/layers/meta-dstack/recipes-core/dstack-guest/dstack-guest.bb +++ b/os/yocto/layers/meta-dstack/recipes-core/dstack-guest/dstack-guest.bb @@ -38,9 +38,9 @@ inherit cargo_bin do_unpack() { install -d "${S}" "${DSTACK_ROOTFS_FILES}" - rsync -a --exclude=".git" --exclude=".worktrees" --exclude="target" \ + rsync -a --no-group --exclude=".git" --exclude=".worktrees" --exclude="target" \ "${DSTACK_CORE_SRC}/" "${S}/" - rsync -a "${DSTACK_ROOTFS_SRC}/" "${DSTACK_ROOTFS_FILES}/" + rsync -a --no-group "${DSTACK_ROOTFS_SRC}/" "${DSTACK_ROOTFS_FILES}/" } do_unpack[cleandirs] = "${UNPACKDIR}/repo" diff --git a/os/yocto/layers/meta-dstack/recipes-core/dstack-tee-simulator/dstack-tee-simulator.bb b/os/yocto/layers/meta-dstack/recipes-core/dstack-tee-simulator/dstack-tee-simulator.bb index de42fa5ee..828a5caa5 100644 --- a/os/yocto/layers/meta-dstack/recipes-core/dstack-tee-simulator/dstack-tee-simulator.bb +++ b/os/yocto/layers/meta-dstack/recipes-core/dstack-tee-simulator/dstack-tee-simulator.bb @@ -28,7 +28,7 @@ inherit cargo_bin do_unpack() { install -d "${S}" - rsync -a --exclude=".git" --exclude=".worktrees" --exclude="target" \ + rsync -a --no-group --exclude=".git" --exclude=".worktrees" --exclude="target" \ "${DSTACK_CORE_SRC}/" "${S}/" } From fdfdfbd081d8369e99fa101f24e81b7f60d6b59f Mon Sep 17 00:00:00 2001 From: Kevin Wang Date: Tue, 15 Sep 2026 20:24:14 -0700 Subject: [PATCH 24/93] fix(yocto): drop source owner as well during unpack staging `--no-group` only covers the receiver-is-not-super-user path. When bitbake runs as (fake) root -- rootless docker/podman, or `sudo os/build.sh` -- archive mode still applies `-o`, and chown to an unmapped source uid fails the same way: rsync: [receiver] chown ".../.f.txt.nQSbHp" failed: Invalid argument (22) rsync error: some files/attrs were not transferred (code 23) The aborted transfer also leaves the file at the temp-file mode (0600) instead of the source mode, so a partially-staged tree can fail later tasks. Verified with rsync 3.2.7 across the three privilege setups (non-root without the source group, non-root with an unmappable gid, userns-root with an unmappable uid): only `--no-owner --no-group` succeeds in all three. --- .../recipes-core/dstack-guest/dstack-guest.bb | 10 ++++++++-- .../dstack-tee-simulator/dstack-tee-simulator.bb | 8 +++++++- 2 files changed, 15 insertions(+), 3 deletions(-) diff --git a/os/yocto/layers/meta-dstack/recipes-core/dstack-guest/dstack-guest.bb b/os/yocto/layers/meta-dstack/recipes-core/dstack-guest/dstack-guest.bb index caed68b31..c2beca373 100644 --- a/os/yocto/layers/meta-dstack/recipes-core/dstack-guest/dstack-guest.bb +++ b/os/yocto/layers/meta-dstack/recipes-core/dstack-guest/dstack-guest.bb @@ -36,11 +36,17 @@ EXTRA_CARGO_FLAGS = "-p dstack-guest-agent -p dstack-util -p dstack-volume" inherit cargo_bin +# The staged source lives outside WORKDIR and carries the host checkout's +# ownership. WORKDIR staging has no use for it, and applying it fails outright +# under container / user-namespace builds where the source uid/gid is not +# settable (rsync aborts with "chown/chgrp ... Invalid argument", exit 23, and +# leaves the partially-attributed temp file behind). Drop owner and group, keep +# the rest of archive mode -- permissions still matter for executable bits. do_unpack() { install -d "${S}" "${DSTACK_ROOTFS_FILES}" - rsync -a --no-group --exclude=".git" --exclude=".worktrees" --exclude="target" \ + rsync -a --no-owner --no-group --exclude=".git" --exclude=".worktrees" --exclude="target" \ "${DSTACK_CORE_SRC}/" "${S}/" - rsync -a --no-group "${DSTACK_ROOTFS_SRC}/" "${DSTACK_ROOTFS_FILES}/" + rsync -a --no-owner --no-group "${DSTACK_ROOTFS_SRC}/" "${DSTACK_ROOTFS_FILES}/" } do_unpack[cleandirs] = "${UNPACKDIR}/repo" diff --git a/os/yocto/layers/meta-dstack/recipes-core/dstack-tee-simulator/dstack-tee-simulator.bb b/os/yocto/layers/meta-dstack/recipes-core/dstack-tee-simulator/dstack-tee-simulator.bb index 828a5caa5..b222818fd 100644 --- a/os/yocto/layers/meta-dstack/recipes-core/dstack-tee-simulator/dstack-tee-simulator.bb +++ b/os/yocto/layers/meta-dstack/recipes-core/dstack-tee-simulator/dstack-tee-simulator.bb @@ -26,9 +26,15 @@ EXTRA_CARGO_FLAGS = "-p dstack-tee-simulator" inherit cargo_bin +# The staged source lives outside WORKDIR and carries the host checkout's +# ownership. WORKDIR staging has no use for it, and applying it fails outright +# under container / user-namespace builds where the source uid/gid is not +# settable (rsync aborts with "chown/chgrp ... Invalid argument", exit 23, and +# leaves the partially-attributed temp file behind). Drop owner and group, keep +# the rest of archive mode -- permissions still matter for executable bits. do_unpack() { install -d "${S}" - rsync -a --no-group --exclude=".git" --exclude=".worktrees" --exclude="target" \ + rsync -a --no-owner --no-group --exclude=".git" --exclude=".worktrees" --exclude="target" \ "${DSTACK_CORE_SRC}/" "${S}/" } From 576b0d2e9430857df3a7e0988a6c0046c0ff5244 Mon Sep 17 00:00:00 2001 From: Kevin Wang Date: Wed, 16 Sep 2026 19:20:22 -0700 Subject: [PATCH 25/93] feat(os): export the guest kernel build tree for out-of-tree modules --- os/common/scripts/export-kernel-devel.sh | 161 +++++++++++++++++++++++ os/tests/kmod-hello/Makefile | 17 +++ os/tests/kmod-hello/hello.c | 29 ++++ 3 files changed, 207 insertions(+) create mode 100755 os/common/scripts/export-kernel-devel.sh create mode 100644 os/tests/kmod-hello/Makefile create mode 100644 os/tests/kmod-hello/hello.c diff --git a/os/common/scripts/export-kernel-devel.sh b/os/common/scripts/export-kernel-devel.sh new file mode 100755 index 000000000..33e47c0c5 --- /dev/null +++ b/os/common/scripts/export-kernel-devel.sh @@ -0,0 +1,161 @@ +#!/bin/bash +# SPDX-License-Identifier: Apache-2.0 +# +# Export the minimal kernel build tree an application needs to compile an +# out-of-tree module (`make -C $KDIR M=$PWD modules`) against the guest kernel. +# +# The guest image ships modules but no build tree, so an application that needs +# its own .ko had to reconstruct a byte-identical kernel build to obtain a +# matching Module.symvers and vermagic. This exports that tree from the build +# that produced the shipped kernel instead. +# +# The file selection is upstream's: scripts/package/install-extmod-build is +# what `make bindeb-pkg` uses to produce Debian's linux-headers package, so the +# set of files carried here tracks Kbuild rather than a list maintained by +# dstack. It is invoked through `make run-command` because it reads srctree, +# SRCARCH, CC and HOSTCC out of the Kbuild environment. +# +# The result is deliberately NOT installed into the guest rootfs: it is ~100 MB +# of build inputs no CVM needs at runtime, and adding it would change the +# rootfs verity hash and therefore the OS image identity. It ships as a +# separate release artifact. +set -euo pipefail + +usage() { + cat <&2; usage >&2; exit 2 ;; + esac +done +if [ -z "$SRC" ] || [ -z "$BUILD" ] || [ -z "$OUT" ]; then + usage >&2 + exit 2 +fi +SRC=$(realpath "$SRC") +BUILD=$(realpath "$BUILD") +OUT=$(realpath -m "$OUT") + +exporter="$SRC/scripts/package/install-extmod-build" +[ -x "$exporter" ] || { + echo "kernel source does not provide scripts/package/install-extmod-build: $SRC" >&2 + exit 1 +} +config="$BUILD/.config" +release_file="$BUILD/include/config/kernel.release" +symvers="$BUILD/Module.symvers" +for required in "$config" "$release_file" "$symvers"; do + [ -f "$required" ] || { + echo "kernel output directory is incomplete, build modules first: $required" >&2 + exit 1 + } +done +release=$(cat "$release_file") +[ -n "$release" ] || { echo "empty kernel release in $release_file" >&2; exit 1; } +version=$(make -C "$SRC" -s kernelversion) + +headers="$OUT/linux-headers-$release" +rm -rf "$OUT" +mkdir -p "$OUT" +# run-command exports the Kbuild environment install-extmod-build reads, and it +# runs in the output directory, which is where the script expects to find +# Module.symvers and include/config. This is exactly how scripts/package/builddeb +# invokes it. +make -C "$SRC" O="$BUILD" run-command \ + KBUILD_RUN_COMMAND="$exporter $headers" +# install-extmod-build carries include/config/auto.conf, which is what Kbuild +# reads for an external module build, but not .config. Ship it anyway: it is +# what an application inspects to find out whether a config option it needs is +# enabled, and kernel-hardening or module-compat questions are answered from it. +install -m0644 "$config" "$headers/.config" +# The one file upstream's list carries that cannot be reproducible: +# scripts/mod/devicetable-offsets.s is a -fverbose-asm intermediate whose +# comments and line markers record the absolute source path of the build that +# produced it, and no prefix map rewrites them (-ffile-prefix-map applies to +# __FILE__ and debug info, not to -S output; measured, the paths survive). +# Dropping it costs nothing: it exists only to generate +# scripts/mod/devicetable-offsets.h, which ships, and so does its .c source. +find "$headers" -name '*.s' -delete + +# Written for the consumer, not for the build: an out-of-tree module is only +# loadable by the image whose kernel produced this tree, so the tree has to say +# which image that is, and a Module.symvers or .config that differs from the +# shipped one silently produces a module that fails to load. +python3 - "$OUT" "$headers" "$release" "$version" "$BACKEND" "$IMAGE_VERSION" \ + "$FLAVOR" <<'PY' +import hashlib +import json +import os +import sys + +out, headers, release, version, backend, image_version, flavor = sys.argv[1:] + + +def digest(path): + with open(path, "rb") as handle: + return hashlib.sha256(handle.read()).hexdigest() + + +def compiler(): + prefix = 'CONFIG_CC_VERSION_TEXT="' + with open(os.path.join(headers, ".config"), encoding="utf-8") as handle: + for line in handle: + if line.startswith(prefix): + return line.rstrip("\n")[len(prefix) : -1] + return "" + + +files = 0 +total = 0 +for root, _, names in os.walk(headers): + for name in names: + path = os.path.join(root, name) + files += 1 + if not os.path.islink(path): + total += os.path.getsize(path) + +metadata = { + "backend": backend, + "compiler": compiler(), + "config_sha256": digest(os.path.join(headers, ".config")), + "file_count": files, + "flavor": flavor, + "image_version": image_version, + "kernel_release": release, + "kernel_version": version, + "module_symvers_sha256": digest(os.path.join(headers, "Module.symvers")), + "uncompressed_bytes": total, +} +with open(os.path.join(out, "kernel-devel.json"), "w", encoding="utf-8") as handle: + json.dump(metadata, handle, indent=2, sort_keys=True) + handle.write("\n") +PY + +if [ -n "${SOURCE_DATE_EPOCH:-}" ]; then + find "$OUT" -print0 | xargs -0r touch --no-dereference \ + --date="@$SOURCE_DATE_EPOCH" +fi +echo "exported kernel build tree for $release to $OUT" diff --git a/os/tests/kmod-hello/Makefile b/os/tests/kmod-hello/Makefile new file mode 100644 index 000000000..fa1ff8121 --- /dev/null +++ b/os/tests/kmod-hello/Makefile @@ -0,0 +1,17 @@ +# SPDX-FileCopyrightText: © 2026 Phala Network +# +# SPDX-License-Identifier: GPL-2.0-only +# Out-of-tree module build, exactly as an application would do it: point KDIR +# at the exported dstack kernel build tree and let Kbuild drive the build. +obj-m += dstack-hello.o +dstack-hello-objs := hello.o + +KDIR ?= /lib/modules/$(shell uname -r)/build + +all: + $(MAKE) -C $(KDIR) M=$(CURDIR) modules + +clean: + $(MAKE) -C $(KDIR) M=$(CURDIR) clean + +.PHONY: all clean diff --git a/os/tests/kmod-hello/hello.c b/os/tests/kmod-hello/hello.c new file mode 100644 index 000000000..035856a7d --- /dev/null +++ b/os/tests/kmod-hello/hello.c @@ -0,0 +1,29 @@ +// SPDX-FileCopyrightText: © 2026 Phala Network +// +// SPDX-License-Identifier: GPL-2.0-only +/* + * Smallest out-of-tree module that still exercises everything an application + * module needs from the exported kernel build tree: the UAPI and internal + * headers, Module.symvers (module_init/module_exit are resolved through it) + * and the vermagic the running guest kernel will check on insmod. + */ + +#include +#include + +static int __init dstack_hello_init(void) +{ + pr_info("dstack-hello: loaded\n"); + return 0; +} + +static void __exit dstack_hello_exit(void) +{ + pr_info("dstack-hello: unloaded\n"); +} + +module_init(dstack_hello_init); +module_exit(dstack_hello_exit); + +MODULE_LICENSE("GPL"); +MODULE_DESCRIPTION("dstack out-of-tree module build smoke test"); From 1581cc910523f6f815ea135cf2867b3d8bdf0912 Mon Sep 17 00:00:00 2001 From: Kevin Wang Date: Wed, 16 Sep 2026 19:20:22 -0700 Subject: [PATCH 26/93] feat(os/image): publish the exported kernel build tree as a release artifact --- os/image/README.md | 13 +++++++++++++ os/image/assemble.sh | 20 +++++++++++++++++++- os/spec/artifact-manifest.schema.json | 4 ++++ 3 files changed, 36 insertions(+), 1 deletion(-) diff --git a/os/image/README.md b/os/image/README.md index b93a39883..6d3b4fd68 100644 --- a/os/image/README.md +++ b/os/image/README.md @@ -47,6 +47,19 @@ the helper lives beside the common assembler. OCI registry. It is likewise independent of the backend that produced the image. +## Kernel build tree + +A backend may export the kernel build tree of the image it built and declare it +as the optional `artifacts.kernel_devel` manifest entry. The assembler +publishes it as `--kernel-devel.tar.gz` beside the image +archives and deliberately keeps it out of `sha256sum.txt`: nothing in it is +measured at boot, so folding it into `os_image_hash` would make the OS identity +depend on a developer artifact. A backend that does not export one omits the +key, and the release is unchanged. + +`kernel-builder/` builds the container image that carries this tree plus a +matching toolchain; see [`kernel-builder/README.md`](kernel-builder/README.md). + ## Kernel setup-header normalization `assemble.sh` runs `normalize-kernel-header.py` over `bzImage` before it diff --git a/os/image/assemble.sh b/os/image/assemble.sh index b27c5a880..670b8a9b4 100755 --- a/os/image/assemble.sh +++ b/os/image/assemble.sh @@ -109,6 +109,7 @@ exact_keys( data["artifacts"], "artifacts", ("initramfs", "kernel", "firmware", "rootfs_verity", "firmware_sev", "uki"), + ("kernel_devel",), ) if "backend_metadata" in data and not isinstance(data["backend_metadata"], dict): raise SystemExit("manifest field must be an object: backend_metadata") @@ -179,6 +180,7 @@ values = [ artifact("rootfs_verity"), artifact("firmware_sev", optional=True), artifact("uki", optional=True), + artifact("kernel_devel", optional=True), ] for value in values: if not isinstance(value, str): @@ -187,7 +189,7 @@ for value in values: PYMANIFEST ) -if [ "${#MANIFEST_VALUES[@]}" -ne 15 ]; then +if [ "${#MANIFEST_VALUES[@]}" -ne 16 ]; then echo "Error: failed to read artifact manifest: $MANIFEST" >&2 exit 1 fi @@ -211,6 +213,7 @@ OVMF_FIRMWARE=${MANIFEST_VALUES[11]} ROOTFS_IMAGE=${MANIFEST_VALUES[12]} OVMF_SEV_FIRMWARE=${MANIFEST_VALUES[13]} UKI_IMAGE=${MANIFEST_VALUES[14]} +KERNEL_DEVEL_ARCHIVE=${MANIFEST_VALUES[15]} SCRIPT_DIR=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd) REPO_ROOT=$(realpath "$SCRIPT_DIR/../..") @@ -230,6 +233,7 @@ case "$OUTPUT_DIR" in esac IMAGE_TAR=${IMAGE_TAR:-"${DIST_DIR}/${DIST_NAME}-${DSTACK_VERSION}.tar.gz"} IMAGE_TAR_UKI=${IMAGE_TAR_UKI:-"${DIST_DIR}/${DIST_NAME}-${DSTACK_VERSION}-uki.tar.gz"} +KERNEL_DEVEL_TAR=${KERNEL_DEVEL_TAR:-"${DIST_DIR}/${DIST_NAME}-${DSTACK_VERSION}-kernel-devel.tar.gz"} TAR_DIR_NAME=$(basename "$OUTPUT_DIR") echo "Assembling ${DIST_NAME} ${DSTACK_VERSION} from ${BACKEND} artifacts (${FLAVOR})" @@ -664,3 +668,17 @@ if [ "$DSTACK_TAR_RELEASE" = "1" ]; then echo fi fi + +# The exported kernel build tree, when the backend produced one. It ships +# beside the image tarballs rather than inside them, and it is deliberately +# absent from sha256sum.txt: nothing in it is measured at boot, so folding it +# into os_image_hash would make the OS identity depend on a developer artifact. +# Applications building an out-of-tree module against this image download it; +# see docs/building-guest-os.md. +if [ -n "$KERNEL_DEVEL_ARCHIVE" ]; then + echo "Publishing the kernel build tree to ${KERNEL_DEVEL_TAR}" + cp "$KERNEL_DEVEL_ARCHIVE" "$KERNEL_DEVEL_TAR" + KERNEL_DEVEL_SHA256=$(sha256sum "$KERNEL_DEVEL_TAR" | awk '{print $1}') + KERNEL_DEVEL_BYTES=$(stat -L -c %s "$KERNEL_DEVEL_TAR") + echo "kernel-devel: ${KERNEL_DEVEL_BYTES} bytes, sha256 ${KERNEL_DEVEL_SHA256}" +fi diff --git a/os/spec/artifact-manifest.schema.json b/os/spec/artifact-manifest.schema.json index 44fe0ab11..d53cafeb0 100644 --- a/os/spec/artifact-manifest.schema.json +++ b/os/spec/artifact-manifest.schema.json @@ -71,6 +71,7 @@ } }, "artifacts": { + "description": "kernel_devel is optional: a backend that does not export the kernel build tree omits the key, so schema version 1 stays compatible.", "type": "object", "additionalProperties": false, "required": [ @@ -91,6 +92,9 @@ }, "uki": { "oneOf": [{"$ref": "#/$defs/artifactPath"}, {"type": "null"}] + }, + "kernel_devel": { + "oneOf": [{"$ref": "#/$defs/artifactPath"}, {"type": "null"}] } } }, From fe8039a9fbd9981f9da10da78ae735c07571a7db Mon Sep 17 00:00:00 2001 From: Kevin Wang Date: Wed, 16 Sep 2026 19:20:22 -0700 Subject: [PATCH 27/93] feat(os/image): add the kernel module builder image --- os/image/kernel-builder/Dockerfile | 86 +++++++++++++++++++++++++++ os/image/kernel-builder/README.md | 59 ++++++++++++++++++ os/tests/test-kernel-builder-image.sh | 43 ++++++++++++++ 3 files changed, 188 insertions(+) create mode 100644 os/image/kernel-builder/Dockerfile create mode 100644 os/image/kernel-builder/README.md create mode 100755 os/tests/test-kernel-builder-image.sh diff --git a/os/image/kernel-builder/Dockerfile b/os/image/kernel-builder/Dockerfile new file mode 100644 index 000000000..d053d160b --- /dev/null +++ b/os/image/kernel-builder/Dockerfile @@ -0,0 +1,86 @@ +# SPDX-FileCopyrightText: © 2026 Phala Network +# +# SPDX-License-Identifier: Apache-2.0 +# +# Toolchain image for building an out-of-tree kernel module against one dstack +# guest OS image. It carries the exported kernel build tree of that image plus +# a compiler from the same pinned Debian snapshot the guest kernel was built +# with, so a module built here matches the guest kernel's vermagic and +# Module.symvers and loads in the CVM. +# +# Build context is the directory holding the kernel-devel tarball: +# +# docker build -f os/image/kernel-builder/Dockerfile \ +# --build-arg DEBIAN_SNAPSHOT= \ +# --build-arg KERNEL_RELEASE= \ +# --build-arg KERNEL_DEVEL_TARBALL=dstack--kernel-devel.tar.gz \ +# -t dstack-kernel-builder: + +ARG BASE_IMAGE=debian:trixie-slim +FROM ${BASE_IMAGE} + +# The snapshot is what pins the compiler, which is the part that has to match +# the guest kernel build; the base image tag only provides the file layout. +ARG DEBIAN_SNAPSHOT +ARG KERNEL_RELEASE +ARG KERNEL_DEVEL_TARBALL +ARG IMAGE_VERSION=unknown +ARG OS_IMAGE_HASH=unknown +ARG GIT_REVISION=unknown + +# Trixie ships deb822 sources in /etc/apt/sources.list.d/, so rewriting +# sources.list alone leaves the live mirror active and the toolchain drifts on +# every rebuild. Wipe both layouts, as dstack/build/shared/pin-packages.sh does. +RUN set -eux; \ + test -n "${DEBIAN_SNAPSHOT}"; \ + suite=$(. /etc/os-release && echo "${VERSION_CODENAME}"); \ + rm -f /etc/apt/sources.list; \ + rm -f /etc/apt/sources.list.d/*.list /etc/apt/sources.list.d/*.sources; \ + printf 'deb [check-valid-until=no] http://snapshot.debian.org/archive/debian/%s %s main\n' \ + "${DEBIAN_SNAPSHOT}" "${suite}" > /etc/apt/sources.list; \ + echo 'Acquire::Check-Valid-Until "false";' > /etc/apt/apt.conf.d/10no-check-valid-until; \ + apt-get update; \ + apt-get install --yes --no-install-recommends \ + bc \ + bison \ + build-essential \ + flex \ + kmod \ + libelf-dev \ + libssl-dev \ + python3 \ + xz-utils; \ + rm -rf /var/lib/apt/lists/* + +COPY ${KERNEL_DEVEL_TARBALL} /tmp/kernel-devel.tar.gz + +# The archive's own metadata is the authority on which kernel it belongs to, so +# a mismatch with the requested release fails the build instead of producing an +# image whose KDIR silently points at a different kernel. +RUN set -eux; \ + test -n "${KERNEL_RELEASE}"; \ + mkdir -p /usr/src; \ + tar -C /usr/src --strip-components=1 -xzf /tmp/kernel-devel.tar.gz; \ + rm /tmp/kernel-devel.tar.gz; \ + declared=$(python3 -c 'import json;print(json.load(open("/usr/src/kernel-devel.json"))["kernel_release"])'); \ + test "${declared}" = "${KERNEL_RELEASE}"; \ + test -f "/usr/src/linux-headers-${KERNEL_RELEASE}/Module.symvers"; \ + mkdir -p "/lib/modules/${KERNEL_RELEASE}"; \ + ln -s "/usr/src/linux-headers-${KERNEL_RELEASE}" "/lib/modules/${KERNEL_RELEASE}/build" + +# KDIR is the variable an out-of-tree Makefile passes to `make -C`. The +# /lib/modules symlink covers Makefiles that derive it from `uname -r` instead, +# which inside a container reports the host kernel and would otherwise point at +# nothing. +ENV KDIR="/usr/src/linux-headers-${KERNEL_RELEASE}" +ENV KERNEL_RELEASE="${KERNEL_RELEASE}" + +LABEL org.opencontainers.image.title="dstack kernel module builder" \ + org.opencontainers.image.description="Kernel build tree and toolchain for dstack guest OS ${IMAGE_VERSION}" \ + org.opencontainers.image.source="https://github.com/Dstack-TEE/dstack" \ + org.opencontainers.image.revision="${GIT_REVISION}" \ + org.opencontainers.image.version="${IMAGE_VERSION}" \ + network.phala.dstack.kernel-release="${KERNEL_RELEASE}" \ + network.phala.dstack.os-image-hash="${OS_IMAGE_HASH}" + +WORKDIR /work diff --git a/os/image/kernel-builder/README.md b/os/image/kernel-builder/README.md new file mode 100644 index 000000000..f47eadd0f --- /dev/null +++ b/os/image/kernel-builder/README.md @@ -0,0 +1,59 @@ +# Kernel module builder image + +Toolchain image for building an out-of-tree kernel module against one dstack +guest OS image. It carries that image's exported kernel build tree +(`dstack--kernel-devel.tar.gz`) and a compiler installed from the same +pinned Debian snapshot the guest kernel was compiled with, so a module built +here matches the guest kernel's `vermagic` and `Module.symvers`. + +Published by `.github/workflows/mkosi-build.yml` on a release tag as +`ghcr.io/dstack-tee/dstack-kernel-builder:`, additionally tagged with +the kernel release (`6.18.x-dstack`). Application usage is documented in +[`../../../docs/building-guest-os.md`](../../../docs/building-guest-os.md#kernel-headers-for-out-of-tree-modules). + +## Build it locally + +The build context is a directory containing only the kernel-devel archive; the +release directory itself holds ~1 GiB of image tarballs that the container +build would otherwise upload to the daemon. + +```bash +source os/mkosi/versions.env +dist=os/mkosi/repro-build/build/out/prod +release=$(tar -xOf "$dist/dstack-$DSTACK_VERSION-kernel-devel.tar.gz" \ + "dstack-$DSTACK_VERSION-kernel-devel/kernel-devel.json" | + python3 -c 'import json,sys; print(json.load(sys.stdin)["kernel_release"])') + +mkdir -p /tmp/kernel-builder-ctx +cp "$dist/dstack-$DSTACK_VERSION-kernel-devel.tar.gz" /tmp/kernel-builder-ctx/ + +docker build \ + --file os/image/kernel-builder/Dockerfile \ + --build-arg "BASE_IMAGE=debian:$DEBIAN_RELEASE-slim" \ + --build-arg "DEBIAN_SNAPSHOT=$DEBIAN_SNAPSHOT" \ + --build-arg "KERNEL_RELEASE=$release" \ + --build-arg "KERNEL_DEVEL_TARBALL=dstack-$DSTACK_VERSION-kernel-devel.tar.gz" \ + --build-arg "IMAGE_VERSION=$DSTACK_VERSION" \ + --tag "dstack-kernel-builder:$DSTACK_VERSION" \ + /tmp/kernel-builder-ctx +``` + +Then check that what it builds would load in the guest: + +```bash +./os/tests/test-kernel-builder-image.sh "dstack-kernel-builder:$DSTACK_VERSION" "$release" +``` + +That script builds `os/tests/kmod-hello/` inside the image and fails unless the +resulting module's `vermagic` names the kernel the image claims to serve. CI +runs it before the image is pushed. + +## What the image pins, and what it does not + +The Debian snapshot pins the compiler, which is the part that has to agree with +the guest kernel build. `BASE_IMAGE` only supplies the file layout that +snapshot is installed into; the tag is mutable, so the image is not +bit-for-bit reproducible the way the guest OS artifacts are. The kernel build +tree it carries is: `os/mkosi/build.sh repro-check` compares +`dstack--kernel-devel.tar.gz` byte for byte across two independent +builds. diff --git a/os/tests/test-kernel-builder-image.sh b/os/tests/test-kernel-builder-image.sh new file mode 100755 index 000000000..02f3d823e --- /dev/null +++ b/os/tests/test-kernel-builder-image.sh @@ -0,0 +1,43 @@ +#!/bin/bash +# SPDX-FileCopyrightText: © 2026 Phala Network +# +# SPDX-License-Identifier: Apache-2.0 +# +# Build the smoke-test module inside a kernel-builder image and assert that the +# result would load in the guest it claims to serve. A builder image whose +# headers, Module.symvers or compiler do not match the shipped kernel still +# produces a .ko; what it does not produce is a matching vermagic, so that is +# what this checks. +set -euo pipefail + +IMAGE=${1:?container image required} +EXPECTED_RELEASE=${2:-} +DOCKER=${DOCKER:-docker} +SELF=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd) + +work=$(mktemp -d) +trap 'rm -rf "$work"' EXIT +cp "$SELF/kmod-hello/hello.c" "$SELF/kmod-hello/Makefile" "$work/" + +# --user keeps the build outputs owned by the caller, so the fixture copy stays +# removable without root. The inner script is single-quoted on purpose: KDIR +# and KERNEL_RELEASE are the image's own environment, not this shell's. +# shellcheck disable=SC2016 +$DOCKER run --rm --user "$(id -u):$(id -g)" \ + -v "$work:/work" -w /work "$IMAGE" \ + bash -eu -o pipefail -c ' + make KDIR="$KDIR" + vermagic=$(modinfo -F vermagic dstack-hello.ko) + echo "vermagic: $vermagic" + if [ "${vermagic%% *}" != "$KERNEL_RELEASE" ]; then + echo "module vermagic ${vermagic%% *} does not match the image kernel $KERNEL_RELEASE" >&2 + exit 1 + fi + printf "%s\n" "$KERNEL_RELEASE" > kernel-release.txt + ' +release=$(cat "$work/kernel-release.txt") +if [ -n "$EXPECTED_RELEASE" ] && [ "$release" != "$EXPECTED_RELEASE" ]; then + echo "builder image serves $release but the release expects $EXPECTED_RELEASE" >&2 + exit 1 +fi +echo "kernel module build accepted: $IMAGE serves $release" From dfb0e46df593f80a3602aa213eb352347323b18a Mon Sep 17 00:00:00 2001 From: Kevin Wang Date: Wed, 16 Sep 2026 19:20:22 -0700 Subject: [PATCH 28/93] feat(os/mkosi): export the kernel build tree of the image it builds --- os/mkosi/build.sh | 6 +++++ os/mkosi/components/kernel/kernel-build.sh | 18 ++++++++++--- os/mkosi/components/kernel/kernel.sh | 13 +++++++--- os/mkosi/mkosi.finalize | 5 ++++ os/mkosi/parity.json | 1 + os/mkosi/scripts/make-release-artifacts.sh | 28 +++++++++++++++++--- os/mkosi/tests/acceptance.sh | 30 +++++++++++++++++++++- os/mkosi/tests/check-output.sh | 18 +++++++++++++ 8 files changed, 109 insertions(+), 10 deletions(-) diff --git a/os/mkosi/build.sh b/os/mkosi/build.sh index 63f567b5a..e3c24cd84 100755 --- a/os/mkosi/build.sh +++ b/os/mkosi/build.sh @@ -208,4 +208,10 @@ build_one "$BUILD_DIR/b" prod "$jobs_b" cmp "$BUILD_DIR/a/dstack-$DSTACK_VERSION.tar.gz" "$BUILD_DIR/b/dstack-$DSTACK_VERSION.tar.gz" cmp "$BUILD_DIR/a/dstack-$DSTACK_VERSION-uki.tar.gz" \ "$BUILD_DIR/b/dstack-$DSTACK_VERSION-uki.tar.gz" +# The kernel build tree publishes parts of the build that nothing shipped +# before: the host programs Kbuild compiled (fixdep, modpost, objtool) and its +# generated intermediates. Comparing it is what proves none of them records the +# build directory or the job count. +cmp "$BUILD_DIR/a/dstack-$DSTACK_VERSION-kernel-devel.tar.gz" \ + "$BUILD_DIR/b/dstack-$DSTACK_VERSION-kernel-devel.tar.gz" echo 'reproducibility check passed' diff --git a/os/mkosi/components/kernel/kernel-build.sh b/os/mkosi/components/kernel/kernel-build.sh index c9031e71f..86b2a9255 100755 --- a/os/mkosi/components/kernel/kernel-build.sh +++ b/os/mkosi/components/kernel/kernel-build.sh @@ -7,8 +7,10 @@ MKOSI_DIR="$ROOT/os/mkosi" source "$MKOSI_DIR/versions.env" BUILD_DIR=${1:?build directory required} STAGING=${2:?staging tree required} +DEVEL_STAGING=${3:?kernel-devel staging tree required} BUILD_DIR=$(realpath -m "$BUILD_DIR") STAGING=$(realpath -m "$STAGING") +DEVEL_STAGING=$(realpath -m "$DEVEL_STAGING") JOBS=${JOBS:-$(nproc)} export KBUILD_BUILD_TIMESTAMP="@${SOURCE_DATE_EPOCH:?SOURCE_DATE_EPOCH required}" export KBUILD_BUILD_USER=dstack KBUILD_BUILD_HOST=reproducible @@ -21,7 +23,7 @@ export KCFLAGS="${KCFLAGS:-} $kernel_map" export KAFLAGS="${KAFLAGS:-} $kernel_map" export KCPPFLAGS="${KCPPFLAGS:-} $kernel_map" -mkdir -p "$BUILD_DIR/downloads" "$BUILD_DIR/kernel-build" "$STAGING" +mkdir -p "$BUILD_DIR/downloads" "$BUILD_DIR/kernel-build" "$STAGING" "$DEVEL_STAGING" # pahole 1.25 produces a different BTF type order when its encoder runs with # Kbuild's parallel job count. Keep compilation parallel, but serialize BTF # encoding so vmlinux and bzImage are byte-for-byte reproducible. @@ -48,7 +50,8 @@ echo "$KERNEL_SHA256 $tarball" | sha256sum --check --status || { echo "kernel source checksum mismatch" >&2; exit 1; } src="$BUILD_DIR/linux-$KERNEL_VERSION" -rm -rf "$src" "$BUILD_DIR/kernel-build" "$STAGING/usr/lib/modules/$KERNEL_VERSION-dstack" +rm -rf "$src" "$BUILD_DIR/kernel-build" "$STAGING/usr/lib/modules/$KERNEL_VERSION-dstack" \ + "$DEVEL_STAGING/usr/lib/dstack/kernel-devel" tar -C "$BUILD_DIR" --no-same-owner -xf "$tarball" for patch in \ "$ROOT/os/yocto/layers/meta-dstack/recipes-kernel/linux/files/0001-x86-tdx-select-dma-direct-remap.patch" \ @@ -87,4 +90,13 @@ install -Dm0644 "$BUILD_DIR/kernel-build/arch/x86/boot/bzImage" \ "$STAGING/usr/lib/modules/$KERNEL_VERSION-dstack/vmlinuz" install -Dm0644 "$BUILD_DIR/kernel-build/.config" \ "$STAGING/usr/lib/modules/$KERNEL_VERSION-dstack/config" -find "$STAGING" -print0 | xargs -0r touch --no-dereference --date="@$SOURCE_DATE_EPOCH" +# Applications that need their own kernel module get the build tree of this +# exact kernel as a release artifact. Staged here and moved out of the rootfs +# by mkosi.finalize before the image is measured: it must not ship in the CVM. +"$ROOT/os/common/scripts/export-kernel-devel.sh" \ + --src "$src" --build "$BUILD_DIR/kernel-build" \ + --out "$DEVEL_STAGING/usr/lib/dstack/kernel-devel" \ + --backend mkosi --image-version "$DSTACK_VERSION" \ + --flavor "${DSTACK_BUILD_FLAVOR:-prod}" +find "$STAGING" "$DEVEL_STAGING" -print0 | \ + xargs -0r touch --no-dereference --date="@$SOURCE_DATE_EPOCH" diff --git a/os/mkosi/components/kernel/kernel.sh b/os/mkosi/components/kernel/kernel.sh index 40dc02848..0c3e8a3b8 100644 --- a/os/mkosi/components/kernel/kernel.sh +++ b/os/mkosi/components/kernel/kernel.sh @@ -1,13 +1,19 @@ # SPDX-License-Identifier: Apache-2.0 # shellcheck shell=bash disable=SC2034 COMPONENT_NAME=kernel -COMPONENT_CACHE_PATHS=("linux-$KERNEL_VERSION" kernel-build component-stages/kernel) -COMPONENT_ROOTFS_TREES=() +COMPONENT_CACHE_PATHS=("linux-$KERNEL_VERSION" kernel-build component-stages/kernel + component-stages/kernel-devel) +# The exported build tree is staged into the rootfs install tree, not because it +# ships in the image but because that is the only writable tree a component +# reaches; mkosi.finalize moves it out before the rootfs is measured, the same +# way it does for the firmware and the image tools. +COMPONENT_ROOTFS_TREES=(component-stages/kernel-devel) COMPONENT_KERNEL_TREES=(component-stages/kernel) component_cache_key() { key_value "$KERNEL_VERSION" "$KERNEL_SHA256" key_file "$COMPONENT_PATH/kernel-build.sh" "$COMPONENT_PATH/kernel.config" \ + "$ROOT/os/common/scripts/export-kernel-devel.sh" \ "$ROOT/os/yocto/layers/meta-dstack/recipes-kernel/linux/files/0001-x86-tdx-select-dma-direct-remap.patch" \ "$ROOT/os/yocto/layers/meta-dstack/recipes-kernel/linux/files/0002-acpi-sandbox-block-aml-systemmemory-ram-access.patch" \ "$ROOT/os/yocto/layers/meta-dstack/recipes-kernel/linux/files/0003-dma-direct-return-struct-page-from-alloc-from-pool.patch" \ @@ -23,5 +29,6 @@ component_cache_key() { } component_build() { - "$COMPONENT_PATH/kernel-build.sh" "$WORK" "$WORK/component-stages/kernel" + "$COMPONENT_PATH/kernel-build.sh" "$WORK" "$WORK/component-stages/kernel" \ + "$WORK/component-stages/kernel-devel" } diff --git a/os/mkosi/mkosi.finalize b/os/mkosi/mkosi.finalize index 10c7e50d0..9c11f81e6 100755 --- a/os/mkosi/mkosi.finalize +++ b/os/mkosi/mkosi.finalize @@ -20,6 +20,11 @@ cp -a "$TREE/usr/lib/modules" "$KSTAGE/usr/lib/" install -m0644 "$TREE/usr/lib/dstack/firmware/ovmf.fd" "$KSTAGE/ovmf.fd" install -m0644 "$TREE/usr/lib/dstack/firmware/ovmf-sev.fd" "$KSTAGE/ovmf-sev.fd" install -m0755 "$TREE/usr/lib/dstack/image-tools/"* "$OUTPUTDIR/.image-tools/" +# The exported kernel build tree is a release artifact, not guest payload: it +# is ~100 MB of build inputs no CVM uses, and leaving it in the tree would put +# it in the measured rootfs. Moved before check-parity.py runs so the parity +# check sees exactly the tree that ships. +mv "$TREE/usr/lib/dstack/kernel-devel" "$KSTAGE/kernel-devel" rm -rf "$TREE/usr/lib/dstack/firmware" "$TREE/usr/lib/dstack/image-tools" rm -f "$TREE/var/cache/ldconfig/aux-cache" diff --git a/os/mkosi/parity.json b/os/mkosi/parity.json index 0b10d4802..2b2a627ef 100644 --- a/os/mkosi/parity.json +++ b/os/mkosi/parity.json @@ -153,6 +153,7 @@ "CONFIG_NETFILTER_XT_MATCH_COMMENT=m" ], "prod_forbidden_paths": [ + "usr/lib/dstack/kernel-devel", "usr/bin/dstack-tee-simulator", "usr/bin/swtpm", "usr/lib/systemd/system/dstack-prepare.service.d/tee-simulator.conf", diff --git a/os/mkosi/scripts/make-release-artifacts.sh b/os/mkosi/scripts/make-release-artifacts.sh index 8752e8069..071d7fef1 100755 --- a/os/mkosi/scripts/make-release-artifacts.sh +++ b/os/mkosi/scripts/make-release-artifacts.sh @@ -124,16 +124,38 @@ touch -d "@$SOURCE_DATE_EPOCH" "$OUT/files/"* is_dev=false; name=dstack if [[ $FLAVOR == dev ]]; then is_dev=true; name=dstack-dev; fi + +# The exported kernel build tree, moved out of the image tree by mkosi.finalize. +# Archived here rather than shipped in either release tarball: applications +# download it to build their own modules, operators deploying the image do not +# need it, and it is not part of the measured OS image identity. Same fixed tar +# options as the release tarballs, so repro-check can compare it byte for byte, +# and skipped on a cached iteration build for the same reason those are. +kernel_devel="$KERNEL_TREE/kernel-devel" +[[ -d $kernel_devel ]] || { echo 'kernel build tree was not exported' >&2; exit 1; } +kernel_devel_artifact= +if [[ ${DSTACK_TAR_RELEASE:-1} == 1 ]]; then + tar -C "$KERNEL_TREE" --sort=name --mtime="@$SOURCE_DATE_EPOCH" \ + --owner=0 --group=0 --numeric-owner \ + --transform="s|^kernel-devel|$name-$DSTACK_VERSION-kernel-devel|" \ + -czf "$OUT/files/kernel-devel.tar.gz" kernel-devel + touch -d "@$SOURCE_DATE_EPOCH" "$OUT/files/kernel-devel.tar.gz" + kernel_devel_artifact=files/kernel-devel.tar.gz +fi + python3 - "$OUT/artifact-manifest.json" "$name" "$FLAVOR" "$is_dev" \ - "${DSTACK_SOURCE_REVISION:?}" "$root_hash" "$data_size" "$DSTACK_VERSION" "$OVMF_VARIANT" <<'PY' + "${DSTACK_SOURCE_REVISION:?}" "$root_hash" "$data_size" "$DSTACK_VERSION" "$OVMF_VARIANT" \ + "$kernel_devel_artifact" <<'PY' import json, sys -path,name,flavor,is_dev,revision,root_hash,data_size,version,ovmf_variant=sys.argv[1:] +(path,name,flavor,is_dev,revision,root_hash,data_size,version,ovmf_variant, + kernel_devel)=sys.argv[1:] data={"schema_version":1,"backend":"mkosi","image":{"name":name,"version":version, "flavor":flavor,"is_dev":is_dev=="true"},"source":{"git_revision":revision}, "boot":{"ovmf_variant":ovmf_variant},"verity":{"root_hash":root_hash,"data_size":data_size}, "artifacts":{"initramfs":"files/initramfs.cpio.gz","kernel":"files/bzImage", "firmware":"files/ovmf.fd","rootfs_verity":"files/rootfs.squashfs.verity", -"firmware_sev":"files/ovmf-sev.fd","uki":"files/dstack-uki.efi"}, +"firmware_sev":"files/ovmf-sev.fd","uki":"files/dstack-uki.efi", +"kernel_devel":kernel_devel or None}, "backend_metadata":{"machine":"dstack","distribution":"debian"}} with open(path,"w") as f: json.dump(data,f,indent=2); f.write("\n") PY diff --git a/os/mkosi/tests/acceptance.sh b/os/mkosi/tests/acceptance.sh index 493315a44..a5d0be4c9 100755 --- a/os/mkosi/tests/acceptance.sh +++ b/os/mkosi/tests/acceptance.sh @@ -182,6 +182,33 @@ grep -q 'AmdSev/AmdSevX64.dsc' "$D/components/ovmf/ovmf-build.sh" grep -q '0006-OvmfPkg-AmdSev-drop-embedded-grub.patch' "$D/components/ovmf/ovmf.sh" grep -q '0005-UefiCpuPkg' "$D/components/ovmf/ovmf.sh" grep -q '0007-OvmfPkg-QemuKernelLoaderFsDxe' "$D/components/ovmf/ovmf.sh" +# The exported kernel build tree is a release artifact, never guest payload. +# Each of these is load-bearing: the component has to export it, finalize has +# to move it out of the tree before the rootfs is measured, and the identity +# preimage must not list it. +grep -q 'export-kernel-devel.sh' "$D/components/kernel/kernel-build.sh" +grep -q 'export-kernel-devel.sh' "$D/components/kernel/kernel.sh" +# shellcheck disable=SC2016 # a grep pattern matching the literal script text +grep -q 'mv "$TREE/usr/lib/dstack/kernel-devel" "$KSTAGE/kernel-devel"' \ + "$D/mkosi.finalize" +grep -q '"kernel_devel":kernel_devel or None' "$D/scripts/make-release-artifacts.sh" +grep -q 'kernel-devel' "$D/tests/check-output.sh" +grep -q 'usr/lib/dstack/kernel-devel' "$D/parity.json" +grep -q 'kernel_devel' "$D/../spec/artifact-manifest.schema.json" +grep -q 'KERNEL_DEVEL_ARCHIVE' "$D/../image/assemble.sh" +# Host programs and generated intermediates reach a published artifact for the +# first time through this tree, so repro-check has to cover it. +grep -q 'kernel-devel.tar.gz' "$D/build.sh" +test -x "$D/../common/scripts/export-kernel-devel.sh" +test -x "$D/../tests/test-kernel-builder-image.sh" +test -f "$D/../image/kernel-builder/Dockerfile" +# The builder image installs its compiler from the snapshot that built the +# guest kernel; a hardcoded one here would drift from versions.env. +grep -q 'ARG DEBIAN_SNAPSHOT' "$D/../image/kernel-builder/Dockerfile" +if grep -q "$DEBIAN_SNAPSHOT" "$D/../image/kernel-builder/Dockerfile"; then + echo 'the builder image must take the Debian snapshot as a build argument' >&2 + exit 1 +fi grep -q 'objcopy --strip-debug' "$D/mkosi.build" grep -q 'depmod -b.*KERNEL_VERSION-dstack' "$D/mkosi.build" grep -q '^CleanPackageMetadata=yes$' "$D/mkosi.conf" @@ -222,7 +249,8 @@ grep -q 'rootfs.img.parted.verity' "$D/../image/assemble.sh" bash -n "$D"/*.sh "$D"/mkosi.build "$D"/mkosi.clean "$D"/mkosi.finalize \ "$D"/mkosi.postinst "$D"/mkosi.postoutput "$D"/scripts/*.sh \ "$D"/components/*/*.sh "$D"/tests/*.sh "$D"/mkosi.skeleton/usr/bin/*.sh \ - "$D"/../image/kernel-cmdline.sh + "$D"/../image/kernel-cmdline.sh "$D"/../common/scripts/export-kernel-devel.sh \ + "$D"/../tests/test-kernel-builder-image.sh # The Python helpers were only ever exercised by a real build. Keep the # bytecode out of the source tree, which is mounted into the build sandbox. pycache=$(mktemp -d) diff --git a/os/mkosi/tests/check-output.sh b/os/mkosi/tests/check-output.sh index 55cbe2178..09d243640 100755 --- a/os/mkosi/tests/check-output.sh +++ b/os/mkosi/tests/check-output.sh @@ -18,4 +18,22 @@ assert d["builder"] == "mkosi" assert d["bios-sev"] == "ovmf-sev.fd" assert d["is_dev"] == (sys.argv[2] == "dev") PY +# The kernel build tree ships beside the image, never inside it. Nothing in it +# is measured at boot, so its presence in the identity preimage would make +# os_image_hash depend on a developer artifact. +if grep -q 'kernel-devel' "$out/sha256sum.txt"; then + echo 'kernel build tree must not be part of the image identity' >&2 + exit 1 +fi +if [[ ${DSTACK_TAR_RELEASE:-1} == 1 ]]; then + [[ -s $out-kernel-devel.tar.gz ]] || { + echo "missing kernel build tree archive: $out-kernel-devel.tar.gz" >&2 + exit 1 + } + tar -tzf "$out-kernel-devel.tar.gz" | \ + grep -qx "$(basename "$out")-kernel-devel/kernel-devel.json" || { + echo 'kernel build tree archive does not declare its kernel' >&2 + exit 1 + } +fi echo "Yocto-compatible release format accepted: $out" From a508fa4fbf45fdb01eb875b7d646f0cb2f93655f Mon Sep 17 00:00:00 2001 From: Kevin Wang Date: Wed, 16 Sep 2026 19:20:22 -0700 Subject: [PATCH 29/93] ci(mkosi): publish the kernel build tree and the builder image --- .github/workflows/mkosi-build.yml | 176 +++++++++++++++++++++++++++++- 1 file changed, 175 insertions(+), 1 deletion(-) diff --git a/.github/workflows/mkosi-build.yml b/.github/workflows/mkosi-build.yml index 258cee143..37c83e01b 100644 --- a/.github/workflows/mkosi-build.yml +++ b/.github/workflows/mkosi-build.yml @@ -63,6 +63,8 @@ jobs: os_image_hash: ${{ steps.hashes.outputs.os_image_hash }} bare_sha256: ${{ steps.hashes.outputs.bare_sha256 }} uki_sha256: ${{ steps.hashes.outputs.uki_sha256 }} + kernel_release: ${{ steps.kernel_devel.outputs.kernel_release }} + kernel_devel_sha256: ${{ steps.kernel_devel.outputs.sha256 }} steps: # GitHub-hosted runners have enough CPU and memory for this build, but @@ -264,6 +266,53 @@ jobs: echo "$uki_sha256 dstack-$version-uki.tar.gz" } | tee "$DIST_DIR/image-hashes.txt" + - name: Report the kernel build tree + id: kernel_devel + env: + DIST_DIR: ${{ steps.build.outputs.dist_dir }} + VERSION: ${{ steps.hashes.outputs.version }} + run: | + set -euo pipefail + # The kernel build tree is what an application compiles its own + # modules against. It is not measured and not part of + # os_image_hash -- it is published so an application does not have to + # reconstruct the kernel build to obtain a matching Module.symvers. + devel="$DIST_DIR/dstack-$VERSION-kernel-devel.tar.gz" + test -f "$devel" + sha256=$(sha256sum "$devel" | awk '{print $1}') + bytes=$(stat -L -c %s "$devel") + # Read the identity out of the archive, not out of versions.env: the + # archive is what ships, and its metadata is what the builder image + # is then built against. + work=$(mktemp -d) + tar -xzf "$devel" -C "$work" \ + "dstack-$VERSION-kernel-devel/kernel-devel.json" + meta="$work/dstack-$VERSION-kernel-devel/kernel-devel.json" + read -r kernel_release files uncompressed < <( + python3 -c 'import json,sys; d=json.load(open(sys.argv[1])); print(d["kernel_release"], d["file_count"], d["uncompressed_bytes"])' \ + "$meta") + compiler=$(python3 -c 'import json,sys; print(json.load(open(sys.argv[1]))["compiler"])' "$meta") + { + echo "kernel_release=$kernel_release" + echo "sha256=$sha256" + } >> "$GITHUB_OUTPUT" + { + echo + echo "### Kernel build tree (out-of-tree modules)" + echo + echo '| Field | Value |' + echo '| --- | --- |' + echo "| Kernel release | \`$kernel_release\` |" + echo "| Compiler | \`$compiler\` |" + echo "| Files | $files |" + echo "| Unpacked | $uncompressed bytes |" + echo "| \`dstack-$VERSION-kernel-devel.tar.gz\` | $bytes bytes, \`$sha256\` |" + } >> "$GITHUB_STEP_SUMMARY" + { + echo "kernel_release $kernel_release" + echo "$sha256 dstack-$VERSION-kernel-devel.tar.gz" + } | tee -a "$DIST_DIR/image-hashes.txt" + - name: Upload guest images uses: actions/upload-artifact@v4 with: @@ -274,11 +323,111 @@ jobs: retention-days: 30 if-no-files-found: error + kernel-builder: + name: Kernel module builder image + # Same gate as the build it consumes. On a dispatch the image is built and + # smoke-tested but not published: only the release tag publishes. + if: >- + github.event_name == 'workflow_dispatch' || + startsWith(github.ref, 'refs/tags/mkosi-os-v') + needs: build + runs-on: ubuntu-latest + timeout-minutes: 30 + permissions: + contents: read + packages: write + + steps: + - name: Checkout + uses: actions/checkout@v5 + + - name: Download guest images + uses: actions/download-artifact@v4 + with: + name: mkosi-guest-images + path: dist + + - name: Build the builder image + id: image + env: + VERSION: ${{ needs.build.outputs.version }} + KERNEL_RELEASE: ${{ needs.build.outputs.kernel_release }} + OS_IMAGE_HASH: ${{ needs.build.outputs.os_image_hash }} + run: | + set -euo pipefail + # shellcheck source=/dev/null + source os/mkosi/versions.env + # GHCR path segments must be lowercase and this repository's owner is + # not; other release workflows hardcode the lowercase owner, which a + # fork cannot publish under. + owner=$(echo "$GITHUB_REPOSITORY_OWNER" | tr '[:upper:]' '[:lower:]') + ref="ghcr.io/$owner/dstack-kernel-builder" + # A build context of its own: dist/ holds the ~1 GiB image tarballs + # and all of it would be uploaded to the daemon. + mkdir -p ctx + cp "dist/dstack-$VERSION-kernel-devel.tar.gz" ctx/ + # The toolchain comes from the same pinned Debian snapshot that + # compiled the guest kernel. A different compiler still builds a + # module; what it does not build is one this kernel accepts. + docker build \ + --file os/image/kernel-builder/Dockerfile \ + --build-arg "BASE_IMAGE=debian:$DEBIAN_RELEASE-slim" \ + --build-arg "DEBIAN_SNAPSHOT=$DEBIAN_SNAPSHOT" \ + --build-arg "KERNEL_RELEASE=$KERNEL_RELEASE" \ + --build-arg "KERNEL_DEVEL_TARBALL=dstack-$VERSION-kernel-devel.tar.gz" \ + --build-arg "IMAGE_VERSION=$VERSION" \ + --build-arg "OS_IMAGE_HASH=$OS_IMAGE_HASH" \ + --build-arg "GIT_REVISION=$GITHUB_SHA" \ + --tag "$ref:$VERSION" \ + --tag "$ref:$KERNEL_RELEASE" \ + ctx + echo "ref=$ref" >> "$GITHUB_OUTPUT" + + # The image is only useful if what it builds loads in the guest, and the + # vermagic is what decides that. Runs before the push, so a mismatched + # toolchain or a stale kernel tree is never published. + - name: Build a module inside the image + env: + IMAGE: ${{ steps.image.outputs.ref }}:${{ needs.build.outputs.version }} + KERNEL_RELEASE: ${{ needs.build.outputs.kernel_release }} + run: ./os/tests/test-kernel-builder-image.sh "$IMAGE" "$KERNEL_RELEASE" + + - name: Log in to GitHub Container Registry + if: startsWith(github.ref, 'refs/tags/mkosi-os-v') + uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9 # v3 + with: + registry: ghcr.io + username: ${{ github.actor }} + password: ${{ secrets.GITHUB_TOKEN }} + + - name: Push the builder image + if: startsWith(github.ref, 'refs/tags/mkosi-os-v') + env: + REF: ${{ steps.image.outputs.ref }} + VERSION: ${{ needs.build.outputs.version }} + KERNEL_RELEASE: ${{ needs.build.outputs.kernel_release }} + run: | + set -euo pipefail + docker push "$REF:$VERSION" + docker push "$REF:$KERNEL_RELEASE" + digest=$(docker inspect --format='{{index .RepoDigests 0}}' "$REF:$VERSION") + echo "::notice title=kernel-builder::$digest" + { + echo "### Kernel module builder image" + echo + echo '```' + echo "$digest" + echo '```' + } >> "$GITHUB_STEP_SUMMARY" + release: name: Publish release # The tag is the publish decision. Nothing else reaches this job. if: startsWith(github.ref, 'refs/tags/mkosi-os-v') - needs: build + # Also needs kernel-builder: the notes point at the builder image, so a + # release must not be created when that image failed to build its smoke + # test module or failed to publish. + needs: [build, kernel-builder] runs-on: ubuntu-latest timeout-minutes: 30 environment: release @@ -311,8 +460,10 @@ jobs: fi bare="dist/dstack-$version.tar.gz" uki="dist/dstack-$version-uki.tar.gz" + kernel_devel="dist/dstack-$version-kernel-devel.tar.gz" test -f "$bare" test -f "$uki" + test -f "$kernel_devel" read -r image_version image_revision < <( tar -xOf "$bare" "dstack-$version/metadata.json" | @@ -339,8 +490,12 @@ jobs: OS_IMAGE_HASH: ${{ needs.build.outputs.os_image_hash }} BARE_SHA256: ${{ needs.build.outputs.bare_sha256 }} UKI_SHA256: ${{ needs.build.outputs.uki_sha256 }} + KERNEL_RELEASE: ${{ needs.build.outputs.kernel_release }} + KERNEL_DEVEL_SHA256: ${{ needs.build.outputs.kernel_devel_sha256 }} run: | set -euo pipefail + owner=$(echo "$GITHUB_REPOSITORY_OWNER" | tr '[:upper:]' '[:lower:]') + builder="ghcr.io/$owner/dstack-kernel-builder:$VERSION" { echo "Guest OS image \`$VERSION\`, built by the Debian/mkosi backend." echo @@ -353,9 +508,27 @@ jobs: echo "| Git revision | \`$GITHUB_SHA\` |" echo "| \`dstack-$VERSION.tar.gz\` | \`$BARE_SHA256\` |" echo "| \`dstack-$VERSION-uki.tar.gz\` | \`$UKI_SHA256\` |" + echo "| \`dstack-$VERSION-kernel-devel.tar.gz\` | \`$KERNEL_DEVEL_SHA256\` |" echo echo "\`image-hashes.txt\` additionally records the TDX \`MRTD\`, the GCP" echo "\`uki_auth\` hash and the AWS \`boot_pcr_digest\` for this image." + echo + echo "### Out-of-tree kernel modules" + echo + echo "This image runs \`$KERNEL_RELEASE\`. To build a module for it, use the" + echo "matching toolchain image, which carries the kernel build tree of this" + echo "release and a compiler from the same pinned Debian snapshot:" + echo + echo '```dockerfile' + echo "FROM $builder AS kmod" + echo "COPY mymod/ /src" + echo "RUN make -C \$KDIR M=/src modules" + echo '```' + echo + echo "\`dstack-$VERSION-kernel-devel.tar.gz\` is the same build tree for use" + echo "outside that image. Neither is measured: they are not part of" + echo "\`os_image_hash\`. See" + echo "[docs/building-guest-os.md](https://github.com/${GITHUB_REPOSITORY}/blob/${TAG}/docs/building-guest-os.md#kernel-headers-for-out-of-tree-modules)." } > release-notes.md # --verify-tag: the tag is the publish trigger, so it must already # exist. This job never creates one. @@ -371,6 +544,7 @@ jobs: gh release create "$TAG" \ "dist/dstack-$VERSION.tar.gz" \ "dist/dstack-$VERSION-uki.tar.gz" \ + "dist/dstack-$VERSION-kernel-devel.tar.gz" \ dist/image-hashes.txt \ --title "dstack OS v$VERSION" \ --notes-file release-notes.md \ From d970febb575593e64093d2f42441c60512ee4721 Mon Sep 17 00:00:00 2001 From: Kevin Wang Date: Wed, 16 Sep 2026 19:20:22 -0700 Subject: [PATCH 30/93] docs: document building out-of-tree kernel modules for the guest OS --- docs/building-guest-os.md | 72 +++++++++++++++++++++++++++++++++++++-- 1 file changed, 70 insertions(+), 2 deletions(-) diff --git a/docs/building-guest-os.md b/docs/building-guest-os.md index 2480b7cc1..4ebc6877a 100644 --- a/docs/building-guest-os.md +++ b/docs/building-guest-os.md @@ -93,7 +93,8 @@ Release artifacts are written under: os/mkosi/repro-build/build/out/prod/ ├── dstack-/ ├── dstack-.tar.gz -└── dstack--uki.tar.gz +├── dstack--uki.tar.gz +└── dstack--kernel-devel.tar.gz ``` To use a different build directory, run @@ -103,7 +104,9 @@ prints the `os_image_hash` and the SHA-256 of both archives. The bare-metal archive includes the kernel, initramfs, OVMF firmware, partitioned dm-verity rootfs, platform measurement CBOR files, `sha256sum.txt`, `digest.txt`, and `metadata.json`. The UKI archive -includes the bootable `disk.raw` plus its identity and measurement files. +includes the bootable `disk.raw` plus its identity and measurement files. The +kernel-devel archive is the build tree for out-of-tree modules; it is not part +of the image identity (see below). Inspect and verify an archive with: @@ -127,6 +130,71 @@ FLAVORS="prod dev" ./os/mkosi/repro-build/repro-build.sh The development archive is written to `out/dev/dstack-dev-.tar.gz` and records `"is_dev": true` in `metadata.json`. +## Kernel headers for out-of-tree modules + +The guest image ships kernel modules but no build tree, so an application that +needs its own `.ko` — a custom network or storage driver, a device shim — +would have to reconstruct the kernel build to obtain a `Module.symvers` and a +`vermagic` the guest kernel accepts. Every release publishes that build tree +instead: + +- `dstack--kernel-devel.tar.gz`, the exported kernel build tree, and +- `ghcr.io/dstack-tee/dstack-kernel-builder:`, that tree plus a + compiler from the same pinned Debian snapshot the guest kernel was built + with, with `KDIR` already pointing at it. + +Build the module where the application image is built, and ship the `.ko` in +the container image: + +```dockerfile +FROM ghcr.io/dstack-tee/dstack-kernel-builder: AS kmod +COPY mymod/ /src +RUN make -C $KDIR M=/src modules + +FROM alpine +COPY --from=kmod /src/mymod.ko /opt/ +``` + +Outside that image, point `KDIR` at the unpacked tree: + +```bash +tar -xzf dstack--kernel-devel.tar.gz +make -C dstack--kernel-devel/linux-headers- M="$PWD" modules +``` + +`kernel-devel.json` in the archive records the kernel release, the compiler, +and the SHA-256 of `.config` and `Module.symvers`, so a module can be traced +back to the image it was built for. `os/tests/kmod-hello/` is a two-file module +that builds against either form and is what CI uses to check a published +builder image. + +Two constraints are worth knowing before debugging a failure: + +- **The compiler matters.** A module built with a different toolchain may still + load, but anything the config derives from `CONFIG_CC_VERSION_TEXT` and the + compiler's own ABI choices is no longer guaranteed to agree. Use the builder + image, or a compiler from the snapshot in `os/mkosi/versions.env`. +- **Modules built this way carry no BTF.** Generating it needs `vmlinux`, which + the archive does not carry because its debug info dwarfs everything else in + the release. Kbuild prints `Skipping BTF generation ... due to unavailability + of vmlinux` and produces a working module. + +### What this means for attestation + +Neither the archive nor the builder image is part of the OS image identity: +they are not in `sha256sum.txt`, so `os_image_hash` does not change when they +do, and a CVM never sees them. + +Loading a module is still a privileged operation and the module is kernel code +the application chose. The container needs `CAP_SYS_MODULE` (in practice, +`privileged: true`), and what a remote verifier can conclude about that module +is exactly what it can conclude about the rest of the application: the module +is part of the container image, the image digest is part of the compose file, +and the compose hash is measured into RTMR3 and enforced by the KMS. The module +itself is not measured separately, and dstack does not enforce module +signatures. Treat a custom module as trusted application code with kernel +privileges, and pin it by digest like any other part of the application. + ## Check reproducibility For a release candidate: From cfd4e1214baa8f70ae3154fee80f92a4faf80e19 Mon Sep 17 00:00:00 2001 From: Kevin Wang Date: Thu, 17 Sep 2026 02:29:50 -0700 Subject: [PATCH 31/93] fix(os): validate kernel-devel archives and simplify builder workflow Replace source-text assertions with artifact behavior tests, including the tar SIGPIPE regression. Share the builder command between CI and local usage, narrow intermediate cleanup, and clarify smoke-test limits. --- .github/workflows/mkosi-build.yml | 37 +---- os/common/scripts/export-kernel-devel.sh | 29 +--- os/image/assemble.sh | 8 +- os/image/kernel-builder/Dockerfile | 6 +- os/image/kernel-builder/README.md | 28 ++-- os/image/kernel-builder/build.sh | 34 +++++ os/mkosi/scripts/make-release-artifacts.sh | 8 +- os/mkosi/tests/acceptance.sh | 31 +--- os/mkosi/tests/check-output.sh | 4 +- os/tests/test-kernel-builder-image.sh | 8 +- os/tests/test-kernel-devel.sh | 158 +++++++++++++++++++++ 11 files changed, 230 insertions(+), 121 deletions(-) create mode 100755 os/image/kernel-builder/build.sh create mode 100755 os/tests/test-kernel-devel.sh diff --git a/.github/workflows/mkosi-build.yml b/.github/workflows/mkosi-build.yml index 37c83e01b..e9c82c05f 100644 --- a/.github/workflows/mkosi-build.yml +++ b/.github/workflows/mkosi-build.yml @@ -273,10 +273,6 @@ jobs: VERSION: ${{ steps.hashes.outputs.version }} run: | set -euo pipefail - # The kernel build tree is what an application compiles its own - # modules against. It is not measured and not part of - # os_image_hash -- it is published so an application does not have to - # reconstruct the kernel build to obtain a matching Module.symvers. devel="$DIST_DIR/dstack-$VERSION-kernel-devel.tar.gz" test -f "$devel" sha256=$(sha256sum "$devel" | awk '{print $1}') @@ -285,6 +281,7 @@ jobs: # archive is what ships, and its metadata is what the builder image # is then built against. work=$(mktemp -d) + trap 'rm -rf "$work"' EXIT tar -xzf "$devel" -C "$work" \ "dstack-$VERSION-kernel-devel/kernel-devel.json" meta="$work/dstack-$VERSION-kernel-devel/kernel-devel.json" @@ -355,37 +352,15 @@ jobs: OS_IMAGE_HASH: ${{ needs.build.outputs.os_image_hash }} run: | set -euo pipefail - # shellcheck source=/dev/null - source os/mkosi/versions.env - # GHCR path segments must be lowercase and this repository's owner is - # not; other release workflows hardcode the lowercase owner, which a - # fork cannot publish under. owner=$(echo "$GITHUB_REPOSITORY_OWNER" | tr '[:upper:]' '[:lower:]') ref="ghcr.io/$owner/dstack-kernel-builder" - # A build context of its own: dist/ holds the ~1 GiB image tarballs - # and all of it would be uploaded to the daemon. - mkdir -p ctx - cp "dist/dstack-$VERSION-kernel-devel.tar.gz" ctx/ - # The toolchain comes from the same pinned Debian snapshot that - # compiled the guest kernel. A different compiler still builds a - # module; what it does not build is one this kernel accepts. - docker build \ - --file os/image/kernel-builder/Dockerfile \ - --build-arg "BASE_IMAGE=debian:$DEBIAN_RELEASE-slim" \ - --build-arg "DEBIAN_SNAPSHOT=$DEBIAN_SNAPSHOT" \ - --build-arg "KERNEL_RELEASE=$KERNEL_RELEASE" \ - --build-arg "KERNEL_DEVEL_TARBALL=dstack-$VERSION-kernel-devel.tar.gz" \ - --build-arg "IMAGE_VERSION=$VERSION" \ - --build-arg "OS_IMAGE_HASH=$OS_IMAGE_HASH" \ - --build-arg "GIT_REVISION=$GITHUB_SHA" \ - --tag "$ref:$VERSION" \ - --tag "$ref:$KERNEL_RELEASE" \ - ctx + IMAGE_VERSION="$VERSION" GIT_REVISION="$GITHUB_SHA" \ + ./os/image/kernel-builder/build.sh \ + "dist/dstack-$VERSION-kernel-devel.tar.gz" "$ref:$VERSION" "$KERNEL_RELEASE" + docker tag "$ref:$VERSION" "$ref:$KERNEL_RELEASE" echo "ref=$ref" >> "$GITHUB_OUTPUT" - # The image is only useful if what it builds loads in the guest, and the - # vermagic is what decides that. Runs before the push, so a mismatched - # toolchain or a stale kernel tree is never published. + # Compile a module and check its target release before publishing. - name: Build a module inside the image env: IMAGE: ${{ steps.image.outputs.ref }}:${{ needs.build.outputs.version }} diff --git a/os/common/scripts/export-kernel-devel.sh b/os/common/scripts/export-kernel-devel.sh index 33e47c0c5..96f9d4db7 100755 --- a/os/common/scripts/export-kernel-devel.sh +++ b/os/common/scripts/export-kernel-devel.sh @@ -1,24 +1,10 @@ #!/bin/bash # SPDX-License-Identifier: Apache-2.0 # -# Export the minimal kernel build tree an application needs to compile an -# out-of-tree module (`make -C $KDIR M=$PWD modules`) against the guest kernel. -# -# The guest image ships modules but no build tree, so an application that needs -# its own .ko had to reconstruct a byte-identical kernel build to obtain a -# matching Module.symvers and vermagic. This exports that tree from the build -# that produced the shipped kernel instead. -# -# The file selection is upstream's: scripts/package/install-extmod-build is -# what `make bindeb-pkg` uses to produce Debian's linux-headers package, so the -# set of files carried here tracks Kbuild rather than a list maintained by -# dstack. It is invoked through `make run-command` because it reads srctree, -# SRCARCH, CC and HOSTCC out of the Kbuild environment. -# -# The result is deliberately NOT installed into the guest rootfs: it is ~100 MB -# of build inputs no CVM needs at runtime, and adding it would change the -# rootfs verity hash and therefore the OS image identity. It ships as a -# separate release artifact. +# Export an external-module build tree from the kernel build that ships in the +# guest. Delegate file selection to upstream's Debian headers exporter through +# make run-command, which supplies its required Kbuild environment. +# This tree is a separate developer artifact, never guest rootfs payload. set -euo pipefail usage() { @@ -98,12 +84,9 @@ install -m0644 "$config" "$headers/.config" # __FILE__ and debug info, not to -S output; measured, the paths survive). # Dropping it costs nothing: it exists only to generate # scripts/mod/devicetable-offsets.h, which ships, and so does its .c source. -find "$headers" -name '*.s' -delete +rm -f "$headers/scripts/mod/devicetable-offsets.s" -# Written for the consumer, not for the build: an out-of-tree module is only -# loadable by the image whose kernel produced this tree, so the tree has to say -# which image that is, and a Module.symvers or .config that differs from the -# shipped one silently produces a module that fails to load. +# Record identity and compatibility inputs for consumers of the archive. python3 - "$OUT" "$headers" "$release" "$version" "$BACKEND" "$IMAGE_VERSION" \ "$FLAVOR" <<'PY' import hashlib diff --git a/os/image/assemble.sh b/os/image/assemble.sh index 670b8a9b4..956b7c8e9 100755 --- a/os/image/assemble.sh +++ b/os/image/assemble.sh @@ -669,12 +669,8 @@ if [ "$DSTACK_TAR_RELEASE" = "1" ]; then fi fi -# The exported kernel build tree, when the backend produced one. It ships -# beside the image tarballs rather than inside them, and it is deliberately -# absent from sha256sum.txt: nothing in it is measured at boot, so folding it -# into os_image_hash would make the OS identity depend on a developer artifact. -# Applications building an out-of-tree module against this image download it; -# see docs/building-guest-os.md. +# Publish the optional developer artifact beside the images, without adding it +# to sha256sum.txt or os_image_hash. See docs/building-guest-os.md. if [ -n "$KERNEL_DEVEL_ARCHIVE" ]; then echo "Publishing the kernel build tree to ${KERNEL_DEVEL_TAR}" cp "$KERNEL_DEVEL_ARCHIVE" "$KERNEL_DEVEL_TAR" diff --git a/os/image/kernel-builder/Dockerfile b/os/image/kernel-builder/Dockerfile index d053d160b..22ddf562f 100644 --- a/os/image/kernel-builder/Dockerfile +++ b/os/image/kernel-builder/Dockerfile @@ -68,10 +68,8 @@ RUN set -eux; \ mkdir -p "/lib/modules/${KERNEL_RELEASE}"; \ ln -s "/usr/src/linux-headers-${KERNEL_RELEASE}" "/lib/modules/${KERNEL_RELEASE}/build" -# KDIR is the variable an out-of-tree Makefile passes to `make -C`. The -# /lib/modules symlink covers Makefiles that derive it from `uname -r` instead, -# which inside a container reports the host kernel and would otherwise point at -# nothing. +# Pass KDIR explicitly; uname -r inside a container still names the host kernel. +# The /lib/modules symlink is for consumers using the target KERNEL_RELEASE. ENV KDIR="/usr/src/linux-headers-${KERNEL_RELEASE}" ENV KERNEL_RELEASE="${KERNEL_RELEASE}" diff --git a/os/image/kernel-builder/README.md b/os/image/kernel-builder/README.md index f47eadd0f..a416489bc 100644 --- a/os/image/kernel-builder/README.md +++ b/os/image/kernel-builder/README.md @@ -13,9 +13,8 @@ the kernel release (`6.18.x-dstack`). Application usage is documented in ## Build it locally -The build context is a directory containing only the kernel-devel archive; the -release directory itself holds ~1 GiB of image tarballs that the container -build would otherwise upload to the daemon. +The build script uses the repository's pinned toolchain snapshot and an isolated +build context containing only the kernel-devel archive. ```bash source os/mkosi/versions.env @@ -24,21 +23,12 @@ release=$(tar -xOf "$dist/dstack-$DSTACK_VERSION-kernel-devel.tar.gz" \ "dstack-$DSTACK_VERSION-kernel-devel/kernel-devel.json" | python3 -c 'import json,sys; print(json.load(sys.stdin)["kernel_release"])') -mkdir -p /tmp/kernel-builder-ctx -cp "$dist/dstack-$DSTACK_VERSION-kernel-devel.tar.gz" /tmp/kernel-builder-ctx/ - -docker build \ - --file os/image/kernel-builder/Dockerfile \ - --build-arg "BASE_IMAGE=debian:$DEBIAN_RELEASE-slim" \ - --build-arg "DEBIAN_SNAPSHOT=$DEBIAN_SNAPSHOT" \ - --build-arg "KERNEL_RELEASE=$release" \ - --build-arg "KERNEL_DEVEL_TARBALL=dstack-$DSTACK_VERSION-kernel-devel.tar.gz" \ - --build-arg "IMAGE_VERSION=$DSTACK_VERSION" \ - --tag "dstack-kernel-builder:$DSTACK_VERSION" \ - /tmp/kernel-builder-ctx +./os/image/kernel-builder/build.sh \ + "$dist/dstack-$DSTACK_VERSION-kernel-devel.tar.gz" \ + "dstack-kernel-builder:$DSTACK_VERSION" "$release" ``` -Then check that what it builds would load in the guest: +Then smoke-test module compilation and the target kernel release: ```bash ./os/tests/test-kernel-builder-image.sh "dstack-kernel-builder:$DSTACK_VERSION" "$release" @@ -46,7 +36,11 @@ Then check that what it builds would load in the guest: That script builds `os/tests/kmod-hello/` inside the image and fails unless the resulting module's `vermagic` names the kernel the image claims to serve. CI -runs it before the image is pushed. +runs it before the image is pushed. This does not prove loadability: symbol +versions and other kernel compatibility requirements need a real guest load test. + +Set `IMAGE_VERSION`, `OS_IMAGE_HASH`, and `GIT_REVISION` to override image labels. +`DOCKER` selects the Docker command for both scripts. ## What the image pins, and what it does not diff --git a/os/image/kernel-builder/build.sh b/os/image/kernel-builder/build.sh new file mode 100755 index 000000000..3cba24500 --- /dev/null +++ b/os/image/kernel-builder/build.sh @@ -0,0 +1,34 @@ +#!/bin/bash +# SPDX-FileCopyrightText: © 2026 Phala Network +# +# SPDX-License-Identifier: Apache-2.0 +set -euo pipefail + +if [[ $# != 3 ]]; then + echo "Usage: $0 ARCHIVE IMAGE_REF KERNEL_RELEASE" >&2 + exit 2 +fi +archive=$(realpath "$1") +image=$2 +release=$3 +SELF=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd) +# shellcheck source=../../mkosi/versions.env +source "$SELF/../../mkosi/versions.env" +DOCKER=${DOCKER:-docker} + +# Do not send the neighboring OS image tarballs to the Docker daemon. +context=$(mktemp -d) +trap 'rm -rf "$context"' EXIT +cp "$archive" "$context/kernel-devel.tar.gz" + +$DOCKER build \ + --file "$SELF/Dockerfile" \ + --build-arg "BASE_IMAGE=debian:$DEBIAN_RELEASE-slim" \ + --build-arg "DEBIAN_SNAPSHOT=$DEBIAN_SNAPSHOT" \ + --build-arg "KERNEL_RELEASE=$release" \ + --build-arg KERNEL_DEVEL_TARBALL=kernel-devel.tar.gz \ + --build-arg "IMAGE_VERSION=${IMAGE_VERSION:-$DSTACK_VERSION}" \ + --build-arg "OS_IMAGE_HASH=${OS_IMAGE_HASH:-unknown}" \ + --build-arg "GIT_REVISION=${GIT_REVISION:-unknown}" \ + --tag "$image" \ + "$context" diff --git a/os/mkosi/scripts/make-release-artifacts.sh b/os/mkosi/scripts/make-release-artifacts.sh index 071d7fef1..69c4696a7 100755 --- a/os/mkosi/scripts/make-release-artifacts.sh +++ b/os/mkosi/scripts/make-release-artifacts.sh @@ -125,12 +125,8 @@ touch -d "@$SOURCE_DATE_EPOCH" "$OUT/files/"* is_dev=false; name=dstack if [[ $FLAVOR == dev ]]; then is_dev=true; name=dstack-dev; fi -# The exported kernel build tree, moved out of the image tree by mkosi.finalize. -# Archived here rather than shipped in either release tarball: applications -# download it to build their own modules, operators deploying the image do not -# need it, and it is not part of the measured OS image identity. Same fixed tar -# options as the release tarballs, so repro-check can compare it byte for byte, -# and skipped on a cached iteration build for the same reason those are. +# Archive the tree removed by mkosi.finalize separately from measured payload. +# Use fixed tar metadata for repro-check; cached iteration builds can skip it. kernel_devel="$KERNEL_TREE/kernel-devel" [[ -d $kernel_devel ]] || { echo 'kernel build tree was not exported' >&2; exit 1; } kernel_devel_artifact= diff --git a/os/mkosi/tests/acceptance.sh b/os/mkosi/tests/acceptance.sh index a5d0be4c9..ec01d101b 100755 --- a/os/mkosi/tests/acceptance.sh +++ b/os/mkosi/tests/acceptance.sh @@ -182,33 +182,7 @@ grep -q 'AmdSev/AmdSevX64.dsc' "$D/components/ovmf/ovmf-build.sh" grep -q '0006-OvmfPkg-AmdSev-drop-embedded-grub.patch' "$D/components/ovmf/ovmf.sh" grep -q '0005-UefiCpuPkg' "$D/components/ovmf/ovmf.sh" grep -q '0007-OvmfPkg-QemuKernelLoaderFsDxe' "$D/components/ovmf/ovmf.sh" -# The exported kernel build tree is a release artifact, never guest payload. -# Each of these is load-bearing: the component has to export it, finalize has -# to move it out of the tree before the rootfs is measured, and the identity -# preimage must not list it. -grep -q 'export-kernel-devel.sh' "$D/components/kernel/kernel-build.sh" -grep -q 'export-kernel-devel.sh' "$D/components/kernel/kernel.sh" -# shellcheck disable=SC2016 # a grep pattern matching the literal script text -grep -q 'mv "$TREE/usr/lib/dstack/kernel-devel" "$KSTAGE/kernel-devel"' \ - "$D/mkosi.finalize" -grep -q '"kernel_devel":kernel_devel or None' "$D/scripts/make-release-artifacts.sh" -grep -q 'kernel-devel' "$D/tests/check-output.sh" -grep -q 'usr/lib/dstack/kernel-devel' "$D/parity.json" -grep -q 'kernel_devel' "$D/../spec/artifact-manifest.schema.json" -grep -q 'KERNEL_DEVEL_ARCHIVE' "$D/../image/assemble.sh" -# Host programs and generated intermediates reach a published artifact for the -# first time through this tree, so repro-check has to cover it. -grep -q 'kernel-devel.tar.gz' "$D/build.sh" -test -x "$D/../common/scripts/export-kernel-devel.sh" -test -x "$D/../tests/test-kernel-builder-image.sh" -test -f "$D/../image/kernel-builder/Dockerfile" -# The builder image installs its compiler from the snapshot that built the -# guest kernel; a hardcoded one here would drift from versions.env. -grep -q 'ARG DEBIAN_SNAPSHOT' "$D/../image/kernel-builder/Dockerfile" -if grep -q "$DEBIAN_SNAPSHOT" "$D/../image/kernel-builder/Dockerfile"; then - echo 'the builder image must take the Debian snapshot as a build argument' >&2 - exit 1 -fi +"$D/../tests/test-kernel-devel.sh" grep -q 'objcopy --strip-debug' "$D/mkosi.build" grep -q 'depmod -b.*KERNEL_VERSION-dstack' "$D/mkosi.build" grep -q '^CleanPackageMetadata=yes$' "$D/mkosi.conf" @@ -250,7 +224,8 @@ bash -n "$D"/*.sh "$D"/mkosi.build "$D"/mkosi.clean "$D"/mkosi.finalize \ "$D"/mkosi.postinst "$D"/mkosi.postoutput "$D"/scripts/*.sh \ "$D"/components/*/*.sh "$D"/tests/*.sh "$D"/mkosi.skeleton/usr/bin/*.sh \ "$D"/../image/kernel-cmdline.sh "$D"/../common/scripts/export-kernel-devel.sh \ - "$D"/../tests/test-kernel-builder-image.sh + "$D"/../tests/test-kernel-builder-image.sh "$D"/../tests/test-kernel-devel.sh \ + "$D"/../image/kernel-builder/build.sh # The Python helpers were only ever exercised by a real build. Keep the # bytecode out of the source tree, which is mounted into the build sandbox. pycache=$(mktemp -d) diff --git a/os/mkosi/tests/check-output.sh b/os/mkosi/tests/check-output.sh index 09d243640..a05c5389e 100755 --- a/os/mkosi/tests/check-output.sh +++ b/os/mkosi/tests/check-output.sh @@ -30,8 +30,10 @@ if [[ ${DSTACK_TAR_RELEASE:-1} == 1 ]]; then echo "missing kernel build tree archive: $out-kernel-devel.tar.gz" >&2 exit 1 } + # Consume the full listing: grep -q would close the pipe on the first match + # and make tar fail with SIGPIPE under pipefail. tar -tzf "$out-kernel-devel.tar.gz" | \ - grep -qx "$(basename "$out")-kernel-devel/kernel-devel.json" || { + grep -Fx "$(basename "$out")-kernel-devel/kernel-devel.json" >/dev/null || { echo 'kernel build tree archive does not declare its kernel' >&2 exit 1 } diff --git a/os/tests/test-kernel-builder-image.sh b/os/tests/test-kernel-builder-image.sh index 02f3d823e..cea15030f 100755 --- a/os/tests/test-kernel-builder-image.sh +++ b/os/tests/test-kernel-builder-image.sh @@ -3,11 +3,9 @@ # # SPDX-License-Identifier: Apache-2.0 # -# Build the smoke-test module inside a kernel-builder image and assert that the -# result would load in the guest it claims to serve. A builder image whose -# headers, Module.symvers or compiler do not match the shipped kernel still -# produces a .ko; what it does not produce is a matching vermagic, so that is -# what this checks. +# Check that the builder compiles a module for the expected kernel release. +# This is not a load test: vermagic's release field does not validate symbol +# versions, compiler compatibility, or the rest of the kernel configuration. set -euo pipefail IMAGE=${1:?container image required} diff --git a/os/tests/test-kernel-devel.sh b/os/tests/test-kernel-devel.sh new file mode 100755 index 000000000..994b30f5c --- /dev/null +++ b/os/tests/test-kernel-devel.sh @@ -0,0 +1,158 @@ +#!/bin/bash +# SPDX-FileCopyrightText: © 2026 Phala Network +# +# SPDX-License-Identifier: Apache-2.0 +set -euo pipefail +ROOT=$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd) +work=$(mktemp -d) +trap 'rm -rf "$work"' EXIT + +python3 - "$ROOT" "$work" <<'PY' +import hashlib +import io +import json +import os +import subprocess +import sys +import tarfile +from pathlib import Path + +root, work = map(Path, sys.argv[1:]) + + +def check(script, *args, success=True, message=None, env=None): + result = subprocess.run( + [str(root / script), *map(str, args)], + env=env, + capture_output=True, + text=True, + ) + output = result.stdout + result.stderr + assert (result.returncode == 0) == success, output + assert message is None or message in output, output + + +# Validate the actual assembler's optional artifact contract without image tools. +payload = work / "payload" +payload.write_bytes(b"fixture\n") +manifest = { + "schema_version": 1, + "backend": "mkosi", + "image": {"name": "dstack", "version": "test", "flavor": "prod", "is_dev": False}, + "source": {"git_revision": "fixture"}, + "boot": {"ovmf_variant": "tdx"}, + "verity": {"root_hash": "0" * 64, "data_size": 4096}, + "artifacts": dict.fromkeys( + ("initramfs", "kernel", "firmware", "rootfs_verity"), "payload" + ), +} +manifest["artifacts"].update(firmware_sev=None, uki=None) +manifest_path = work / "artifact-manifest.json" +manifest_path.write_text(json.dumps(manifest)) +check("os/image/assemble.sh", "--manifest", manifest_path, "--validate-only") +for value, success in ( + (None, True), + ("payload", True), + (str(payload), False), + ("../payload", False), + ("missing", False), +): + manifest["artifacts"]["kernel_devel"] = value + manifest_path.write_text(json.dumps(manifest)) + check( + "os/image/assemble.sh", + "--manifest", + manifest_path, + "--validate-only", + success=success, + ) + +# Only partition inspection is stubbed; checksums, tar, grep and the output +# checker run for real. The fixture deliberately is not a bootable disk image. +bin_dir = work / "bin" +bin_dir.mkdir() +sgdisk = bin_dir / "sgdisk" +sgdisk.write_text("#!/bin/sh\nprintf 'dstack-rootfs\\n'\n") +sgdisk.chmod(0o755) +env = dict(os.environ, PATH=f"{bin_dir}:{os.environ['PATH']}", DSTACK_TAR_RELEASE="1") +out = work / "dstack-test" +out.mkdir() +for name in ( + "bzImage", + "initramfs.cpio.gz", + "ovmf.fd", + "ovmf-sev.fd", + "rootfs.img.parted.verity", + "measurement.tdx.cbor", + "measurement.snp.cbor", + "digest.txt", +): + (out / name).write_bytes(payload.read_bytes()) +metadata = dict.fromkeys( + ( + "bios", + "kernel", + "cmdline", + "initrd", + "rootfs", + "version", + "git_revision", + "ovmf_variant", + ), + "fixture", +) +metadata.update({"bios-sev": "ovmf-sev.fd", "builder": "mkosi", "is_dev": False}) +(out / "metadata.json").write_text(json.dumps(metadata)) +hashes = "".join( + f"{hashlib.sha256(path.read_bytes()).hexdigest()} {path.name}\n" + for path in sorted(out.iterdir()) +) +(out / "sha256sum.txt").write_text(hashes) +archive = work / "dstack-test-kernel-devel.tar.gz" + + +def write_archive(with_metadata): + prefix = "dstack-test-kernel-devel/" + with tarfile.open(archive, "w:gz") as tar: + if with_metadata: + data = b'{"kernel_release": "test"}\n' + member = tarfile.TarInfo(prefix + "kernel-devel.json") + member.size = len(data) + tar.addfile(member, io.BytesIO(data)) + # Exceed pipe capacity after the match to expose grep -q's SIGPIPE. + for index in range(3000): + tar.addfile( + tarfile.TarInfo( + prefix + f"linux-headers-test/include/header-{index:04}.h" + ) + ) + + +checker = "os/mkosi/tests/check-output.sh" +write_archive(True) +check(checker, out, env=env) +valid_archive = archive.read_bytes() +write_archive(False) +check(checker, out, env=env, success=False, message="does not declare its kernel") +archive.write_bytes(valid_archive[: len(valid_archive) // 2]) +check(checker, out, env=env, success=False) +archive.unlink() +check(checker, out, env=env, success=False, message="missing kernel build tree archive") +check(checker, out, env=dict(env, DSTACK_TAR_RELEASE="0")) +archive.write_bytes(valid_archive) + +# A valid checksum must not make a developer artifact part of image identity. +name = "kernel-devel.tar.gz" +(out / name).write_bytes(valid_archive) +(out / "sha256sum.txt").write_text( + hashes + f"{hashlib.sha256(valid_archive).hexdigest()} {name}\n" +) +check( + checker, + out, + env=env, + success=False, + message="must not be part of the image identity", +) +print("kernel-devel artifact tests passed") +PY From 7e76996f972b4e5b1cca516949bbea7a53c02735 Mon Sep 17 00:00:00 2001 From: Kevin Wang Date: Thu, 17 Sep 2026 02:44:20 -0700 Subject: [PATCH 32/93] refactor(os): simplify the kernel-devel export and builder image Drop the fixture tests, the kernel-devel.json metadata and the standalone export script; inline the export into the kernel component. Trim the workflow reporting, the builder image and the duplicated docs, and stop tagging the builder image with the kernel release, which is not unique across dstack versions. --- .github/workflows/mkosi-build.yml | 116 ++------------- docs/building-guest-os.md | 63 ++------ os/common/scripts/export-kernel-devel.sh | 144 ------------------- os/image/README.md | 14 +- os/image/assemble.sh | 4 - os/image/kernel-builder/Dockerfile | 71 +++------ os/image/kernel-builder/README.md | 53 ------- os/image/kernel-builder/build.sh | 19 +-- os/mkosi/build.sh | 4 - os/mkosi/components/kernel/kernel-build.sh | 19 +-- os/mkosi/components/kernel/kernel.sh | 6 +- os/mkosi/mkosi.finalize | 5 +- os/mkosi/scripts/make-release-artifacts.sh | 5 +- os/mkosi/tests/acceptance.sh | 5 +- os/mkosi/tests/check-output.sh | 20 --- os/spec/artifact-manifest.schema.json | 1 - os/tests/kmod-hello/Makefile | 17 --- os/tests/kmod-hello/hello.c | 29 ---- os/tests/test-kernel-builder-image.sh | 41 ------ os/tests/test-kernel-devel.sh | 158 --------------------- 20 files changed, 65 insertions(+), 729 deletions(-) delete mode 100755 os/common/scripts/export-kernel-devel.sh delete mode 100644 os/image/kernel-builder/README.md delete mode 100644 os/tests/kmod-hello/Makefile delete mode 100644 os/tests/kmod-hello/hello.c delete mode 100755 os/tests/test-kernel-builder-image.sh delete mode 100755 os/tests/test-kernel-devel.sh diff --git a/.github/workflows/mkosi-build.yml b/.github/workflows/mkosi-build.yml index e9c82c05f..22386e50f 100644 --- a/.github/workflows/mkosi-build.yml +++ b/.github/workflows/mkosi-build.yml @@ -63,8 +63,7 @@ jobs: os_image_hash: ${{ steps.hashes.outputs.os_image_hash }} bare_sha256: ${{ steps.hashes.outputs.bare_sha256 }} uki_sha256: ${{ steps.hashes.outputs.uki_sha256 }} - kernel_release: ${{ steps.kernel_devel.outputs.kernel_release }} - kernel_devel_sha256: ${{ steps.kernel_devel.outputs.sha256 }} + kernel_devel_sha256: ${{ steps.hashes.outputs.kernel_devel_sha256 }} steps: # GitHub-hosted runners have enough CPU and memory for this build, but @@ -208,6 +207,7 @@ jobs: bare_sha256=$(sha256sum "$bare" | awk '{print $1}') uki_sha256=$(sha256sum "$uki" | awk '{print $1}') + kernel_devel_sha256=$(sha256sum "$DIST_DIR/dstack-$version-kernel-devel.tar.gz" | awk '{print $1}') git_revision=$(python3 -c 'import json,sys; print(json.load(open(sys.argv[1]))["git_revision"])' "$img/metadata.json") mrtd=$("$DSTACK_MR" inspect-measurement "$img/measurement.tdx.cbor" \ | python3 -c 'import json,sys; print(json.load(sys.stdin)["tdvf"]["mrtd"]["single_pass"])') @@ -221,6 +221,7 @@ jobs: echo "os_image_hash=$os_image_hash" echo "bare_sha256=$bare_sha256" echo "uki_sha256=$uki_sha256" + echo "kernel_devel_sha256=$kernel_devel_sha256" } >> "$GITHUB_OUTPUT" # os_image_hash is the value operators need most, so surface it in the @@ -238,6 +239,7 @@ jobs: echo "| AWS \`boot_pcr_digest\` | \`$aws_pcr\` |" echo "| \`dstack-$version.tar.gz\` | \`$bare_sha256\` |" echo "| \`dstack-$version-uki.tar.gz\` | \`$uki_sha256\` |" + echo "| \`dstack-$version-kernel-devel.tar.gz\` | \`$kernel_devel_sha256\` |" echo echo '
sha256sum.txt (os_image_hash preimage)' echo @@ -264,52 +266,9 @@ jobs: echo "aws_boot_pcr $aws_pcr" echo "$bare_sha256 dstack-$version.tar.gz" echo "$uki_sha256 dstack-$version-uki.tar.gz" + echo "$kernel_devel_sha256 dstack-$version-kernel-devel.tar.gz" } | tee "$DIST_DIR/image-hashes.txt" - - name: Report the kernel build tree - id: kernel_devel - env: - DIST_DIR: ${{ steps.build.outputs.dist_dir }} - VERSION: ${{ steps.hashes.outputs.version }} - run: | - set -euo pipefail - devel="$DIST_DIR/dstack-$VERSION-kernel-devel.tar.gz" - test -f "$devel" - sha256=$(sha256sum "$devel" | awk '{print $1}') - bytes=$(stat -L -c %s "$devel") - # Read the identity out of the archive, not out of versions.env: the - # archive is what ships, and its metadata is what the builder image - # is then built against. - work=$(mktemp -d) - trap 'rm -rf "$work"' EXIT - tar -xzf "$devel" -C "$work" \ - "dstack-$VERSION-kernel-devel/kernel-devel.json" - meta="$work/dstack-$VERSION-kernel-devel/kernel-devel.json" - read -r kernel_release files uncompressed < <( - python3 -c 'import json,sys; d=json.load(open(sys.argv[1])); print(d["kernel_release"], d["file_count"], d["uncompressed_bytes"])' \ - "$meta") - compiler=$(python3 -c 'import json,sys; print(json.load(open(sys.argv[1]))["compiler"])' "$meta") - { - echo "kernel_release=$kernel_release" - echo "sha256=$sha256" - } >> "$GITHUB_OUTPUT" - { - echo - echo "### Kernel build tree (out-of-tree modules)" - echo - echo '| Field | Value |' - echo '| --- | --- |' - echo "| Kernel release | \`$kernel_release\` |" - echo "| Compiler | \`$compiler\` |" - echo "| Files | $files |" - echo "| Unpacked | $uncompressed bytes |" - echo "| \`dstack-$VERSION-kernel-devel.tar.gz\` | $bytes bytes, \`$sha256\` |" - } >> "$GITHUB_STEP_SUMMARY" - { - echo "kernel_release $kernel_release" - echo "$sha256 dstack-$VERSION-kernel-devel.tar.gz" - } | tee -a "$DIST_DIR/image-hashes.txt" - - name: Upload guest images uses: actions/upload-artifact@v4 with: @@ -322,8 +281,6 @@ jobs: kernel-builder: name: Kernel module builder image - # Same gate as the build it consumes. On a dispatch the image is built and - # smoke-tested but not published: only the release tag publishes. if: >- github.event_name == 'workflow_dispatch' || startsWith(github.ref, 'refs/tags/mkosi-os-v') @@ -345,27 +302,13 @@ jobs: path: dist - name: Build the builder image - id: image env: VERSION: ${{ needs.build.outputs.version }} - KERNEL_RELEASE: ${{ needs.build.outputs.kernel_release }} - OS_IMAGE_HASH: ${{ needs.build.outputs.os_image_hash }} run: | - set -euo pipefail owner=$(echo "$GITHUB_REPOSITORY_OWNER" | tr '[:upper:]' '[:lower:]') - ref="ghcr.io/$owner/dstack-kernel-builder" - IMAGE_VERSION="$VERSION" GIT_REVISION="$GITHUB_SHA" \ - ./os/image/kernel-builder/build.sh \ - "dist/dstack-$VERSION-kernel-devel.tar.gz" "$ref:$VERSION" "$KERNEL_RELEASE" - docker tag "$ref:$VERSION" "$ref:$KERNEL_RELEASE" - echo "ref=$ref" >> "$GITHUB_OUTPUT" - - # Compile a module and check its target release before publishing. - - name: Build a module inside the image - env: - IMAGE: ${{ steps.image.outputs.ref }}:${{ needs.build.outputs.version }} - KERNEL_RELEASE: ${{ needs.build.outputs.kernel_release }} - run: ./os/tests/test-kernel-builder-image.sh "$IMAGE" "$KERNEL_RELEASE" + echo "IMAGE=ghcr.io/$owner/dstack-kernel-builder:$VERSION" >> "$GITHUB_ENV" + GIT_REVISION="$GITHUB_SHA" ./os/image/kernel-builder/build.sh \ + "dist/dstack-$VERSION-kernel-devel.tar.gz" "ghcr.io/$owner/dstack-kernel-builder:$VERSION" - name: Log in to GitHub Container Registry if: startsWith(github.ref, 'refs/tags/mkosi-os-v') @@ -377,31 +320,12 @@ jobs: - name: Push the builder image if: startsWith(github.ref, 'refs/tags/mkosi-os-v') - env: - REF: ${{ steps.image.outputs.ref }} - VERSION: ${{ needs.build.outputs.version }} - KERNEL_RELEASE: ${{ needs.build.outputs.kernel_release }} - run: | - set -euo pipefail - docker push "$REF:$VERSION" - docker push "$REF:$KERNEL_RELEASE" - digest=$(docker inspect --format='{{index .RepoDigests 0}}' "$REF:$VERSION") - echo "::notice title=kernel-builder::$digest" - { - echo "### Kernel module builder image" - echo - echo '```' - echo "$digest" - echo '```' - } >> "$GITHUB_STEP_SUMMARY" + run: docker push "$IMAGE" release: name: Publish release # The tag is the publish decision. Nothing else reaches this job. if: startsWith(github.ref, 'refs/tags/mkosi-os-v') - # Also needs kernel-builder: the notes point at the builder image, so a - # release must not be created when that image failed to build its smoke - # test module or failed to publish. needs: [build, kernel-builder] runs-on: ubuntu-latest timeout-minutes: 30 @@ -435,10 +359,8 @@ jobs: fi bare="dist/dstack-$version.tar.gz" uki="dist/dstack-$version-uki.tar.gz" - kernel_devel="dist/dstack-$version-kernel-devel.tar.gz" test -f "$bare" test -f "$uki" - test -f "$kernel_devel" read -r image_version image_revision < <( tar -xOf "$bare" "dstack-$version/metadata.json" | @@ -465,12 +387,9 @@ jobs: OS_IMAGE_HASH: ${{ needs.build.outputs.os_image_hash }} BARE_SHA256: ${{ needs.build.outputs.bare_sha256 }} UKI_SHA256: ${{ needs.build.outputs.uki_sha256 }} - KERNEL_RELEASE: ${{ needs.build.outputs.kernel_release }} KERNEL_DEVEL_SHA256: ${{ needs.build.outputs.kernel_devel_sha256 }} run: | set -euo pipefail - owner=$(echo "$GITHUB_REPOSITORY_OWNER" | tr '[:upper:]' '[:lower:]') - builder="ghcr.io/$owner/dstack-kernel-builder:$VERSION" { echo "Guest OS image \`$VERSION\`, built by the Debian/mkosi backend." echo @@ -488,21 +407,8 @@ jobs: echo "\`image-hashes.txt\` additionally records the TDX \`MRTD\`, the GCP" echo "\`uki_auth\` hash and the AWS \`boot_pcr_digest\` for this image." echo - echo "### Out-of-tree kernel modules" - echo - echo "This image runs \`$KERNEL_RELEASE\`. To build a module for it, use the" - echo "matching toolchain image, which carries the kernel build tree of this" - echo "release and a compiler from the same pinned Debian snapshot:" - echo - echo '```dockerfile' - echo "FROM $builder AS kmod" - echo "COPY mymod/ /src" - echo "RUN make -C \$KDIR M=/src modules" - echo '```' - echo - echo "\`dstack-$VERSION-kernel-devel.tar.gz\` is the same build tree for use" - echo "outside that image. Neither is measured: they are not part of" - echo "\`os_image_hash\`. See" + echo "To build out-of-tree kernel modules, use" + echo "\`ghcr.io/dstack-tee/dstack-kernel-builder:$VERSION\`; see" echo "[docs/building-guest-os.md](https://github.com/${GITHUB_REPOSITORY}/blob/${TAG}/docs/building-guest-os.md#kernel-headers-for-out-of-tree-modules)." } > release-notes.md # --verify-tag: the tag is the publish trigger, so it must already diff --git a/docs/building-guest-os.md b/docs/building-guest-os.md index 4ebc6877a..4521380eb 100644 --- a/docs/building-guest-os.md +++ b/docs/building-guest-os.md @@ -132,19 +132,13 @@ and records `"is_dev": true` in `metadata.json`. ## Kernel headers for out-of-tree modules -The guest image ships kernel modules but no build tree, so an application that -needs its own `.ko` — a custom network or storage driver, a device shim — -would have to reconstruct the kernel build to obtain a `Module.symvers` and a -`vermagic` the guest kernel accepts. Every release publishes that build tree -instead: +Each release publishes the guest kernel's build tree for building out-of-tree +modules: -- `dstack--kernel-devel.tar.gz`, the exported kernel build tree, and -- `ghcr.io/dstack-tee/dstack-kernel-builder:`, that tree plus a - compiler from the same pinned Debian snapshot the guest kernel was built - with, with `KDIR` already pointing at it. - -Build the module where the application image is built, and ship the `.ko` in -the container image: +- `dstack--kernel-devel.tar.gz`, the exported build tree, and +- `ghcr.io/dstack-tee/dstack-kernel-builder:`, the same tree plus a + compiler from the pinned Debian snapshot the kernel was built with, with + `KDIR` set. ```dockerfile FROM ghcr.io/dstack-tee/dstack-kernel-builder: AS kmod @@ -155,45 +149,14 @@ FROM alpine COPY --from=kmod /src/mymod.ko /opt/ ``` -Outside that image, point `KDIR` at the unpacked tree: - -```bash -tar -xzf dstack--kernel-devel.tar.gz -make -C dstack--kernel-devel/linux-headers- M="$PWD" modules -``` +To build the image locally from a release archive, run +`os/image/kernel-builder/build.sh `. -`kernel-devel.json` in the archive records the kernel release, the compiler, -and the SHA-256 of `.config` and `Module.symvers`, so a module can be traced -back to the image it was built for. `os/tests/kmod-hello/` is a two-file module -that builds against either form and is what CI uses to check a published -builder image. - -Two constraints are worth knowing before debugging a failure: - -- **The compiler matters.** A module built with a different toolchain may still - load, but anything the config derives from `CONFIG_CC_VERSION_TEXT` and the - compiler's own ABI choices is no longer guaranteed to agree. Use the builder - image, or a compiler from the snapshot in `os/mkosi/versions.env`. -- **Modules built this way carry no BTF.** Generating it needs `vmlinux`, which - the archive does not carry because its debug info dwarfs everything else in - the release. Kbuild prints `Skipping BTF generation ... due to unavailability - of vmlinux` and produces a working module. - -### What this means for attestation - -Neither the archive nor the builder image is part of the OS image identity: -they are not in `sha256sum.txt`, so `os_image_hash` does not change when they -do, and a CVM never sees them. - -Loading a module is still a privileged operation and the module is kernel code -the application chose. The container needs `CAP_SYS_MODULE` (in practice, -`privileged: true`), and what a remote verifier can conclude about that module -is exactly what it can conclude about the rest of the application: the module -is part of the container image, the image digest is part of the compose file, -and the compose hash is measured into RTMR3 and enforced by the KMS. The module -itself is not measured separately, and dstack does not enforce module -signatures. Treat a custom module as trusted application code with kernel -privileges, and pin it by digest like any other part of the application. +Neither artifact is part of `sha256sum.txt`, so they do not affect +`os_image_hash`. A module loaded by an application (which needs +`CAP_SYS_MODULE`) is covered by the container image digest and therefore the +compose hash, but it is not measured separately and dstack does not enforce +module signatures. ## Check reproducibility diff --git a/os/common/scripts/export-kernel-devel.sh b/os/common/scripts/export-kernel-devel.sh deleted file mode 100755 index 96f9d4db7..000000000 --- a/os/common/scripts/export-kernel-devel.sh +++ /dev/null @@ -1,144 +0,0 @@ -#!/bin/bash -# SPDX-License-Identifier: Apache-2.0 -# -# Export an external-module build tree from the kernel build that ships in the -# guest. Delegate file selection to upstream's Debian headers exporter through -# make run-command, which supplies its required Kbuild environment. -# This tree is a separate developer artifact, never guest rootfs payload. -set -euo pipefail - -usage() { - cat <&2; usage >&2; exit 2 ;; - esac -done -if [ -z "$SRC" ] || [ -z "$BUILD" ] || [ -z "$OUT" ]; then - usage >&2 - exit 2 -fi -SRC=$(realpath "$SRC") -BUILD=$(realpath "$BUILD") -OUT=$(realpath -m "$OUT") - -exporter="$SRC/scripts/package/install-extmod-build" -[ -x "$exporter" ] || { - echo "kernel source does not provide scripts/package/install-extmod-build: $SRC" >&2 - exit 1 -} -config="$BUILD/.config" -release_file="$BUILD/include/config/kernel.release" -symvers="$BUILD/Module.symvers" -for required in "$config" "$release_file" "$symvers"; do - [ -f "$required" ] || { - echo "kernel output directory is incomplete, build modules first: $required" >&2 - exit 1 - } -done -release=$(cat "$release_file") -[ -n "$release" ] || { echo "empty kernel release in $release_file" >&2; exit 1; } -version=$(make -C "$SRC" -s kernelversion) - -headers="$OUT/linux-headers-$release" -rm -rf "$OUT" -mkdir -p "$OUT" -# run-command exports the Kbuild environment install-extmod-build reads, and it -# runs in the output directory, which is where the script expects to find -# Module.symvers and include/config. This is exactly how scripts/package/builddeb -# invokes it. -make -C "$SRC" O="$BUILD" run-command \ - KBUILD_RUN_COMMAND="$exporter $headers" -# install-extmod-build carries include/config/auto.conf, which is what Kbuild -# reads for an external module build, but not .config. Ship it anyway: it is -# what an application inspects to find out whether a config option it needs is -# enabled, and kernel-hardening or module-compat questions are answered from it. -install -m0644 "$config" "$headers/.config" -# The one file upstream's list carries that cannot be reproducible: -# scripts/mod/devicetable-offsets.s is a -fverbose-asm intermediate whose -# comments and line markers record the absolute source path of the build that -# produced it, and no prefix map rewrites them (-ffile-prefix-map applies to -# __FILE__ and debug info, not to -S output; measured, the paths survive). -# Dropping it costs nothing: it exists only to generate -# scripts/mod/devicetable-offsets.h, which ships, and so does its .c source. -rm -f "$headers/scripts/mod/devicetable-offsets.s" - -# Record identity and compatibility inputs for consumers of the archive. -python3 - "$OUT" "$headers" "$release" "$version" "$BACKEND" "$IMAGE_VERSION" \ - "$FLAVOR" <<'PY' -import hashlib -import json -import os -import sys - -out, headers, release, version, backend, image_version, flavor = sys.argv[1:] - - -def digest(path): - with open(path, "rb") as handle: - return hashlib.sha256(handle.read()).hexdigest() - - -def compiler(): - prefix = 'CONFIG_CC_VERSION_TEXT="' - with open(os.path.join(headers, ".config"), encoding="utf-8") as handle: - for line in handle: - if line.startswith(prefix): - return line.rstrip("\n")[len(prefix) : -1] - return "" - - -files = 0 -total = 0 -for root, _, names in os.walk(headers): - for name in names: - path = os.path.join(root, name) - files += 1 - if not os.path.islink(path): - total += os.path.getsize(path) - -metadata = { - "backend": backend, - "compiler": compiler(), - "config_sha256": digest(os.path.join(headers, ".config")), - "file_count": files, - "flavor": flavor, - "image_version": image_version, - "kernel_release": release, - "kernel_version": version, - "module_symvers_sha256": digest(os.path.join(headers, "Module.symvers")), - "uncompressed_bytes": total, -} -with open(os.path.join(out, "kernel-devel.json"), "w", encoding="utf-8") as handle: - json.dump(metadata, handle, indent=2, sort_keys=True) - handle.write("\n") -PY - -if [ -n "${SOURCE_DATE_EPOCH:-}" ]; then - find "$OUT" -print0 | xargs -0r touch --no-dereference \ - --date="@$SOURCE_DATE_EPOCH" -fi -echo "exported kernel build tree for $release to $OUT" diff --git a/os/image/README.md b/os/image/README.md index 6d3b4fd68..4084af001 100644 --- a/os/image/README.md +++ b/os/image/README.md @@ -49,16 +49,10 @@ image. ## Kernel build tree -A backend may export the kernel build tree of the image it built and declare it -as the optional `artifacts.kernel_devel` manifest entry. The assembler -publishes it as `--kernel-devel.tar.gz` beside the image -archives and deliberately keeps it out of `sha256sum.txt`: nothing in it is -measured at boot, so folding it into `os_image_hash` would make the OS identity -depend on a developer artifact. A backend that does not export one omits the -key, and the release is unchanged. - -`kernel-builder/` builds the container image that carries this tree plus a -matching toolchain; see [`kernel-builder/README.md`](kernel-builder/README.md). +The optional `artifacts.kernel_devel` manifest entry is published as +`--kernel-devel.tar.gz` beside the image archives. It is not in +`sha256sum.txt`, so it does not affect `os_image_hash`. `kernel-builder/` +builds the matching module-builder container image. ## Kernel setup-header normalization diff --git a/os/image/assemble.sh b/os/image/assemble.sh index 956b7c8e9..554e54b0c 100755 --- a/os/image/assemble.sh +++ b/os/image/assemble.sh @@ -672,9 +672,5 @@ fi # Publish the optional developer artifact beside the images, without adding it # to sha256sum.txt or os_image_hash. See docs/building-guest-os.md. if [ -n "$KERNEL_DEVEL_ARCHIVE" ]; then - echo "Publishing the kernel build tree to ${KERNEL_DEVEL_TAR}" cp "$KERNEL_DEVEL_ARCHIVE" "$KERNEL_DEVEL_TAR" - KERNEL_DEVEL_SHA256=$(sha256sum "$KERNEL_DEVEL_TAR" | awk '{print $1}') - KERNEL_DEVEL_BYTES=$(stat -L -c %s "$KERNEL_DEVEL_TAR") - echo "kernel-devel: ${KERNEL_DEVEL_BYTES} bytes, sha256 ${KERNEL_DEVEL_SHA256}" fi diff --git a/os/image/kernel-builder/Dockerfile b/os/image/kernel-builder/Dockerfile index 22ddf562f..31b2d7f23 100644 --- a/os/image/kernel-builder/Dockerfile +++ b/os/image/kernel-builder/Dockerfile @@ -2,83 +2,44 @@ # # SPDX-License-Identifier: Apache-2.0 # -# Toolchain image for building an out-of-tree kernel module against one dstack -# guest OS image. It carries the exported kernel build tree of that image plus -# a compiler from the same pinned Debian snapshot the guest kernel was built -# with, so a module built here matches the guest kernel's vermagic and -# Module.symvers and loads in the CVM. -# -# Build context is the directory holding the kernel-devel tarball: -# -# docker build -f os/image/kernel-builder/Dockerfile \ -# --build-arg DEBIAN_SNAPSHOT= \ -# --build-arg KERNEL_RELEASE= \ -# --build-arg KERNEL_DEVEL_TARBALL=dstack--kernel-devel.tar.gz \ -# -t dstack-kernel-builder: +# Toolchain image for building out-of-tree modules against one dstack guest OS +# kernel: its exported build tree plus a compiler from the same pinned Debian +# snapshot the kernel was built with. Built by build.sh. ARG BASE_IMAGE=debian:trixie-slim FROM ${BASE_IMAGE} -# The snapshot is what pins the compiler, which is the part that has to match -# the guest kernel build; the base image tag only provides the file layout. ARG DEBIAN_SNAPSHOT ARG KERNEL_RELEASE -ARG KERNEL_DEVEL_TARBALL ARG IMAGE_VERSION=unknown -ARG OS_IMAGE_HASH=unknown ARG GIT_REVISION=unknown -# Trixie ships deb822 sources in /etc/apt/sources.list.d/, so rewriting -# sources.list alone leaves the live mirror active and the toolchain drifts on -# every rebuild. Wipe both layouts, as dstack/build/shared/pin-packages.sh does. +# Pin apt to the snapshot; trixie also ships deb822 sources that must go. RUN set -eux; \ - test -n "${DEBIAN_SNAPSHOT}"; \ suite=$(. /etc/os-release && echo "${VERSION_CODENAME}"); \ - rm -f /etc/apt/sources.list; \ - rm -f /etc/apt/sources.list.d/*.list /etc/apt/sources.list.d/*.sources; \ - printf 'deb [check-valid-until=no] http://snapshot.debian.org/archive/debian/%s %s main\n' \ - "${DEBIAN_SNAPSHOT}" "${suite}" > /etc/apt/sources.list; \ - echo 'Acquire::Check-Valid-Until "false";' > /etc/apt/apt.conf.d/10no-check-valid-until; \ + rm -f /etc/apt/sources.list /etc/apt/sources.list.d/*; \ + echo "deb [check-valid-until=no] http://snapshot.debian.org/archive/debian/${DEBIAN_SNAPSHOT:?} ${suite} main" \ + > /etc/apt/sources.list; \ apt-get update; \ apt-get install --yes --no-install-recommends \ - bc \ - bison \ - build-essential \ - flex \ - kmod \ - libelf-dev \ - libssl-dev \ - python3 \ - xz-utils; \ + bc bison build-essential flex kmod libelf-dev libssl-dev; \ rm -rf /var/lib/apt/lists/* -COPY ${KERNEL_DEVEL_TARBALL} /tmp/kernel-devel.tar.gz +ENV KERNEL_RELEASE=${KERNEL_RELEASE} +ENV KDIR=/usr/src/linux-headers-${KERNEL_RELEASE} -# The archive's own metadata is the authority on which kernel it belongs to, so -# a mismatch with the requested release fails the build instead of producing an -# image whose KDIR silently points at a different kernel. +ADD kernel-devel.tar.gz /tmp/ RUN set -eux; \ - test -n "${KERNEL_RELEASE}"; \ - mkdir -p /usr/src; \ - tar -C /usr/src --strip-components=1 -xzf /tmp/kernel-devel.tar.gz; \ - rm /tmp/kernel-devel.tar.gz; \ - declared=$(python3 -c 'import json;print(json.load(open("/usr/src/kernel-devel.json"))["kernel_release"])'); \ - test "${declared}" = "${KERNEL_RELEASE}"; \ - test -f "/usr/src/linux-headers-${KERNEL_RELEASE}/Module.symvers"; \ + mv /tmp/*-kernel-devel/linux-headers-* /usr/src/; \ + rm -rf /tmp/*-kernel-devel; \ + test "$(cat "$KDIR/include/config/kernel.release")" = "${KERNEL_RELEASE:?}"; \ mkdir -p "/lib/modules/${KERNEL_RELEASE}"; \ - ln -s "/usr/src/linux-headers-${KERNEL_RELEASE}" "/lib/modules/${KERNEL_RELEASE}/build" - -# Pass KDIR explicitly; uname -r inside a container still names the host kernel. -# The /lib/modules symlink is for consumers using the target KERNEL_RELEASE. -ENV KDIR="/usr/src/linux-headers-${KERNEL_RELEASE}" -ENV KERNEL_RELEASE="${KERNEL_RELEASE}" + ln -s "$KDIR" "/lib/modules/${KERNEL_RELEASE}/build" LABEL org.opencontainers.image.title="dstack kernel module builder" \ - org.opencontainers.image.description="Kernel build tree and toolchain for dstack guest OS ${IMAGE_VERSION}" \ org.opencontainers.image.source="https://github.com/Dstack-TEE/dstack" \ org.opencontainers.image.revision="${GIT_REVISION}" \ org.opencontainers.image.version="${IMAGE_VERSION}" \ - network.phala.dstack.kernel-release="${KERNEL_RELEASE}" \ - network.phala.dstack.os-image-hash="${OS_IMAGE_HASH}" + network.phala.dstack.kernel-release="${KERNEL_RELEASE}" WORKDIR /work diff --git a/os/image/kernel-builder/README.md b/os/image/kernel-builder/README.md deleted file mode 100644 index a416489bc..000000000 --- a/os/image/kernel-builder/README.md +++ /dev/null @@ -1,53 +0,0 @@ -# Kernel module builder image - -Toolchain image for building an out-of-tree kernel module against one dstack -guest OS image. It carries that image's exported kernel build tree -(`dstack--kernel-devel.tar.gz`) and a compiler installed from the same -pinned Debian snapshot the guest kernel was compiled with, so a module built -here matches the guest kernel's `vermagic` and `Module.symvers`. - -Published by `.github/workflows/mkosi-build.yml` on a release tag as -`ghcr.io/dstack-tee/dstack-kernel-builder:`, additionally tagged with -the kernel release (`6.18.x-dstack`). Application usage is documented in -[`../../../docs/building-guest-os.md`](../../../docs/building-guest-os.md#kernel-headers-for-out-of-tree-modules). - -## Build it locally - -The build script uses the repository's pinned toolchain snapshot and an isolated -build context containing only the kernel-devel archive. - -```bash -source os/mkosi/versions.env -dist=os/mkosi/repro-build/build/out/prod -release=$(tar -xOf "$dist/dstack-$DSTACK_VERSION-kernel-devel.tar.gz" \ - "dstack-$DSTACK_VERSION-kernel-devel/kernel-devel.json" | - python3 -c 'import json,sys; print(json.load(sys.stdin)["kernel_release"])') - -./os/image/kernel-builder/build.sh \ - "$dist/dstack-$DSTACK_VERSION-kernel-devel.tar.gz" \ - "dstack-kernel-builder:$DSTACK_VERSION" "$release" -``` - -Then smoke-test module compilation and the target kernel release: - -```bash -./os/tests/test-kernel-builder-image.sh "dstack-kernel-builder:$DSTACK_VERSION" "$release" -``` - -That script builds `os/tests/kmod-hello/` inside the image and fails unless the -resulting module's `vermagic` names the kernel the image claims to serve. CI -runs it before the image is pushed. This does not prove loadability: symbol -versions and other kernel compatibility requirements need a real guest load test. - -Set `IMAGE_VERSION`, `OS_IMAGE_HASH`, and `GIT_REVISION` to override image labels. -`DOCKER` selects the Docker command for both scripts. - -## What the image pins, and what it does not - -The Debian snapshot pins the compiler, which is the part that has to agree with -the guest kernel build. `BASE_IMAGE` only supplies the file layout that -snapshot is installed into; the tag is mutable, so the image is not -bit-for-bit reproducible the way the guest OS artifacts are. The kernel build -tree it carries is: `os/mkosi/build.sh repro-check` compares -`dstack--kernel-devel.tar.gz` byte for byte across two independent -builds. diff --git a/os/image/kernel-builder/build.sh b/os/image/kernel-builder/build.sh index 3cba24500..b179f976d 100755 --- a/os/image/kernel-builder/build.sh +++ b/os/image/kernel-builder/build.sh @@ -2,33 +2,26 @@ # SPDX-FileCopyrightText: © 2026 Phala Network # # SPDX-License-Identifier: Apache-2.0 +# Usage: build.sh KERNEL_DEVEL_ARCHIVE IMAGE_REF set -euo pipefail -if [[ $# != 3 ]]; then - echo "Usage: $0 ARCHIVE IMAGE_REF KERNEL_RELEASE" >&2 - exit 2 -fi -archive=$(realpath "$1") -image=$2 -release=$3 +archive=$(realpath "${1:?kernel-devel archive required}") +image=${2:?image ref required} SELF=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd) # shellcheck source=../../mkosi/versions.env source "$SELF/../../mkosi/versions.env" -DOCKER=${DOCKER:-docker} -# Do not send the neighboring OS image tarballs to the Docker daemon. +# Keep the build context to the one archive. context=$(mktemp -d) trap 'rm -rf "$context"' EXIT cp "$archive" "$context/kernel-devel.tar.gz" -$DOCKER build \ +${DOCKER:-docker} build \ --file "$SELF/Dockerfile" \ --build-arg "BASE_IMAGE=debian:$DEBIAN_RELEASE-slim" \ --build-arg "DEBIAN_SNAPSHOT=$DEBIAN_SNAPSHOT" \ - --build-arg "KERNEL_RELEASE=$release" \ - --build-arg KERNEL_DEVEL_TARBALL=kernel-devel.tar.gz \ + --build-arg "KERNEL_RELEASE=$KERNEL_VERSION-dstack" \ --build-arg "IMAGE_VERSION=${IMAGE_VERSION:-$DSTACK_VERSION}" \ - --build-arg "OS_IMAGE_HASH=${OS_IMAGE_HASH:-unknown}" \ --build-arg "GIT_REVISION=${GIT_REVISION:-unknown}" \ --tag "$image" \ "$context" diff --git a/os/mkosi/build.sh b/os/mkosi/build.sh index e3c24cd84..8d13cfed4 100755 --- a/os/mkosi/build.sh +++ b/os/mkosi/build.sh @@ -208,10 +208,6 @@ build_one "$BUILD_DIR/b" prod "$jobs_b" cmp "$BUILD_DIR/a/dstack-$DSTACK_VERSION.tar.gz" "$BUILD_DIR/b/dstack-$DSTACK_VERSION.tar.gz" cmp "$BUILD_DIR/a/dstack-$DSTACK_VERSION-uki.tar.gz" \ "$BUILD_DIR/b/dstack-$DSTACK_VERSION-uki.tar.gz" -# The kernel build tree publishes parts of the build that nothing shipped -# before: the host programs Kbuild compiled (fixdep, modpost, objtool) and its -# generated intermediates. Comparing it is what proves none of them records the -# build directory or the job count. cmp "$BUILD_DIR/a/dstack-$DSTACK_VERSION-kernel-devel.tar.gz" \ "$BUILD_DIR/b/dstack-$DSTACK_VERSION-kernel-devel.tar.gz" echo 'reproducibility check passed' diff --git a/os/mkosi/components/kernel/kernel-build.sh b/os/mkosi/components/kernel/kernel-build.sh index 86b2a9255..48c986e97 100755 --- a/os/mkosi/components/kernel/kernel-build.sh +++ b/os/mkosi/components/kernel/kernel-build.sh @@ -90,13 +90,16 @@ install -Dm0644 "$BUILD_DIR/kernel-build/arch/x86/boot/bzImage" \ "$STAGING/usr/lib/modules/$KERNEL_VERSION-dstack/vmlinuz" install -Dm0644 "$BUILD_DIR/kernel-build/.config" \ "$STAGING/usr/lib/modules/$KERNEL_VERSION-dstack/config" -# Applications that need their own kernel module get the build tree of this -# exact kernel as a release artifact. Staged here and moved out of the rootfs -# by mkosi.finalize before the image is measured: it must not ship in the CVM. -"$ROOT/os/common/scripts/export-kernel-devel.sh" \ - --src "$src" --build "$BUILD_DIR/kernel-build" \ - --out "$DEVEL_STAGING/usr/lib/dstack/kernel-devel" \ - --backend mkosi --image-version "$DSTACK_VERSION" \ - --flavor "${DSTACK_BUILD_FLAVOR:-prod}" +# Export the external-module build tree as a release artifact, using the same +# upstream helper `make bindeb-pkg` uses for linux-headers. It is staged into +# the rootfs install tree and moved out by mkosi.finalize before measurement. +devel="$DEVEL_STAGING/usr/lib/dstack/kernel-devel" +headers="$devel/linux-headers-$(cat "$BUILD_DIR/kernel-build/include/config/kernel.release")" +make -C "$src" O="$BUILD_DIR/kernel-build" run-command \ + KBUILD_RUN_COMMAND="$src/scripts/package/install-extmod-build $headers" +install -m0644 "$BUILD_DIR/kernel-build/.config" "$headers/.config" +# A -fverbose-asm intermediate that records the absolute build path, which no +# prefix map rewrites. Only its generated .h is needed. +rm -f "$headers/scripts/mod/devicetable-offsets.s" find "$STAGING" "$DEVEL_STAGING" -print0 | \ xargs -0r touch --no-dereference --date="@$SOURCE_DATE_EPOCH" diff --git a/os/mkosi/components/kernel/kernel.sh b/os/mkosi/components/kernel/kernel.sh index 0c3e8a3b8..9217b66b0 100644 --- a/os/mkosi/components/kernel/kernel.sh +++ b/os/mkosi/components/kernel/kernel.sh @@ -3,17 +3,13 @@ COMPONENT_NAME=kernel COMPONENT_CACHE_PATHS=("linux-$KERNEL_VERSION" kernel-build component-stages/kernel component-stages/kernel-devel) -# The exported build tree is staged into the rootfs install tree, not because it -# ships in the image but because that is the only writable tree a component -# reaches; mkosi.finalize moves it out before the rootfs is measured, the same -# way it does for the firmware and the image tools. +# Staged via the rootfs tree; mkosi.finalize moves it out before measurement. COMPONENT_ROOTFS_TREES=(component-stages/kernel-devel) COMPONENT_KERNEL_TREES=(component-stages/kernel) component_cache_key() { key_value "$KERNEL_VERSION" "$KERNEL_SHA256" key_file "$COMPONENT_PATH/kernel-build.sh" "$COMPONENT_PATH/kernel.config" \ - "$ROOT/os/common/scripts/export-kernel-devel.sh" \ "$ROOT/os/yocto/layers/meta-dstack/recipes-kernel/linux/files/0001-x86-tdx-select-dma-direct-remap.patch" \ "$ROOT/os/yocto/layers/meta-dstack/recipes-kernel/linux/files/0002-acpi-sandbox-block-aml-systemmemory-ram-access.patch" \ "$ROOT/os/yocto/layers/meta-dstack/recipes-kernel/linux/files/0003-dma-direct-return-struct-page-from-alloc-from-pool.patch" \ diff --git a/os/mkosi/mkosi.finalize b/os/mkosi/mkosi.finalize index 9c11f81e6..38032a15c 100755 --- a/os/mkosi/mkosi.finalize +++ b/os/mkosi/mkosi.finalize @@ -20,10 +20,7 @@ cp -a "$TREE/usr/lib/modules" "$KSTAGE/usr/lib/" install -m0644 "$TREE/usr/lib/dstack/firmware/ovmf.fd" "$KSTAGE/ovmf.fd" install -m0644 "$TREE/usr/lib/dstack/firmware/ovmf-sev.fd" "$KSTAGE/ovmf-sev.fd" install -m0755 "$TREE/usr/lib/dstack/image-tools/"* "$OUTPUTDIR/.image-tools/" -# The exported kernel build tree is a release artifact, not guest payload: it -# is ~100 MB of build inputs no CVM uses, and leaving it in the tree would put -# it in the measured rootfs. Moved before check-parity.py runs so the parity -# check sees exactly the tree that ships. +# The kernel build tree is a release artifact, never part of the measured rootfs. mv "$TREE/usr/lib/dstack/kernel-devel" "$KSTAGE/kernel-devel" rm -rf "$TREE/usr/lib/dstack/firmware" "$TREE/usr/lib/dstack/image-tools" diff --git a/os/mkosi/scripts/make-release-artifacts.sh b/os/mkosi/scripts/make-release-artifacts.sh index 69c4696a7..c5971da38 100755 --- a/os/mkosi/scripts/make-release-artifacts.sh +++ b/os/mkosi/scripts/make-release-artifacts.sh @@ -125,10 +125,7 @@ touch -d "@$SOURCE_DATE_EPOCH" "$OUT/files/"* is_dev=false; name=dstack if [[ $FLAVOR == dev ]]; then is_dev=true; name=dstack-dev; fi -# Archive the tree removed by mkosi.finalize separately from measured payload. -# Use fixed tar metadata for repro-check; cached iteration builds can skip it. -kernel_devel="$KERNEL_TREE/kernel-devel" -[[ -d $kernel_devel ]] || { echo 'kernel build tree was not exported' >&2; exit 1; } +# The kernel build tree is archived beside, not inside, the measured payload. kernel_devel_artifact= if [[ ${DSTACK_TAR_RELEASE:-1} == 1 ]]; then tar -C "$KERNEL_TREE" --sort=name --mtime="@$SOURCE_DATE_EPOCH" \ diff --git a/os/mkosi/tests/acceptance.sh b/os/mkosi/tests/acceptance.sh index ec01d101b..493315a44 100755 --- a/os/mkosi/tests/acceptance.sh +++ b/os/mkosi/tests/acceptance.sh @@ -182,7 +182,6 @@ grep -q 'AmdSev/AmdSevX64.dsc' "$D/components/ovmf/ovmf-build.sh" grep -q '0006-OvmfPkg-AmdSev-drop-embedded-grub.patch' "$D/components/ovmf/ovmf.sh" grep -q '0005-UefiCpuPkg' "$D/components/ovmf/ovmf.sh" grep -q '0007-OvmfPkg-QemuKernelLoaderFsDxe' "$D/components/ovmf/ovmf.sh" -"$D/../tests/test-kernel-devel.sh" grep -q 'objcopy --strip-debug' "$D/mkosi.build" grep -q 'depmod -b.*KERNEL_VERSION-dstack' "$D/mkosi.build" grep -q '^CleanPackageMetadata=yes$' "$D/mkosi.conf" @@ -223,9 +222,7 @@ grep -q 'rootfs.img.parted.verity' "$D/../image/assemble.sh" bash -n "$D"/*.sh "$D"/mkosi.build "$D"/mkosi.clean "$D"/mkosi.finalize \ "$D"/mkosi.postinst "$D"/mkosi.postoutput "$D"/scripts/*.sh \ "$D"/components/*/*.sh "$D"/tests/*.sh "$D"/mkosi.skeleton/usr/bin/*.sh \ - "$D"/../image/kernel-cmdline.sh "$D"/../common/scripts/export-kernel-devel.sh \ - "$D"/../tests/test-kernel-builder-image.sh "$D"/../tests/test-kernel-devel.sh \ - "$D"/../image/kernel-builder/build.sh + "$D"/../image/kernel-cmdline.sh # The Python helpers were only ever exercised by a real build. Keep the # bytecode out of the source tree, which is mounted into the build sandbox. pycache=$(mktemp -d) diff --git a/os/mkosi/tests/check-output.sh b/os/mkosi/tests/check-output.sh index a05c5389e..55cbe2178 100755 --- a/os/mkosi/tests/check-output.sh +++ b/os/mkosi/tests/check-output.sh @@ -18,24 +18,4 @@ assert d["builder"] == "mkosi" assert d["bios-sev"] == "ovmf-sev.fd" assert d["is_dev"] == (sys.argv[2] == "dev") PY -# The kernel build tree ships beside the image, never inside it. Nothing in it -# is measured at boot, so its presence in the identity preimage would make -# os_image_hash depend on a developer artifact. -if grep -q 'kernel-devel' "$out/sha256sum.txt"; then - echo 'kernel build tree must not be part of the image identity' >&2 - exit 1 -fi -if [[ ${DSTACK_TAR_RELEASE:-1} == 1 ]]; then - [[ -s $out-kernel-devel.tar.gz ]] || { - echo "missing kernel build tree archive: $out-kernel-devel.tar.gz" >&2 - exit 1 - } - # Consume the full listing: grep -q would close the pipe on the first match - # and make tar fail with SIGPIPE under pipefail. - tar -tzf "$out-kernel-devel.tar.gz" | \ - grep -Fx "$(basename "$out")-kernel-devel/kernel-devel.json" >/dev/null || { - echo 'kernel build tree archive does not declare its kernel' >&2 - exit 1 - } -fi echo "Yocto-compatible release format accepted: $out" diff --git a/os/spec/artifact-manifest.schema.json b/os/spec/artifact-manifest.schema.json index d53cafeb0..1a75eadc6 100644 --- a/os/spec/artifact-manifest.schema.json +++ b/os/spec/artifact-manifest.schema.json @@ -71,7 +71,6 @@ } }, "artifacts": { - "description": "kernel_devel is optional: a backend that does not export the kernel build tree omits the key, so schema version 1 stays compatible.", "type": "object", "additionalProperties": false, "required": [ diff --git a/os/tests/kmod-hello/Makefile b/os/tests/kmod-hello/Makefile deleted file mode 100644 index fa1ff8121..000000000 --- a/os/tests/kmod-hello/Makefile +++ /dev/null @@ -1,17 +0,0 @@ -# SPDX-FileCopyrightText: © 2026 Phala Network -# -# SPDX-License-Identifier: GPL-2.0-only -# Out-of-tree module build, exactly as an application would do it: point KDIR -# at the exported dstack kernel build tree and let Kbuild drive the build. -obj-m += dstack-hello.o -dstack-hello-objs := hello.o - -KDIR ?= /lib/modules/$(shell uname -r)/build - -all: - $(MAKE) -C $(KDIR) M=$(CURDIR) modules - -clean: - $(MAKE) -C $(KDIR) M=$(CURDIR) clean - -.PHONY: all clean diff --git a/os/tests/kmod-hello/hello.c b/os/tests/kmod-hello/hello.c deleted file mode 100644 index 035856a7d..000000000 --- a/os/tests/kmod-hello/hello.c +++ /dev/null @@ -1,29 +0,0 @@ -// SPDX-FileCopyrightText: © 2026 Phala Network -// -// SPDX-License-Identifier: GPL-2.0-only -/* - * Smallest out-of-tree module that still exercises everything an application - * module needs from the exported kernel build tree: the UAPI and internal - * headers, Module.symvers (module_init/module_exit are resolved through it) - * and the vermagic the running guest kernel will check on insmod. - */ - -#include -#include - -static int __init dstack_hello_init(void) -{ - pr_info("dstack-hello: loaded\n"); - return 0; -} - -static void __exit dstack_hello_exit(void) -{ - pr_info("dstack-hello: unloaded\n"); -} - -module_init(dstack_hello_init); -module_exit(dstack_hello_exit); - -MODULE_LICENSE("GPL"); -MODULE_DESCRIPTION("dstack out-of-tree module build smoke test"); diff --git a/os/tests/test-kernel-builder-image.sh b/os/tests/test-kernel-builder-image.sh deleted file mode 100755 index cea15030f..000000000 --- a/os/tests/test-kernel-builder-image.sh +++ /dev/null @@ -1,41 +0,0 @@ -#!/bin/bash -# SPDX-FileCopyrightText: © 2026 Phala Network -# -# SPDX-License-Identifier: Apache-2.0 -# -# Check that the builder compiles a module for the expected kernel release. -# This is not a load test: vermagic's release field does not validate symbol -# versions, compiler compatibility, or the rest of the kernel configuration. -set -euo pipefail - -IMAGE=${1:?container image required} -EXPECTED_RELEASE=${2:-} -DOCKER=${DOCKER:-docker} -SELF=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd) - -work=$(mktemp -d) -trap 'rm -rf "$work"' EXIT -cp "$SELF/kmod-hello/hello.c" "$SELF/kmod-hello/Makefile" "$work/" - -# --user keeps the build outputs owned by the caller, so the fixture copy stays -# removable without root. The inner script is single-quoted on purpose: KDIR -# and KERNEL_RELEASE are the image's own environment, not this shell's. -# shellcheck disable=SC2016 -$DOCKER run --rm --user "$(id -u):$(id -g)" \ - -v "$work:/work" -w /work "$IMAGE" \ - bash -eu -o pipefail -c ' - make KDIR="$KDIR" - vermagic=$(modinfo -F vermagic dstack-hello.ko) - echo "vermagic: $vermagic" - if [ "${vermagic%% *}" != "$KERNEL_RELEASE" ]; then - echo "module vermagic ${vermagic%% *} does not match the image kernel $KERNEL_RELEASE" >&2 - exit 1 - fi - printf "%s\n" "$KERNEL_RELEASE" > kernel-release.txt - ' -release=$(cat "$work/kernel-release.txt") -if [ -n "$EXPECTED_RELEASE" ] && [ "$release" != "$EXPECTED_RELEASE" ]; then - echo "builder image serves $release but the release expects $EXPECTED_RELEASE" >&2 - exit 1 -fi -echo "kernel module build accepted: $IMAGE serves $release" diff --git a/os/tests/test-kernel-devel.sh b/os/tests/test-kernel-devel.sh deleted file mode 100755 index 994b30f5c..000000000 --- a/os/tests/test-kernel-devel.sh +++ /dev/null @@ -1,158 +0,0 @@ -#!/bin/bash -# SPDX-FileCopyrightText: © 2026 Phala Network -# -# SPDX-License-Identifier: Apache-2.0 -set -euo pipefail -ROOT=$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd) -work=$(mktemp -d) -trap 'rm -rf "$work"' EXIT - -python3 - "$ROOT" "$work" <<'PY' -import hashlib -import io -import json -import os -import subprocess -import sys -import tarfile -from pathlib import Path - -root, work = map(Path, sys.argv[1:]) - - -def check(script, *args, success=True, message=None, env=None): - result = subprocess.run( - [str(root / script), *map(str, args)], - env=env, - capture_output=True, - text=True, - ) - output = result.stdout + result.stderr - assert (result.returncode == 0) == success, output - assert message is None or message in output, output - - -# Validate the actual assembler's optional artifact contract without image tools. -payload = work / "payload" -payload.write_bytes(b"fixture\n") -manifest = { - "schema_version": 1, - "backend": "mkosi", - "image": {"name": "dstack", "version": "test", "flavor": "prod", "is_dev": False}, - "source": {"git_revision": "fixture"}, - "boot": {"ovmf_variant": "tdx"}, - "verity": {"root_hash": "0" * 64, "data_size": 4096}, - "artifacts": dict.fromkeys( - ("initramfs", "kernel", "firmware", "rootfs_verity"), "payload" - ), -} -manifest["artifacts"].update(firmware_sev=None, uki=None) -manifest_path = work / "artifact-manifest.json" -manifest_path.write_text(json.dumps(manifest)) -check("os/image/assemble.sh", "--manifest", manifest_path, "--validate-only") -for value, success in ( - (None, True), - ("payload", True), - (str(payload), False), - ("../payload", False), - ("missing", False), -): - manifest["artifacts"]["kernel_devel"] = value - manifest_path.write_text(json.dumps(manifest)) - check( - "os/image/assemble.sh", - "--manifest", - manifest_path, - "--validate-only", - success=success, - ) - -# Only partition inspection is stubbed; checksums, tar, grep and the output -# checker run for real. The fixture deliberately is not a bootable disk image. -bin_dir = work / "bin" -bin_dir.mkdir() -sgdisk = bin_dir / "sgdisk" -sgdisk.write_text("#!/bin/sh\nprintf 'dstack-rootfs\\n'\n") -sgdisk.chmod(0o755) -env = dict(os.environ, PATH=f"{bin_dir}:{os.environ['PATH']}", DSTACK_TAR_RELEASE="1") -out = work / "dstack-test" -out.mkdir() -for name in ( - "bzImage", - "initramfs.cpio.gz", - "ovmf.fd", - "ovmf-sev.fd", - "rootfs.img.parted.verity", - "measurement.tdx.cbor", - "measurement.snp.cbor", - "digest.txt", -): - (out / name).write_bytes(payload.read_bytes()) -metadata = dict.fromkeys( - ( - "bios", - "kernel", - "cmdline", - "initrd", - "rootfs", - "version", - "git_revision", - "ovmf_variant", - ), - "fixture", -) -metadata.update({"bios-sev": "ovmf-sev.fd", "builder": "mkosi", "is_dev": False}) -(out / "metadata.json").write_text(json.dumps(metadata)) -hashes = "".join( - f"{hashlib.sha256(path.read_bytes()).hexdigest()} {path.name}\n" - for path in sorted(out.iterdir()) -) -(out / "sha256sum.txt").write_text(hashes) -archive = work / "dstack-test-kernel-devel.tar.gz" - - -def write_archive(with_metadata): - prefix = "dstack-test-kernel-devel/" - with tarfile.open(archive, "w:gz") as tar: - if with_metadata: - data = b'{"kernel_release": "test"}\n' - member = tarfile.TarInfo(prefix + "kernel-devel.json") - member.size = len(data) - tar.addfile(member, io.BytesIO(data)) - # Exceed pipe capacity after the match to expose grep -q's SIGPIPE. - for index in range(3000): - tar.addfile( - tarfile.TarInfo( - prefix + f"linux-headers-test/include/header-{index:04}.h" - ) - ) - - -checker = "os/mkosi/tests/check-output.sh" -write_archive(True) -check(checker, out, env=env) -valid_archive = archive.read_bytes() -write_archive(False) -check(checker, out, env=env, success=False, message="does not declare its kernel") -archive.write_bytes(valid_archive[: len(valid_archive) // 2]) -check(checker, out, env=env, success=False) -archive.unlink() -check(checker, out, env=env, success=False, message="missing kernel build tree archive") -check(checker, out, env=dict(env, DSTACK_TAR_RELEASE="0")) -archive.write_bytes(valid_archive) - -# A valid checksum must not make a developer artifact part of image identity. -name = "kernel-devel.tar.gz" -(out / name).write_bytes(valid_archive) -(out / "sha256sum.txt").write_text( - hashes + f"{hashlib.sha256(valid_archive).hexdigest()} {name}\n" -) -check( - checker, - out, - env=env, - success=False, - message="must not be part of the image identity", -) -print("kernel-devel artifact tests passed") -PY From b94c6bf728fc457f0451926cf7c54838001590df Mon Sep 17 00:00:00 2001 From: Kevin Wang Date: Thu, 17 Sep 2026 07:17:03 -0700 Subject: [PATCH 33/93] fix(scripts): remove the installer's temporary checkout on exit Signed-off-by: Kevin Wang --- dstack/scripts/install.sh | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/dstack/scripts/install.sh b/dstack/scripts/install.sh index 736c18745..1540f2346 100755 --- a/dstack/scripts/install.sh +++ b/dstack/scripts/install.sh @@ -163,14 +163,16 @@ script_checkout() { return 1 } +# Sets `checkout`. Runs in the calling shell rather than a command +# substitution so the temporary checkout it may create stays visible to the +# exit trap that removes it. resolve_source() { if is_checkout "."; then - abs_dir "." + checkout=$(abs_dir ".") return 0 fi if checkout=$(script_checkout); then - echo "$checkout" return 0 fi @@ -211,7 +213,7 @@ resolve_source() { ) >&2 fi - abs_dir "$src" + checkout=$(abs_dir "$src") } validate_prefix() { @@ -246,7 +248,7 @@ fi need_cmd cargo need_cmd install -checkout=$(resolve_source) +resolve_source core_checkout=$(core_dir "$checkout") bin_dir="$prefix/bin" From 73f61b62d68f8b6f057b9d8e8053c57653d4041b Mon Sep 17 00:00:00 2001 From: Kevin Wang Date: Thu, 17 Sep 2026 07:24:18 -0700 Subject: [PATCH 34/93] fix(kms): sync auth backend bun lockfiles with package.json Signed-off-by: Kevin Wang --- dstack/kms/auth-eth-bun/bun.lock | 202 +++++++++++-------------------- dstack/kms/auth-mock/bun.lock | 29 +++-- dstack/kms/auth-simple/bun.lock | 4 +- 3 files changed, 88 insertions(+), 147 deletions(-) diff --git a/dstack/kms/auth-eth-bun/bun.lock b/dstack/kms/auth-eth-bun/bun.lock index 0af400847..a471f6bc8 100644 --- a/dstack/kms/auth-eth-bun/bun.lock +++ b/dstack/kms/auth-eth-bun/bun.lock @@ -5,7 +5,7 @@ "name": "auth-eth-bun", "dependencies": { "@hono/zod-validator": "0.2.2", - "hono": "4.12.34", + "hono": "4.13.5", "viem": "2.31.7", "zod": "3.25.76", }, @@ -16,64 +16,18 @@ "openapi-types": "12.1.3", "oxlint": "0.9.10", "typescript": "5.8.3", - "vitest": "3.2.6", + "vitest": "4.1.11", }, }, }, "packages": { "@adraffy/ens-normalize": ["@adraffy/ens-normalize@1.11.0", "", {}, "sha512-/3DDPKHqqIqxUULp8yP4zODUY1i+2xvVWsv8A79xGWdCAG+8sb0hRh0Rk2QyOJUnnbyPUAZYcpBuRe3nS2OIUg=="], - "@esbuild/aix-ppc64": ["@esbuild/aix-ppc64@0.21.5", "", { "os": "aix", "cpu": "ppc64" }, "sha512-1SDgH6ZSPTlggy1yI6+Dbkiz8xzpHJEVAlF/AM1tHPLsf5STom9rwtjE4hKAF20FfXXNTFqEYXyJNWh1GiZedQ=="], - - "@esbuild/android-arm": ["@esbuild/android-arm@0.21.5", "", { "os": "android", "cpu": "arm" }, "sha512-vCPvzSjpPHEi1siZdlvAlsPxXl7WbOVUBBAowWug4rJHb68Ox8KualB+1ocNvT5fjv6wpkX6o/iEpbDrf68zcg=="], - - "@esbuild/android-arm64": ["@esbuild/android-arm64@0.21.5", "", { "os": "android", "cpu": "arm64" }, "sha512-c0uX9VAUBQ7dTDCjq+wdyGLowMdtR/GoC2U5IYk/7D1H1JYC0qseD7+11iMP2mRLN9RcCMRcjC4YMclCzGwS/A=="], - - "@esbuild/android-x64": ["@esbuild/android-x64@0.21.5", "", { "os": "android", "cpu": "x64" }, "sha512-D7aPRUUNHRBwHxzxRvp856rjUHRFW1SdQATKXH2hqA0kAZb1hKmi02OpYRacl0TxIGz/ZmXWlbZgjwWYaCakTA=="], - - "@esbuild/darwin-arm64": ["@esbuild/darwin-arm64@0.21.5", "", { "os": "darwin", "cpu": "arm64" }, "sha512-DwqXqZyuk5AiWWf3UfLiRDJ5EDd49zg6O9wclZ7kUMv2WRFr4HKjXp/5t8JZ11QbQfUS6/cRCKGwYhtNAY88kQ=="], - - "@esbuild/darwin-x64": ["@esbuild/darwin-x64@0.21.5", "", { "os": "darwin", "cpu": "x64" }, "sha512-se/JjF8NlmKVG4kNIuyWMV/22ZaerB+qaSi5MdrXtd6R08kvs2qCN4C09miupktDitvh8jRFflwGFBQcxZRjbw=="], - - "@esbuild/freebsd-arm64": ["@esbuild/freebsd-arm64@0.21.5", "", { "os": "freebsd", "cpu": "arm64" }, "sha512-5JcRxxRDUJLX8JXp/wcBCy3pENnCgBR9bN6JsY4OmhfUtIHe3ZW0mawA7+RDAcMLrMIZaf03NlQiX9DGyB8h4g=="], - - "@esbuild/freebsd-x64": ["@esbuild/freebsd-x64@0.21.5", "", { "os": "freebsd", "cpu": "x64" }, "sha512-J95kNBj1zkbMXtHVH29bBriQygMXqoVQOQYA+ISs0/2l3T9/kj42ow2mpqerRBxDJnmkUDCaQT/dfNXWX/ZZCQ=="], - - "@esbuild/linux-arm": ["@esbuild/linux-arm@0.21.5", "", { "os": "linux", "cpu": "arm" }, "sha512-bPb5AHZtbeNGjCKVZ9UGqGwo8EUu4cLq68E95A53KlxAPRmUyYv2D6F0uUI65XisGOL1hBP5mTronbgo+0bFcA=="], - - "@esbuild/linux-arm64": ["@esbuild/linux-arm64@0.21.5", "", { "os": "linux", "cpu": "arm64" }, "sha512-ibKvmyYzKsBeX8d8I7MH/TMfWDXBF3db4qM6sy+7re0YXya+K1cem3on9XgdT2EQGMu4hQyZhan7TeQ8XkGp4Q=="], - - "@esbuild/linux-ia32": ["@esbuild/linux-ia32@0.21.5", "", { "os": "linux", "cpu": "ia32" }, "sha512-YvjXDqLRqPDl2dvRODYmmhz4rPeVKYvppfGYKSNGdyZkA01046pLWyRKKI3ax8fbJoK5QbxblURkwK/MWY18Tg=="], - - "@esbuild/linux-loong64": ["@esbuild/linux-loong64@0.21.5", "", { "os": "linux", "cpu": "none" }, "sha512-uHf1BmMG8qEvzdrzAqg2SIG/02+4/DHB6a9Kbya0XDvwDEKCoC8ZRWI5JJvNdUjtciBGFQ5PuBlpEOXQj+JQSg=="], - - "@esbuild/linux-mips64el": ["@esbuild/linux-mips64el@0.21.5", "", { "os": "linux", "cpu": "none" }, "sha512-IajOmO+KJK23bj52dFSNCMsz1QP1DqM6cwLUv3W1QwyxkyIWecfafnI555fvSGqEKwjMXVLokcV5ygHW5b3Jbg=="], - - "@esbuild/linux-ppc64": ["@esbuild/linux-ppc64@0.21.5", "", { "os": "linux", "cpu": "ppc64" }, "sha512-1hHV/Z4OEfMwpLO8rp7CvlhBDnjsC3CttJXIhBi+5Aj5r+MBvy4egg7wCbe//hSsT+RvDAG7s81tAvpL2XAE4w=="], - - "@esbuild/linux-riscv64": ["@esbuild/linux-riscv64@0.21.5", "", { "os": "linux", "cpu": "none" }, "sha512-2HdXDMd9GMgTGrPWnJzP2ALSokE/0O5HhTUvWIbD3YdjME8JwvSCnNGBnTThKGEB91OZhzrJ4qIIxk/SBmyDDA=="], - - "@esbuild/linux-s390x": ["@esbuild/linux-s390x@0.21.5", "", { "os": "linux", "cpu": "s390x" }, "sha512-zus5sxzqBJD3eXxwvjN1yQkRepANgxE9lgOW2qLnmr8ikMTphkjgXu1HR01K4FJg8h1kEEDAqDcZQtbrRnB41A=="], - - "@esbuild/linux-x64": ["@esbuild/linux-x64@0.21.5", "", { "os": "linux", "cpu": "x64" }, "sha512-1rYdTpyv03iycF1+BhzrzQJCdOuAOtaqHTWJZCWvijKD2N5Xu0TtVC8/+1faWqcP9iBCWOmjmhoH94dH82BxPQ=="], - - "@esbuild/netbsd-x64": ["@esbuild/netbsd-x64@0.21.5", "", { "os": "none", "cpu": "x64" }, "sha512-Woi2MXzXjMULccIwMnLciyZH4nCIMpWQAs049KEeMvOcNADVxo0UBIQPfSmxB3CWKedngg7sWZdLvLczpe0tLg=="], - - "@esbuild/openbsd-x64": ["@esbuild/openbsd-x64@0.21.5", "", { "os": "openbsd", "cpu": "x64" }, "sha512-HLNNw99xsvx12lFBUwoT8EVCsSvRNDVxNpjZ7bPn947b8gJPzeHWyNVhFsaerc0n3TsbOINvRP2byTZ5LKezow=="], - - "@esbuild/sunos-x64": ["@esbuild/sunos-x64@0.21.5", "", { "os": "sunos", "cpu": "x64" }, "sha512-6+gjmFpfy0BHU5Tpptkuh8+uw3mnrvgs+dSPQXQOv3ekbordwnzTVEb4qnIvQcYXq6gzkyTnoZ9dZG+D4garKg=="], - - "@esbuild/win32-arm64": ["@esbuild/win32-arm64@0.21.5", "", { "os": "win32", "cpu": "arm64" }, "sha512-Z0gOTd75VvXqyq7nsl93zwahcTROgqvuAcYDUr+vOv8uHhNSKROyU961kgtCD1e95IqPKSQKH7tBTslnS3tA8A=="], - - "@esbuild/win32-ia32": ["@esbuild/win32-ia32@0.21.5", "", { "os": "win32", "cpu": "ia32" }, "sha512-SWXFF1CL2RVNMaVs+BBClwtfZSvDgtL//G/smwAc5oVK/UPu2Gu9tIaRgFmYFFKrmg3SyAjSrElf0TiJ1v8fYA=="], - - "@esbuild/win32-x64": ["@esbuild/win32-x64@0.21.5", "", { "os": "win32", "cpu": "x64" }, "sha512-tQd/1efJuzPC6rCFwEvLtci/xNFcTZknmXs98FYDfGE4wP9ClFV98nyKrzJKVPMhdDnjzLhdUyMX4PsQAPjwIw=="], - "@hono/zod-validator": ["@hono/zod-validator@0.2.2", "", { "peerDependencies": { "hono": ">=3.9.0", "zod": "^3.19.1" } }, "sha512-dSDxaPV70Py8wuIU2QNpoVEIOSzSXZ/6/B/h4xA7eOMz7+AarKTSGV8E6QwrdcCbBLkpqfJ4Q2TmBO0eP1tCBQ=="], "@jest/schemas": ["@jest/schemas@29.6.3", "", { "dependencies": { "@sinclair/typebox": "^0.27.8" } }, "sha512-mo5j5X+jIZmJQveBKeS/clAueipV7KgiX1vMgCxam1RNYiqE1w62n0/tJJnHtjW8ZHcQco5gY85jA3mi0L+nSA=="], - "@jridgewell/sourcemap-codec": ["@jridgewell/sourcemap-codec@1.5.4", "", {}, "sha512-VT2+G1VQs/9oz078bLrYbecdZKs912zQlkelYpuf+SXF+QvZDYJlbx/LSx+meSAwdDFnF8FVXW92AVjjkVmgFw=="], + "@jridgewell/sourcemap-codec": ["@jridgewell/sourcemap-codec@1.6.0", "", {}, "sha512-T7jf+5zgsZHwNJ4lvQ7/aezbyk0nNX+zJVWpmHA7VYsEx7a7qr5Rg5IbtJFqkgze5Y2sruq1RUY8Q837Od7iFw=="], "@noble/ciphers": ["@noble/ciphers@1.3.0", "", {}, "sha512-2I0gnIVPtfnMw9ee9h1dJG7tp81+8Ob3OJb3Mv37rx5L40/b0i7djjCVvGOVqc9AEIQyvyu1i6ypKdFw8R8gQw=="], @@ -87,6 +41,8 @@ "@nodelib/fs.walk": ["@nodelib/fs.walk@1.2.8", "", { "dependencies": { "@nodelib/fs.scandir": "2.1.5", "fastq": "^1.6.0" } }, "sha512-oGB+UxlgWcgQkgwo8GcEGwemoTFt3FIO9ababBmaGwXIoBKZ+GTy0pP185beGg7Llih/NSHSV2XAs1lnznocSg=="], + "@oxc-project/types": ["@oxc-project/types@0.150.0", "", {}, "sha512-rDS5/31E9HfPl/CIzGrn0DOlvBbXFseQ5URJ9sYMfstbKLD/c6Gm9vmRzRGDdAXyOIL4zmO37lc9RIwYqVruZw=="], + "@oxlint/darwin-arm64": ["@oxlint/darwin-arm64@0.9.10", "", { "os": "darwin", "cpu": "arm64" }, "sha512-eOXKZYq5bnCSgDefgM5bzAg+4Fc//Rc4yjgKN8iDWUARweCaChiQXb6TXX8MfEfs6qayEMy6yVj0pqoFz0B1aw=="], "@oxlint/darwin-x64": ["@oxlint/darwin-x64@0.9.10", "", { "os": "darwin", "cpu": "x64" }, "sha512-UeYICDvLUaUOcY+0ugZUEmBMRLP+x8iTgL7TeY6BlpGw2ahbtUOTbyIIRWtr/0O++TnjZ+v8TzhJ9crw6Ij6dg=="], @@ -105,45 +61,37 @@ "@polka/url": ["@polka/url@1.0.0-next.29", "", {}, "sha512-wwQAWhWSuHaag8c4q/KN/vCoeOJYshAIvMQwD4GpSb3OiZklFfvAgmj0VCBBImRpuF/aFgIRzllXlVX93Jevww=="], - "@rollup/rollup-android-arm-eabi": ["@rollup/rollup-android-arm-eabi@4.45.1", "", { "os": "android", "cpu": "arm" }, "sha512-NEySIFvMY0ZQO+utJkgoMiCAjMrGvnbDLHvcmlA33UXJpYBCvlBEbMMtV837uCkS+plG2umfhn0T5mMAxGrlRA=="], - - "@rollup/rollup-android-arm64": ["@rollup/rollup-android-arm64@4.45.1", "", { "os": "android", "cpu": "arm64" }, "sha512-ujQ+sMXJkg4LRJaYreaVx7Z/VMgBBd89wGS4qMrdtfUFZ+TSY5Rs9asgjitLwzeIbhwdEhyj29zhst3L1lKsRQ=="], - - "@rollup/rollup-darwin-arm64": ["@rollup/rollup-darwin-arm64@4.45.1", "", { "os": "darwin", "cpu": "arm64" }, "sha512-FSncqHvqTm3lC6Y13xncsdOYfxGSLnP+73k815EfNmpewPs+EyM49haPS105Rh4aF5mJKywk9X0ogzLXZzN9lA=="], - - "@rollup/rollup-darwin-x64": ["@rollup/rollup-darwin-x64@4.45.1", "", { "os": "darwin", "cpu": "x64" }, "sha512-2/vVn/husP5XI7Fsf/RlhDaQJ7x9zjvC81anIVbr4b/f0xtSmXQTFcGIQ/B1cXIYM6h2nAhJkdMHTnD7OtQ9Og=="], - - "@rollup/rollup-freebsd-arm64": ["@rollup/rollup-freebsd-arm64@4.45.1", "", { "os": "freebsd", "cpu": "arm64" }, "sha512-4g1kaDxQItZsrkVTdYQ0bxu4ZIQ32cotoQbmsAnW1jAE4XCMbcBPDirX5fyUzdhVCKgPcrwWuucI8yrVRBw2+g=="], + "@rolldown/binding-android-arm-eabi": ["@rolldown/binding-android-arm-eabi@1.2.9", "", { "os": "android", "cpu": "arm" }, "sha512-tNISae1QEf/vkb3xkRcjV5SEdzPE97We5IVaa2Z8jSszQPZ8U60B/YCYpw4QI7VidYsBtKavczXf+DyDs9WGxw=="], - "@rollup/rollup-freebsd-x64": ["@rollup/rollup-freebsd-x64@4.45.1", "", { "os": "freebsd", "cpu": "x64" }, "sha512-L/6JsfiL74i3uK1Ti2ZFSNsp5NMiM4/kbbGEcOCps99aZx3g8SJMO1/9Y0n/qKlWZfn6sScf98lEOUe2mBvW9A=="], + "@rolldown/binding-android-arm64": ["@rolldown/binding-android-arm64@1.2.9", "", { "os": "android", "cpu": "arm64" }, "sha512-YC8YsI30o606GTZi0VyzYlsDKFP8W61i/QzayHDkLbNEz/IShqAmTa+hsJRj13xTHA0H+6fk4b2UmGn+Q/cMlg=="], - "@rollup/rollup-linux-arm-gnueabihf": ["@rollup/rollup-linux-arm-gnueabihf@4.45.1", "", { "os": "linux", "cpu": "arm" }, "sha512-RkdOTu2jK7brlu+ZwjMIZfdV2sSYHK2qR08FUWcIoqJC2eywHbXr0L8T/pONFwkGukQqERDheaGTeedG+rra6Q=="], + "@rolldown/binding-darwin-arm64": ["@rolldown/binding-darwin-arm64@1.2.9", "", { "os": "darwin", "cpu": "arm64" }, "sha512-IwhlH3qK5urrY8hZiEgGkHKEFN901p/p2bjxCxJlr4GyNnF7wYpUvK+Y43uaRYuC4hpfjzbR3SJC3arX1jGvmw=="], - "@rollup/rollup-linux-arm-musleabihf": ["@rollup/rollup-linux-arm-musleabihf@4.45.1", "", { "os": "linux", "cpu": "arm" }, "sha512-3kJ8pgfBt6CIIr1o+HQA7OZ9mp/zDk3ctekGl9qn/pRBgrRgfwiffaUmqioUGN9hv0OHv2gxmvdKOkARCtRb8Q=="], + "@rolldown/binding-darwin-x64": ["@rolldown/binding-darwin-x64@1.2.9", "", { "os": "darwin", "cpu": "x64" }, "sha512-XxpJfVzFh+jilRxIXUqcfYAYcunIc/XEzIizsOL1fcJee5Sf7H3mH8WlLmfHfluz5amqR88QQo9izKtmMlavAw=="], - "@rollup/rollup-linux-arm64-gnu": ["@rollup/rollup-linux-arm64-gnu@4.45.1", "", { "os": "linux", "cpu": "arm64" }, "sha512-k3dOKCfIVixWjG7OXTCOmDfJj3vbdhN0QYEqB+OuGArOChek22hn7Uy5A/gTDNAcCy5v2YcXRJ/Qcnm4/ma1xw=="], + "@rolldown/binding-freebsd-x64": ["@rolldown/binding-freebsd-x64@1.2.9", "", { "os": "freebsd", "cpu": "x64" }, "sha512-kSfvhmgeWyfkbT3p/1s5vSgboogoah2zkm9fX2zjg2hHxSV7T4KhMWRUUaRk4OXNqoD3QAUeRqLcs1aZOK4U1g=="], - "@rollup/rollup-linux-arm64-musl": ["@rollup/rollup-linux-arm64-musl@4.45.1", "", { "os": "linux", "cpu": "arm64" }, "sha512-PmI1vxQetnM58ZmDFl9/Uk2lpBBby6B6rF4muJc65uZbxCs0EA7hhKCk2PKlmZKuyVSHAyIw3+/SiuMLxKxWog=="], + "@rolldown/binding-linux-arm-gnueabihf": ["@rolldown/binding-linux-arm-gnueabihf@1.2.9", "", { "os": "linux", "cpu": "arm" }, "sha512-1RVzG17pxqbTfYLC352JlLt6kKLG+6Hr30n8DlIJqsnV5luUDd2Qdx9Ayw1Cabfyb1K9k0jXEZ7evxkRoT+uiw=="], - "@rollup/rollup-linux-loongarch64-gnu": ["@rollup/rollup-linux-loongarch64-gnu@4.45.1", "", { "os": "linux", "cpu": "none" }, "sha512-9UmI0VzGmNJ28ibHW2GpE2nF0PBQqsyiS4kcJ5vK+wuwGnV5RlqdczVocDSUfGX/Na7/XINRVoUgJyFIgipoRg=="], + "@rolldown/binding-linux-arm64-gnu": ["@rolldown/binding-linux-arm64-gnu@1.2.9", "", { "os": "linux", "cpu": "arm64" }, "sha512-BXqPvZ2drqVD+/Z8UpKwcs4Mp7grM+eGFku4CAEKrEtcbAsUpzREphK1sogCRZGreVPiMkiiBtw0n3TPteuqvw=="], - "@rollup/rollup-linux-powerpc64le-gnu": ["@rollup/rollup-linux-powerpc64le-gnu@4.45.1", "", { "os": "linux", "cpu": "ppc64" }, "sha512-7nR2KY8oEOUTD3pBAxIBBbZr0U7U+R9HDTPNy+5nVVHDXI4ikYniH1oxQz9VoB5PbBU1CZuDGHkLJkd3zLMWsg=="], + "@rolldown/binding-linux-arm64-musl": ["@rolldown/binding-linux-arm64-musl@1.2.9", "", { "os": "linux", "cpu": "arm64" }, "sha512-11vWvo8YDwLzukt27J3aYDWU+gg2P7J+ZOmiJ0hkF5BXZDW7pVya7r40MXDy6ya0i9KamoENSVKIugvJNgFXIA=="], - "@rollup/rollup-linux-riscv64-gnu": ["@rollup/rollup-linux-riscv64-gnu@4.45.1", "", { "os": "linux", "cpu": "none" }, "sha512-nlcl3jgUultKROfZijKjRQLUu9Ma0PeNv/VFHkZiKbXTBQXhpytS8CIj5/NfBeECZtY2FJQubm6ltIxm/ftxpw=="], + "@rolldown/binding-linux-ppc64-gnu": ["@rolldown/binding-linux-ppc64-gnu@1.2.9", "", { "os": "linux", "cpu": "ppc64" }, "sha512-a1tijMkdwsIARtc0F39ApURROkf3NwqinI6TOiSSWCTR7dT96dffNvMUtDHnq64wKNTIZOIlzKrFvvFUznJiyw=="], - "@rollup/rollup-linux-riscv64-musl": ["@rollup/rollup-linux-riscv64-musl@4.45.1", "", { "os": "linux", "cpu": "none" }, "sha512-HJV65KLS51rW0VY6rvZkiieiBnurSzpzore1bMKAhunQiECPuxsROvyeaot/tcK3A3aGnI+qTHqisrpSgQrpgA=="], + "@rolldown/binding-linux-s390x-gnu": ["@rolldown/binding-linux-s390x-gnu@1.2.9", "", { "os": "linux", "cpu": "s390x" }, "sha512-x6SQNdAvv4c3hWqTMaWuawzMX9myaCs/yEmlGsxJzkdClnHW7FbrjQuSiRDhuSYzEYoEMhsaJy9qHG/XNemJPQ=="], - "@rollup/rollup-linux-s390x-gnu": ["@rollup/rollup-linux-s390x-gnu@4.45.1", "", { "os": "linux", "cpu": "s390x" }, "sha512-NITBOCv3Qqc6hhwFt7jLV78VEO/il4YcBzoMGGNxznLgRQf43VQDae0aAzKiBeEPIxnDrACiMgbqjuihx08OOw=="], + "@rolldown/binding-linux-x64-gnu": ["@rolldown/binding-linux-x64-gnu@1.2.9", "", { "os": "linux", "cpu": "x64" }, "sha512-9s0AZ8BFK5/n7B/TBoa2yJE3gI3KURrbXcPBlsAsvjU4VeJKgE90y1YtNxyEUIcHPQkg6/yfF3qihUrcM/Kf0Q=="], - "@rollup/rollup-linux-x64-gnu": ["@rollup/rollup-linux-x64-gnu@4.45.1", "", { "os": "linux", "cpu": "x64" }, "sha512-+E/lYl6qu1zqgPEnTrs4WysQtvc/Sh4fC2nByfFExqgYrqkKWp1tWIbe+ELhixnenSpBbLXNi6vbEEJ8M7fiHw=="], + "@rolldown/binding-linux-x64-musl": ["@rolldown/binding-linux-x64-musl@1.2.9", "", { "os": "linux", "cpu": "x64" }, "sha512-P7VWAmV+WdJluH7ovnRGoiv2i8To7GAZ+kGzfGup635cyL7SyYl3lSUaA3Gp5THf0n/Co5EyEqb2zbqq+nMOHQ=="], - "@rollup/rollup-linux-x64-musl": ["@rollup/rollup-linux-x64-musl@4.45.1", "", { "os": "linux", "cpu": "x64" }, "sha512-a6WIAp89p3kpNoYStITT9RbTbTnqarU7D8N8F2CV+4Cl9fwCOZraLVuVFvlpsW0SbIiYtEnhCZBPLoNdRkjQFw=="], + "@rolldown/binding-openharmony-arm64": ["@rolldown/binding-openharmony-arm64@1.2.9", "", { "os": "none", "cpu": "arm64" }, "sha512-1qixtsE4BK8h+yS3BfmZ09UhA7O/N4IACva6YBr7EBvCJraByTuRcgOTaiA62Tm0vey3UcKXLOaoGHtYmNGEVg=="], - "@rollup/rollup-win32-arm64-msvc": ["@rollup/rollup-win32-arm64-msvc@4.45.1", "", { "os": "win32", "cpu": "arm64" }, "sha512-T5Bi/NS3fQiJeYdGvRpTAP5P02kqSOpqiopwhj0uaXB6nzs5JVi2XMJb18JUSKhCOX8+UE1UKQufyD6Or48dJg=="], + "@rolldown/binding-win32-arm64-msvc": ["@rolldown/binding-win32-arm64-msvc@1.2.9", "", { "os": "win32", "cpu": "arm64" }, "sha512-ok8IQjcEPs1AKZfuEUznVBrJw+gK4soq+bx8b1X2XoMqVClarc1q5JDmVtWXY1xfr6ZuHTAsPXHTgTrqKTZeww=="], - "@rollup/rollup-win32-ia32-msvc": ["@rollup/rollup-win32-ia32-msvc@4.45.1", "", { "os": "win32", "cpu": "ia32" }, "sha512-lxV2Pako3ujjuUe9jiU3/s7KSrDfH6IgTSQOnDWr9aJ92YsFd7EurmClK0ly/t8dzMkDtd04g60WX6yl0sGfdw=="], + "@rolldown/binding-win32-x64-msvc": ["@rolldown/binding-win32-x64-msvc@1.2.9", "", { "os": "win32", "cpu": "x64" }, "sha512-Ip2mXoU0hM0boq3Rf+ekuT653OROSo6aSYcPT1VHE4q52KvyxgFkQgrgb/IEsxOuvQ2fZZbs8khJAyCEPM24/g=="], - "@rollup/rollup-win32-x64-msvc": ["@rollup/rollup-win32-x64-msvc@4.45.1", "", { "os": "win32", "cpu": "x64" }, "sha512-M/fKi4sasCdM8i0aWJjCSFm2qEnYRR8AMLG2kxp6wD13+tMGA4Z1tVAuHkNRjud5SW2EM3naLuK35w9twvf6aA=="], + "@rolldown/pluginutils": ["@rolldown/pluginutils@1.0.1", "", {}, "sha512-2j9bGt5Jh8hj+vPtgzPtl72j0yRxHAyumoo6TNfAjsLB04UtpSvPbPcDcBMxz7n+9CYB0c1GxQFxYRg2jimqGw=="], "@scure/base": ["@scure/base@1.2.6", "", {}, "sha512-g/nm5FgUa//MCj1gV09zTJTaM6KBAHqLN907YVQqf7zC49+DcO4B1so4ZX07Ef10Twr6nuqYEH9GEggFXA4Fmg=="], @@ -153,6 +101,8 @@ "@sinclair/typebox": ["@sinclair/typebox@0.27.8", "", {}, "sha512-+Fj43pSMwJs4KRrH/938Uf+uAELIgVBmQzg/q1YG10djyfA3TnrU8N8XzqCh/okZdszqBQTZf96idMfE5lnwTA=="], + "@standard-schema/spec": ["@standard-schema/spec@1.1.0", "", {}, "sha512-l2aFy5jALhniG5HgqrD6jXLi/rUWrKvqN/qJx6yoJsgKhblVd+iqqU4RCXavm/jPityDo5TCvKMnpjKnOriy0w=="], + "@types/bun": ["@types/bun@1.2.18", "", { "dependencies": { "bun-types": "1.2.18" } }, "sha512-Xf6RaWVheyemaThV0kUfaAUvCNokFr+bH8Jxp+tTZfx7dAPA8z9ePnP9S9+Vspzuxxx9JRAXhnyccRj3GyCMdQ=="], "@types/chai": ["@types/chai@5.2.3", "", { "dependencies": { "@types/deep-eql": "*", "assertion-error": "^2.0.1" } }, "sha512-Mw558oeA9fFbv65/y4mHtXDs9bPnFMZAL/jxdPFUpOHHIXX91mcgEHbS5Lahr+pwZFR8A7GQleRWeI6cGFC2UA=="], @@ -165,17 +115,17 @@ "@types/react": ["@types/react@19.1.8", "", { "dependencies": { "csstype": "^3.0.2" } }, "sha512-AwAfQ2Wa5bCx9WP8nZL2uMZWod7J7/JSplxbTmBQ5ms6QpqNYm672H0Vu9ZVKVngQ+ii4R/byguVEUZQyeg44g=="], - "@vitest/expect": ["@vitest/expect@3.2.6", "", { "dependencies": { "@types/chai": "^5.2.2", "@vitest/spy": "3.2.6", "@vitest/utils": "3.2.6", "chai": "^5.2.0", "tinyrainbow": "^2.0.0" } }, "sha512-1+7q9BtaKzEmO+fmNT3kYvoNn5Y71XWAx2Q5HRim4tTVRQVRv4uJFAQ5FbK0OPUeNP/WmVCpxYxoJdvuHVjzBQ=="], + "@vitest/expect": ["@vitest/expect@4.1.11", "", { "dependencies": { "@standard-schema/spec": "^1.1.0", "@types/chai": "^5.2.2", "@vitest/spy": "4.1.11", "@vitest/utils": "4.1.11", "chai": "^6.2.2", "tinyrainbow": "^3.1.0" } }, "sha512-VX2x5vNJXET47KAFzwERI+KRMtTTCSWTfSMKsW7JsUsXV4psq++e3DvZpuTDOpHcxytiDs6p2nhVb2tVDiiUYw=="], - "@vitest/mocker": ["@vitest/mocker@3.2.6", "", { "dependencies": { "@vitest/spy": "3.2.6", "estree-walker": "^3.0.3", "magic-string": "^0.30.17" }, "peerDependencies": { "msw": "^2.4.9", "vite": "^5.0.0 || ^6.0.0 || ^7.0.0-0" }, "optionalPeers": ["msw", "vite"] }, "sha512-EZOrpDbkKotFAP7wPAQV1UIyoGOk4oX7ynWhBhLB7v+meMHbQhU16oPpIYGTTe4oFlhpryGpgpcZP/sin3hYuw=="], + "@vitest/mocker": ["@vitest/mocker@4.1.11", "", { "dependencies": { "@vitest/spy": "4.1.11", "estree-walker": "^3.0.3", "magic-string": "^0.30.21" }, "peerDependencies": { "msw": "^2.4.9", "vite": "^6.0.0 || ^7.0.0 || ^8.0.0" }, "optionalPeers": ["msw", "vite"] }, "sha512-2XJVD55d1o5AZous5CCGKS74g/riOj9odEt2bQpCVZeblHyHdnMeFl4jl0XjU21stf4mbjUkew2eXQZt65g5CQ=="], - "@vitest/pretty-format": ["@vitest/pretty-format@3.2.7", "", { "dependencies": { "tinyrainbow": "^2.0.0" } }, "sha512-KUHlwqVu0sRlhCdyPdQ/wBoTfRahjUky1MubOmYw9fWfIZy1gNoHpuaaQBPAaMaVYdQYHJLurzj8ECCj5OwTqA=="], + "@vitest/pretty-format": ["@vitest/pretty-format@4.1.11", "", { "dependencies": { "tinyrainbow": "^3.1.0" } }, "sha512-yiZzPbGTS9Sr/JpFl8zHrcIkAofNbFV6k21vIgQN/cY/oxZeXhJv5sc/MBJ5jFKWmWs+oJHw0UXLZjmf931+Vw=="], - "@vitest/runner": ["@vitest/runner@3.2.6", "", { "dependencies": { "@vitest/utils": "3.2.6", "pathe": "^2.0.3", "strip-literal": "^3.0.0" } }, "sha512-HYcoSj1w5tcgUnzoF0HcyaAQjpA1gj9ftUJ7iSJSuipc02jW9gKkigwZbjFldAfYHA1fa8UZVRftdMY5msWM9Q=="], + "@vitest/runner": ["@vitest/runner@4.1.11", "", { "dependencies": { "@vitest/utils": "4.1.11", "pathe": "^2.0.3" } }, "sha512-LztvUgdwMNJMIkj3hQnnxiC2Xy1zNxq928W/xhjCLaNCzqTZOudjwbQf6v9IntZGPw132i2Lq2rgTRZHD3JHNw=="], - "@vitest/snapshot": ["@vitest/snapshot@3.2.6", "", { "dependencies": { "@vitest/pretty-format": "3.2.6", "magic-string": "^0.30.17", "pathe": "^2.0.3" } }, "sha512-H+ZjNTWGpObenh0YnlBctAPnJSI20P81PL8BPzWpx54YXLLTm8hEsWawtcYLMrwvpK48hGxLLbCS+1KRXhsKhw=="], + "@vitest/snapshot": ["@vitest/snapshot@4.1.11", "", { "dependencies": { "@vitest/pretty-format": "4.1.11", "@vitest/utils": "4.1.11", "magic-string": "^0.30.21", "pathe": "^2.0.3" } }, "sha512-pN7ikn1ON7h8ee4gIAp4AzyK+zBtJPzVbqOgu5LCEh4VaJVbPQcgYQYJIMGQPXVeJJq1fnfazis7a5pFNPahog=="], - "@vitest/spy": ["@vitest/spy@3.2.6", "", { "dependencies": { "tinyspy": "^4.0.3" } }, "sha512-oq6BbH68WzcWmwtBrU9nqLeaXTR4XwJF7FSLkKEZo4i6eoXcrxjcwSuTvWBIRUTC6VC72nXYunzqgZA+IKdtxg=="], + "@vitest/spy": ["@vitest/spy@4.1.11", "", {}, "sha512-apNa/prQy2qCeywhnixOHPRCgGNhvg7T4Dapfl1GahLp/R+uhBm5cPyFoNVyqsNd2h1nJxL6BqqdIjiABL60YA=="], "@vitest/ui": ["@vitest/ui@1.6.1", "", { "dependencies": { "@vitest/utils": "1.6.1", "fast-glob": "^3.3.2", "fflate": "^0.8.1", "flatted": "^3.2.9", "pathe": "^1.1.1", "picocolors": "^1.0.0", "sirv": "^2.0.4" }, "peerDependencies": { "vitest": "1.6.1" } }, "sha512-xa57bCPGuzEFqGjPs3vVLyqareG8DX0uMkr5U/v5vLv5/ZUrBrPL7gzxzTJedEyZxFMfsozwTIbbYfEQVo3kgg=="], @@ -191,23 +141,17 @@ "bun-types": ["bun-types@1.2.18", "", { "dependencies": { "@types/node": "*" }, "peerDependencies": { "@types/react": "^19" } }, "sha512-04+Eha5NP7Z0A9YgDAzMk5PHR16ZuLVa83b26kH5+cp1qZW4F6FmAURngE7INf4tKOvCE69vYvDEwoNl1tGiWw=="], - "cac": ["cac@6.7.14", "", {}, "sha512-b6Ilus+c3RrdDk+JhLKUAQfzzgLEPy6wcXqS7f/xe1EETvsDP6GORG7SFuOs6cID5YkqchW/LXZbX5bc8j7ZcQ=="], - - "chai": ["chai@5.3.3", "", { "dependencies": { "assertion-error": "^2.0.1", "check-error": "^2.1.1", "deep-eql": "^5.0.1", "loupe": "^3.1.0", "pathval": "^2.0.0" } }, "sha512-4zNhdJD/iOjSH0A05ea+Ke6MU5mmpQcbQsSOkgdaUMJ9zTlDTD/GYlwohmIE2u0gaxHYiVHEn1Fw9mZ/ktJWgw=="], + "chai": ["chai@6.2.2", "", {}, "sha512-NUPRluOfOiTKBKvWPtSD4PhFvWCqOi0BGStNWs57X9js7XGTprSmFoz5F0tWhR4WPjNeR9jXqdC7/UpSJTnlRg=="], - "check-error": ["check-error@2.1.3", "", {}, "sha512-PAJdDJusoxnwm1VwW07VWwUN1sl7smmC3OKggvndJFadxxDRyFJBX/ggnu/KE4kQAB7a3Dp8f/YXC1FlUprWmA=="], + "convert-source-map": ["convert-source-map@2.0.0", "", {}, "sha512-Kvp459HrV2FEJ1CAsi1Ku+MY3kasH19TFykTz2xWmMeq6bk2NU3XXvfJ+Q61m0xktWwt+1HSYf3JZsTms3aRJg=="], "csstype": ["csstype@3.1.3", "", {}, "sha512-M1uQkMl8rQK/szD0LNhtqxIPLpimGm8sOBwU7lLnCpSbTyY3yeU1Vc7l4KT5zT4s/yOxHH5O7tIuuLOCnLADRw=="], - "debug": ["debug@4.4.1", "", { "dependencies": { "ms": "^2.1.3" } }, "sha512-KcKCqiftBJcZr++7ykoDIEwSa3XWowTfNPo92BYxjXiyYEVrUQh2aLyhxBCwww+heortUFxEJYcRzosstTEBYQ=="], - - "deep-eql": ["deep-eql@5.0.2", "", {}, "sha512-h5k/5U50IJJFpzfL6nO9jaaumfjO/f2NjK/oYB2Djzm4p9L+3T9qWpZqZ2hAbLPuuYq9wrU08WQyBTL5GbPk5Q=="], + "detect-libc": ["detect-libc@2.1.2", "", {}, "sha512-Btj2BOOO83o3WyH59e8MgXsxEQVcarkUOpEYrubB0urwnN10yQ364rsiByU11nZlqWYZm05i/of7io4mzihBtQ=="], "diff-sequences": ["diff-sequences@29.6.3", "", {}, "sha512-EjePK1srD3P08o2j4f0ExnylqRs5B9tJjcp9t1krH2qRi8CCdsYfwe9JgSLurFBWwq4uOlipzfk5fHNvwFKr8Q=="], - "es-module-lexer": ["es-module-lexer@1.7.0", "", {}, "sha512-jEQoCwk8hyb2AZziIOLhDqpm5+2ww5uIE6lkO/6jcOCusfk6LhMHpXXfBLXTZ7Ydyt0j4VoUQv6uGNYbdW+kBA=="], - - "esbuild": ["esbuild@0.21.5", "", { "optionalDependencies": { "@esbuild/aix-ppc64": "0.21.5", "@esbuild/android-arm": "0.21.5", "@esbuild/android-arm64": "0.21.5", "@esbuild/android-x64": "0.21.5", "@esbuild/darwin-arm64": "0.21.5", "@esbuild/darwin-x64": "0.21.5", "@esbuild/freebsd-arm64": "0.21.5", "@esbuild/freebsd-x64": "0.21.5", "@esbuild/linux-arm": "0.21.5", "@esbuild/linux-arm64": "0.21.5", "@esbuild/linux-ia32": "0.21.5", "@esbuild/linux-loong64": "0.21.5", "@esbuild/linux-mips64el": "0.21.5", "@esbuild/linux-ppc64": "0.21.5", "@esbuild/linux-riscv64": "0.21.5", "@esbuild/linux-s390x": "0.21.5", "@esbuild/linux-x64": "0.21.5", "@esbuild/netbsd-x64": "0.21.5", "@esbuild/openbsd-x64": "0.21.5", "@esbuild/sunos-x64": "0.21.5", "@esbuild/win32-arm64": "0.21.5", "@esbuild/win32-ia32": "0.21.5", "@esbuild/win32-x64": "0.21.5" }, "bin": { "esbuild": "bin/esbuild" } }, "sha512-mg3OPMV4hXywwpoDxu3Qda5xCKQi+vCTZq8S9J/EpkhB2HzKXq4SNFZE3+NK93JYxc8VMSep+lOUSC/RVKaBqw=="], + "es-module-lexer": ["es-module-lexer@2.3.2", "", {}, "sha512-poHGpORABojJJucnV9KbOavETW8lBVnphkW77ER5/BQ5Fz7oXSoCNek7IH3vR5nRjdsEz926ibFYX8KtLQmdyw=="], "estree-walker": ["estree-walker@3.0.3", "", { "dependencies": { "@types/estree": "^1.0.0" } }, "sha512-7RUKfXgSMMkzt6ZuXmqapOurLGPPfgj6l9uRZ7lRGolvk0y2yocc35LdcxKC5PQZdn2DMqioAQ2NoWcrTKmm6g=="], @@ -233,7 +177,7 @@ "glob-parent": ["glob-parent@5.1.2", "", { "dependencies": { "is-glob": "^4.0.1" } }, "sha512-AOIgSQCepiJYwP3ARnGx+5VnTu2HBYdzbGP45eLw1vr3zB3vZLeyed1sC9hnbcOc9/SrMyM5RPQrkGz4aS9Zow=="], - "hono": ["hono@4.12.34", "", {}, "sha512-GqXJqY/xJkJmuloTrnV1ZEXG3fqte+VjkUqoRNZXcrUidiUOP4fMSIHHY4tsqZBK++kVyWmt/AAfSUuy57/eSA=="], + "hono": ["hono@4.13.5", "", {}, "sha512-O6+/eCYRkzzzy0rPWwKLiGBR1nFuUPZynnwjxN1MBA62NNqbT0wQEzQyK2gSO5yDIDB336sXQleAhOHrzlYyKw=="], "is-extglob": ["is-extglob@2.1.1", "", {}, "sha512-SbKbANkN603Vi4jEZv49LeVJMn4yGwsbzZworEoyEiutsN3nJYdbO36zfhGJ6QEDpOZIFkDtnq5JRxmvl3jsoQ=="], @@ -243,11 +187,33 @@ "isows": ["isows@1.0.7", "", { "peerDependencies": { "ws": "*" } }, "sha512-I1fSfDCZL5P0v33sVqeTDSpcstAg/N+wF5HS033mogOVIp4B+oHC7oOCsA3axAbBSGTJ8QubbNmnIRN/h8U7hg=="], - "js-tokens": ["js-tokens@9.0.1", "", {}, "sha512-mxa9E9ITFOt0ban3j6L5MpjwegGz6lBQmM1IJkWeBZGcMxto50+eWdjC/52xDbS2vy0k7vIMK0Fe2wfL9OQSpQ=="], + "lightningcss": ["lightningcss@1.33.0", "", { "dependencies": { "detect-libc": "^2.0.3" }, "optionalDependencies": { "lightningcss-android-arm64": "1.33.0", "lightningcss-darwin-arm64": "1.33.0", "lightningcss-darwin-x64": "1.33.0", "lightningcss-freebsd-x64": "1.33.0", "lightningcss-linux-arm-gnueabihf": "1.33.0", "lightningcss-linux-arm64-gnu": "1.33.0", "lightningcss-linux-arm64-musl": "1.33.0", "lightningcss-linux-x64-gnu": "1.33.0", "lightningcss-linux-x64-musl": "1.33.0", "lightningcss-win32-arm64-msvc": "1.33.0", "lightningcss-win32-x64-msvc": "1.33.0" } }, "sha512-WkUDrojuJs0xkgGf2udWxa3yGBRxPtxUkB79i6aCZLRgc7PM8fZe9TosfPDcvEpQZbuFASnHYmRLBLUbmLOIIA=="], + + "lightningcss-android-arm64": ["lightningcss-android-arm64@1.33.0", "", { "os": "android", "cpu": "arm64" }, "sha512-gEpRTalKdosp4Bb8qWtc2iOgE5SeIHlpS1up9bFq2wAyYhl1UdTObYiHe98zEM9SQvSoqQZ1IQD0JNpg3Ml5pg=="], + + "lightningcss-darwin-arm64": ["lightningcss-darwin-arm64@1.33.0", "", { "os": "darwin", "cpu": "arm64" }, "sha512-Sciaz8eenNTKn9b3t7+xr0ipTp9YxKQY4npwQ3mrRuL0BAVHBLyZxofhaKBAVtzmtRZ/zTyo0/to4B1uWG/Djg=="], + + "lightningcss-darwin-x64": ["lightningcss-darwin-x64@1.33.0", "", { "os": "darwin", "cpu": "x64" }, "sha512-Z5UPAxzrjlWNNyGy6i65cJzzvgJ5D3T6wMvs+gWpY9d7qRhANrxqAp6LhxIgZhWEw18RfJTGcRxjuLIBr+m8XQ=="], + + "lightningcss-freebsd-x64": ["lightningcss-freebsd-x64@1.33.0", "", { "os": "freebsd", "cpu": "x64" }, "sha512-QQM/Ti/hQajJwCY+RiWuCZ9sdtI/XQk7nDK5vC8kkdwixezOlDgvDx7+RT+QjK6FcFT4MpsuoBnHIo/O3StRRg=="], + + "lightningcss-linux-arm-gnueabihf": ["lightningcss-linux-arm-gnueabihf@1.33.0", "", { "os": "linux", "cpu": "arm" }, "sha512-N7FVBe6iS24MlM6R/4RBTxGhQheZGs7tiQ9U32UtF75NzP5Q7xWPRqLBCKxlRQRk3rY1jCIPLzx7WzOhuUIRLQ=="], + + "lightningcss-linux-arm64-gnu": ["lightningcss-linux-arm64-gnu@1.33.0", "", { "os": "linux", "cpu": "arm64" }, "sha512-j2v/itmy4HlNxlc6voKXYgBqNi0Ng2LShg4z7GufpEgs05P+2suBVyi9I6YHq5uoVFx9ETin3eCEhLVyXGQnKg=="], + + "lightningcss-linux-arm64-musl": ["lightningcss-linux-arm64-musl@1.33.0", "", { "os": "linux", "cpu": "arm64" }, "sha512-yiO5ROMuYQgXbC60yjZU5CYSFZGKXL0HFATXt9mHJn1+zW55oCtMI9NfcVhYLMFDL7gV7oBPon/EmMMGg2OvtQ=="], + + "lightningcss-linux-x64-gnu": ["lightningcss-linux-x64-gnu@1.33.0", "", { "os": "linux", "cpu": "x64" }, "sha512-ar+Ju7LmcN0Jo4FpL4hpFybwNG9/3A/Br5KW2n2jyODg3MEZXaDYADdemoNS+BDNfMgKvylJLj4S5tyRActuAg=="], + + "lightningcss-linux-x64-musl": ["lightningcss-linux-x64-musl@1.33.0", "", { "os": "linux", "cpu": "x64" }, "sha512-RYiYbkokw0trfKqqzfF55lginwEPrD3OJDfTuJzFs1MK6iFnDenaz1fqLLtX4ITG3OktJQXOeTaw1awrBAlZPw=="], + + "lightningcss-win32-arm64-msvc": ["lightningcss-win32-arm64-msvc@1.33.0", "", { "os": "win32", "cpu": "arm64" }, "sha512-1K+MPfLSFVpphzpdbfkhlWk6wBrTObBzS2T6db10PNOZgR9GoVsAWzwNyuhUYYbTp23j+4RrncfujZ4uAzXvwA=="], + + "lightningcss-win32-x64-msvc": ["lightningcss-win32-x64-msvc@1.33.0", "", { "os": "win32", "cpu": "x64" }, "sha512-OlEICDx/Xl0FqSp4bry8zFnCvGpig3Gl4gCquvYwHuqJKEC1+n9NgDniFvqHGmMv1ZkqDJrDqKKSykTDX+ehuA=="], "loupe": ["loupe@2.3.7", "", { "dependencies": { "get-func-name": "^2.0.1" } }, "sha512-zSMINGVYkdpYSOBmLi0D1Uo7JU9nVdQKrHxC8eYlV+9YKK9WePqAlL7lSlorG/U2Fw1w0hTBmaa/jrQ3UbPHtA=="], - "magic-string": ["magic-string@0.30.17", "", { "dependencies": { "@jridgewell/sourcemap-codec": "^1.5.0" } }, "sha512-sNPKHvyjVf7gyjwS4xGTaW/mCnF8wnjtifKBEhxfZ7E/S8tQ0rssrwGNn6q8JH/ohItJfSQp9mBtQYuTlH5QnA=="], + "magic-string": ["magic-string@0.30.21", "", { "dependencies": { "@jridgewell/sourcemap-codec": "^1.5.5" } }, "sha512-vd2F4YUyEXKGcLHoq+TEyCjxueSeHnFxyyjNp80yg0XV4vUhnDer/lvvlqM/arB5bXQN5K2/3oinyCRyx8T2CQ=="], "merge2": ["merge2@1.4.1", "", {}, "sha512-8q7VEgMJW4J8tcfVPy8g09NcQwZdbwFEqhe/WZkoIzjn/3TGDwtOCYtXGxA3O8tPzpczCCDgv+P2P5y00ZJOOg=="], @@ -255,9 +221,9 @@ "mrmime": ["mrmime@2.0.1", "", {}, "sha512-Y3wQdFg2Va6etvQ5I82yUhGdsKrcYox6p7FfL1LbK2J4V01F9TGlepTIhnK24t7koZibmg82KGglhA1XK5IsLQ=="], - "ms": ["ms@2.1.3", "", {}, "sha512-6FlzubTLZG3J2a/NVCAleEhjzq5oxgHyaCU9yYXvcLsvoVaHJq/s5xXI6/XXP6tz7R9xAOtHnSO/tXtF3WRTlA=="], + "nanoid": ["nanoid@3.3.19", "", { "bin": { "nanoid": "bin/nanoid.cjs" } }, "sha512-Y2tUNy4ouw6tq5oDSKeQYGOyhkUBhNOcGV/02KC+6kd9eDGqdZd++mjMiIDilrBYvjEnCYvVtsuHCuP+okSfug=="], - "nanoid": ["nanoid@3.3.11", "", { "bin": { "nanoid": "bin/nanoid.cjs" } }, "sha512-N8SpfPUnUp1bK+PMYW8qSWdl9U+wwNWI4QKxOYDy9JAro3WMX7p2OeVRF9v+347pnakNevPmiHhNmZ2HbFA76w=="], + "obug": ["obug@2.2.1", "", {}, "sha512-XrsrhT5sybtKI6wakr2SPOlGZWWYbUXZ7a0jT8/QOeAPau+1X/bSegNe5YR75oJmEZQbKningirmGOEJCIk61Q=="], "openapi-types": ["openapi-types@12.1.3", "", {}, "sha512-N4YtSYJqghVu4iek2ZUvcN/0aqH1kRDuNqzcycDxhOUpg7GdvLa2F3DgS6yBNhInhv2r/6I0Flkn7CqL8+nIcw=="], @@ -267,13 +233,11 @@ "pathe": ["pathe@1.1.2", "", {}, "sha512-whLdWMYL2TwI08hn8/ZqAbrVemu0LNaNNJZX73O6qaIdCTfXutsLhMkjdENX0qhsQ9uIimo4/aQOmXkoon2nDQ=="], - "pathval": ["pathval@2.0.1", "", {}, "sha512-//nshmD55c46FuFw26xV/xFAaB5HF9Xdap7HJBBnrKdAd6/GxDBaNA1870O79+9ueg61cZLSVc+OaFlfmObYVQ=="], - "picocolors": ["picocolors@1.1.1", "", {}, "sha512-xceH2snhtb5M9liqDsmEw56le376mTZkEX/jEb/RxNFyegNul7eNslCXP9FDj/Lcu0X8KEyMceP2ntpaHrDEVA=="], "picomatch": ["picomatch@4.0.5", "", {}, "sha512-RvwwcruNjI1ncT5xRakeyS9Lf8lcItv34KD+aif+VH9kduAyfYBipGh12274xtenIPZ119/R9BdTBa8gAwSh0A=="], - "postcss": ["postcss@8.5.6", "", { "dependencies": { "nanoid": "^3.3.11", "picocolors": "^1.1.1", "source-map-js": "^1.2.1" } }, "sha512-3Ybi1tAuwAP9s0r1UQ2J4n5Y0G05bJkpUIO0/bI9MhwmD70S5aTWbXGBwxHrelT+XM1k6dM0pk+SwNkpTRN7Pg=="], + "postcss": ["postcss@8.5.28", "", { "dependencies": { "nanoid": "^3.3.18", "picocolors": "^1.1.1", "source-map-js": "^1.2.1" } }, "sha512-RRuzqDtt5Y9h3quz5hWhK+TPnsmVs6WwSU6LkJMeY4HstUEDuYTG8UJSdawMRzmzAtV+KEoG8N3Qg2qLy5vM/A=="], "pretty-format": ["pretty-format@29.7.0", "", { "dependencies": { "@jest/schemas": "^29.6.3", "ansi-styles": "^5.0.0", "react-is": "^18.0.0" } }, "sha512-Pdlw/oPxN+aXdmM9R00JVC9WVFoCLTKJvDVLgmJ+qAffBMxsV85l/Lu7sNx4zSzPyoL2euImuEwHhOXdEgNFZQ=="], @@ -283,7 +247,7 @@ "reusify": ["reusify@1.1.0", "", {}, "sha512-g6QUff04oZpHs0eG5p83rFLhHeV00ug/Yf9nZM6fLeUrPguBTkTQOdpAWWspMh55TZfVQDPaN3NQJfbVRAxdIw=="], - "rollup": ["rollup@4.45.1", "", { "dependencies": { "@types/estree": "1.0.8" }, "optionalDependencies": { "@rollup/rollup-android-arm-eabi": "4.45.1", "@rollup/rollup-android-arm64": "4.45.1", "@rollup/rollup-darwin-arm64": "4.45.1", "@rollup/rollup-darwin-x64": "4.45.1", "@rollup/rollup-freebsd-arm64": "4.45.1", "@rollup/rollup-freebsd-x64": "4.45.1", "@rollup/rollup-linux-arm-gnueabihf": "4.45.1", "@rollup/rollup-linux-arm-musleabihf": "4.45.1", "@rollup/rollup-linux-arm64-gnu": "4.45.1", "@rollup/rollup-linux-arm64-musl": "4.45.1", "@rollup/rollup-linux-loongarch64-gnu": "4.45.1", "@rollup/rollup-linux-powerpc64le-gnu": "4.45.1", "@rollup/rollup-linux-riscv64-gnu": "4.45.1", "@rollup/rollup-linux-riscv64-musl": "4.45.1", "@rollup/rollup-linux-s390x-gnu": "4.45.1", "@rollup/rollup-linux-x64-gnu": "4.45.1", "@rollup/rollup-linux-x64-musl": "4.45.1", "@rollup/rollup-win32-arm64-msvc": "4.45.1", "@rollup/rollup-win32-ia32-msvc": "4.45.1", "@rollup/rollup-win32-x64-msvc": "4.45.1", "fsevents": "~2.3.2" }, "bin": { "rollup": "dist/bin/rollup" } }, "sha512-4iya7Jb76fVpQyLoiVpzUrsjQ12r3dM7fIVz+4NwoYvZOShknRmiv+iu9CClZml5ZLGb0XMcYLutK6w9tgxHDw=="], + "rolldown": ["rolldown@1.2.9", "", { "dependencies": { "@oxc-project/types": "=0.150.0", "@rolldown/pluginutils": "^1.0.0" }, "optionalDependencies": { "@rolldown/binding-android-arm-eabi": "1.2.9", "@rolldown/binding-android-arm64": "1.2.9", "@rolldown/binding-darwin-arm64": "1.2.9", "@rolldown/binding-darwin-x64": "1.2.9", "@rolldown/binding-freebsd-x64": "1.2.9", "@rolldown/binding-linux-arm-gnueabihf": "1.2.9", "@rolldown/binding-linux-arm64-gnu": "1.2.9", "@rolldown/binding-linux-arm64-musl": "1.2.9", "@rolldown/binding-linux-ppc64-gnu": "1.2.9", "@rolldown/binding-linux-s390x-gnu": "1.2.9", "@rolldown/binding-linux-x64-gnu": "1.2.9", "@rolldown/binding-linux-x64-musl": "1.2.9", "@rolldown/binding-openharmony-arm64": "1.2.9", "@rolldown/binding-win32-arm64-msvc": "1.2.9", "@rolldown/binding-win32-x64-msvc": "1.2.9" }, "bin": { "rolldown": "./bin/cli.mjs" } }, "sha512-hx/Pv0N1haXRb11qkfnK5MXB/iqr7i0yjWQqmO9uHqZpBgQSqzc8UsSnEpalsh+j1I8qQ2CkXAkJC8Br3dKSlg=="], "run-parallel": ["run-parallel@1.2.0", "", { "dependencies": { "queue-microtask": "^1.2.2" } }, "sha512-5l4VyZR86LZ/lDxZTR6jqL8AFE2S0IFLMP26AbjsLVADxHdhB/c0GUsH+y39UfCi3dzz8OlQuPmnaJOMoDHQBA=="], @@ -295,21 +259,15 @@ "stackback": ["stackback@0.0.2", "", {}, "sha512-1XMJE5fQo1jGH6Y/7ebnwPOBEkIEnT4QF32d5R1+VXdXveM0IBMJt8zfaxX1P3QhVwrYe+576+jkANtSS2mBbw=="], - "std-env": ["std-env@3.9.0", "", {}, "sha512-UGvjygr6F6tpH7o2qyqR6QYpwraIjKSdtzyBdyytFOHmPZY917kwdwLG0RbOjWOnKmnm3PeHjaoLLMie7kPLQw=="], - - "strip-literal": ["strip-literal@3.1.0", "", { "dependencies": { "js-tokens": "^9.0.1" } }, "sha512-8r3mkIM/2+PpjHoOtiAW8Rg3jJLHaV7xPwG+YRGrv6FP0wwk/toTpATxWYOW0BKdWwl82VT2tFYi5DlROa0Mxg=="], + "std-env": ["std-env@4.2.0", "", {}, "sha512-oCUKSupKTHX53EyjDtuZQ64pjLJ6yYCtpmEw0goYxtjG9KpbRe8KAsl2tBUGU9DyMcJ0RwJ8GqJAFzMXcXW1Rw=="], "tinybench": ["tinybench@2.9.0", "", {}, "sha512-0+DUvqWMValLmha6lr4kD8iAMK1HzV0/aKnCtWb9v9641TnP/MFb7Pc2bxoxQjTXAErryXVgUOfv2YqNllqGeg=="], - "tinyexec": ["tinyexec@0.3.2", "", {}, "sha512-KQQR9yN7R5+OSwaK0XQoj22pwHoTlgYqmUscPYoknOoWCWfj/5/ABTMRi69FrKU5ffPVh5QcFikpWJI/P1ocHA=="], + "tinyexec": ["tinyexec@1.3.1", "", {}, "sha512-GCvB3aoys96IuDFBMcTB46JOR6mdMtAToqwiW8JlWhsoh1mhHi/xn9ss/Dg7N555GiJyEt2qzoG/NHCwM6h1EA=="], "tinyglobby": ["tinyglobby@0.2.17", "", { "dependencies": { "fdir": "^6.5.0", "picomatch": "^4.0.4" } }, "sha512-wXR/dYpcqKmfWpEdZjiKJOwCNFndD0DMnrW/cYjVGttEkBfVgcLFHoNrlj47mjOVic9yyNu65alsgF4NQyTa2g=="], - "tinypool": ["tinypool@1.1.1", "", {}, "sha512-Zba82s87IFq9A9XmjiX5uZA/ARWDrB03OHlq+Vw1fSdt0I+4/Kutwy8BP4Y/y/aORMo61FQ0vIb5j44vSo5Pkg=="], - - "tinyrainbow": ["tinyrainbow@2.0.0", "", {}, "sha512-op4nsTR47R6p0vMUUoYl/a+ljLFVtlfaXkLQmqfLR1qHma1h/ysYk4hEXZ880bf2CYgTskvTa/e196Vd5dDQXw=="], - - "tinyspy": ["tinyspy@4.0.4", "", {}, "sha512-azl+t0z7pw/z958Gy9svOTuzqIk6xq+NSheJzn5MMWtWTFywIacg2wUlzKFGtt3cthx0r2SxMK0yzJOR0IES7Q=="], + "tinyrainbow": ["tinyrainbow@3.1.1", "", {}, "sha512-yau8yJdTt989Mm0Bd/236QnzEiPf2xLLTqUZRUJOo/3CB078LSwzei343DgtJVmfJKJE3TMINY1u42SQsP6mXw=="], "to-regex-range": ["to-regex-range@5.0.1", "", { "dependencies": { "is-number": "^7.0.0" } }, "sha512-65P7iz6X5yEr1cwcgvQxbbIw7Uk3gOy5dIdtZ4rDveLqhrdJP+Li/Hx6tyK0NEb+2GCyneCMJiGqrADCSNk8sQ=="], @@ -321,11 +279,9 @@ "viem": ["viem@2.31.7", "", { "dependencies": { "@noble/curves": "1.9.2", "@noble/hashes": "1.8.0", "@scure/bip32": "1.7.0", "@scure/bip39": "1.6.0", "abitype": "1.0.8", "isows": "1.0.7", "ox": "0.8.1", "ws": "8.18.2" }, "peerDependencies": { "typescript": ">=5.0.4" }, "optionalPeers": ["typescript"] }, "sha512-mpB8Hp6xK77E/b/yJmpAIQcxcOfpbrwWNItjnXaIA8lxZYt4JS433Pge2gg6Hp3PwyFtaUMh01j5L8EXnLTjQQ=="], - "vite": ["vite@5.4.19", "", { "dependencies": { "esbuild": "^0.21.3", "postcss": "^8.4.43", "rollup": "^4.20.0" }, "optionalDependencies": { "fsevents": "~2.3.3" }, "peerDependencies": { "@types/node": "^18.0.0 || >=20.0.0", "less": "*", "lightningcss": "^1.21.0", "sass": "*", "sass-embedded": "*", "stylus": "*", "sugarss": "*", "terser": "^5.4.0" }, "optionalPeers": ["@types/node", "less", "lightningcss", "sass", "sass-embedded", "stylus", "sugarss", "terser"], "bin": { "vite": "bin/vite.js" } }, "sha512-qO3aKv3HoQC8QKiNSTuUM1l9o/XX3+c+VTgLHbJWHZGeTPVAg2XwazI9UWzoxjIJCGCV2zU60uqMzjeLZuULqA=="], - - "vite-node": ["vite-node@3.2.4", "", { "dependencies": { "cac": "^6.7.14", "debug": "^4.4.1", "es-module-lexer": "^1.7.0", "pathe": "^2.0.3", "vite": "^5.0.0 || ^6.0.0 || ^7.0.0-0" }, "bin": { "vite-node": "vite-node.mjs" } }, "sha512-EbKSKh+bh1E1IFxeO0pg1n4dvoOTt0UDiXMd/qn++r98+jPO1xtJilvXldeuQ8giIB5IkpjCgMleHMNEsGH6pg=="], + "vite": ["vite@8.3.0", "", { "dependencies": { "lightningcss": "^1.33.0", "picomatch": "^4.0.7", "postcss": "^8.5.28", "rolldown": "~1.2.6", "tinyglobby": "^0.2.17" }, "optionalDependencies": { "fsevents": "~2.3.3" }, "peerDependencies": { "@types/node": "^20.19.0 || >=22.12.0", "@vitejs/devtools": "^0.7.1", "esbuild": "^0.27.0 || ^0.28.0", "jiti": ">=1.21.0", "less": "^4.0.0", "sass": "^1.70.0", "sass-embedded": "^1.70.0", "stylus": ">=0.54.8", "sugarss": "^5.0.0", "terser": "^5.16.0", "tsx": "^4.8.1", "yaml": "^2.4.2" }, "optionalPeers": ["@types/node", "@vitejs/devtools", "esbuild", "jiti", "less", "sass", "sass-embedded", "stylus", "sugarss", "terser", "tsx", "yaml"], "bin": { "vite": "bin/vite.js" } }, "sha512-lhZBVvEHefgE+HQZC9O7EBJgCU/nVzFNl7vkS4RE0APtWLP02/8QVIkQtzBxPquh7lq5/78NHipTj7ODQ6XuyQ=="], - "vitest": ["vitest@3.2.6", "", { "dependencies": { "@types/chai": "^5.2.2", "@vitest/expect": "3.2.6", "@vitest/mocker": "3.2.6", "@vitest/pretty-format": "^3.2.6", "@vitest/runner": "3.2.6", "@vitest/snapshot": "3.2.6", "@vitest/spy": "3.2.6", "@vitest/utils": "3.2.6", "chai": "^5.2.0", "debug": "^4.4.1", "expect-type": "^1.2.1", "magic-string": "^0.30.17", "pathe": "^2.0.3", "picomatch": "^4.0.2", "std-env": "^3.9.0", "tinybench": "^2.9.0", "tinyexec": "^0.3.2", "tinyglobby": "^0.2.14", "tinypool": "^1.1.1", "tinyrainbow": "^2.0.0", "vite": "^5.0.0 || ^6.0.0 || ^7.0.0-0", "vite-node": "3.2.4", "why-is-node-running": "^2.3.0" }, "peerDependencies": { "@edge-runtime/vm": "*", "@types/debug": "^4.1.12", "@types/node": "^18.0.0 || ^20.0.0 || >=22.0.0", "@vitest/browser": "3.2.6", "@vitest/ui": "3.2.6", "happy-dom": "*", "jsdom": "*" }, "optionalPeers": ["@edge-runtime/vm", "@types/debug", "@types/node", "@vitest/browser", "@vitest/ui", "happy-dom", "jsdom"], "bin": { "vitest": "vitest.mjs" } }, "sha512-xejya+bT/j/+R/AGa1XOfRxLmNUlLtlwjRsFUILF+xHfzElmGcmFydy2gqqIrd62ptIEfwVMofd19uNWD9L7Nw=="], + "vitest": ["vitest@4.1.11", "", { "dependencies": { "@vitest/expect": "4.1.11", "@vitest/mocker": "4.1.11", "@vitest/pretty-format": "4.1.11", "@vitest/runner": "4.1.11", "@vitest/snapshot": "4.1.11", "@vitest/spy": "4.1.11", "@vitest/utils": "4.1.11", "es-module-lexer": "^2.0.0", "expect-type": "^1.3.0", "magic-string": "^0.30.21", "obug": "^2.1.1", "pathe": "^2.0.3", "picomatch": "^4.0.3", "std-env": "^4.0.0-rc.1", "tinybench": "^2.9.0", "tinyexec": "^1.0.2", "tinyglobby": "^0.2.15", "tinyrainbow": "^3.1.0", "vite": "^6.0.0 || ^7.0.0 || ^8.0.0", "why-is-node-running": "^2.3.0" }, "peerDependencies": { "@edge-runtime/vm": "*", "@opentelemetry/api": "^1.9.0", "@types/node": "^20.0.0 || ^22.0.0 || >=24.0.0", "@vitest/browser-playwright": "4.1.11", "@vitest/browser-preview": "4.1.11", "@vitest/browser-webdriverio": "4.1.11", "@vitest/coverage-istanbul": "4.1.11", "@vitest/coverage-v8": "4.1.11", "@vitest/ui": "4.1.11", "happy-dom": "*", "jsdom": "*" }, "optionalPeers": ["@edge-runtime/vm", "@opentelemetry/api", "@types/node", "@vitest/browser-playwright", "@vitest/browser-preview", "@vitest/browser-webdriverio", "@vitest/coverage-istanbul", "@vitest/coverage-v8", "@vitest/ui", "happy-dom", "jsdom"], "bin": { "vitest": "./vitest.mjs" } }, "sha512-fhACrNXUidIbGSBr5FlbuBkO7VWC1ZyLl0DO4CU2DrQoAPxX84Ysxs+HeGQpii5lZWV1Q4gBZTTu49mF+A6Edw=="], "why-is-node-running": ["why-is-node-running@2.3.0", "", { "dependencies": { "siginfo": "^2.0.0", "stackback": "0.0.2" }, "bin": { "why-is-node-running": "cli.js" } }, "sha512-hUrmaWBdVDcxvYqnyh09zunKzROWjbZTiNy8dBEjkS7ehEDQibXJ7XvlmtbwuTclUiIyN+CyXQD4Vmko8fNm8w=="], @@ -333,40 +289,26 @@ "zod": ["zod@3.25.76", "", {}, "sha512-gzUt/qt81nXsFGKIFcC3YnfEAx5NkunCfnDlvuBSSFS02bcXu4Lmea0AFIUwbLWxWPx3d9p8S5QoaujKcNQxcQ=="], - "@vitest/expect/@vitest/utils": ["@vitest/utils@3.2.6", "", { "dependencies": { "@vitest/pretty-format": "3.2.6", "loupe": "^3.1.4", "tinyrainbow": "^2.0.0" } }, "sha512-lI23nIs4bnT3T8NIoh+vFaz5s2/DdP0Jgt2jxwgWljvwn82cLJtyi/If+fjFyoLMGIOz0U/fKvWE0d4jsNQEfg=="], + "@vitest/expect/@vitest/utils": ["@vitest/utils@4.1.11", "", { "dependencies": { "@vitest/pretty-format": "4.1.11", "convert-source-map": "^2.0.0", "tinyrainbow": "^3.1.0" } }, "sha512-zTCVGpyFsGWBhllOyKlTw/vnr6D9qxsfSDyfbyZmTyjHw5N/VuvzHpHoQjm2ZJzn4RJgx5w4r7V0er69CmLgPQ=="], - "@vitest/runner/@vitest/utils": ["@vitest/utils@3.2.6", "", { "dependencies": { "@vitest/pretty-format": "3.2.6", "loupe": "^3.1.4", "tinyrainbow": "^2.0.0" } }, "sha512-lI23nIs4bnT3T8NIoh+vFaz5s2/DdP0Jgt2jxwgWljvwn82cLJtyi/If+fjFyoLMGIOz0U/fKvWE0d4jsNQEfg=="], + "@vitest/runner/@vitest/utils": ["@vitest/utils@4.1.11", "", { "dependencies": { "@vitest/pretty-format": "4.1.11", "convert-source-map": "^2.0.0", "tinyrainbow": "^3.1.0" } }, "sha512-zTCVGpyFsGWBhllOyKlTw/vnr6D9qxsfSDyfbyZmTyjHw5N/VuvzHpHoQjm2ZJzn4RJgx5w4r7V0er69CmLgPQ=="], "@vitest/runner/pathe": ["pathe@2.0.3", "", {}, "sha512-WUjGcAqP1gQacoQe+OBJsFA7Ld4DyXuUIjZ5cc75cLHvJ7dtNsTugphxIADwspS+AraAUePCKrSVtPLFj/F88w=="], - "@vitest/snapshot/@vitest/pretty-format": ["@vitest/pretty-format@3.2.6", "", { "dependencies": { "tinyrainbow": "^2.0.0" } }, "sha512-lb7XXXzmm2h2ASzFnRvQpDo6onT1NmMJA3tkGTWiBFtRJ9lxGY3d3mm/Apt36gej2bkkOVLL/yTOtufDaFa/jA=="], + "@vitest/snapshot/@vitest/utils": ["@vitest/utils@4.1.11", "", { "dependencies": { "@vitest/pretty-format": "4.1.11", "convert-source-map": "^2.0.0", "tinyrainbow": "^3.1.0" } }, "sha512-zTCVGpyFsGWBhllOyKlTw/vnr6D9qxsfSDyfbyZmTyjHw5N/VuvzHpHoQjm2ZJzn4RJgx5w4r7V0er69CmLgPQ=="], "@vitest/snapshot/pathe": ["pathe@2.0.3", "", {}, "sha512-WUjGcAqP1gQacoQe+OBJsFA7Ld4DyXuUIjZ5cc75cLHvJ7dtNsTugphxIADwspS+AraAUePCKrSVtPLFj/F88w=="], "bun-types/@types/node": ["@types/node@22.7.5", "", { "dependencies": { "undici-types": "~6.19.2" } }, "sha512-jML7s2NAzMWc//QSJ1a3prpk78cOPchGvXJsC3C6R6PSMoooztvRVQEz89gmBTBY1SPMaqo5teB4uNHPdetShQ=="], - "chai/loupe": ["loupe@3.2.1", "", {}, "sha512-CdzqowRJCeLU72bHvWqwRBBlLcMEtIvGrlvef74kMnV2AolS9Y8xUv1I0U/MNAWMhBlKIoyuEgoJ0t/bbwHbLQ=="], - "micromatch/picomatch": ["picomatch@2.3.1", "", {}, "sha512-JU3teHTNjmE2VCGFzuY8EXzCDVwEqB2a8fsIvwaStHhAWJEeVd1o1QD80CU6+ZdEXXSLbSsuLwJjkCBWqRQUVA=="], - "vite-node/pathe": ["pathe@2.0.3", "", {}, "sha512-WUjGcAqP1gQacoQe+OBJsFA7Ld4DyXuUIjZ5cc75cLHvJ7dtNsTugphxIADwspS+AraAUePCKrSVtPLFj/F88w=="], + "vite/picomatch": ["picomatch@4.0.7", "", {}, "sha512-qcJu88Q2IWqJsDD529JKMdwGm/dvInW4HvQnRwiH9JtihJvzGOscDtHE3x1pBKeUOTysQ8kVmLnJ2kJu7yhcGA=="], - "vitest/@vitest/utils": ["@vitest/utils@3.2.6", "", { "dependencies": { "@vitest/pretty-format": "3.2.6", "loupe": "^3.1.4", "tinyrainbow": "^2.0.0" } }, "sha512-lI23nIs4bnT3T8NIoh+vFaz5s2/DdP0Jgt2jxwgWljvwn82cLJtyi/If+fjFyoLMGIOz0U/fKvWE0d4jsNQEfg=="], + "vitest/@vitest/utils": ["@vitest/utils@4.1.11", "", { "dependencies": { "@vitest/pretty-format": "4.1.11", "convert-source-map": "^2.0.0", "tinyrainbow": "^3.1.0" } }, "sha512-zTCVGpyFsGWBhllOyKlTw/vnr6D9qxsfSDyfbyZmTyjHw5N/VuvzHpHoQjm2ZJzn4RJgx5w4r7V0er69CmLgPQ=="], "vitest/pathe": ["pathe@2.0.3", "", {}, "sha512-WUjGcAqP1gQacoQe+OBJsFA7Ld4DyXuUIjZ5cc75cLHvJ7dtNsTugphxIADwspS+AraAUePCKrSVtPLFj/F88w=="], - "@vitest/expect/@vitest/utils/@vitest/pretty-format": ["@vitest/pretty-format@3.2.6", "", { "dependencies": { "tinyrainbow": "^2.0.0" } }, "sha512-lb7XXXzmm2h2ASzFnRvQpDo6onT1NmMJA3tkGTWiBFtRJ9lxGY3d3mm/Apt36gej2bkkOVLL/yTOtufDaFa/jA=="], - - "@vitest/expect/@vitest/utils/loupe": ["loupe@3.2.1", "", {}, "sha512-CdzqowRJCeLU72bHvWqwRBBlLcMEtIvGrlvef74kMnV2AolS9Y8xUv1I0U/MNAWMhBlKIoyuEgoJ0t/bbwHbLQ=="], - - "@vitest/runner/@vitest/utils/@vitest/pretty-format": ["@vitest/pretty-format@3.2.6", "", { "dependencies": { "tinyrainbow": "^2.0.0" } }, "sha512-lb7XXXzmm2h2ASzFnRvQpDo6onT1NmMJA3tkGTWiBFtRJ9lxGY3d3mm/Apt36gej2bkkOVLL/yTOtufDaFa/jA=="], - - "@vitest/runner/@vitest/utils/loupe": ["loupe@3.2.1", "", {}, "sha512-CdzqowRJCeLU72bHvWqwRBBlLcMEtIvGrlvef74kMnV2AolS9Y8xUv1I0U/MNAWMhBlKIoyuEgoJ0t/bbwHbLQ=="], - "bun-types/@types/node/undici-types": ["undici-types@6.19.8", "", {}, "sha512-ve2KP6f/JnbPBFyobGHuerC9g1FYGn/F8n1LWTwNxCEzd6IfqTwUQcNXgEtmmQ6DlRrC1hrSrBnCZPokRrDHjw=="], - - "vitest/@vitest/utils/@vitest/pretty-format": ["@vitest/pretty-format@3.2.6", "", { "dependencies": { "tinyrainbow": "^2.0.0" } }, "sha512-lb7XXXzmm2h2ASzFnRvQpDo6onT1NmMJA3tkGTWiBFtRJ9lxGY3d3mm/Apt36gej2bkkOVLL/yTOtufDaFa/jA=="], - - "vitest/@vitest/utils/loupe": ["loupe@3.2.1", "", {}, "sha512-CdzqowRJCeLU72bHvWqwRBBlLcMEtIvGrlvef74kMnV2AolS9Y8xUv1I0U/MNAWMhBlKIoyuEgoJ0t/bbwHbLQ=="], } } diff --git a/dstack/kms/auth-mock/bun.lock b/dstack/kms/auth-mock/bun.lock index 35fa3ece6..955a0dda3 100644 --- a/dstack/kms/auth-mock/bun.lock +++ b/dstack/kms/auth-mock/bun.lock @@ -1,12 +1,11 @@ { "lockfileVersion": 1, - "configVersion": 0, "workspaces": { "": { "name": "auth-mock", "dependencies": { "@hono/zod-validator": "0.2.2", - "hono": "4.12.34", + "hono": "4.13.5", "zod": "3.25.76", }, "devDependencies": { @@ -16,7 +15,7 @@ "openapi-types": "12.1.3", "oxlint": "0.9.10", "typescript": "5.8.3", - "vitest": "4.1.0", + "vitest": "4.1.11", }, }, }, @@ -111,17 +110,17 @@ "@types/react": ["@types/react@19.1.8", "", { "dependencies": { "csstype": "^3.0.2" } }, "sha512-AwAfQ2Wa5bCx9WP8nZL2uMZWod7J7/JSplxbTmBQ5ms6QpqNYm672H0Vu9ZVKVngQ+ii4R/byguVEUZQyeg44g=="], - "@vitest/expect": ["@vitest/expect@4.1.0", "", { "dependencies": { "@standard-schema/spec": "^1.1.0", "@types/chai": "^5.2.2", "@vitest/spy": "4.1.0", "@vitest/utils": "4.1.0", "chai": "^6.2.2", "tinyrainbow": "^3.0.3" } }, "sha512-EIxG7k4wlWweuCLG9Y5InKFwpMEOyrMb6ZJ1ihYu02LVj/bzUwn2VMU+13PinsjRW75XnITeFrQBMH5+dLvCDA=="], + "@vitest/expect": ["@vitest/expect@4.1.11", "", { "dependencies": { "@standard-schema/spec": "^1.1.0", "@types/chai": "^5.2.2", "@vitest/spy": "4.1.11", "@vitest/utils": "4.1.11", "chai": "^6.2.2", "tinyrainbow": "^3.1.0" } }, "sha512-VX2x5vNJXET47KAFzwERI+KRMtTTCSWTfSMKsW7JsUsXV4psq++e3DvZpuTDOpHcxytiDs6p2nhVb2tVDiiUYw=="], - "@vitest/mocker": ["@vitest/mocker@4.1.0", "", { "dependencies": { "@vitest/spy": "4.1.0", "estree-walker": "^3.0.3", "magic-string": "^0.30.21" }, "peerDependencies": { "msw": "^2.4.9", "vite": "^6.0.0 || ^7.0.0 || ^8.0.0-0" }, "optionalPeers": ["msw", "vite"] }, "sha512-evxREh+Hork43+Y4IOhTo+h5lGmVRyjqI739Rz4RlUPqwrkFFDF6EMvOOYjTx4E8Tl6gyCLRL8Mu7Ry12a13Tw=="], + "@vitest/mocker": ["@vitest/mocker@4.1.11", "", { "dependencies": { "@vitest/spy": "4.1.11", "estree-walker": "^3.0.3", "magic-string": "^0.30.21" }, "peerDependencies": { "msw": "^2.4.9", "vite": "^6.0.0 || ^7.0.0 || ^8.0.0" }, "optionalPeers": ["msw", "vite"] }, "sha512-2XJVD55d1o5AZous5CCGKS74g/riOj9odEt2bQpCVZeblHyHdnMeFl4jl0XjU21stf4mbjUkew2eXQZt65g5CQ=="], - "@vitest/pretty-format": ["@vitest/pretty-format@4.1.0", "", { "dependencies": { "tinyrainbow": "^3.0.3" } }, "sha512-3RZLZlh88Ib0J7NQTRATfc/3ZPOnSUn2uDBUoGNn5T36+bALixmzphN26OUD3LRXWkJu4H0s5vvUeqBiw+kS0A=="], + "@vitest/pretty-format": ["@vitest/pretty-format@4.1.11", "", { "dependencies": { "tinyrainbow": "^3.1.0" } }, "sha512-yiZzPbGTS9Sr/JpFl8zHrcIkAofNbFV6k21vIgQN/cY/oxZeXhJv5sc/MBJ5jFKWmWs+oJHw0UXLZjmf931+Vw=="], - "@vitest/runner": ["@vitest/runner@4.1.0", "", { "dependencies": { "@vitest/utils": "4.1.0", "pathe": "^2.0.3" } }, "sha512-Duvx2OzQ7d6OjchL+trw+aSrb9idh7pnNfxrklo14p3zmNL4qPCDeIJAK+eBKYjkIwG96Bc6vYuxhqDXQOWpoQ=="], + "@vitest/runner": ["@vitest/runner@4.1.11", "", { "dependencies": { "@vitest/utils": "4.1.11", "pathe": "^2.0.3" } }, "sha512-LztvUgdwMNJMIkj3hQnnxiC2Xy1zNxq928W/xhjCLaNCzqTZOudjwbQf6v9IntZGPw132i2Lq2rgTRZHD3JHNw=="], - "@vitest/snapshot": ["@vitest/snapshot@4.1.0", "", { "dependencies": { "@vitest/pretty-format": "4.1.0", "@vitest/utils": "4.1.0", "magic-string": "^0.30.21", "pathe": "^2.0.3" } }, "sha512-0Vy9euT1kgsnj1CHttwi9i9o+4rRLEaPRSOJ5gyv579GJkNpgJK+B4HSv/rAWixx2wdAFci1X4CEPjiu2bXIMg=="], + "@vitest/snapshot": ["@vitest/snapshot@4.1.11", "", { "dependencies": { "@vitest/pretty-format": "4.1.11", "@vitest/utils": "4.1.11", "magic-string": "^0.30.21", "pathe": "^2.0.3" } }, "sha512-pN7ikn1ON7h8ee4gIAp4AzyK+zBtJPzVbqOgu5LCEh4VaJVbPQcgYQYJIMGQPXVeJJq1fnfazis7a5pFNPahog=="], - "@vitest/spy": ["@vitest/spy@4.1.0", "", {}, "sha512-pz77k+PgNpyMDv2FV6qmk5ZVau6c3R8HC8v342T2xlFxQKTrSeYw9waIJG8KgV9fFwAtTu4ceRzMivPTH6wSxw=="], + "@vitest/spy": ["@vitest/spy@4.1.11", "", {}, "sha512-apNa/prQy2qCeywhnixOHPRCgGNhvg7T4Dapfl1GahLp/R+uhBm5cPyFoNVyqsNd2h1nJxL6BqqdIjiABL60YA=="], "@vitest/ui": ["@vitest/ui@1.6.1", "", { "dependencies": { "@vitest/utils": "1.6.1", "fast-glob": "^3.3.2", "fflate": "^0.8.1", "flatted": "^3.2.9", "pathe": "^1.1.1", "picocolors": "^1.0.0", "sirv": "^2.0.4" }, "peerDependencies": { "vitest": "1.6.1" } }, "sha512-xa57bCPGuzEFqGjPs3vVLyqareG8DX0uMkr5U/v5vLv5/ZUrBrPL7gzxzTJedEyZxFMfsozwTIbbYfEQVo3kgg=="], @@ -169,7 +168,7 @@ "glob-parent": ["glob-parent@5.1.2", "", { "dependencies": { "is-glob": "^4.0.1" } }, "sha512-AOIgSQCepiJYwP3ARnGx+5VnTu2HBYdzbGP45eLw1vr3zB3vZLeyed1sC9hnbcOc9/SrMyM5RPQrkGz4aS9Zow=="], - "hono": ["hono@4.12.34", "", {}, "sha512-GqXJqY/xJkJmuloTrnV1ZEXG3fqte+VjkUqoRNZXcrUidiUOP4fMSIHHY4tsqZBK++kVyWmt/AAfSUuy57/eSA=="], + "hono": ["hono@4.13.5", "", {}, "sha512-O6+/eCYRkzzzy0rPWwKLiGBR1nFuUPZynnwjxN1MBA62NNqbT0wQEzQyK2gSO5yDIDB336sXQleAhOHrzlYyKw=="], "is-extglob": ["is-extglob@2.1.1", "", {}, "sha512-SbKbANkN603Vi4jEZv49LeVJMn4yGwsbzZworEoyEiutsN3nJYdbO36zfhGJ6QEDpOZIFkDtnq5JRxmvl3jsoQ=="], @@ -269,19 +268,19 @@ "vite": ["vite@8.1.5", "", { "dependencies": { "lightningcss": "^1.32.0", "picomatch": "^4.0.5", "postcss": "^8.5.17", "rolldown": "~1.1.5", "tinyglobby": "^0.2.17" }, "optionalDependencies": { "fsevents": "~2.3.3" }, "peerDependencies": { "@types/node": "^20.19.0 || >=22.12.0", "@vitejs/devtools": "^0.3.0", "esbuild": "^0.27.0 || ^0.28.0", "jiti": ">=1.21.0", "less": "^4.0.0", "sass": "^1.70.0", "sass-embedded": "^1.70.0", "stylus": ">=0.54.8", "sugarss": "^5.0.0", "terser": "^5.16.0", "tsx": "^4.8.1", "yaml": "^2.4.2" }, "optionalPeers": ["@types/node", "@vitejs/devtools", "esbuild", "jiti", "less", "sass", "sass-embedded", "stylus", "sugarss", "terser", "tsx", "yaml"], "bin": { "vite": "bin/vite.js" } }, "sha512-7ULLwsCdYx/nRyrpiEwvqb5TFHrMVZyBt+rg/OAXT7rgj/z+DtTDyKFeLAdDkubDVDKD8jOsndmy7m55XcfUsw=="], - "vitest": ["vitest@4.1.0", "", { "dependencies": { "@vitest/expect": "4.1.0", "@vitest/mocker": "4.1.0", "@vitest/pretty-format": "4.1.0", "@vitest/runner": "4.1.0", "@vitest/snapshot": "4.1.0", "@vitest/spy": "4.1.0", "@vitest/utils": "4.1.0", "es-module-lexer": "^2.0.0", "expect-type": "^1.3.0", "magic-string": "^0.30.21", "obug": "^2.1.1", "pathe": "^2.0.3", "picomatch": "^4.0.3", "std-env": "^4.0.0-rc.1", "tinybench": "^2.9.0", "tinyexec": "^1.0.2", "tinyglobby": "^0.2.15", "tinyrainbow": "^3.0.3", "vite": "^6.0.0 || ^7.0.0 || ^8.0.0-0", "why-is-node-running": "^2.3.0" }, "peerDependencies": { "@edge-runtime/vm": "*", "@opentelemetry/api": "^1.9.0", "@types/node": "^20.0.0 || ^22.0.0 || >=24.0.0", "@vitest/browser-playwright": "4.1.0", "@vitest/browser-preview": "4.1.0", "@vitest/browser-webdriverio": "4.1.0", "@vitest/ui": "4.1.0", "happy-dom": "*", "jsdom": "*" }, "optionalPeers": ["@edge-runtime/vm", "@opentelemetry/api", "@types/node", "@vitest/browser-playwright", "@vitest/browser-preview", "@vitest/browser-webdriverio", "@vitest/ui", "happy-dom", "jsdom"], "bin": { "vitest": "vitest.mjs" } }, "sha512-YbDrMF9jM2Lqc++2530UourxZHmkKLxrs4+mYhEwqWS97WJ7wOYEkcr+QfRgJ3PW9wz3odRijLZjHEaRLTNbqw=="], + "vitest": ["vitest@4.1.11", "", { "dependencies": { "@vitest/expect": "4.1.11", "@vitest/mocker": "4.1.11", "@vitest/pretty-format": "4.1.11", "@vitest/runner": "4.1.11", "@vitest/snapshot": "4.1.11", "@vitest/spy": "4.1.11", "@vitest/utils": "4.1.11", "es-module-lexer": "^2.0.0", "expect-type": "^1.3.0", "magic-string": "^0.30.21", "obug": "^2.1.1", "pathe": "^2.0.3", "picomatch": "^4.0.3", "std-env": "^4.0.0-rc.1", "tinybench": "^2.9.0", "tinyexec": "^1.0.2", "tinyglobby": "^0.2.15", "tinyrainbow": "^3.1.0", "vite": "^6.0.0 || ^7.0.0 || ^8.0.0", "why-is-node-running": "^2.3.0" }, "peerDependencies": { "@edge-runtime/vm": "*", "@opentelemetry/api": "^1.9.0", "@types/node": "^20.0.0 || ^22.0.0 || >=24.0.0", "@vitest/browser-playwright": "4.1.11", "@vitest/browser-preview": "4.1.11", "@vitest/browser-webdriverio": "4.1.11", "@vitest/coverage-istanbul": "4.1.11", "@vitest/coverage-v8": "4.1.11", "@vitest/ui": "4.1.11", "happy-dom": "*", "jsdom": "*" }, "optionalPeers": ["@edge-runtime/vm", "@opentelemetry/api", "@types/node", "@vitest/browser-playwright", "@vitest/browser-preview", "@vitest/browser-webdriverio", "@vitest/coverage-istanbul", "@vitest/coverage-v8", "@vitest/ui", "happy-dom", "jsdom"], "bin": { "vitest": "./vitest.mjs" } }, "sha512-fhACrNXUidIbGSBr5FlbuBkO7VWC1ZyLl0DO4CU2DrQoAPxX84Ysxs+HeGQpii5lZWV1Q4gBZTTu49mF+A6Edw=="], "why-is-node-running": ["why-is-node-running@2.3.0", "", { "dependencies": { "siginfo": "^2.0.0", "stackback": "0.0.2" }, "bin": { "why-is-node-running": "cli.js" } }, "sha512-hUrmaWBdVDcxvYqnyh09zunKzROWjbZTiNy8dBEjkS7ehEDQibXJ7XvlmtbwuTclUiIyN+CyXQD4Vmko8fNm8w=="], "zod": ["zod@3.25.76", "", {}, "sha512-gzUt/qt81nXsFGKIFcC3YnfEAx5NkunCfnDlvuBSSFS02bcXu4Lmea0AFIUwbLWxWPx3d9p8S5QoaujKcNQxcQ=="], - "@vitest/expect/@vitest/utils": ["@vitest/utils@4.1.0", "", { "dependencies": { "@vitest/pretty-format": "4.1.0", "convert-source-map": "^2.0.0", "tinyrainbow": "^3.0.3" } }, "sha512-XfPXT6a8TZY3dcGY8EdwsBulFCIw+BeeX0RZn2x/BtiY/75YGh8FeWGG8QISN/WhaqSrE2OrlDgtF8q5uhOTmw=="], + "@vitest/expect/@vitest/utils": ["@vitest/utils@4.1.11", "", { "dependencies": { "@vitest/pretty-format": "4.1.11", "convert-source-map": "^2.0.0", "tinyrainbow": "^3.1.0" } }, "sha512-zTCVGpyFsGWBhllOyKlTw/vnr6D9qxsfSDyfbyZmTyjHw5N/VuvzHpHoQjm2ZJzn4RJgx5w4r7V0er69CmLgPQ=="], - "@vitest/runner/@vitest/utils": ["@vitest/utils@4.1.0", "", { "dependencies": { "@vitest/pretty-format": "4.1.0", "convert-source-map": "^2.0.0", "tinyrainbow": "^3.0.3" } }, "sha512-XfPXT6a8TZY3dcGY8EdwsBulFCIw+BeeX0RZn2x/BtiY/75YGh8FeWGG8QISN/WhaqSrE2OrlDgtF8q5uhOTmw=="], + "@vitest/runner/@vitest/utils": ["@vitest/utils@4.1.11", "", { "dependencies": { "@vitest/pretty-format": "4.1.11", "convert-source-map": "^2.0.0", "tinyrainbow": "^3.1.0" } }, "sha512-zTCVGpyFsGWBhllOyKlTw/vnr6D9qxsfSDyfbyZmTyjHw5N/VuvzHpHoQjm2ZJzn4RJgx5w4r7V0er69CmLgPQ=="], "@vitest/runner/pathe": ["pathe@2.0.3", "", {}, "sha512-WUjGcAqP1gQacoQe+OBJsFA7Ld4DyXuUIjZ5cc75cLHvJ7dtNsTugphxIADwspS+AraAUePCKrSVtPLFj/F88w=="], - "@vitest/snapshot/@vitest/utils": ["@vitest/utils@4.1.0", "", { "dependencies": { "@vitest/pretty-format": "4.1.0", "convert-source-map": "^2.0.0", "tinyrainbow": "^3.0.3" } }, "sha512-XfPXT6a8TZY3dcGY8EdwsBulFCIw+BeeX0RZn2x/BtiY/75YGh8FeWGG8QISN/WhaqSrE2OrlDgtF8q5uhOTmw=="], + "@vitest/snapshot/@vitest/utils": ["@vitest/utils@4.1.11", "", { "dependencies": { "@vitest/pretty-format": "4.1.11", "convert-source-map": "^2.0.0", "tinyrainbow": "^3.1.0" } }, "sha512-zTCVGpyFsGWBhllOyKlTw/vnr6D9qxsfSDyfbyZmTyjHw5N/VuvzHpHoQjm2ZJzn4RJgx5w4r7V0er69CmLgPQ=="], "@vitest/snapshot/pathe": ["pathe@2.0.3", "", {}, "sha512-WUjGcAqP1gQacoQe+OBJsFA7Ld4DyXuUIjZ5cc75cLHvJ7dtNsTugphxIADwspS+AraAUePCKrSVtPLFj/F88w=="], @@ -289,7 +288,7 @@ "micromatch/picomatch": ["picomatch@2.3.1", "", {}, "sha512-JU3teHTNjmE2VCGFzuY8EXzCDVwEqB2a8fsIvwaStHhAWJEeVd1o1QD80CU6+ZdEXXSLbSsuLwJjkCBWqRQUVA=="], - "vitest/@vitest/utils": ["@vitest/utils@4.1.0", "", { "dependencies": { "@vitest/pretty-format": "4.1.0", "convert-source-map": "^2.0.0", "tinyrainbow": "^3.0.3" } }, "sha512-XfPXT6a8TZY3dcGY8EdwsBulFCIw+BeeX0RZn2x/BtiY/75YGh8FeWGG8QISN/WhaqSrE2OrlDgtF8q5uhOTmw=="], + "vitest/@vitest/utils": ["@vitest/utils@4.1.11", "", { "dependencies": { "@vitest/pretty-format": "4.1.11", "convert-source-map": "^2.0.0", "tinyrainbow": "^3.1.0" } }, "sha512-zTCVGpyFsGWBhllOyKlTw/vnr6D9qxsfSDyfbyZmTyjHw5N/VuvzHpHoQjm2ZJzn4RJgx5w4r7V0er69CmLgPQ=="], "vitest/pathe": ["pathe@2.0.3", "", {}, "sha512-WUjGcAqP1gQacoQe+OBJsFA7Ld4DyXuUIjZ5cc75cLHvJ7dtNsTugphxIADwspS+AraAUePCKrSVtPLFj/F88w=="], diff --git a/dstack/kms/auth-simple/bun.lock b/dstack/kms/auth-simple/bun.lock index dc65b4382..211959aa6 100644 --- a/dstack/kms/auth-simple/bun.lock +++ b/dstack/kms/auth-simple/bun.lock @@ -5,7 +5,7 @@ "name": "auth-simple", "dependencies": { "@hono/zod-validator": "0.2.2", - "hono": "4.12.34", + "hono": "4.13.5", "zod": "3.25.76", }, "devDependencies": { @@ -218,7 +218,7 @@ "glob-parent": ["glob-parent@5.1.2", "", { "dependencies": { "is-glob": "^4.0.1" } }, "sha512-AOIgSQCepiJYwP3ARnGx+5VnTu2HBYdzbGP45eLw1vr3zB3vZLeyed1sC9hnbcOc9/SrMyM5RPQrkGz4aS9Zow=="], - "hono": ["hono@4.12.34", "", {}, "sha512-GqXJqY/xJkJmuloTrnV1ZEXG3fqte+VjkUqoRNZXcrUidiUOP4fMSIHHY4tsqZBK++kVyWmt/AAfSUuy57/eSA=="], + "hono": ["hono@4.13.5", "", {}, "sha512-O6+/eCYRkzzzy0rPWwKLiGBR1nFuUPZynnwjxN1MBA62NNqbT0wQEzQyK2gSO5yDIDB336sXQleAhOHrzlYyKw=="], "is-extglob": ["is-extglob@2.1.1", "", {}, "sha512-SbKbANkN603Vi4jEZv49LeVJMn4yGwsbzZworEoyEiutsN3nJYdbO36zfhGJ6QEDpOZIFkDtnq5JRxmvl3jsoQ=="], From 3f451c3c963deaf62630038f9a6b92b68cb725f3 Mon Sep 17 00:00:00 2001 From: Kevin Wang Date: Thu, 17 Sep 2026 23:59:34 -0700 Subject: [PATCH 35/93] fix(os): normalize the kernel setup header to QEMU's layout, not zeros Signed-off-by: Kevin Wang --- docs/security/security-model.md | 28 ++- os/image/README.md | 31 ++- os/image/assemble.sh | 5 +- os/image/normalize-kernel-header.py | 167 +++++++++---- os/tests/test-kernel-header-normalization.sh | 126 ++++++---- ...elLoaderFsDxe-normalize-setup-header.patch | 227 +++++++++++++----- 6 files changed, 419 insertions(+), 165 deletions(-) diff --git a/docs/security/security-model.md b/docs/security/security-model.md index 751dbdc79..9381a8cbf 100644 --- a/docs/security/security-model.md +++ b/docs/security/security-model.md @@ -326,12 +326,20 @@ kernel header for CoCo VMs", first released in 10.2.0) stopped rewriting the header for confidential guests, so the same kernel would otherwise measure differently depending on which QEMU the host chose to run. -dstack removes that dependency instead of modelling it. The image build zeroes -the boot-loader-written fields in the kernel it ships, and dstack's OVMF zeroes -them again before the kernel blob is measured and loaded. RTMR[1] is therefore -the plain Authenticode hash of the `bzImage` listed in `sha256sum.txt`, and the -verifier needs nothing from the host to predict it -- not a QEMU version, not a -memory size. +dstack removes that dependency instead of modelling it. The image build writes +the boot-loader-written fields in the kernel it ships, with the values QEMU +<= 10.1 writes, and dstack's OVMF writes the same values again before the +kernel blob is measured and loaded. RTMR[1] is therefore the plain Authenticode +hash of the `bzImage` listed in `sha256sum.txt`, and the verifier needs nothing +from the host to predict it -- not a QEMU version, not a memory size. + +Normalizing to QEMU's layout rather than to zeros keeps an earlier release able +to verify these images: `dstack-mr` computes that layout for an image that does +not declare the flag, so a KMS that predates the declaration can still verify a +guest booted on a QEMU that no longer patches the header, and a root-key +handover does not have to bypass image verification. Recomputing it needs the +guest RAM size, which limits such a verifier to guests with exactly 2 GiB or at +least 2816 MiB. Images built before this landed keep their original behavior: their firmware does not normalize, so their digest still covers QEMU's rewritten copy. Which @@ -351,10 +359,10 @@ deployments, since the previous QEMU-patched digest varied with guest RAM and was only reproducible at specific memory sizes. The normalized field set comes from the boot protocol rather than from QEMU's -behavior: every field `Documentation/arch/x86/boot.rst` types as `write` is one -the boot loader fills in and the kernel supplies no value for, so zeroing it -discards nothing the kernel provided. Fields typed `modify` carry real -kernel-supplied values and are left measured. +behavior: every field written is one `Documentation/arch/x86/boot.rst` types as +`write`, which the boot loader fills in and the kernel supplies no value for. +Fields typed `modify` carry real kernel-supplied values and are left as the +kernel built them. ### TCB status is surfaced, not gated, during verification diff --git a/os/image/README.md b/os/image/README.md index 4084af001..723659e31 100644 --- a/os/image/README.md +++ b/os/image/README.md @@ -71,14 +71,25 @@ host's QEMU version — and the host is the one that declares that version. The fix has two halves that must stay in sync: -- this script zeroes those fields in the kernel we ship; -- `0007-OvmfPkg-QemuKernelLoaderFsDxe-normalize-setup-header.patch` zeroes them - again in OVMF, before the kernel blob is measured and loaded. +- this script writes those fields in the kernel we ship, with the values + QEMU <= 10.1 writes; +- `0007-OvmfPkg-QemuKernelLoaderFsDxe-normalize-setup-header.patch` writes the + same values in OVMF, before the kernel blob is measured and loaded. The result is that RTMR[1] is the plain Authenticode hash of `bzImage` as listed in `sha256sum.txt`, on every QEMU version and at every guest memory size. +Normalizing to QEMU's layout rather than to zeros is what keeps an earlier +release able to verify these images: `dstack-mr` already computes that layout +(`patch_kernel`) for an image that does not declare the flag, so a 0.5.x KMS +can still onboard a 0.6.0 root with image verification on. Recomputing it needs +the guest RAM size, which limits such a verifier to guests with exactly 2 GiB +or at least 2816 MiB -- the one range QEMU places the initrd differently in, +and the range the no-image-download path already excludes. The values are the +ones QEMU writes above that threshold, and the shipped initrd's size is an +input, since it decides `ramdisk_image`. + `assemble.sh` records this in `metadata.json` as `"kernel_header_normalized": true`, and re-runs the script with `--check` first so the build fails rather than shipping a kernel that disagrees with what the image declares. Images @@ -91,16 +102,14 @@ fills in and the kernel supplies no value for. Fields typed `modify` carry real kernel-supplied values — `code32_start` is the protected-mode entry point — and are deliberately left alone. -In practice this rewrites **two bytes**: `heap_end_ptr` (0x224) is the only -`write` field a built kernel leaves non-zero. That is safe for every boot path: -the boot protocol types it `write (obligatory)`, `init_heap()` reads it only -when the boot loader has set `CAN_USE_HEAP` (which the kernel builds clear and -this script also clears), and on the EFI-stub path the real-mode setup code -never runs at all. The PE headers sit at 0x40..0x170, so no setup-header field -overlaps them and the EFI entry point is untouched. +That is safe for every boot path: these are the values QEMU itself wrote for +every release before 10.2, and on the EFI-stub path the real-mode setup code +never runs at all -- the stub takes the initrd through LoadFile2 and the +command line through its load options. The PE headers sit at 0x40..0x170, so no +setup-header field overlaps them and the EFI entry point is untouched. To check an image without modifying it: ```bash -./normalize-kernel-header.py --check /path/to/bzImage +./normalize-kernel-header.py --check /path/to/bzImage /path/to/initramfs.cpio.gz ``` diff --git a/os/image/assemble.sh b/os/image/assemble.sh index 554e54b0c..aa7e11b34 100755 --- a/os/image/assemble.sh +++ b/os/image/assemble.sh @@ -420,7 +420,8 @@ verbose cp "$KERNEL_IMAGE" "${OUTPUT_DIR}/bzImage" # result into RTMR[1]. QEMU >= 10.2 stopped doing that for confidential guests, # so leaving the header as built would make the same image measure differently # per QEMU version. Normalizing here, and again in OVMF before it measures, -# makes RTMR[1] the plain Authenticode hash of this file. Runs before +# makes RTMR[1] the plain Authenticode hash of this file. The initrd size is an +# input, so this runs once both files are in place, and before # tdx-measurement-cbor and sha256sum.txt below, so both cover the normalized # kernel. The matching half is in OVMF: metadata.json declares # kernel_header_normalized below, and what makes that declaration true is @@ -428,7 +429,7 @@ verbose cp "$KERNEL_IMAGE" "${OUTPUT_DIR}/bzImage" # same build applies -- ovmf-build.sh and the bitbake recipe both fail if it # does not apply. See os/image/README.md. verbose "$(dirname "${BASH_SOURCE[0]}")/normalize-kernel-header.py" \ - "${OUTPUT_DIR}/bzImage" + "${OUTPUT_DIR}/bzImage" "${OUTPUT_DIR}/initramfs.cpio.gz" verbose cp "$OVMF_FIRMWARE" "${OUTPUT_DIR}/ovmf.fd" # AMD SEV firmware (additive). Shipped alongside the TDX firmware so a SEV-SNP diff --git a/os/image/normalize-kernel-header.py b/os/image/normalize-kernel-header.py index d628e5f4f..b2682f61a 100755 --- a/os/image/normalize-kernel-header.py +++ b/os/image/normalize-kernel-header.py @@ -9,39 +9,60 @@ whether QEMU rewrote the header -- which changed in QEMU 10.2 (commit a7542a38f399, "x86/loader: Don't update kernel header for CoCo VMs"). -Zeroing those fields in the shipped kernel, and having OVMF zero them again +Writing those fields in the shipped kernel, and having OVMF write them again before measuring, makes RTMR[1] the plain Authenticode hash of the file we ship, on every QEMU version. -The field set comes from the boot protocol, not from QEMU: every field -`Documentation/arch/x86/boot.rst` types as `write` is one the boot loader -fills in and the kernel supplies no value for. Fields typed `modify` carry -real kernel-supplied values (`code32_start` is the protected-mode entry point) -and are left alone. - -In a freshly built kernel every one of these fields is already zero except -`heap_end_ptr`, so this normally rewrites exactly two bytes. +The values are the ones QEMU <= 10.1 writes, for a guest with 2 GiB or more of +RAM below 4G -- the only guest memory layout modelled, and the one `dstack-mr` +computes (`patch_kernel` in dstack/dstack-mr/src/kernel.rs) for an image that +does not declare `kernel_header_normalized`. Normalizing to QEMU's layout +rather than to zeros is what keeps a verifier or KMS from an earlier release +able to measure these images. + +The field set comes from the boot protocol, not from QEMU: every field written +is one `Documentation/arch/x86/boot.rst` types as `write`, which the boot +loader fills in and the kernel supplies no value for. Fields typed `modify` +carry real kernel-supplied values (`code32_start` is the protected-mode entry +point) and are left alone. Fields QEMU leaves alone are left alone too: a built +kernel ships them zero, so writing them would only add a way to disagree. + +The initrd size decides `ramdisk_image`, so the initrd the image ships is an +input. """ import argparse import sys -# Offset, size, name -- every boot.rst field typed `write`. -WRITE_FIELDS = [ - (0x210, 1, "type_of_loader"), - (0x218, 4, "ramdisk_image"), - (0x21C, 4, "ramdisk_size"), - (0x224, 2, "heap_end_ptr"), - (0x226, 1, "ext_loader_ver"), - (0x227, 1, "ext_loader_type"), - (0x228, 4, "cmd_line_ptr"), - (0x23C, 4, "hardware_subarch"), - (0x240, 8, "hardware_subarch_data"), - (0x250, 8, "setup_data"), -] +# Offsets of the boot.rst `write` fields QEMU fills in. +TYPE_OF_LOADER_OFFSET = 0x210 +RAMDISK_IMAGE_OFFSET = 0x218 +RAMDISK_SIZE_OFFSET = 0x21C +HEAP_END_PTR_OFFSET = 0x224 +CMD_LINE_PTR_OFFSET = 0x228 +INITRD_ADDR_MAX_OFFSET = 0x22C +XLOADFLAGS_OFFSET = 0x236 LOADFLAGS_OFFSET = 0x211 +LOADED_HIGH = 0x01 CAN_USE_HEAP = 0x80 +XLF_CAN_BE_LOADED_ABOVE_4G = 0x40 + +# What QEMU writes: "Qemu" version 0, the real-mode block and command line it +# loads a kernel at, the 0x200-byte gap it leaves below the command line for +# the setup heap, the below-4G window it reserves for ACPI tables with RAM +# split at 2 GiB, the initrd ceiling a kernel that declares none gets, and the +# alignment it rounds the initrd address down to. +TYPE_OF_LOADER_QEMU = 0xB0 +REAL_ADDR_HIGH = 0x10000 +CMDLINE_ADDR_HIGH = 0x20000 +REAL_ADDR_LOW = 0x90000 +CMDLINE_ADDR_LOW = 0x9A000 +SETUP_HEAP_GAP = 0x200 +LOW_MEMORY_SPLIT = 0x80000000 +ACPI_DATA_SIZE = 0x28000 +INITRD_ADDR_MAX_DEFAULT = 0x37FFFFFF +INITRD_ALIGNMENT = 0x1000 HEADER_MAGIC_OFFSET = 0x202 HEADER_MAGIC = b"HdrS" @@ -50,12 +71,15 @@ # header, and slicing past the end would silently grow the image instead of # failing. The OVMF side applies the same bound. HEADER_END_OFFSET = 0x258 -# `setup_data` (0x250) requires 2.09+; every field above exists by then. -MIN_PROTOCOL = 0x0209 +# `cmd_line_ptr` moved into the setup header in 2.02, `initrd_addr_max` +# arrived in 2.03 and `xloadflags` in 2.12. +MIN_PROTOCOL = 0x0202 +IAM_PROTOCOL = 0x0203 +XLF_PROTOCOL = 0x020C -def normalize(image: bytearray) -> list: - """Zero the boot-loader-written fields. Returns the fields it changed.""" +def target_fields(image: bytes, initrd_size: int) -> list: + """Return the (offset, bytes, name) QEMU writes for this kernel.""" if len(image) < HEADER_END_OFFSET: raise ValueError( f"image is {len(image)} bytes, shorter than the " @@ -71,25 +95,73 @@ def normalize(image: bytearray) -> list: "script normalizes is not guaranteed" ) - changed = [] - for offset, size, name in WRITE_FIELDS: - old = bytes(image[offset : offset + size]) - if old != bytes(size): - changed.append((name, offset, old.hex(), "0" * (size * 2))) - image[offset : offset + size] = bytes(size) - - loadflags = image[LOADFLAGS_OFFSET] - if loadflags & CAN_USE_HEAP: - changed.append( - ( - "loadflags", - LOADFLAGS_OFFSET, - f"{loadflags:02x}", - f"{loadflags & ~CAN_USE_HEAP:02x}", - ) + if image[LOADFLAGS_OFFSET] & LOADED_HIGH: + real_addr, cmdline_addr = REAL_ADDR_HIGH, CMDLINE_ADDR_HIGH + else: + real_addr, cmdline_addr = REAL_ADDR_LOW, CMDLINE_ADDR_LOW + + fields = [ + (TYPE_OF_LOADER_OFFSET, bytes([TYPE_OF_LOADER_QEMU]), "type_of_loader"), + ( + LOADFLAGS_OFFSET, + bytes([image[LOADFLAGS_OFFSET] | CAN_USE_HEAP]), + "loadflags", + ), + # heap_end_ptr is two bytes, but the value never exceeds 16 bits and + # the two bytes above it (ext_loader_ver, ext_loader_type) are zero in + # a built kernel and untouched by QEMU, so writing the word + # zero-extended to 32 bits is what OVMF and dstack-mr also write. + ( + HEAP_END_PTR_OFFSET, + (cmdline_addr - real_addr - SETUP_HEAP_GAP).to_bytes(4, "little"), + "heap_end_ptr", + ), + (CMD_LINE_PTR_OFFSET, cmdline_addr.to_bytes(4, "little"), "cmd_line_ptr"), + ] + if initrd_size <= 0: + return fields + + if protocol >= XLF_PROTOCOL: + xloadflags = int.from_bytes( + image[XLOADFLAGS_OFFSET : XLOADFLAGS_OFFSET + 2], "little" + ) + initrd_max = ( + 0xFFFFFFFF + if xloadflags & XLF_CAN_BE_LOADED_ABOVE_4G + else INITRD_ADDR_MAX_DEFAULT + ) + elif protocol >= IAM_PROTOCOL: + declared = int.from_bytes( + image[INITRD_ADDR_MAX_OFFSET : INITRD_ADDR_MAX_OFFSET + 4], "little" + ) + initrd_max = declared or INITRD_ADDR_MAX_DEFAULT + else: + initrd_max = INITRD_ADDR_MAX_DEFAULT + available = LOW_MEMORY_SPLIT - ACPI_DATA_SIZE + if initrd_max >= available: + initrd_max = available - 1 + if initrd_size >= initrd_max: + raise ValueError( + f"an initrd of {initrd_size} bytes does not fit below 0x{initrd_max:x}" ) - image[LOADFLAGS_OFFSET] = loadflags & ~CAN_USE_HEAP + initrd_addr = (initrd_max - initrd_size) & ~(INITRD_ALIGNMENT - 1) + fields.append( + (RAMDISK_IMAGE_OFFSET, initrd_addr.to_bytes(4, "little"), "ramdisk_image") + ) + fields.append( + (RAMDISK_SIZE_OFFSET, initrd_size.to_bytes(4, "little"), "ramdisk_size") + ) + return fields + +def normalize(image: bytearray, initrd_size: int) -> list: + """Write the boot-loader-written fields. Returns the fields it changed.""" + changed = [] + for offset, value, name in target_fields(bytes(image), initrd_size): + old = bytes(image[offset : offset + len(value)]) + if old != value: + changed.append((name, offset, old.hex(), value.hex())) + image[offset : offset + len(value)] = value return changed @@ -97,6 +169,10 @@ def main() -> int: """Normalize the image named on the command line, or check it in place.""" parser = argparse.ArgumentParser(description=__doc__) parser.add_argument("bzimage", help="kernel image to normalize in place") + parser.add_argument( + "initrd", + help="initrd the image ships, whose size decides the initrd address", + ) parser.add_argument( "--check", action="store_true", @@ -106,8 +182,11 @@ def main() -> int: with open(args.bzimage, "rb") as f: image = bytearray(f.read()) + with open(args.initrd, "rb") as f: + f.seek(0, 2) + initrd_size = f.tell() - changed = normalize(image) + changed = normalize(image, initrd_size) for name, offset, old, new in changed: print(f"{args.bzimage}: {name} (0x{offset:03x}) {old} -> {new}") diff --git a/os/tests/test-kernel-header-normalization.sh b/os/tests/test-kernel-header-normalization.sh index 73bb9e710..24e71aabd 100755 --- a/os/tests/test-kernel-header-normalization.sh +++ b/os/tests/test-kernel-header-normalization.sh @@ -11,9 +11,11 @@ # 0007-OvmfPkg-...-normalize-setup-header.patch (firmware) # # If they ever disagree, every CVM fails attestation with an RTMR[1] mismatch -# and nothing else points at why. So this test parses the field table out of -# both and compares them, then exercises the Python side against a synthetic -# bzImage. +# and nothing else points at why. So this test checks that both write the same +# fields with the same constants, then exercises the Python side against a +# synthetic bzImage. dstack-mr is the third implementation -- it predicts this +# layout for an image that does not declare the flag -- so its constants are +# checked too. set -euo pipefail here=$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd) @@ -24,45 +26,71 @@ patch=$root/os/yocto/layers/meta-dstack/recipes-core/dstack-ovmf/dstack-ovmf/000 [ -x "$script" ] || { echo "missing $script" >&2; exit 1; } [ -f "$patch" ] || { echo "missing $patch" >&2; exit 1; } -python3 - "$script" "$patch" <<'PYEOF' +kernel_rs=$root/dstack/dstack-mr/src/kernel.rs +[ -f "$kernel_rs" ] || { echo "missing $kernel_rs" >&2; exit 1; } + +python3 - "$script" "$patch" "$kernel_rs" <<'PYEOF' import importlib.util import re import sys -script_path, patch_path = sys.argv[1], sys.argv[2] +script_path, patch_path, kernel_rs_path = sys.argv[1:4] spec = importlib.util.spec_from_file_location("normalize", script_path) normalize = importlib.util.module_from_spec(spec) spec.loader.exec_module(normalize) -py_fields = sorted((offset, size) for offset, size, _ in normalize.WRITE_FIELDS) +patch_text = open(patch_path, encoding="utf-8").read() +kernel_rs = open(kernel_rs_path, encoding="utf-8").read() -# Added lines of the form " { 0x210, 1 }, // type_of_loader". -patch_text = open(patch_path, encoding="ascii").read() -ovmf_fields = sorted( - (int(offset, 16), int(size)) - for offset, size in re.findall( - r"^\+\s*\{\s*(0x[0-9A-Fa-f]+),\s*(\d+)\s*\},", patch_text, re.M - ) -) -assert ovmf_fields, "no field table found in the OVMF patch" -assert py_fields == ovmf_fields, ( - "normalization field tables disagree\n" - f" image build: {[(hex(o), s) for o, s in py_fields]}\n" - f" OVMF patch: {[(hex(o), s) for o, s in ovmf_fields]}" -) +def ovmf(name): + match = re.search(rf"^\+#define {name}\s+(0x[0-9A-Fa-f]+)", patch_text, re.M) + assert match, f"the OVMF patch no longer defines {name}" + return int(match.group(1), 16) -for token in ("LINUX_HDR_MIN_PROTOCOL 0x0209", "LINUX_LOADFLAGS_CAN_USE_HEAP 0x80"): - assert token in patch_text, f"OVMF patch no longer defines {token}" -assert normalize.MIN_PROTOCOL == 0x0209 -assert normalize.CAN_USE_HEAP == 0x80 -# `modify` fields carry kernel-supplied values and must never be normalized. -# code32_start is the protected-mode entry point; zeroing it bricks the kernel. -for forbidden in (0x1F2, 0x1FA, 0x211, 0x212, 0x214): - assert all(offset != forbidden for offset, _ in py_fields), ( - f"0x{forbidden:x} is a `modify` field and must not be zeroed" +# Both sides write these fields, and only these. +py_offsets = {0x210, 0x211, 0x218, 0x21C, 0x224, 0x228} +ovmf_offsets = { + ovmf(f"LINUX_HDR_{name}_OFFSET") + for name in ( + "TYPE_OF_LOADER", + "LOADFLAGS", + "RAMDISK_IMAGE", + "RAMDISK_SIZE", + "HEAP_END_PTR", + "CMD_LINE_PTR", + ) +} +assert py_offsets == ovmf_offsets, (py_offsets, ovmf_offsets) + +# Every value the layout depends on, in all three implementations. +for name, value, rust in ( + ("QEMU_LINUX_TYPE_OF_LOADER", normalize.TYPE_OF_LOADER_QEMU, "0xb0"), + ("QEMU_LINUX_REAL_ADDR_LOW", normalize.REAL_ADDR_LOW, "0x90000_u32"), + ("QEMU_LINUX_CMDLINE_ADDR_LOW", normalize.CMDLINE_ADDR_LOW, "0x9a000_u32"), + ("QEMU_LINUX_REAL_ADDR_HIGH", normalize.REAL_ADDR_HIGH, "0x10000_u32"), + ("QEMU_LINUX_CMDLINE_ADDR_HIGH", normalize.CMDLINE_ADDR_HIGH, "0x20000_u32"), + ("QEMU_LINUX_SETUP_HEAP_GAP", normalize.SETUP_HEAP_GAP, "0x200"), + ("QEMU_LOW_MEMORY_SPLIT", normalize.LOW_MEMORY_SPLIT, "0x80000000"), + ("QEMU_ACPI_DATA_SIZE", normalize.ACPI_DATA_SIZE, "0x28000"), + ("LINUX_INITRD_ADDR_MAX_DEFAULT", normalize.INITRD_ADDR_MAX_DEFAULT, "0x37ffffff"), + ("QEMU_INITRD_ALIGNMENT", normalize.INITRD_ALIGNMENT, "4095"), +): + assert ovmf(name) == value, f"{name} is {ovmf(name):#x} in the OVMF patch" + assert rust in kernel_rs, f"dstack-mr no longer spells {rust}" + +assert ovmf("LINUX_HDR_MIN_PROTOCOL") == normalize.MIN_PROTOCOL == 0x0202 +assert ovmf("LINUX_LOADFLAGS_CAN_USE_HEAP") == normalize.CAN_USE_HEAP == 0x80 +assert ovmf("LINUX_LOADFLAGS_LOADED_HIGH") == normalize.LOADED_HIGH == 0x01 + +# `modify` fields carry kernel-supplied values and must never be written. +# code32_start is the protected-mode entry point; overwriting it bricks the +# kernel. loadflags is only OR-ed with CAN_USE_HEAP, never replaced. +for forbidden in (0x1F2, 0x1FA, 0x212, 0x214): + assert forbidden not in py_offsets, ( + f"0x{forbidden:x} is a `modify` field and must not be written" ) # Synthetic bzImage: a plausible built kernel, then the same image with every @@ -78,30 +106,46 @@ def make_image(patched: bool) -> bytearray: image[0x214:0x218] = (0x100000).to_bytes(4, "little") # code32_start image[0x224:0x226] = (0x50A0).to_bytes(2, "little") # heap_end_ptr if patched: + # What QEMU writes for a 1 GiB guest, whose initrd sits lower than the + # layout normalized to. image[0x210] = 0xB0 image[0x211] |= 0x80 - image[0x218:0x21C] = (0xA97FC000).to_bytes(4, "little") - image[0x21C:0x220] = (0x0062A954).to_bytes(4, "little") + image[0x218:0x21C] = (0x37BF3000).to_bytes(4, "little") + image[0x21C:0x220] = (INITRD_SIZE).to_bytes(4, "little") image[0x224:0x226] = (0xFE00).to_bytes(2, "little") image[0x228:0x22C] = (0x20000).to_bytes(4, "little") return image + +INITRD_SIZE = 6_454_799 + built, patched = make_image(False), make_image(True) assert built != patched -normalize.normalize(built) -normalize.normalize(patched) +normalize.normalize(built, INITRD_SIZE) +normalize.normalize(patched, INITRD_SIZE) assert built == patched, "normalizing a QEMU-patched kernel must reproduce the shipped one" -# Idempotent, so OVMF re-running it over an already normalized kernel is a no-op. +# The layout QEMU writes, which is what dstack-mr predicts for an image that +# does not declare the flag. +assert built[0x210] == 0xB0 +assert built[0x211] == 0x81 +assert built[0x218:0x21C] == (0x379D8000).to_bytes(4, "little") +assert built[0x21C:0x220] == (INITRD_SIZE).to_bytes(4, "little") +assert built[0x224:0x228] == (0xFE00).to_bytes(4, "little") +assert built[0x228:0x22C] == (0x20000).to_bytes(4, "little") + +# Independent of guest RAM: an initrd address QEMU derived for another guest +# size normalizes to the same bytes, which is what makes the host's QEMU +# version and memory size irrelevant. again = bytearray(built) -assert normalize.normalize(again) == [] +assert normalize.normalize(again, INITRD_SIZE) == [] assert again == built -# The `modify` fields survived. +# The `modify` fields survived, and loadflags kept LOADED_HIGH. assert built[0x1F2:0x1F4] == (0x0001).to_bytes(2, "little") assert built[0x1FA:0x1FC] == (0xFFFF).to_bytes(2, "little") -assert built[0x211] == 0x01 +assert built[0x211] & 0x01 assert built[0x212:0x214] == (0x8000).to_bytes(2, "little") assert built[0x214:0x218] == (0x100000).to_bytes(4, "little") @@ -112,7 +156,7 @@ assert built[0x214:0x218] == (0x100000).to_bytes(4, "little") other = bytearray(0x1000) other[0x210] = 0xB0 try: - normalize.normalize(other) + normalize.normalize(other, INITRD_SIZE) except ValueError: pass else: @@ -125,13 +169,13 @@ assert "return;" in patch_text.split('CompareMem (Data + LINUX_HDR_MAGIC_OFFSET' # normalizes. The image build still fails loudly about it. ancient = bytearray(0x1000) ancient[0x202:0x206] = b"HdrS" -ancient[0x206:0x208] = (0x0208).to_bytes(2, "little") +ancient[0x206:0x208] = (0x0201).to_bytes(2, "little") try: - normalize.normalize(ancient) + normalize.normalize(ancient, INITRD_SIZE) except ValueError: pass else: - raise AssertionError("the image build must reject boot protocols below 2.09") + raise AssertionError("the image build must reject boot protocols below 2.02") print("kernel setup-header normalization: OK") PYEOF diff --git a/os/yocto/layers/meta-dstack/recipes-core/dstack-ovmf/dstack-ovmf/0007-OvmfPkg-QemuKernelLoaderFsDxe-normalize-setup-header.patch b/os/yocto/layers/meta-dstack/recipes-core/dstack-ovmf/dstack-ovmf/0007-OvmfPkg-QemuKernelLoaderFsDxe-normalize-setup-header.patch index 30f41be5b..2d054d08c 100644 --- a/os/yocto/layers/meta-dstack/recipes-core/dstack-ovmf/dstack-ovmf/0007-OvmfPkg-QemuKernelLoaderFsDxe-normalize-setup-header.patch +++ b/os/yocto/layers/meta-dstack/recipes-core/dstack-ovmf/dstack-ovmf/0007-OvmfPkg-QemuKernelLoaderFsDxe-normalize-setup-header.patch @@ -12,75 +12,121 @@ guests, so from 10.2 on the same kernel measures differently than it does on 10.1 and earlier. Attestation has to know the host's QEMU version to predict its own RTMR[1], and the host declares that version. -Zero the fields the boot protocol says a boot loader owns, which makes the -published bytes independent of what QEMU did. The dstack image build applies -the same normalization to the kernel it ships, so the measurement is the plain -Authenticode hash of that file on every QEMU version. +Write the fields the boot protocol says a boot loader owns, with the values +QEMU <= 10.1 writes, which makes the published bytes independent of what QEMU +did. The dstack image build writes the same values into the kernel it ships, so +the measurement is the plain Authenticode hash of that file on every QEMU +version. + +The values follow from the kernel file and the initrd size -- never from the +QEMU version, and never from guest RAM: the initrd address QEMU derives from +RAM is fixed at the value it resolves to for a guest with 2 GiB or more below +4G. Normalizing to QEMU's layout rather than to zeros is what keeps a verifier +or KMS from an earlier release able to measure these images: dstack-mr already +computes this layout (patch_kernel in dstack/dstack-mr/src/kernel.rs) for an +image that does not declare kernel_header_normalized. The field set is taken from Documentation/arch/x86/boot.rst rather than from QEMU's behavior: fields typed `write` are ones the boot loader fills in and the kernel supplies no value for. Fields typed `modify` are left alone because they -carry real kernel-supplied values. +carry real kernel-supplied values. Fields QEMU leaves alone are left alone too. Booting is unaffected. On the EFI-stub path the real-mode setup code never -runs, and heap_end_ptr -- the only one of these fields a built kernel leaves -non-zero -- is read only when the boot loader has set CAN_USE_HEAP, which this -also clears. +runs, the stub takes the initrd through LoadFile2 and the command line through +its load options, and these are the values QEMU itself wrote for every release +before 10.2. Signed-off-by: dstack --- --- a/OvmfPkg/QemuKernelLoaderFsDxe/QemuKernelLoaderFsDxe.c +++ b/OvmfPkg/QemuKernelLoaderFsDxe/QemuKernelLoaderFsDxe.c -@@ -49,6 +49,111 @@ +@@ -49,6 +49,215 @@ struct KERNEL_BLOB { KERNEL_BLOB *Next; }; +// -+// Offsets into the Linux x86 setup header, from ++// Linux x86 setup-header offsets and flags, from +// Documentation/arch/x86/boot.rst. +// -+#define LINUX_HDR_MAGIC_OFFSET 0x202 -+#define LINUX_HDR_VERSION_OFFSET 0x206 -+#define LINUX_HDR_LOADFLAGS_OFFSET 0x211 -+#define LINUX_HDR_END_OFFSET 0x258 ++#define LINUX_HDR_MAGIC_OFFSET 0x202 ++#define LINUX_HDR_VERSION_OFFSET 0x206 ++#define LINUX_HDR_TYPE_OF_LOADER_OFFSET 0x210 ++#define LINUX_HDR_LOADFLAGS_OFFSET 0x211 ++#define LINUX_HDR_RAMDISK_IMAGE_OFFSET 0x218 ++#define LINUX_HDR_RAMDISK_SIZE_OFFSET 0x21C ++#define LINUX_HDR_HEAP_END_PTR_OFFSET 0x224 ++#define LINUX_HDR_CMD_LINE_PTR_OFFSET 0x228 ++#define LINUX_HDR_INITRD_ADDR_MAX_OFFSET 0x22C ++#define LINUX_HDR_XLOADFLAGS_OFFSET 0x236 ++#define LINUX_HDR_END_OFFSET 0x258 ++#define LINUX_LOADFLAGS_LOADED_HIGH 0x01 ++#define LINUX_LOADFLAGS_CAN_USE_HEAP 0x80 ++#define LINUX_XLF_CAN_BE_LOADED_ABOVE_4G 0x40 + +// -+// CAN_USE_HEAP in loadflags. The kernel builds it clear; a boot loader sets it -+// when it has also filled in heap_end_ptr. ++// cmd_line_ptr moved into the setup header in boot protocol 2.02, ++// initrd_addr_max arrived in 2.03 and xloadflags in 2.12. +// -+#define LINUX_LOADFLAGS_CAN_USE_HEAP 0x80 ++#define LINUX_HDR_MIN_PROTOCOL 0x0202 ++#define LINUX_HDR_IAM_PROTOCOL 0x0203 ++#define LINUX_HDR_XLF_PROTOCOL 0x020C + +// -+// setup_data (0x250) arrived in boot protocol 2.09. Below that the field -+// layout normalized here is not guaranteed, so the image is left alone. ++// What QEMU writes as boot loader: "Qemu" version 0, the real-mode block and ++// command line it loads a kernel at, the 0x200-byte gap it leaves below the ++// command line for the setup heap, the below-4G window it reserves for ACPI ++// tables with RAM split at 2 GiB, the initrd ceiling a kernel that declares ++// none gets, and the alignment it rounds the initrd address down to. Same ++// values as os/image/normalize-kernel-header.py and dstack-mr's patch_kernel. +// -+#define LINUX_HDR_MIN_PROTOCOL 0x0209 ++#define QEMU_LINUX_TYPE_OF_LOADER 0xB0 ++#define QEMU_LINUX_REAL_ADDR_HIGH 0x10000 ++#define QEMU_LINUX_CMDLINE_ADDR_HIGH 0x20000 ++#define QEMU_LINUX_REAL_ADDR_LOW 0x90000 ++#define QEMU_LINUX_CMDLINE_ADDR_LOW 0x9A000 ++#define QEMU_LINUX_SETUP_HEAP_GAP 0x200 ++#define QEMU_LOW_MEMORY_SPLIT 0x80000000 ++#define QEMU_ACPI_DATA_SIZE 0x28000 ++#define LINUX_INITRD_ADDR_MAX_DEFAULT 0x37FFFFFF ++#define QEMU_INITRD_ALIGNMENT 0x1000 + -+// -+// Every setup-header field boot.rst types as `write`: the boot loader fills it -+// in and the kernel supplies no value, so zeroing it discards nothing. Fields -+// typed `modify` carry real kernel-supplied values -- code32_start (0x214) is -+// the protected-mode entry point -- and are deliberately absent. -+// -+STATIC CONST struct { -+ UINT16 Offset; -+ UINT16 Size; -+} mLinuxBootLoaderWrittenFields[] = { -+ { 0x210, 1 }, // type_of_loader -+ { 0x218, 4 }, // ramdisk_image -+ { 0x21C, 4 }, // ramdisk_size -+ { 0x224, 2 }, // heap_end_ptr -+ { 0x226, 1 }, // ext_loader_ver -+ { 0x227, 1 }, // ext_loader_type -+ { 0x228, 4 }, // cmd_line_ptr -+ { 0x23C, 4 }, // hardware_subarch -+ { 0x240, 8 }, // hardware_subarch_data -+ { 0x250, 8 }, // setup_data -+}; ++STATIC ++UINT32 ++LinuxHdrRead ( ++ IN CONST UINT8 *Data, ++ IN UINTN Offset, ++ IN UINTN Size ++ ) ++{ ++ UINT32 Value; ++ UINTN Index; ++ ++ Value = 0; ++ for (Index = 0; Index < Size; Index++) { ++ Value |= (UINT32)Data[Offset + Index] << (8 * Index); ++ } ++ ++ return Value; ++} ++ ++STATIC ++VOID ++LinuxHdrWrite32 ( ++ IN OUT UINT8 *Data, ++ IN UINTN Offset, ++ IN UINT32 Value ++ ) ++{ ++ UINTN Index; ++ ++ for (Index = 0; Index < 4; Index++) { ++ Data[Offset + Index] = (UINT8)(Value >> (8 * Index)); ++ } ++} + +/** -+ Clear the setup-header fields a boot loader owns, so the measured kernel does -+ not depend on what QEMU wrote there. ++ Normalize the setup-header fields a boot loader owns, so the measured kernel ++ does not depend on what QEMU wrote there. + + QEMU acts as the boot loader for -kernel and fills these fields in before + serving the image over fw_cfg, which changes the bytes this driver publishes @@ -88,9 +134,13 @@ Signed-off-by: dstack + doing that for confidential guests (commit a7542a38f399), so without this the + same kernel measures differently depending on the host's QEMU. + -+ Zeroing is idempotent and matches what the image build already writes into -+ the shipped kernel, so the measurement equals the Authenticode hash of the -+ file on disk on every QEMU version. ++ The values written are the ones QEMU <= 10.1 writes, for a guest with 2 GiB ++ or more of RAM below 4G: they follow from the kernel file and the initrd ++ size, never from the QEMU version or from guest RAM. The dstack image build ++ writes the same values into the kernel it ships, so the measurement is the ++ Authenticode hash of that file on every QEMU version, and dstack-mr computes ++ them when it predicts RTMR[1] for an image that does not declare ++ kernel_header_normalized. Writing them is idempotent. + + @param[in,out] Data The kernel blob, starting at the setup header. + @param[in] Size Size of the blob in bytes. @@ -102,8 +152,12 @@ Signed-off-by: dstack + IN UINT32 Size + ) +{ -+ UINTN Idx; -+ UINT16 Protocol; ++ UINT32 Protocol; ++ UINT32 RealAddr; ++ UINT32 CmdlineAddr; ++ UINT32 InitrdSize; ++ UINT32 InitrdMax; ++ UINT32 AvailableMem; + + if (Size < LINUX_HDR_END_OFFSET) { + return; @@ -117,8 +171,7 @@ Signed-off-by: dstack + return; + } + -+ Protocol = (UINT16)(Data[LINUX_HDR_VERSION_OFFSET] | -+ (Data[LINUX_HDR_VERSION_OFFSET + 1] << 8)); ++ Protocol = LinuxHdrRead (Data, LINUX_HDR_VERSION_OFFSET, 2); + if (Protocol < LINUX_HDR_MIN_PROTOCOL) { + DEBUG (( + DEBUG_WARN, @@ -130,20 +183,80 @@ Signed-off-by: dstack + return; + } + -+ for (Idx = 0; Idx < ARRAY_SIZE (mLinuxBootLoaderWrittenFields); Idx++) { -+ ZeroMem ( -+ Data + mLinuxBootLoaderWrittenFields[Idx].Offset, -+ mLinuxBootLoaderWrittenFields[Idx].Size -+ ); ++ if ((Data[LINUX_HDR_LOADFLAGS_OFFSET] & LINUX_LOADFLAGS_LOADED_HIGH) == 0) { ++ RealAddr = QEMU_LINUX_REAL_ADDR_LOW; ++ CmdlineAddr = QEMU_LINUX_CMDLINE_ADDR_LOW; ++ } else { ++ RealAddr = QEMU_LINUX_REAL_ADDR_HIGH; ++ CmdlineAddr = QEMU_LINUX_CMDLINE_ADDR_HIGH; ++ } ++ ++ Data[LINUX_HDR_TYPE_OF_LOADER_OFFSET] = QEMU_LINUX_TYPE_OF_LOADER; ++ Data[LINUX_HDR_LOADFLAGS_OFFSET] |= LINUX_LOADFLAGS_CAN_USE_HEAP; ++ // ++ // heap_end_ptr is two bytes, but the value never exceeds 16 bits and the two ++ // bytes above it (ext_loader_ver, ext_loader_type) are zero in a built kernel ++ // and untouched by QEMU, so writing the word zero-extended to 32 bits is what ++ // the image build and dstack-mr also write. ++ // ++ LinuxHdrWrite32 ( ++ Data, ++ LINUX_HDR_HEAP_END_PTR_OFFSET, ++ CmdlineAddr - RealAddr - QEMU_LINUX_SETUP_HEAP_GAP ++ ); ++ LinuxHdrWrite32 (Data, LINUX_HDR_CMD_LINE_PTR_OFFSET, CmdlineAddr); ++ ++ QemuFwCfgSelectItem (QemuFwCfgItemInitrdSize); ++ InitrdSize = QemuFwCfgRead32 (); ++ if (InitrdSize == 0) { ++ // ++ // No initrd: QEMU leaves ramdisk_image and ramdisk_size alone, and a built ++ // kernel ships them zero. ++ // ++ return; ++ } ++ ++ if (Protocol >= LINUX_HDR_XLF_PROTOCOL) { ++ InitrdMax = (LinuxHdrRead (Data, LINUX_HDR_XLOADFLAGS_OFFSET, 2) & ++ LINUX_XLF_CAN_BE_LOADED_ABOVE_4G) != 0 ? ++ MAX_UINT32 : LINUX_INITRD_ADDR_MAX_DEFAULT; ++ } else if (Protocol >= LINUX_HDR_IAM_PROTOCOL) { ++ InitrdMax = LinuxHdrRead (Data, LINUX_HDR_INITRD_ADDR_MAX_OFFSET, 4); ++ if (InitrdMax == 0) { ++ InitrdMax = LINUX_INITRD_ADDR_MAX_DEFAULT; ++ } ++ } else { ++ InitrdMax = LINUX_INITRD_ADDR_MAX_DEFAULT; ++ } ++ ++ AvailableMem = QEMU_LOW_MEMORY_SPLIT - QEMU_ACPI_DATA_SIZE; ++ if (InitrdMax >= AvailableMem) { ++ InitrdMax = AvailableMem - 1; ++ } ++ ++ if (InitrdSize >= InitrdMax) { ++ DEBUG (( ++ DEBUG_WARN, ++ "%a: an initrd of 0x%x bytes does not fit below 0x%x\n", ++ __func__, ++ InitrdSize, ++ InitrdMax ++ )); ++ return; + } + -+ Data[LINUX_HDR_LOADFLAGS_OFFSET] &= (UINT8)~LINUX_LOADFLAGS_CAN_USE_HEAP; ++ LinuxHdrWrite32 ( ++ Data, ++ LINUX_HDR_RAMDISK_IMAGE_OFFSET, ++ (InitrdMax - InitrdSize) & ~(UINT32)(QEMU_INITRD_ALIGNMENT - 1) ++ ); ++ LinuxHdrWrite32 (Data, LINUX_HDR_RAMDISK_SIZE_OFFSET, InitrdSize); +} + STATIC KERNEL_BLOB_ITEMS mKernelBlobItems[] = { { L"kernel", -@@ -1063,6 +1168,10 @@ +@@ -1063,6 +1272,10 @@ QemuKernelFetchBlob ( ChunkData += BlobItems->FwCfgItem[Idx].Size; } From 7727bd1cea8b601b48ee990f003c5a2b0919c26c Mon Sep 17 00:00:00 2001 From: Kevin Wang Date: Fri, 18 Sep 2026 06:44:16 -0700 Subject: [PATCH 36/93] docs(os): correct stale wording about how the setup header is normalized Signed-off-by: Kevin Wang --- docs/security/security-model.md | 6 +++--- dstack/dstack-mr/src/kernel.rs | 4 ++-- os/image/README.md | 10 ++++++---- 3 files changed, 11 insertions(+), 9 deletions(-) diff --git a/docs/security/security-model.md b/docs/security/security-model.md index 9381a8cbf..596699fa2 100644 --- a/docs/security/security-model.md +++ b/docs/security/security-model.md @@ -326,9 +326,9 @@ kernel header for CoCo VMs", first released in 10.2.0) stopped rewriting the header for confidential guests, so the same kernel would otherwise measure differently depending on which QEMU the host chose to run. -dstack removes that dependency instead of modelling it. The image build writes -the boot-loader-written fields in the kernel it ships, with the values QEMU -<= 10.1 writes, and dstack's OVMF writes the same values again before the +dstack removes that dependency instead of modelling it. The image build fills +in the boot-loader-owned fields of the kernel it ships with the values QEMU +<= 10.1 would write, and dstack's OVMF writes the same values again before the kernel blob is measured and loaded. RTMR[1] is therefore the plain Authenticode hash of the `bzImage` listed in `sha256sum.txt`, and the verifier needs nothing from the host to predict it -- not a QEMU version, not a memory size. diff --git a/dstack/dstack-mr/src/kernel.rs b/dstack/dstack-mr/src/kernel.rs index 7a130321d..74d6d5e8d 100644 --- a/dstack/dstack-mr/src/kernel.rs +++ b/dstack/dstack-mr/src/kernel.rs @@ -240,8 +240,8 @@ pub(crate) fn patched_kernel_authenticode_sha384( /// Compute the first RTMR[1] event digest for an image whose OVMF normalizes /// the Linux setup header: the Authenticode SHA-384 of the kernel file itself. /// -/// Both sides zero the boot-loader-written fields -- -/// `os/image/normalize-kernel-header.py` in the image build and +/// Both sides fill in the boot-loader-owned fields with the values QEMU <= 10.1 +/// writes -- `os/image/normalize-kernel-header.py` in the image build and /// `0007-OvmfPkg-QemuKernelLoaderFsDxe-normalize-setup-header.patch` in the /// firmware -- so what OVMF measures is the file, on every QEMU version and at /// every guest memory size. diff --git a/os/image/README.md b/os/image/README.md index 723659e31..b54033527 100644 --- a/os/image/README.md +++ b/os/image/README.md @@ -71,8 +71,8 @@ host's QEMU version — and the host is the one that declares that version. The fix has two halves that must stay in sync: -- this script writes those fields in the kernel we ship, with the values - QEMU <= 10.1 writes; +- this script fills in those fields in the kernel we ship, with the values + QEMU <= 10.1 would write; - `0007-OvmfPkg-QemuKernelLoaderFsDxe-normalize-setup-header.patch` writes the same values in OVMF, before the kernel blob is measured and loaded. @@ -91,8 +91,10 @@ ones QEMU writes above that threshold, and the shipped initrd's size is an input, since it decides `ramdisk_image`. `assemble.sh` records this in `metadata.json` as `"kernel_header_normalized": -true`, and re-runs the script with `--check` first so the build fails rather -than shipping a kernel that disagrees with what the image declares. Images +true`. What makes that declaration true is the OVMF half, which the same build +applies -- `ovmf-build.sh` and the bitbake recipe both fail if the patch does +not apply, so an image cannot ship the flag with firmware that ignores it. +Images without the field are the ones built before this existed; `dstack-mr` measures those the old way, against QEMU's rewritten header. From b4197820896bb5b407942250d506c88e8517b7fc Mon Sep 17 00:00:00 2001 From: Kevin Wang Date: Fri, 18 Sep 2026 08:23:05 -0700 Subject: [PATCH 37/93] fix(os): read XLF_CAN_BE_LOADED_ABOVE_4G as bit 1, not 0x40 0x40 is XLF_5LEVEL_ENABLED. All three implementations of the setup-header layout tested it as XLF_CAN_BE_LOADED_ABOVE_4G, so a kernel built without 5-level paging would get the 0x37ffffff initrd ceiling where QEMU raises it to 4G, putting the initrd about 1.1 GiB from where QEMU puts it and breaking the RTMR[1] prediction an earlier release's verifier or KMS makes. Every kernel dstack has shipped sets both bits, so no released image changes. A kernel loaded low is now refused rather than normalized: QEMU puts its command line at 0x9a000 - cmdline_size, which depends on the command line the host passes, so there is no single value to pin. The firmware leaves such a header as served. The cross-check test no longer restates what the script writes. It derives the written fields and their widths from both implementations and compares them, matches dstack-mr's constants where it computes the layout rather than anywhere in the file, and covers the ceiling every shipped kernel actually takes -- which the previous vector, with xloadflags = 0, never reached. Signed-off-by: Kevin Wang --- dstack/dstack-mr/src/kernel.rs | 33 ++- os/image/README.md | 12 +- os/image/normalize-kernel-header.py | 34 +-- os/tests/test-kernel-header-normalization.sh | 208 +++++++++++++----- ...elLoaderFsDxe-normalize-setup-header.patch | 45 ++-- .../dstack-ovmf/dstack-ovmf_git.bb | 9 +- 6 files changed, 249 insertions(+), 92 deletions(-) diff --git a/dstack/dstack-mr/src/kernel.rs b/dstack/dstack-mr/src/kernel.rs index 74d6d5e8d..38977feee 100644 --- a/dstack/dstack-mr/src/kernel.rs +++ b/dstack/dstack-mr/src/kernel.rs @@ -173,7 +173,10 @@ fn patch_kernel( let mut initrd_max = if protocol >= 0x20c { let xlf = u16::from_le_bytes(kd[0x236..0x238].try_into().context("impossible failure")?); - if (xlf & 0x40) != 0 { + // XLF_CAN_BE_LOADED_ABOVE_4G is bit 1. Bit 6, 0x40, is + // XLF_5LEVEL_ENABLED; a kernel built without 5-level paging sets + // one and not the other, and QEMU tests bit 1. + if (xlf & 0x02) != 0 { u32::MAX } else { 0x37ffffff @@ -308,7 +311,7 @@ mod tests { kernel[0x206..0x208].copy_from_slice(&0x020cu16.to_le_bytes()); // XLF_CAN_BE_LOADED_ABOVE_4G, so QEMU derives the initrd address from // available low memory and the patched digest moves with guest RAM. - kernel[0x236..0x238].copy_from_slice(&0x0040u16.to_le_bytes()); + kernel[0x236..0x238].copy_from_slice(&0x0002u16.to_le_bytes()); kernel[0x224..0x226].copy_from_slice(&0x50a0u16.to_le_bytes()); kernel } @@ -340,13 +343,37 @@ mod tests { assert_ne!(patched_at(0x8000_0000), patched_at(0xA000_0000)); } + /// QEMU derives the initrd ceiling from XLF_CAN_BE_LOADED_ABOVE_4G, + /// which is bit 1; bit 6 is XLF_5LEVEL_ENABLED and must not be read as it. + #[test] + fn only_xlf_bit_1_raises_the_initrd_ceiling() { + let mut kernel = vec![0u8; 0x1000]; + kernel[0x206..0x208].copy_from_slice(&0x020cu16.to_le_bytes()); + + // The initrd the 0.6.0 image ships, so these are the addresses QEMU + // puts it at and the image build writes into the shipped kernel. + let initrd_size = 6_454_798; + let with_xlf = |xlf: u16| { + let mut k = kernel.clone(); + k[0x236..0x238].copy_from_slice(&xlf.to_le_bytes()); + initrd_addr(&patch_kernel(&k, initrd_size, 0x8000_0000, 0x28000).unwrap()) + }; + + // 0x37ffffff is the ceiling for a kernel that cannot be loaded above + // 4G; the raised ceiling is the below-4G window minus the ACPI area. + assert_eq!(with_xlf(0x0000), 0x379D_8000); + assert_eq!(with_xlf(0x0040), 0x379D_8000, "the 5-level flag raised it"); + assert_eq!(with_xlf(0x0002), 0x7F9B_0000, "the above-4G flag did not"); + assert_eq!(with_xlf(0x0042), 0x7F9B_0000); + } + #[test] fn tdx_kernel_patch_uses_precomputed_digest_at_2g_and_high_memory() { let mut kernel = vec![0u8; 0x1000]; // Linux boot protocol >= 2.12 with XLF_CAN_BE_LOADED_ABOVE_4G makes // QEMU derive the initrd address from available low memory. kernel[0x206..0x208].copy_from_slice(&0x020cu16.to_le_bytes()); - kernel[0x236..0x238].copy_from_slice(&0x0040u16.to_le_bytes()); + kernel[0x236..0x238].copy_from_slice(&0x0002u16.to_le_bytes()); let below_2g = patch_kernel(&kernel, 0x100000, 0x80000000 - 0x1000, 0x28000).unwrap(); let at_2g = patch_kernel(&kernel, 0x100000, 0x80000000, 0x28000).unwrap(); diff --git a/os/image/README.md b/os/image/README.md index b54033527..61123f834 100644 --- a/os/image/README.md +++ b/os/image/README.md @@ -88,7 +88,17 @@ the guest RAM size, which limits such a verifier to guests with exactly 2 GiB or at least 2816 MiB -- the one range QEMU places the initrd differently in, and the range the no-image-download path already excludes. The values are the ones QEMU writes above that threshold, and the shipped initrd's size is an -input, since it decides `ramdisk_image`. +input, since it decides `ramdisk_image`: QEMU packs the initrd against a +ceiling, which is 4G for a kernel that sets `XLF_CAN_BE_LOADED_ABOVE_4G` +(bit 1 of `xloadflags` — bit 6, `0x40`, is `XLF_5LEVEL_ENABLED`, a different +flag) and `0x37ffffff` for one that does not, then clamped to the below-4G +window QEMU leaves free of ACPI tables. + +The kernel must be loaded high, which every kernel this build ships is. QEMU +puts a low-loaded kernel's command line at `0x9a000 - cmdline_size`, a value +that depends on the command line the host passes, so there is nothing to +normalize to: the script refuses such a kernel and OVMF leaves its header as +served. `assemble.sh` records this in `metadata.json` as `"kernel_header_normalized": true`. What makes that declaration true is the OVMF half, which the same build diff --git a/os/image/normalize-kernel-header.py b/os/image/normalize-kernel-header.py index b2682f61a..90e0ea7e8 100755 --- a/os/image/normalize-kernel-header.py +++ b/os/image/normalize-kernel-header.py @@ -13,8 +13,8 @@ before measuring, makes RTMR[1] the plain Authenticode hash of the file we ship, on every QEMU version. -The values are the ones QEMU <= 10.1 writes, for a guest with 2 GiB or more of -RAM below 4G -- the only guest memory layout modelled, and the one `dstack-mr` +The values are the ones QEMU <= 10.1 writes, for a kernel loaded high in a +guest with 2 GiB or more of RAM below 4G -- the only guest memory layout modelled, and the one `dstack-mr` computes (`patch_kernel` in dstack/dstack-mr/src/kernel.rs) for an image that does not declare `kernel_header_normalized`. Normalizing to QEMU's layout rather than to zeros is what keeps a verifier or KMS from an earlier release @@ -46,18 +46,18 @@ LOADFLAGS_OFFSET = 0x211 LOADED_HIGH = 0x01 CAN_USE_HEAP = 0x80 -XLF_CAN_BE_LOADED_ABOVE_4G = 0x40 +# xloadflags bit 1. Bit 6, 0x40, is XLF_5LEVEL_ENABLED, not this flag. +XLF_CAN_BE_LOADED_ABOVE_4G = 0x02 # What QEMU writes: "Qemu" version 0, the real-mode block and command line it -# loads a kernel at, the 0x200-byte gap it leaves below the command line for +# loads a kernel loaded high at, the 0x200-byte gap it leaves below the command +# line for # the setup heap, the below-4G window it reserves for ACPI tables with RAM # split at 2 GiB, the initrd ceiling a kernel that declares none gets, and the # alignment it rounds the initrd address down to. TYPE_OF_LOADER_QEMU = 0xB0 -REAL_ADDR_HIGH = 0x10000 -CMDLINE_ADDR_HIGH = 0x20000 -REAL_ADDR_LOW = 0x90000 -CMDLINE_ADDR_LOW = 0x9A000 +REAL_ADDR = 0x10000 +CMDLINE_ADDR = 0x20000 SETUP_HEAP_GAP = 0x200 LOW_MEMORY_SPLIT = 0x80000000 ACPI_DATA_SIZE = 0x28000 @@ -95,10 +95,16 @@ def target_fields(image: bytes, initrd_size: int) -> list: "script normalizes is not guaranteed" ) - if image[LOADFLAGS_OFFSET] & LOADED_HIGH: - real_addr, cmdline_addr = REAL_ADDR_HIGH, CMDLINE_ADDR_HIGH - else: - real_addr, cmdline_addr = REAL_ADDR_LOW, CMDLINE_ADDR_LOW + if not image[LOADFLAGS_OFFSET] & LOADED_HIGH: + # For a kernel loaded low, QEMU's command line sits at + # 0x9a000 - cmdline_size, which depends on the command line the host + # passes: there is no single value to normalize to. Every kernel this + # build ships is loaded high, so refuse rather than write a value QEMU + # never writes. The OVMF side leaves such a header as served. + raise ValueError( + "the kernel is not loaded high; its cmd_line_ptr depends on the " + "command line and cannot be normalized" + ) fields = [ (TYPE_OF_LOADER_OFFSET, bytes([TYPE_OF_LOADER_QEMU]), "type_of_loader"), @@ -113,10 +119,10 @@ def target_fields(image: bytes, initrd_size: int) -> list: # zero-extended to 32 bits is what OVMF and dstack-mr also write. ( HEAP_END_PTR_OFFSET, - (cmdline_addr - real_addr - SETUP_HEAP_GAP).to_bytes(4, "little"), + (CMDLINE_ADDR - REAL_ADDR - SETUP_HEAP_GAP).to_bytes(4, "little"), "heap_end_ptr", ), - (CMD_LINE_PTR_OFFSET, cmdline_addr.to_bytes(4, "little"), "cmd_line_ptr"), + (CMD_LINE_PTR_OFFSET, CMDLINE_ADDR.to_bytes(4, "little"), "cmd_line_ptr"), ] if initrd_size <= 0: return fields diff --git a/os/tests/test-kernel-header-normalization.sh b/os/tests/test-kernel-header-normalization.sh index 24e71aabd..b5f959a6c 100755 --- a/os/tests/test-kernel-header-normalization.sh +++ b/os/tests/test-kernel-header-normalization.sh @@ -27,21 +27,31 @@ patch=$root/os/yocto/layers/meta-dstack/recipes-core/dstack-ovmf/dstack-ovmf/000 [ -f "$patch" ] || { echo "missing $patch" >&2; exit 1; } kernel_rs=$root/dstack/dstack-mr/src/kernel.rs +machine_rs=$root/dstack/dstack-mr/src/machine.rs [ -f "$kernel_rs" ] || { echo "missing $kernel_rs" >&2; exit 1; } +[ -f "$machine_rs" ] || { echo "missing $machine_rs" >&2; exit 1; } -python3 - "$script" "$patch" "$kernel_rs" <<'PYEOF' +python3 - "$script" "$patch" "$kernel_rs" "$machine_rs" <<'PYEOF' import importlib.util import re import sys -script_path, patch_path, kernel_rs_path = sys.argv[1:4] +script_path, patch_path, kernel_rs_path, machine_rs_path = sys.argv[1:5] + +# Importing the script must not leave a .pyc beside it: a cached one whose +# source changed within the same second, without changing size, is reused, and +# this test would then check bytecode that is no longer on disk. +sys.dont_write_bytecode = True spec = importlib.util.spec_from_file_location("normalize", script_path) normalize = importlib.util.module_from_spec(spec) spec.loader.exec_module(normalize) patch_text = open(patch_path, encoding="utf-8").read() -kernel_rs = open(kernel_rs_path, encoding="utf-8").read() +machine_rs = open(machine_rs_path, encoding="utf-8").read() +# Only what dstack-mr compiles into a release counts: a constant that survives +# in `#[cfg(test)]` alone would let the two sides drift while this test passes. +kernel_rs = open(kernel_rs_path, encoding="utf-8").read().split("mod tests {")[0] def ovmf(name): @@ -50,52 +60,12 @@ def ovmf(name): return int(match.group(1), 16) -# Both sides write these fields, and only these. -py_offsets = {0x210, 0x211, 0x218, 0x21C, 0x224, 0x228} -ovmf_offsets = { - ovmf(f"LINUX_HDR_{name}_OFFSET") - for name in ( - "TYPE_OF_LOADER", - "LOADFLAGS", - "RAMDISK_IMAGE", - "RAMDISK_SIZE", - "HEAP_END_PTR", - "CMD_LINE_PTR", - ) -} -assert py_offsets == ovmf_offsets, (py_offsets, ovmf_offsets) - -# Every value the layout depends on, in all three implementations. -for name, value, rust in ( - ("QEMU_LINUX_TYPE_OF_LOADER", normalize.TYPE_OF_LOADER_QEMU, "0xb0"), - ("QEMU_LINUX_REAL_ADDR_LOW", normalize.REAL_ADDR_LOW, "0x90000_u32"), - ("QEMU_LINUX_CMDLINE_ADDR_LOW", normalize.CMDLINE_ADDR_LOW, "0x9a000_u32"), - ("QEMU_LINUX_REAL_ADDR_HIGH", normalize.REAL_ADDR_HIGH, "0x10000_u32"), - ("QEMU_LINUX_CMDLINE_ADDR_HIGH", normalize.CMDLINE_ADDR_HIGH, "0x20000_u32"), - ("QEMU_LINUX_SETUP_HEAP_GAP", normalize.SETUP_HEAP_GAP, "0x200"), - ("QEMU_LOW_MEMORY_SPLIT", normalize.LOW_MEMORY_SPLIT, "0x80000000"), - ("QEMU_ACPI_DATA_SIZE", normalize.ACPI_DATA_SIZE, "0x28000"), - ("LINUX_INITRD_ADDR_MAX_DEFAULT", normalize.INITRD_ADDR_MAX_DEFAULT, "0x37ffffff"), - ("QEMU_INITRD_ALIGNMENT", normalize.INITRD_ALIGNMENT, "4095"), -): - assert ovmf(name) == value, f"{name} is {ovmf(name):#x} in the OVMF patch" - assert rust in kernel_rs, f"dstack-mr no longer spells {rust}" - -assert ovmf("LINUX_HDR_MIN_PROTOCOL") == normalize.MIN_PROTOCOL == 0x0202 -assert ovmf("LINUX_LOADFLAGS_CAN_USE_HEAP") == normalize.CAN_USE_HEAP == 0x80 -assert ovmf("LINUX_LOADFLAGS_LOADED_HIGH") == normalize.LOADED_HIGH == 0x01 +INITRD_SIZE = 6_454_798 -# `modify` fields carry kernel-supplied values and must never be written. -# code32_start is the protected-mode entry point; overwriting it bricks the -# kernel. loadflags is only OR-ed with CAN_USE_HEAP, never replaced. -for forbidden in (0x1F2, 0x1FA, 0x212, 0x214): - assert forbidden not in py_offsets, ( - f"0x{forbidden:x} is a `modify` field and must not be written" - ) # Synthetic bzImage: a plausible built kernel, then the same image with every # boot-loader-written field filled in the way QEMU fills them. -def make_image(patched: bool) -> bytearray: +def make_image(patched: bool, xloadflags: int = 0, initrd_addr: int = 0) -> bytearray: image = bytearray(0x1000) image[0x202:0x206] = b"HdrS" image[0x206:0x208] = (0x020F).to_bytes(2, "little") @@ -105,21 +75,121 @@ def make_image(patched: bool) -> bytearray: image[0x212:0x214] = (0x8000).to_bytes(2, "little") # setup_move_size image[0x214:0x218] = (0x100000).to_bytes(4, "little") # code32_start image[0x224:0x226] = (0x50A0).to_bytes(2, "little") # heap_end_ptr + image[0x236:0x238] = xloadflags.to_bytes(2, "little") if patched: - # What QEMU writes for a 1 GiB guest, whose initrd sits lower than the - # layout normalized to. + # What QEMU wrote for some other guest: the initrd address is the only + # field its memory size reaches, so it is the caller's to vary. image[0x210] = 0xB0 image[0x211] |= 0x80 - image[0x218:0x21C] = (0x37BF3000).to_bytes(4, "little") + image[0x218:0x21C] = initrd_addr.to_bytes(4, "little") image[0x21C:0x220] = (INITRD_SIZE).to_bytes(4, "little") image[0x224:0x226] = (0xFE00).to_bytes(2, "little") image[0x228:0x22C] = (0x20000).to_bytes(4, "little") return image -INITRD_SIZE = 6_454_799 +# Both sides must write the same fields at the same widths. Read the Python +# side out of the script instead of restating it here, so that a field added or +# widened on one side alone fails this test. +py_fields = { + (offset, len(value)) + for offset, value, _ in normalize.target_fields(bytes(make_image(False)), INITRD_SIZE) +} + + +def ovmf_written_fields(): + """The (offset, size) the OVMF patch assigns to, parsed from its code.""" + written = {} + depth = 0 + for line in patch_text.splitlines(): + if not line.startswith("+"): + continue + body = line[1:] + for match in re.finditer(r"Data\[LINUX_HDR_(\w+)_OFFSET\]\s*\|?=", body): + written[match.group(1)] = 1 + if depth: + for match in re.finditer(r"LINUX_HDR_(\w+)_OFFSET", body): + written[match.group(1)] = 4 + if "LinuxHdrWrite32 (" in body: + # A call reaching the closing `);` on its own line spans several. + after = body.split("LinuxHdrWrite32 (", 1)[1] + for match in re.finditer(r"LINUX_HDR_(\w+)_OFFSET", after): + written[match.group(1)] = 4 + depth = 0 if ");" in after else 1 + elif depth and ");" in body: + depth = 0 + return {(ovmf(f"LINUX_HDR_{name}_OFFSET"), size) for name, size in written.items()} + + +ovmf_fields = ovmf_written_fields() +assert py_fields == ovmf_fields, (sorted(py_fields), sorted(ovmf_fields)) +assert len(py_fields) == 6, sorted(py_fields) + +# Every value the layout depends on, in the firmware. +for name, value in ( + ("QEMU_LINUX_TYPE_OF_LOADER", normalize.TYPE_OF_LOADER_QEMU), + ("QEMU_LINUX_REAL_ADDR", normalize.REAL_ADDR), + ("QEMU_LINUX_CMDLINE_ADDR", normalize.CMDLINE_ADDR), + ("QEMU_LINUX_SETUP_HEAP_GAP", normalize.SETUP_HEAP_GAP), + ("QEMU_LOW_MEMORY_SPLIT", normalize.LOW_MEMORY_SPLIT), + ("QEMU_ACPI_DATA_SIZE", normalize.ACPI_DATA_SIZE), + ("LINUX_INITRD_ADDR_MAX_DEFAULT", normalize.INITRD_ADDR_MAX_DEFAULT), + ("QEMU_INITRD_ALIGNMENT", normalize.INITRD_ALIGNMENT), + ("LINUX_XLF_CAN_BE_LOADED_ABOVE_4G", normalize.XLF_CAN_BE_LOADED_ABOVE_4G), + ("LINUX_HDR_MIN_PROTOCOL", normalize.MIN_PROTOCOL), + ("LINUX_LOADFLAGS_CAN_USE_HEAP", normalize.CAN_USE_HEAP), + ("LINUX_LOADFLAGS_LOADED_HIGH", normalize.LOADED_HIGH), +): + assert ovmf(name) == value, f"{name} is {ovmf(name):#x} in the OVMF patch" + +assert normalize.MIN_PROTOCOL == 0x0202 +assert normalize.CAN_USE_HEAP == 0x80 +assert normalize.LOADED_HIGH == 0x01 + +# The same values in dstack-mr, matched where it computes the layout rather +# than anywhere in the file: it is the implementation an earlier release's +# verifier or KMS runs against these images. +for text, spelling, what in ( + (kernel_rs, f"kd[0x210] = {normalize.TYPE_OF_LOADER_QEMU:#x}", "type_of_loader"), + ( + kernel_rs, + f"({normalize.REAL_ADDR:#x}_u32, {normalize.CMDLINE_ADDR:#x}_u32)", + "the real-mode and command-line addresses", + ), + ( + kernel_rs, + f"saturating_sub({normalize.SETUP_HEAP_GAP:#x})", + "the setup-heap gap", + ), + ( + kernel_rs, + f"xlf & {normalize.XLF_CAN_BE_LOADED_ABOVE_4G:#04x}", + "XLF_CAN_BE_LOADED_ABOVE_4G", + ), + ( + kernel_rs, + f"{normalize.INITRD_ADDR_MAX_DEFAULT:#x}", + "the default initrd ceiling", + ), + (kernel_rs, f"& !{normalize.INITRD_ALIGNMENT - 1}", "the initrd alignment"), + ( + kernel_rs, + f"TDX_KERNEL_HASH_COMPAT_2G_MEMORY: u64 = {normalize.LOW_MEMORY_SPLIT:#x}", + "the 2 GiB split", + ), + (machine_rs, f"{normalize.ACPI_DATA_SIZE:#x}", "the ACPI window"), +): + assert spelling in text, f"dstack-mr no longer spells {what} as {spelling}" + +# `modify` fields carry kernel-supplied values and must never be written. +# code32_start is the protected-mode entry point; overwriting it bricks the +# kernel. loadflags is only OR-ed with CAN_USE_HEAP, never replaced. +for forbidden in (0x1F2, 0x1FA, 0x212, 0x214): + assert forbidden not in {offset for offset, _ in py_fields}, ( + f"0x{forbidden:x} is a `modify` field and must not be written" + ) -built, patched = make_image(False), make_image(True) +built, patched = make_image(False), make_image(True, initrd_addr=0x37BF3000) assert built != patched normalize.normalize(built, INITRD_SIZE) @@ -127,7 +197,8 @@ normalize.normalize(patched, INITRD_SIZE) assert built == patched, "normalizing a QEMU-patched kernel must reproduce the shipped one" # The layout QEMU writes, which is what dstack-mr predicts for an image that -# does not declare the flag. +# does not declare the flag. This kernel declares no xloadflags, so QEMU caps +# the initrd at the 0x37ffffff a kernel that cannot be loaded above 4G gets. assert built[0x210] == 0xB0 assert built[0x211] == 0x81 assert built[0x218:0x21C] == (0x379D8000).to_bytes(4, "little") @@ -135,6 +206,41 @@ assert built[0x21C:0x220] == (INITRD_SIZE).to_bytes(4, "little") assert built[0x224:0x228] == (0xFE00).to_bytes(4, "little") assert built[0x228:0x22C] == (0x20000).to_bytes(4, "little") +# The branch every kernel this build ships takes: XLF_CAN_BE_LOADED_ABOVE_4G +# lifts QEMU's ceiling to 4G, and what caps the initrd is then the below-4G +# window it reserves for ACPI tables with RAM split at 2 GiB. +above_4g = make_image(False, xloadflags=normalize.XLF_CAN_BE_LOADED_ABOVE_4G) +normalize.normalize(above_4g, INITRD_SIZE) +assert above_4g[0x218:0x21C] == (0x7F9B0000).to_bytes(4, "little"), ( + f"initrd address is {int.from_bytes(above_4g[0x218:0x21C], 'little'):#x}" +) +assert above_4g[0x21C:0x220] == (INITRD_SIZE).to_bytes(4, "little") + +# 0x40 is XLF_5LEVEL_ENABLED, not XLF_CAN_BE_LOADED_ABOVE_4G. Reading it as the +# latter -- which all three implementations once did -- moves the initrd about +# 1.1 GiB away from where QEMU puts it, for any kernel built without 5-level +# paging. +five_level = make_image(False, xloadflags=0x0040) +normalize.normalize(five_level, INITRD_SIZE) +assert five_level[0x218:0x21C] == built[0x218:0x21C], ( + "the 5-level flag must not raise the initrd ceiling" +) + +# A kernel loaded low takes QEMU's other command-line address, which depends on +# the command line the host passes: there is nothing to normalize to, and the +# build has to say so rather than write a value QEMU never writes. +low = make_image(False) +low[0x211] &= ~normalize.LOADED_HIGH +try: + normalize.normalize(low, INITRD_SIZE) +except ValueError: + pass +else: + raise AssertionError("the image build must reject a kernel not loaded high") +assert "kernel is not loaded high" in patch_text, ( + "the OVMF side must leave a low-loaded kernel as served" +) + # Independent of guest RAM: an initrd address QEMU derived for another guest # size normalizes to the same bytes, which is what makes the host's QEMU # version and memory size irrelevant. diff --git a/os/yocto/layers/meta-dstack/recipes-core/dstack-ovmf/dstack-ovmf/0007-OvmfPkg-QemuKernelLoaderFsDxe-normalize-setup-header.patch b/os/yocto/layers/meta-dstack/recipes-core/dstack-ovmf/dstack-ovmf/0007-OvmfPkg-QemuKernelLoaderFsDxe-normalize-setup-header.patch index 2d054d08c..525d5fea1 100644 --- a/os/yocto/layers/meta-dstack/recipes-core/dstack-ovmf/dstack-ovmf/0007-OvmfPkg-QemuKernelLoaderFsDxe-normalize-setup-header.patch +++ b/os/yocto/layers/meta-dstack/recipes-core/dstack-ovmf/dstack-ovmf/0007-OvmfPkg-QemuKernelLoaderFsDxe-normalize-setup-header.patch @@ -21,7 +21,8 @@ version. The values follow from the kernel file and the initrd size -- never from the QEMU version, and never from guest RAM: the initrd address QEMU derives from RAM is fixed at the value it resolves to for a guest with 2 GiB or more below -4G. Normalizing to QEMU's layout rather than to zeros is what keeps a verifier +4G. A kernel loaded low is left alone, because QEMU's command-line address on +that path depends on the command line the host passes. Normalizing to QEMU's layout rather than to zeros is what keeps a verifier or KMS from an earlier release able to measure these images: dstack-mr already computes this layout (patch_kernel in dstack/dstack-mr/src/kernel.rs) for an image that does not declare kernel_header_normalized. @@ -40,7 +41,7 @@ Signed-off-by: dstack --- --- a/OvmfPkg/QemuKernelLoaderFsDxe/QemuKernelLoaderFsDxe.c +++ b/OvmfPkg/QemuKernelLoaderFsDxe/QemuKernelLoaderFsDxe.c -@@ -49,6 +49,215 @@ struct KERNEL_BLOB { +@@ -49,6 +49,221 @@ struct KERNEL_BLOB { KERNEL_BLOB *Next; }; @@ -59,9 +60,12 @@ Signed-off-by: dstack +#define LINUX_HDR_INITRD_ADDR_MAX_OFFSET 0x22C +#define LINUX_HDR_XLOADFLAGS_OFFSET 0x236 +#define LINUX_HDR_END_OFFSET 0x258 ++// ++// xloadflags bit 1. Bit 6, 0x40, is XLF_5LEVEL_ENABLED, not this flag. ++// +#define LINUX_LOADFLAGS_LOADED_HIGH 0x01 +#define LINUX_LOADFLAGS_CAN_USE_HEAP 0x80 -+#define LINUX_XLF_CAN_BE_LOADED_ABOVE_4G 0x40 ++#define LINUX_XLF_CAN_BE_LOADED_ABOVE_4G 0x02 + +// +// cmd_line_ptr moved into the setup header in boot protocol 2.02, @@ -80,10 +84,8 @@ Signed-off-by: dstack +// values as os/image/normalize-kernel-header.py and dstack-mr's patch_kernel. +// +#define QEMU_LINUX_TYPE_OF_LOADER 0xB0 -+#define QEMU_LINUX_REAL_ADDR_HIGH 0x10000 -+#define QEMU_LINUX_CMDLINE_ADDR_HIGH 0x20000 -+#define QEMU_LINUX_REAL_ADDR_LOW 0x90000 -+#define QEMU_LINUX_CMDLINE_ADDR_LOW 0x9A000 ++#define QEMU_LINUX_REAL_ADDR 0x10000 ++#define QEMU_LINUX_CMDLINE_ADDR 0x20000 +#define QEMU_LINUX_SETUP_HEAP_GAP 0x200 +#define QEMU_LOW_MEMORY_SPLIT 0x80000000 +#define QEMU_ACPI_DATA_SIZE 0x28000 @@ -134,8 +136,8 @@ Signed-off-by: dstack + doing that for confidential guests (commit a7542a38f399), so without this the + same kernel measures differently depending on the host's QEMU. + -+ The values written are the ones QEMU <= 10.1 writes, for a guest with 2 GiB -+ or more of RAM below 4G: they follow from the kernel file and the initrd ++ The values written are the ones QEMU <= 10.1 writes, for a kernel loaded ++ high in a guest with 2 GiB or more of RAM below 4G: they follow from the kernel file and the initrd + size, never from the QEMU version or from guest RAM. The dstack image build + writes the same values into the kernel it ships, so the measurement is the + Authenticode hash of that file on every QEMU version, and dstack-mr computes @@ -153,8 +155,6 @@ Signed-off-by: dstack + ) +{ + UINT32 Protocol; -+ UINT32 RealAddr; -+ UINT32 CmdlineAddr; + UINT32 InitrdSize; + UINT32 InitrdMax; + UINT32 AvailableMem; @@ -184,11 +184,18 @@ Signed-off-by: dstack + } + + if ((Data[LINUX_HDR_LOADFLAGS_OFFSET] & LINUX_LOADFLAGS_LOADED_HIGH) == 0) { -+ RealAddr = QEMU_LINUX_REAL_ADDR_LOW; -+ CmdlineAddr = QEMU_LINUX_CMDLINE_ADDR_LOW; -+ } else { -+ RealAddr = QEMU_LINUX_REAL_ADDR_HIGH; -+ CmdlineAddr = QEMU_LINUX_CMDLINE_ADDR_HIGH; ++ // ++ // A kernel loaded low takes QEMU's other command-line address, ++ // 0x9a000 - cmdline_size, which depends on the command line the host ++ // passes. There is no single value to normalize to, so leave the header ++ // alone. The dstack image build refuses to ship such a kernel. ++ // ++ DEBUG (( ++ DEBUG_WARN, ++ "%a: kernel is not loaded high; setup header left as served\n", ++ __func__ ++ )); ++ return; + } + + Data[LINUX_HDR_TYPE_OF_LOADER_OFFSET] = QEMU_LINUX_TYPE_OF_LOADER; @@ -202,9 +209,9 @@ Signed-off-by: dstack + LinuxHdrWrite32 ( + Data, + LINUX_HDR_HEAP_END_PTR_OFFSET, -+ CmdlineAddr - RealAddr - QEMU_LINUX_SETUP_HEAP_GAP ++ QEMU_LINUX_CMDLINE_ADDR - QEMU_LINUX_REAL_ADDR - QEMU_LINUX_SETUP_HEAP_GAP + ); -+ LinuxHdrWrite32 (Data, LINUX_HDR_CMD_LINE_PTR_OFFSET, CmdlineAddr); ++ LinuxHdrWrite32 (Data, LINUX_HDR_CMD_LINE_PTR_OFFSET, QEMU_LINUX_CMDLINE_ADDR); + + QemuFwCfgSelectItem (QemuFwCfgItemInitrdSize); + InitrdSize = QemuFwCfgRead32 (); @@ -256,7 +263,7 @@ Signed-off-by: dstack STATIC KERNEL_BLOB_ITEMS mKernelBlobItems[] = { { L"kernel", -@@ -1063,6 +1272,10 @@ QemuKernelFetchBlob ( +@@ -1063,6 +1278,10 @@ QemuKernelFetchBlob ( ChunkData += BlobItems->FwCfgItem[Idx].Size; } diff --git a/os/yocto/layers/meta-dstack/recipes-core/dstack-ovmf/dstack-ovmf_git.bb b/os/yocto/layers/meta-dstack/recipes-core/dstack-ovmf/dstack-ovmf_git.bb index 0e5d89ed2..c946f87d0 100644 --- a/os/yocto/layers/meta-dstack/recipes-core/dstack-ovmf/dstack-ovmf_git.bb +++ b/os/yocto/layers/meta-dstack/recipes-core/dstack-ovmf/dstack-ovmf_git.bb @@ -89,10 +89,11 @@ DEPENDS = "nasm-native acpica-native ovmf-native util-linux-native" # modules / no sevsecret). The patch fails loud if a future edk2 bump changes # the AmdSev layout. # -# 0007-OvmfPkg-QemuKernelLoaderFsDxe-normalize-setup-header.patch zeroes the -# setup-header fields QEMU writes as boot loader, so RTMR[1] no longer depends -# on the host's QEMU version. It pairs with os/image/normalize-kernel-header.py, -# which applies the same normalization to the shipped bzImage. +# 0007-OvmfPkg-QemuKernelLoaderFsDxe-normalize-setup-header.patch writes the +# setup-header fields QEMU writes as boot loader, with the values QEMU <= 10.1 +# writes, so RTMR[1] no longer depends on the host's QEMU version. It pairs with +# os/image/normalize-kernel-header.py, which writes the same values into the +# shipped bzImage. OVMF_BUILD_SEV ??= "1" EDK_TOOLS_DIR="edk2_basetools" From aa4e3d05589edb0e77588c3dda4635467fd697ad Mon Sep 17 00:00:00 2001 From: Kevin Wang Date: Fri, 18 Sep 2026 22:33:27 -0700 Subject: [PATCH 38/93] docs(os): state that one fixed header is written, not QEMU's logic The rationale paragraph carried over from the zeroing version said the field set comes from the boot protocol rather than from QEMU's behavior, that every field written is typed `write` in Documentation/arch/x86/boot.rst, and that fields typed `modify` are left alone. Two of those are false: loadflags is typed `modify (obligatory)` and is written, and the set is now exactly what QEMU fills in, with the write-typed fields QEMU never touches deliberately dropped. A reader following the paragraph would either drop the CAN_USE_HEAP write or add hardware_subarch and setup_data back, desyncing the script from the firmware. It appeared verbatim in the script, the image README, the security model and the OVMF patch's commit message; all four now say what the code does, including that only the CAN_USE_HEAP bit of loadflags is set, because the boot protocol assigns that bit to the boot loader. They also now say what this is: one fixed header, chosen because dstack-mr already computes it, not a reimplementation of QEMU's loader. Zeros would serve the measurement goal equally well; reproducing a header QEMU once wrote is what lets an earlier release's verifier or KMS keep measuring these images. The README's guest-memory note had its appositives attached to the allowed range rather than its complement, so it read as naming the sizes to avoid. It now names both sides. The HEADER_END_OFFSET comment still justified 0x258 as one past the last field touched, which was true when setup_data was zeroed; nothing above 0x238 is read now, so the constant is stated as what it is, the end of the protocol 2.09 header kept as a minimum length. Also joins a paragraph broken mid-sentence in the README, rewraps a comment broken mid-phrase and a 106-character docstring line. No behavior change. The OVMF patch's diff is untouched -- only the commit message above it changes -- so the file it applies to is byte-identical and the firmware does not move. Signed-off-by: Kevin Wang --- docs/security/security-model.md | 13 +++-- os/image/README.md | 40 +++++++++------- os/image/normalize-kernel-header.py | 48 +++++++++++-------- ...elLoaderFsDxe-normalize-setup-header.patch | 22 +++++---- 4 files changed, 72 insertions(+), 51 deletions(-) diff --git a/docs/security/security-model.md b/docs/security/security-model.md index 596699fa2..ca5647a71 100644 --- a/docs/security/security-model.md +++ b/docs/security/security-model.md @@ -358,11 +358,14 @@ here there is no knob. It also removes a class of correct-but-rejected deployments, since the previous QEMU-patched digest varied with guest RAM and was only reproducible at specific memory sizes. -The normalized field set comes from the boot protocol rather than from QEMU's -behavior: every field written is one `Documentation/arch/x86/boot.rst` types as -`write`, which the boot loader fills in and the kernel supplies no value for. -Fields typed `modify` carry real kernel-supplied values and are left as the -kernel built them. +What is written is one fixed header -- the one QEMU <= 10.1 in fact wrote for +these kernels -- and not a reimplementation of QEMU's loader. Every field +written is one QEMU fills in as boot loader; all but one are typed `write` in +`Documentation/arch/x86/boot.rst`, which the boot loader supplies and the +kernel has no value for. The exception is `loadflags`, typed +`modify (obligatory)`, of which only the `CAN_USE_HEAP` bit is set, a bit the +protocol assigns to the boot loader. Fields that carry real kernel-supplied +values are left as the kernel built them. ### TCB status is surfaced, not gated, during verification diff --git a/os/image/README.md b/os/image/README.md index 61123f834..d1f4e1256 100644 --- a/os/image/README.md +++ b/os/image/README.md @@ -80,15 +80,16 @@ The result is that RTMR[1] is the plain Authenticode hash of `bzImage` as listed in `sha256sum.txt`, on every QEMU version and at every guest memory size. -Normalizing to QEMU's layout rather than to zeros is what keeps an earlier -release able to verify these images: `dstack-mr` already computes that layout -(`patch_kernel`) for an image that does not declare the flag, so a 0.5.x KMS -can still onboard a 0.6.0 root with image verification on. Recomputing it needs -the guest RAM size, which limits such a verifier to guests with exactly 2 GiB -or at least 2816 MiB -- the one range QEMU places the initrd differently in, -and the range the no-image-download path already excludes. The values are the -ones QEMU writes above that threshold, and the shipped initrd's size is an -input, since it decides `ramdisk_image`: QEMU packs the initrd against a +Normalizing to a header QEMU once wrote, rather than to zeros, is what keeps an +earlier release able to verify these images: `dstack-mr` already computes that +header (`patch_kernel`) for an image that does not declare the flag, so a 0.5.x +KMS can still onboard a 0.6.0 root with image verification on. That +recomputation takes the guest RAM size as an input, so it reproduces the +normalized bytes only for guests of exactly 2 GiB or at least 2816 MiB. Below +2 GiB, and between 2 GiB and 2816 MiB, QEMU placed the initrd somewhere else, +so such a verifier computes a different RTMR[1]; the no-image-download path +refuses that same range for the same reason. The shipped initrd's size is an +input too, since it decides `ramdisk_image`: QEMU packs the initrd against a ceiling, which is 4G for a kernel that sets `XLF_CAN_BE_LOADED_ABOVE_4G` (bit 1 of `xloadflags` — bit 6, `0x40`, is `XLF_5LEVEL_ENABLED`, a different flag) and `0x37ffffff` for one that does not, then clamped to the below-4G @@ -104,15 +105,18 @@ served. true`. What makes that declaration true is the OVMF half, which the same build applies -- `ovmf-build.sh` and the bitbake recipe both fail if the patch does not apply, so an image cannot ship the flag with firmware that ignores it. -Images -without the field are the ones built before this existed; `dstack-mr` measures -those the old way, against QEMU's rewritten header. - -The field set comes from the boot protocol, not from QEMU's behavior: every -field `Documentation/arch/x86/boot.rst` types as `write` is one the boot loader -fills in and the kernel supplies no value for. Fields typed `modify` carry real -kernel-supplied values — `code32_start` is the protected-mode entry point — and -are deliberately left alone. +Images without the field are the ones built before this existed; `dstack-mr` +measures those the old way, against QEMU's rewritten header. + +What gets written is one fixed header, not a reimplementation of QEMU's loader: +every field written is one QEMU fills in as boot loader, and the script refuses +any kernel outside the shape it reproduces. All but one of those fields are +typed `write` in `Documentation/arch/x86/boot.rst`, meaning the boot loader +supplies them and the kernel has no value there. The exception is `loadflags`, +typed `modify (obligatory)`, of which only the `CAN_USE_HEAP` bit is set — a +bit the protocol assigns to the boot loader. Fields that carry real +kernel-supplied values, such as `code32_start`, the protected-mode entry point, +are deliberately left alone, and so are the `write` fields QEMU never touches. That is safe for every boot path: these are the values QEMU itself wrote for every release before 10.2, and on the EFI-stub path the real-mode setup code diff --git a/os/image/normalize-kernel-header.py b/os/image/normalize-kernel-header.py index 90e0ea7e8..01509fe93 100755 --- a/os/image/normalize-kernel-header.py +++ b/os/image/normalize-kernel-header.py @@ -13,19 +13,26 @@ before measuring, makes RTMR[1] the plain Authenticode hash of the file we ship, on every QEMU version. -The values are the ones QEMU <= 10.1 writes, for a kernel loaded high in a -guest with 2 GiB or more of RAM below 4G -- the only guest memory layout modelled, and the one `dstack-mr` -computes (`patch_kernel` in dstack/dstack-mr/src/kernel.rs) for an image that -does not declare `kernel_header_normalized`. Normalizing to QEMU's layout -rather than to zeros is what keeps a verifier or KMS from an earlier release -able to measure these images. - -The field set comes from the boot protocol, not from QEMU: every field written -is one `Documentation/arch/x86/boot.rst` types as `write`, which the boot -loader fills in and the kernel supplies no value for. Fields typed `modify` -carry real kernel-supplied values (`code32_start` is the protected-mode entry -point) and are left alone. Fields QEMU leaves alone are left alone too: a built -kernel ships them zero, so writing them would only add a way to disagree. +The values are the ones QEMU <= 10.1 wrote for a kernel loaded high in a guest +with 2 GiB or more of RAM below 4G. Zeros would serve the measurement goal +equally well; these particular values are chosen because `dstack-mr` already +computes them (`patch_kernel` in dstack/dstack-mr/src/kernel.rs) for an image +that does not declare `kernel_header_normalized`, so a verifier or KMS from an +earlier dstack release can still measure these images. + +This writes one fixed header. It is not a reimplementation of QEMU's loader: +the branches below reproduce the header QEMU in fact wrote for the kernels +dstack ships, and a kernel outside that shape is refused rather than guessed +at. + +Every field written is one QEMU fills in as boot loader. All but one are typed +`write` in `Documentation/arch/x86/boot.rst`: the boot loader supplies them and +the kernel has no value there. The exception is `loadflags`, typed +`modify (obligatory)`, of which only `CAN_USE_HEAP` is set -- a bit the +protocol assigns to the boot loader. Fields carrying real kernel-supplied +values, such as `code32_start`, are left alone, and so are the `write` fields +QEMU never touches: a built kernel ships them zero, so writing them would only +add a way to disagree. The initrd size decides `ramdisk_image`, so the initrd the image ships is an input. @@ -51,10 +58,9 @@ # What QEMU writes: "Qemu" version 0, the real-mode block and command line it # loads a kernel loaded high at, the 0x200-byte gap it leaves below the command -# line for -# the setup heap, the below-4G window it reserves for ACPI tables with RAM -# split at 2 GiB, the initrd ceiling a kernel that declares none gets, and the -# alignment it rounds the initrd address down to. +# line for the setup heap, the below-4G window it reserves for ACPI tables with +# RAM split at 2 GiB, the initrd ceiling a kernel that declares none gets, and +# the alignment it rounds the initrd address down to. TYPE_OF_LOADER_QEMU = 0xB0 REAL_ADDR = 0x10000 CMDLINE_ADDR = 0x20000 @@ -67,9 +73,11 @@ HEADER_MAGIC_OFFSET = 0x202 HEADER_MAGIC = b"HdrS" VERSION_OFFSET = 0x206 -# One past the last field this touches. Anything shorter cannot carry a setup -# header, and slicing past the end would silently grow the image instead of -# failing. The OVMF side applies the same bound. +# Minimum length for an image to be treated as carrying a setup header. The +# fields below all sit under 0x238; 0x258 is the end of the protocol 2.09 +# header, kept as the bound so a truncated image is rejected outright instead +# of being sliced past its end, which would silently grow it. The OVMF side +# applies the same bound. HEADER_END_OFFSET = 0x258 # `cmd_line_ptr` moved into the setup header in 2.02, `initrd_addr_max` # arrived in 2.03 and `xloadflags` in 2.12. diff --git a/os/yocto/layers/meta-dstack/recipes-core/dstack-ovmf/dstack-ovmf/0007-OvmfPkg-QemuKernelLoaderFsDxe-normalize-setup-header.patch b/os/yocto/layers/meta-dstack/recipes-core/dstack-ovmf/dstack-ovmf/0007-OvmfPkg-QemuKernelLoaderFsDxe-normalize-setup-header.patch index 525d5fea1..098002ed5 100644 --- a/os/yocto/layers/meta-dstack/recipes-core/dstack-ovmf/dstack-ovmf/0007-OvmfPkg-QemuKernelLoaderFsDxe-normalize-setup-header.patch +++ b/os/yocto/layers/meta-dstack/recipes-core/dstack-ovmf/dstack-ovmf/0007-OvmfPkg-QemuKernelLoaderFsDxe-normalize-setup-header.patch @@ -22,15 +22,21 @@ The values follow from the kernel file and the initrd size -- never from the QEMU version, and never from guest RAM: the initrd address QEMU derives from RAM is fixed at the value it resolves to for a guest with 2 GiB or more below 4G. A kernel loaded low is left alone, because QEMU's command-line address on -that path depends on the command line the host passes. Normalizing to QEMU's layout rather than to zeros is what keeps a verifier -or KMS from an earlier release able to measure these images: dstack-mr already -computes this layout (patch_kernel in dstack/dstack-mr/src/kernel.rs) for an -image that does not declare kernel_header_normalized. +that path depends on the command line the host passes. -The field set is taken from Documentation/arch/x86/boot.rst rather than from -QEMU's behavior: fields typed `write` are ones the boot loader fills in and the -kernel supplies no value for. Fields typed `modify` are left alone because they -carry real kernel-supplied values. Fields QEMU leaves alone are left alone too. +This writes one fixed header, not a reimplementation of QEMU's loader. Zeros +would serve the measurement goal equally well; these particular values are +chosen because dstack-mr already computes them (patch_kernel in +dstack/dstack-mr/src/kernel.rs) for an image that does not declare +kernel_header_normalized, so a verifier or KMS from an earlier dstack release +can still measure these images. + +Every field written is one QEMU fills in as boot loader. All but one are typed +`write` in Documentation/arch/x86/boot.rst: the boot loader supplies them and +the kernel has no value there. The exception is loadflags, typed `modify +(obligatory)`, of which only CAN_USE_HEAP is set -- a bit the protocol assigns +to the boot loader. Fields carrying real kernel-supplied values are left alone, +and so are the `write` fields QEMU never touches. Booting is unaffected. On the EFI-stub path the real-mode setup code never runs, the stub takes the initrd through LoadFile2 and the command line through From a15f083cde3a6502343ef22465d52b156441fe3f Mon Sep 17 00:00:00 2001 From: Kevin Wang Date: Fri, 18 Sep 2026 23:57:43 -0700 Subject: [PATCH 39/93] feat(vmm): add a host-side data disk preallocation option CVM data disks were always thin qcow2 images: the size a VM is deployed with is virtual, nothing is reserved on the host, and an oversubscribed host turns into an ENOSPC inside a guest, mid-run, on a write the app had every reason to expect to succeed. Add a preallocation mode -- off (default, unchanged), metadata, falloc or full, mapped one-to-one onto qemu-img's own preallocation= -- settable per node as cvm.disk_prealloc and per VM through the deploy request. It is resolved once at deploy time into the host-side manifest, so a later change to the node default leaves existing VMs alone, and ResizeVm grows a disk with the mode it was created with. The switch stays out of app-compose on purpose: how the host backs a disk has no guest-visible semantics, and anything in app-compose enters the compose hash and therefore the app identity. qcow2 refuses preallocation on top of a backing file unless subcluster allocation is on, so a preallocated disk that has a base image is created with extended_l2=on (QEMU 5.2+). falloc and full reserve host blocks, which a guest that may discard hands straight back on its first trim, so they are rejected unless app-compose sets storage_discard: false; metadata reserves no data blocks and is left alone by that check. --- dstack/vmm/rpc/proto/vmm_rpc.proto | 6 + dstack/vmm/src/app/qemu.rs | 166 ++++++++++++++++++++++--- dstack/vmm/src/app/vm_info.rs | 1 + dstack/vmm/src/config.rs | 86 +++++++++++++ dstack/vmm/src/main_service.rs | 188 ++++++++++++++++++++++++++++- dstack/vmm/vmm.toml | 24 ++++ 6 files changed, 455 insertions(+), 16 deletions(-) diff --git a/dstack/vmm/rpc/proto/vmm_rpc.proto b/dstack/vmm/rpc/proto/vmm_rpc.proto index 209c350a5..60eec34b9 100644 --- a/dstack/vmm/rpc/proto/vmm_rpc.proto +++ b/dstack/vmm/rpc/proto/vmm_rpc.proto @@ -137,6 +137,12 @@ message VmConfiguration { // Development-only TEE ABI to simulate for this VM. When set, the VMM // launches a non-TEE VM and provides an instance-specific simulator config. optional string simulated_tee = 21; + // Host-side preallocation for the VM data disk: "off", "metadata", + // "falloc" or "full", matching qemu-img's preallocation modes. Unset uses + // the node default (cvm.disk_prealloc). The field itself never enters the + // app identity, but "falloc" and "full" reserve host blocks and are only + // accepted with storage_discard=false in compose_file. + optional string disk_prealloc = 22; } // Per-VM networking configuration. diff --git a/dstack/vmm/src/app/qemu.rs b/dstack/vmm/src/app/qemu.rs index f0652dd13..c527305a0 100644 --- a/dstack/vmm/src/app/qemu.rs +++ b/dstack/vmm/src/app/qemu.rs @@ -17,7 +17,7 @@ use super::{ }; use crate::{ app::Manifest, - config::{CvmConfig, CvmPlatform, Networking, NetworkingMode, ProcessAnnotation}, + config::{CvmConfig, CvmPlatform, DiskPrealloc, Networking, NetworkingMode, ProcessAnnotation}, netd::{tap_name, InterfaceIdentity}, vm_launcher::{ChildCommand, LaunchSpec, OpenFile}, }; @@ -143,22 +143,51 @@ pub struct VmConfig { pub gateway_enabled: bool, } +/// Build the `qemu-img create` arguments for a CVM data disk. +fn qemu_img_create_args( + image_file: &Path, + backing_file: Option<&Path>, + size: &str, + prealloc: DiskPrealloc, +) -> Vec { + let mut args = vec!["create".to_string(), "-f".to_string(), "qcow2".to_string()]; + if let Some(backing_file) = backing_file { + args.push("-o".to_string()); + args.push(format!("backing_file={}", backing_file.display())); + args.push("-o".to_string()); + args.push("backing_fmt=qcow2".to_string()); + } + if !prealloc.is_off() { + // qcow2 rejects preallocation on top of a backing file unless + // subcluster allocation is on, because without it a preallocated + // cluster cannot record which of its parts still read through to the + // backing image. Images that ship a base hda always land here, so the + // choice is extended_l2 (QEMU 5.2+) or no preallocation at all. + if backing_file.is_some() { + args.push("-o".to_string()); + args.push("extended_l2=on".to_string()); + } + args.push("-o".to_string()); + args.push(format!("preallocation={}", prealloc.as_str())); + } + args.push(image_file.display().to_string()); + args.push(size.to_string()); + args +} + fn create_hd( image_file: impl AsRef, backing_file: Option>, size: &str, + prealloc: DiskPrealloc, ) -> Result<()> { - let mut command = Command::new("qemu-img"); - command.arg("create").arg("-f").arg("qcow2"); - if let Some(backing_file) = backing_file { - command - .arg("-o") - .arg(format!("backing_file={}", backing_file.as_ref().display())); - command.arg("-o").arg("backing_fmt=qcow2"); - } - command.arg(image_file.as_ref()); - command.arg(size); - let output = command.output()?; + let args = qemu_img_create_args( + image_file.as_ref(), + backing_file.as_ref().map(AsRef::as_ref), + size, + prealloc, + ); + let output = Command::new("qemu-img").args(&args).output()?; if !output.status.success() { bail!( "Failed to create disk: {}", @@ -336,10 +365,19 @@ impl PreparedQemuLaunch { fn prepare_data_disk(vm: &VmConfig, workdir: &VmWorkDir) -> Result<()> { let hda_path = workdir.hda_path(); if !hda_path.exists() { + let prealloc = vm.manifest.disk_prealloc; + if !prealloc.is_off() { + tracing::info!( + "creating {}GB data disk with preallocation={}", + vm.manifest.disk_size, + prealloc.as_str() + ); + } create_hd( &hda_path, vm.image.hda.as_ref(), &format!("{}G", vm.manifest.disk_size), + prealloc, )?; } Ok(()) @@ -1097,9 +1135,12 @@ mod tests { Figment, }; + use crate::config::DiskPrealloc; + use super::{ - amd_sev_snp_memory_backend_arg, macvtap_fd_layout, parse_amd_sev_snp_qmp_capabilities, - virtio_pci_device, PreparedQemuLaunch, PreparedVolume, QemuCommandBuilder, VmConfig, + amd_sev_snp_memory_backend_arg, create_hd, fs, macvtap_fd_layout, + parse_amd_sev_snp_qmp_capabilities, qemu_img_create_args, virtio_pci_device, Command, Path, + PreparedQemuLaunch, PreparedVolume, QemuCommandBuilder, VmConfig, }; use crate::app::image::{Image, ImageInfo}; use crate::app::{needs_swtpm, GpuConfig, GpuSpec, Manifest, PortMapping, VmVolume, VmWorkDir}; @@ -1157,6 +1198,102 @@ mod tests { ); } + #[test] + fn disk_creation_args_carry_preallocation_only_when_requested() { + let image = PathBuf::from("/vms/vm-1/hda.img"); + let backing = PathBuf::from("/images/base/hda.img"); + + assert_eq!( + qemu_img_create_args(&image, None, "20G", DiskPrealloc::Off), + vec!["create", "-f", "qcow2", "/vms/vm-1/hda.img", "20G"] + ); + + // No backing file: qcow2 takes preallocation on its own. + assert_eq!( + qemu_img_create_args(&image, None, "20G", DiskPrealloc::Falloc), + vec![ + "create", + "-f", + "qcow2", + "-o", + "preallocation=falloc", + "/vms/vm-1/hda.img", + "20G" + ] + ); + + // With a backing file, preallocation additionally needs extended_l2. + assert_eq!( + qemu_img_create_args(&image, Some(&backing), "20G", DiskPrealloc::Full), + vec![ + "create", + "-f", + "qcow2", + "-o", + "backing_file=/images/base/hda.img", + "-o", + "backing_fmt=qcow2", + "-o", + "extended_l2=on", + "-o", + "preallocation=full", + "/vms/vm-1/hda.img", + "20G" + ] + ); + + // An off disk keeps the exact arguments it had before the option + // existed, so existing deployments create byte-identical images. + assert_eq!( + qemu_img_create_args(&image, Some(&backing), "20G", DiskPrealloc::Off), + vec![ + "create", + "-f", + "qcow2", + "-o", + "backing_file=/images/base/hda.img", + "-o", + "backing_fmt=qcow2", + "/vms/vm-1/hda.img", + "20G" + ] + ); + } + + /// The option combination is the risky part: qemu-img rejects + /// preallocation on top of a backing file unless subcluster allocation is + /// on, and that rejection only shows up when qemu-img actually runs. + /// Skipped where qemu-img is not installed. + #[test] + fn preallocated_disk_reserves_host_space_on_top_of_a_backing_file() { + use std::os::unix::fs::MetadataExt; + + if Command::new("qemu-img").arg("--version").output().is_err() { + eprintln!("qemu-img not installed, skipping"); + return; + } + + let root = tempfile::TempDir::new().unwrap(); + let base = root.path().join("base.img"); + let hda = root.path().join("hda.img"); + create_hd(&base, None::<&Path>, "8M", DiskPrealloc::Off).unwrap(); + + create_hd(&hda, Some(&base), "8M", DiskPrealloc::Falloc).unwrap(); + let allocated = fs::metadata(&hda).unwrap().blocks() * 512; + assert!( + allocated >= 8 * 1024 * 1024, + "expected the 8M disk to be reserved, got {allocated} bytes" + ); + + fs::remove_file(&hda).unwrap(); + create_hd(&hda, Some(&base), "8M", DiskPrealloc::Off).unwrap(); + let allocated = fs::metadata(&hda).unwrap().blocks() * 512; + assert!( + allocated < 8 * 1024 * 1024, + "expected a thin disk, got {allocated} bytes" + ); + } + /// Minimal launch fixture. Nothing it points at has to exist on disk; every /// test overrides the fields it asserts on. fn test_launch_fixture() -> (Config, VmConfig, PreparedQemuLaunch) { @@ -1196,6 +1333,7 @@ mod tests { volumes: vec![VmVolume { source: "/does-not-exist/volume.img".into(), }], + disk_prealloc: DiskPrealloc::Off, }, image: Image { info: ImageInfo { diff --git a/dstack/vmm/src/app/vm_info.rs b/dstack/vmm/src/app/vm_info.rs index 8b5b524d4..cdd4f18e4 100644 --- a/dstack/vmm/src/app/vm_info.rs +++ b/dstack/vmm/src/app/vm_info.rs @@ -219,6 +219,7 @@ impl VmInfo { }) .collect(), app_id: Some(self.manifest.app_id.clone()), + disk_prealloc: Some(self.manifest.disk_prealloc.as_str().to_string()), hugepages: self.manifest.hugepages, pin_numa: self.manifest.pin_numa, gpus: self.manifest.gpus.as_ref().map(|config| pb::GpuConfig { diff --git a/dstack/vmm/src/config.rs b/dstack/vmm/src/config.rs index 49f013f57..23684737c 100644 --- a/dstack/vmm/src/config.rs +++ b/dstack/vmm/src/config.rs @@ -109,6 +109,63 @@ impl Protocol { } } +/// Host-side preallocation policy for a CVM data disk. The variants map +/// one-to-one onto `qemu-img`'s `preallocation=` option, so the semantics are +/// qcow2's, not ours. +#[derive(Debug, Clone, Copy, Default, PartialEq, Eq, Deserialize, Serialize)] +#[serde(rename_all = "lowercase")] +pub enum DiskPrealloc { + /// Allocate nothing up front. The image grows as the guest writes, so the + /// host may be oversubscribed and a guest write can fail with ENOSPC. + #[default] + Off, + /// Allocate the qcow2 metadata only. Cheap, and it removes the metadata + /// growth cost on first write, but it reserves no space for guest data. + Metadata, + /// Reserve the full disk with fallocate(2). Fast, and it moves the ENOSPC + /// failure to VM creation time where the operator sees it. + Falloc, + /// Reserve the full disk and write it out. Slowest to create -- a 1 TB + /// disk writes 1 TB of zeros -- but it leaves nothing to allocate later. + Full, +} + +impl FromStr for DiskPrealloc { + type Err = anyhow::Error; + + fn from_str(s: &str) -> Result { + Ok(match s { + "off" => DiskPrealloc::Off, + "metadata" => DiskPrealloc::Metadata, + "falloc" => DiskPrealloc::Falloc, + "full" => DiskPrealloc::Full, + _ => bail!("invalid disk preallocation mode: {s}"), + }) + } +} + +impl DiskPrealloc { + pub fn as_str(&self) -> &'static str { + match self { + DiskPrealloc::Off => "off", + DiskPrealloc::Metadata => "metadata", + DiskPrealloc::Falloc => "falloc", + DiskPrealloc::Full => "full", + } + } + + pub fn is_off(&self) -> bool { + matches!(self, DiskPrealloc::Off) + } + + /// Whether the mode reserves host blocks for guest data. `metadata` does + /// not -- it only builds the qcow2 tables -- so a guest that discards + /// takes nothing away from it. + pub fn reserves_data_blocks(&self) -> bool { + matches!(self, DiskPrealloc::Falloc | DiskPrealloc::Full) + } +} + #[derive(Debug, Clone, Copy, PartialEq, Eq, Deserialize, Serialize)] #[serde(rename_all = "kebab-case")] pub enum CvmPlatform { @@ -438,6 +495,14 @@ pub struct CvmConfig { /// Host sharing mode. (9p, vhd, vvfat) pub host_share_mode: String, + /// Default preallocation for newly created CVM data disks. A deploy + /// request may pick another mode per VM. This is a host storage policy: + /// the guest cannot observe it and it never enters the measurements. + /// It applies to disks created from here on; a VM that already has its + /// data disk keeps whatever it was created with. + #[serde(default)] + pub disk_prealloc: DiskPrealloc, + /// QGS (Quote Generation Service) vsock port for kernel-level TSM support. /// When set, QEMU will pass this port to tdx-guest for configfs-tsm quote generation. /// The guest kernel will use this vsock port to communicate with the host QGS. @@ -1283,6 +1348,27 @@ impl Config { mod tests { use super::*; + /// The four names are spelled out three times -- serde, `FromStr` and + /// `as_str` -- so a new variant that misses one of them fails here. + #[test] + fn disk_prealloc_names_round_trip_through_every_encoding() { + for mode in [ + DiskPrealloc::Off, + DiskPrealloc::Metadata, + DiskPrealloc::Falloc, + DiskPrealloc::Full, + ] { + let name = mode.as_str(); + assert_eq!(name.parse::().unwrap(), mode); + assert_eq!(serde_json::to_string(&mode).unwrap(), format!("\"{name}\"")); + assert_eq!( + serde_json::from_str::(&format!("\"{name}\"")).unwrap(), + mode + ); + } + assert!("sparse".parse::().is_err()); + } + #[cfg(unix)] fn cvm_with_qemu(dir: &tempfile::TempDir, script: &str) -> CvmConfig { use std::os::unix::fs::PermissionsExt; diff --git a/dstack/vmm/src/main_service.rs b/dstack/vmm/src/main_service.rs index 5d72e4f20..17be060c7 100644 --- a/dstack/vmm/src/main_service.rs +++ b/dstack/vmm/src/main_service.rs @@ -29,7 +29,7 @@ use crate::app::{ validate_resolved_networks, App, AttachMode, GpuConfig, GpuSpec, Manifest, PortMapping, VmWorkDir, }; -use crate::config::{CvmConfig, Networking, NetworkingMode, NicNetworking}; +use crate::config::{CvmConfig, DiskPrealloc, Networking, NetworkingMode, NicNetworking}; fn hex_sha256(data: &str) -> String { use sha2::Digest; @@ -312,6 +312,8 @@ pub fn create_manifest_from_vm_config( } let key_provider = key_provider_from_compose(&request.compose_file)?; let swtpm = needs_swtpm(key_provider, simulated_tee); + let disk_prealloc = disk_prealloc_from_vm_config(&request, cvm_config)?; + validate_disk_prealloc_against_compose(disk_prealloc, &request.compose_file)?; Ok(Manifest { id, @@ -333,9 +335,66 @@ pub fn create_manifest_from_vm_config( swtpm, networks, volumes, + disk_prealloc, }) } +/// Resolve the data disk preallocation for a deployment. This is a host +/// storage decision, so it comes from the request or the node default and +/// never from app-compose. +fn disk_prealloc_from_vm_config( + request: &VmConfiguration, + cvm_config: &CvmConfig, +) -> Result { + let Some(mode) = request + .disk_prealloc + .as_deref() + .map(str::trim) + .filter(|mode| !mode.is_empty()) + else { + return Ok(cvm_config.disk_prealloc); + }; + mode.parse() + .context("invalid disk_prealloc, expected off, metadata, falloc or full") +} + +/// Reserving data blocks and discarding them pull in opposite directions: the +/// host reserves the space up front, then the guest hands it straight back on +/// the first fstrim, leaving a disk that is neither reserved nor thin. Reject +/// the pair at deployment instead of letting the operator find out from `du` +/// later. +/// +/// `metadata` is deliberately not covered. It reserves no data blocks, so +/// discard costs it nothing, and rejecting it would force a compose change -- +/// a new compose hash, a new app id, another on-chain whitelist entry -- for a +/// combination that is not actually contradictory. +fn validate_disk_prealloc_against_compose( + prealloc: DiskPrealloc, + compose_file: &str, +) -> Result<()> { + if !prealloc.reserves_data_blocks() || !storage_discard_from_compose(compose_file)? { + return Ok(()); + } + bail!( + "disk preallocation ({}) reserves host blocks and requires storage_discard = false \ + in app-compose; either disable discard in the compose file or deploy with \ + disk_prealloc = \"off\" or \"metadata\"", + prealloc.as_str() + ) +} + +/// Read `storage_discard` out of app-compose. Everything else in the document +/// stays opaque, and a missing field means the app-compose default, which is +/// discard enabled. +fn storage_discard_from_compose(compose_file: &str) -> Result { + let compose: serde_json::Value = + serde_json::from_str(compose_file).context("invalid app compose JSON")?; + Ok(compose + .get("storage_discard") + .and_then(serde_json::Value::as_bool) + .unwrap_or(true)) +} + /// Extract only the field understood by this VMM. Keep every other app-compose /// field opaque so newer guest schemas and legacy third-party clients remain /// compatible with older VMMs. @@ -810,8 +869,19 @@ impl RpcHandler { if hda_path.exists() { info!("Resizing disk to {}GB", disk_size); let new_size_str = format!("{}G", disk_size); + // Grow the disk the same way it was created, so a VM + // deployed with preallocation keeps its space reserved. + let mut args = vec!["resize".to_string()]; + if !manifest.disk_prealloc.is_off() { + args.push(format!( + "--preallocation={}", + manifest.disk_prealloc.as_str() + )); + } + args.push(hda_path.display().to_string()); + args.push(new_size_str); let output = std::process::Command::new("qemu-img") - .args(["resize", &hda_path.display().to_string(), &new_size_str]) + .args(&args) .output() .context("Failed to resize disk")?; if !output.status.success() { @@ -953,6 +1023,14 @@ impl VmmRpc for RpcHandler { if !compose_file_path.exists() { bail!("The instance {} not found", request.id); } + // Read the manifest here rather than reusing the one below, so a + // rejected update leaves the stored compose file untouched. + let manifest = self + .app + .work_dir(&request.id)? + .manifest() + .context("Failed to read manifest")?; + validate_disk_prealloc_against_compose(manifest.disk_prealloc, &request.compose_file)?; fs::write(compose_file_path, &request.compose_file) .context("Failed to write compose file")?; @@ -1442,6 +1520,7 @@ mod tests { simulated_tee: None, networking: None, networks: vec![], + disk_prealloc: None, } } @@ -1514,6 +1593,111 @@ mod tests { assert!(manifest.networks.is_empty()); } + /// Preallocation is only accepted together with discard disabled, so a + /// preallocating request needs a compose file that says so. + fn test_vm_configuration_without_discard() -> VmConfiguration { + VmConfiguration { + compose_file: r#"{"storage_discard": false}"#.to_string(), + ..test_vm_configuration() + } + } + + #[test] + fn disk_prealloc_defaults_to_node_config_and_accepts_request_override() { + let mut cvm_config = test_cvm_config(); + assert_eq!(cvm_config.disk_prealloc, DiskPrealloc::Off); + + let manifest = + create_manifest_from_vm_config(test_vm_configuration(), &cvm_config).unwrap(); + assert_eq!(manifest.disk_prealloc, DiskPrealloc::Off); + + cvm_config.disk_prealloc = DiskPrealloc::Falloc; + let manifest = + create_manifest_from_vm_config(test_vm_configuration_without_discard(), &cvm_config) + .unwrap(); + assert_eq!(manifest.disk_prealloc, DiskPrealloc::Falloc); + + // `metadata` reserves no data blocks, so it coexists with discard. + let mut request = test_vm_configuration(); + request.disk_prealloc = Some("metadata".to_string()); + let manifest = create_manifest_from_vm_config(request, &cvm_config).unwrap(); + assert_eq!(manifest.disk_prealloc, DiskPrealloc::Metadata); + + // An empty string is the same as an unset field, as older clients and + // JSON round-trips produce it. + let mut request = test_vm_configuration_without_discard(); + request.disk_prealloc = Some(String::new()); + let manifest = create_manifest_from_vm_config(request, &cvm_config).unwrap(); + assert_eq!(manifest.disk_prealloc, DiskPrealloc::Falloc); + + // Opting one VM out of a preallocating node default needs no compose + // change, so the app identity stays put. + let mut request = test_vm_configuration(); + request.disk_prealloc = Some("off".to_string()); + let manifest = create_manifest_from_vm_config(request, &cvm_config).unwrap(); + assert_eq!(manifest.disk_prealloc, DiskPrealloc::Off); + } + + #[test] + fn create_rejects_unknown_disk_prealloc_mode() { + let mut request = test_vm_configuration(); + request.disk_prealloc = Some("sparse".to_string()); + let err = create_manifest_from_vm_config(request, &test_cvm_config()).unwrap_err(); + assert!(err.to_string().contains("disk_prealloc"), "{err}"); + } + + #[test] + fn preallocation_is_rejected_while_the_guest_may_discard() { + let cvm_config = test_cvm_config(); + + // storage_discard defaults to true in app-compose, so an unset field + // conflicts just as an explicit `true` does. + for compose in ["{}", r#"{"storage_discard": true}"#] { + let request = VmConfiguration { + compose_file: compose.to_string(), + disk_prealloc: Some("falloc".to_string()), + ..test_vm_configuration() + }; + let err = create_manifest_from_vm_config(request, &cvm_config).unwrap_err(); + assert!( + err.to_string().contains("storage_discard"), + "{compose}: {err}" + ); + } + + // The same conflict, reached through the node default instead of the + // request. + let preallocating_node = CvmConfig { + disk_prealloc: DiskPrealloc::Full, + ..cvm_config + }; + let err = create_manifest_from_vm_config(test_vm_configuration(), &preallocating_node) + .unwrap_err(); + assert!(err.to_string().contains("storage_discard"), "{err}"); + } + + #[test] + fn metadata_preallocation_coexists_with_discard() { + // Rejecting it would cost a compose change -- and with it a new app id + // -- to avoid a conflict that does not exist: no data block is + // reserved, so there is none for the guest to hand back. + let mut request = test_vm_configuration(); + request.disk_prealloc = Some("metadata".to_string()); + let manifest = create_manifest_from_vm_config(request, &test_cvm_config()).unwrap(); + assert_eq!(manifest.disk_prealloc, DiskPrealloc::Metadata); + } + + #[test] + fn a_malformed_compose_is_reported_as_such() { + // Not reachable today (the callers parse the compose first), but the + // discard lookup must not turn "broken JSON" into "turn discard off". + let mut request = test_vm_configuration(); + request.compose_file = "{not json".to_string(); + request.disk_prealloc = Some("falloc".to_string()); + let err = create_manifest_from_vm_config(request, &test_cvm_config()).unwrap_err(); + assert!(!err.to_string().contains("storage_discard"), "{err}"); + } + #[test] fn resize_request_rejects_empty_zero_and_empty_image_updates() { let mut request = ResizeVmRequest { diff --git a/dstack/vmm/vmm.toml b/dstack/vmm/vmm.toml index 61107342e..8a52a3826 100644 --- a/dstack/vmm/vmm.toml +++ b/dstack/vmm/vmm.toml @@ -105,6 +105,30 @@ tdx_attestation_variant = "auto" host_share_mode = "9p" +# Default preallocation for a new CVM data disk. This is a host storage +# policy -- the guest cannot see it and it does not affect measurements -- and +# a deploy request may pick another mode per VM. +# "off" : allocate nothing; the image grows as the guest writes, so the +# host is oversubscribed and a guest write can hit ENOSPC. +# "metadata" : allocate qcow2 metadata only; no space is reserved for data. +# "falloc" : reserve the whole disk with fallocate(2). Fast, and it turns a +# full host into a VM start failure instead of a guest ENOSPC. +# "full" : reserve and zero the whole disk. Creation writes the full disk +# size, so a large disk takes minutes. +# "falloc" and "full" reserve host blocks, so they require +# `storage_discard: false` in the app's app-compose.json -- a guest that may +# discard hands the reserved blocks straight back -- and deployments that ask +# for both are rejected. Note what that costs the app: editing app-compose +# changes its hash and therefore its app id, so those two modes are not free +# of the identity for an app that ships with discard on. "metadata" reserves +# nothing and needs no compose change. +# Preallocation on top of a base image needs qcow2 subcluster allocation, so +# the VMM creates such disks with extended_l2=on (QEMU 5.2+). +# This value applies to disks created from here on. A VM that already has a +# data disk keeps the mode it was created with; there is no migration path +# short of editing its vm-manifest.json. +disk_prealloc = "off" + # QGS (Quote Generation Service) vsock port for kernel-level TSM support. # When set, QEMU will pass this port to tdx-guest for configfs-tsm quote generation. # The guest kernel will use this vsock port to communicate with the host QGS. From bea7451064ecf6169e2bcb056e97749eb671f663 Mon Sep 17 00:00:00 2001 From: Kevin Wang Date: Fri, 18 Sep 2026 23:57:43 -0700 Subject: [PATCH 40/93] fix(vmm): build the QEMU launch off the async executor config_qemu writes the guest config files and creates the data disk with a blocking qemu-img call, but ran directly in start_vm's async context. With thin disks that was microseconds, so it never mattered; preallocation makes it real work. Measured on this ext4-on-NVMe host, qemu-img create of a 4 GB disk takes 0.45 s with falloc and 11.7 s with full, so a 200 GB full disk pins a worker thread for roughly ten minutes while StartVm does not return. Move it into spawn_blocking, the way sanitize_on_attach ten lines above already does. --- dstack/vmm/src/app.rs | 41 ++++++++++++++++++++++++++++++++++------- 1 file changed, 34 insertions(+), 7 deletions(-) diff --git a/dstack/vmm/src/app.rs b/dstack/vmm/src/app.rs index 1920a0a14..5a29a5ce0 100644 --- a/dstack/vmm/src/app.rs +++ b/dstack/vmm/src/app.rs @@ -3,7 +3,10 @@ // SPDX-License-Identifier: Apache-2.0 use crate::{ - config::{Config, Networking, NetworkingMode, NicNetworking, ProcessAnnotation, Protocol}, + config::{ + Config, DiskPrealloc, Networking, NetworkingMode, NicNetworking, ProcessAnnotation, + Protocol, + }, logrotate, netd::{self, InterfaceIdentity, PrepareBridgeRequest, PrepareMacvtapRequest}, }; @@ -136,6 +139,11 @@ pub struct Manifest { pub networks: Vec, #[serde(default)] pub volumes: Vec, + /// Host-side preallocation used when the data disk is created. Resolved + /// once at deployment from the request and the node default, so a later + /// change to `cvm.disk_prealloc` leaves existing VMs alone. + #[serde(default)] + pub disk_prealloc: DiskPrealloc, } impl Manifest { @@ -537,12 +545,27 @@ impl App { .context("GPU sanitization task failed")??; self.prepare_netd_networks(&vm_config, &mut runtime_networks) .await?; - let processes = match vm_config.config_qemu( - &work_dir, - &self.config.cvm, - &devices, - &runtime_networks, - ) { + // Off the async executor: this writes the guest config files and, + // when the VM preallocates, creates the data disk. `full` on a + // large disk writes the whole thing, which is minutes of blocking + // work, and even `falloc` is not instant. + let qemu_config = vm_config.clone(); + let app_config = self.config.clone(); + let qemu_workdir = work_dir.path().to_path_buf(); + let qemu_devices = devices.clone(); + let qemu_networks = runtime_networks.clone(); + let configured = tokio::task::spawn_blocking(move || { + qemu_config.config_qemu( + &qemu_workdir, + &app_config.cvm, + &qemu_devices, + &qemu_networks, + ) + }) + .await + .context("QEMU configuration task failed") + .and_then(|result| result); + let processes = match configured { Ok(processes) => processes, Err(error) => { self.release_vm_interfaces(&vm_config.manifest.id).await; @@ -3012,6 +3035,8 @@ mod tests { assert_eq!(manifest.networks.len(), 1); assert_eq!(manifest.networks[0].mode, NetworkingMode::Bridge); assert_eq!(manifest.networks[0].bridge, "dstack-br0"); + // A manifest written before the option existed keeps thin disks. + assert_eq!(manifest.disk_prealloc, DiskPrealloc::Off); } fn write_u16_le_at(buf: &mut [u8], off: usize, value: u16) { @@ -3108,6 +3133,7 @@ mod tests { swtpm: false, networks: vec![], volumes: vec![], + disk_prealloc: DiskPrealloc::Off, } } @@ -3462,6 +3488,7 @@ mod tests { swtpm: false, networks: vec![], volumes: vec![], + disk_prealloc: DiskPrealloc::Off, }; let mr_config = MrConfigV3::new( From 46f3eb8a34ca108711144b81c20123971108207d Mon Sep 17 00:00:00 2001 From: Kevin Wang Date: Fri, 18 Sep 2026 23:57:43 -0700 Subject: [PATCH 41/93] feat(vmm-cli): expose disk preallocation and storage_discard deploy --disk-prealloc picks the host-side preallocation mode for a new VM, and vm info reports it alongside the disk size. compose --no-storage-discard emits storage_discard: false, which the block-reserving modes require; the field is written only when the flag is set, so default compose output -- and with it every existing compose hash -- is unchanged. --- dstack/vmm/src/vmm-cli.py | 33 ++++++++++++++++++++++++++++++++- 1 file changed, 32 insertions(+), 1 deletion(-) diff --git a/dstack/vmm/src/vmm-cli.py b/dstack/vmm/src/vmm-cli.py index 41674726c..70d399729 100755 --- a/dstack/vmm/src/vmm-cli.py +++ b/dstack/vmm/src/vmm-cli.py @@ -882,6 +882,11 @@ def create_app_compose(self, args) -> None: "no_instance_id": args.no_instance_id, "secure_time": args.secure_time, } + # Only emit the field when it is turned off, so a default compose file + # keeps the exact bytes -- and therefore the compose hash -- it had + # before this option existed. + if args.no_storage_discard: + app_compose["storage_discard"] = False if args.key_provider: app_compose["key_provider"] = args.key_provider if args.prelaunch_script: @@ -954,6 +959,8 @@ def create_vm(self, args) -> None: "stopped": args.stopped, "no_tee": args.no_tee, } + if args.disk_prealloc: + params["disk_prealloc"] = args.disk_prealloc if args.simulated_tee: params["simulated_tee"] = args.simulated_tee if args.swap is not None: @@ -1373,7 +1380,9 @@ def show_info(self, vm_id: str, json_output: bool = False) -> None: print(f"Image Version: {info.get('image_version', '-')}") print(f"vCPU: {config.get('vcpu', '-')}") print(f"Memory: {config.get('memory', '-')}MB") - print(f"Disk: {config.get('disk_size', '-')}GB") + prealloc = config.get("disk_prealloc") or "off" + disk_note = "" if prealloc == "off" else f" (prealloc: {prealloc})" + print(f"Disk: {config.get('disk_size', '-')}GB{disk_note}") print(f"GPUs: {self._format_gpu_info(config.get('gpus'))}") print(f"Boot Progress: {info.get('boot_progress', '-')}") if info.get("boot_error"): @@ -1909,6 +1918,15 @@ def _patched_format_help(): compose_parser.add_argument( "--secure-time", action="store_true", help="Enable secure time" ) + compose_parser.add_argument( + "--no-storage-discard", + action="store_true", + help=( + "Keep deleted data-disk blocks instead of returning them to the " + "host. Required by 'deploy --disk-prealloc falloc|full', and it " + "also stops the host from seeing filesystem allocation patterns" + ), + ) compose_parser.add_argument( "--swap", type=parse_memory_size, @@ -1979,6 +1997,19 @@ def _patched_format_help(): deploy_parser.add_argument( "--hugepages", action="store_true", help="Enable hugepages for the VM" ) + deploy_parser.add_argument( + "--disk-prealloc", + choices=["off", "metadata", "falloc", "full"], + default=None, + help=( + "Preallocate the data disk on the host: metadata (qcow2 metadata " + "only), falloc (reserve space, fast) or full (reserve and zero). " + "falloc and full reserve host blocks and require " + "storage_discard=false in the compose file (see " + "'compose --no-storage-discard'). Defaults to the VMM's " + "cvm.disk_prealloc setting" + ), + ) deploy_parser.add_argument("--kms-url", action="append", type=str, help="KMS URL") deploy_parser.add_argument( "--kms-encrypt-url", From b6ea12379047fccc46461973e083235611318e98 Mon Sep 17 00:00:00 2001 From: Kevin Wang Date: Fri, 18 Sep 2026 23:57:43 -0700 Subject: [PATCH 42/93] feat(vmm-ui): offer disk preallocation in the create dialog Adds the mode select next to the storage size and carries it through the create and clone-config payloads. When a block-reserving mode is paired with Storage Discard still on -- the combination the VMM rejects -- the dialog says so instead of waiting for the deploy to fail. --- .../vmm/ui/src/components/CreateVmDialog.ts | 21 +++++++++++++++++++ dstack/vmm/ui/src/composables/useVmManager.ts | 10 +++++++++ 2 files changed, 31 insertions(+) diff --git a/dstack/vmm/ui/src/components/CreateVmDialog.ts b/dstack/vmm/ui/src/components/CreateVmDialog.ts index 4334d8844..8325a6210 100644 --- a/dstack/vmm/ui/src/components/CreateVmDialog.ts +++ b/dstack/vmm/ui/src/components/CreateVmDialog.ts @@ -61,6 +61,13 @@ const CreateVmDialogComponent = { return `queues: auto${cap}`; }, }, + computed: { + // Only falloc and full reserve host blocks; metadata does not, so it does + // not conflict with discard and the VMM accepts it either way. + reservesDiskBlocks(): boolean { + return ['falloc', 'full'].includes((this as any).form.disk_prealloc); + }, + }, emits: ['close', 'submit', 'load-compose'], template: /* html */ `
@@ -116,6 +123,20 @@ const CreateVmDialogComponent = {
+
+ + + Turn Storage Discard off, or this deployment is rejected: the guest would return the reserved blocks to the host on its first trim. +
+