Repository navigation
chore: sync upstream dstack and validate alpha fork - #2
Merged
Merged
Conversation
Split out of the vhost/multiqueue change (Dstack-TEE#1145); carries the interface lifecycle parts of Dstack-TEE#1154 and its follow-ups. - netd records each interface's owner in the kernel interface alias - new `RemoveVm`, `ListInterfaces` and `RemoveInterfaceByName` netd RPCs, plus the `dstack-vmm netd list|remove-vm|remove-interface` subcommands - prepare requests carry the VM's workdir, which netd logs - the VMM sweeps a VM's interfaces before every launch and on stop, update, and removal, and tracks outstanding cleanup in `.netd-pending` - removal keeps the VM directory until netd confirms the release - a per-VM launch lock serializes launch, stop, update, and removal - the fake netd used by tests serves the real Netd RPC
- `.netd-pending` is the only record that netd may hold interfaces for a VM. The runtime snapshot no longer takes part in cleanup decisions, and a VM without the marker never contacts netd to release anything. - A removal retries the release with backoff inside its background task until netd answers, instead of returning and waiting for the next VMM start. The VM stays marked as being removed meanwhile, so the removal mark is handed to the task and no longer needs to be retained by hand. - The VM directory is deleted with `.removing` last, and `.removing` is written durably, so a removal interrupted by a crash is resumed. - A periodic reconciliation releases what stopped VMs still hold, which covers a stop that could not reach netd and a VM that exited on its own. - `create_vm` removes a VM whose first start failed through the normal removal path, so interfaces netd could not release keep their marker.
…ing removal - A launch whose first netd request never reached a netd clears the `.netd-pending` marker it just wrote, unless an earlier boot still left one. Without this, a bridge deployment on a node without netd left the VM stuck in removal, holding its CID, until netd was installed. - `RemoveInterfaceByName` reports failure when a binding that exists, or is all the name could still hold, was not deleted, following the node's filter policy like the sweep. It previously printed "removed" regardless. - Listing nwfilter bindings uses a 10s timeout, and a sweep whose listing timed out skips binding deletion instead of waiting on the same hung libvirtd again, so a sweep fits inside the VMM's request timeout. - Reconciliation skips running VMs from one supervisor listing instead of taking every running VM's launch lock each pass, and does not burst after a slow pass. - Drop comments that described the removed legacy marker backfill.
- `run_command_with_timeout` drains stdout and stderr (and feeds stdin) on their own threads while it waits. Waiting first turned any output larger than a pipe buffer into a timeout, so on a host with a couple of thousand nwfilter bindings every sweep read as a hung libvirtd. - A sweep and a manual removal set a present interface down before deleting its binding, so a QEMU a stop only signalled does not reach the bridge unfiltered while the binding goes. - A listed binding at a name that is now a macvtap is deleted too, instead of being skipped while the sweep reports success. - A manual removal whose binding listing timed out does not also wait for a deletion from the same libvirtd. - `resize_vm` refuses a VM being removed and holds its launch lock, like `update_vm`, so it cannot recreate a directory a removal just deleted. `remove_all` puts `.removing` back if the directory could not be removed. - Correct the netd trust-boundary doc now that `RemoveInterfaceByName` accepts a netd-shaped TAP name.
- A sweep leaves the binding alone at a name held by a device netd did not create, and lets `remove_interface` refuse it, as a manual removal does. - A manual removal of a macvtap on a filtering node does not report success when the binding listing failed; like a sweep, it requires confirmation.
Add two guest kernel patches for many-vCPU TDX guests: wake_up_q() notification IPI batching (tdx_wake_q_batch) and PV single-target function IPIs (tdx_pv_single_ipi). Both are runtime switches, on by default in TDX guests and unavailable elsewhere; boot parameters or sysfs writes can turn them off. Build the haltpoll cpuidle driver as a module so guest halt polling can be enabled and disabled at runtime. It stays opt-in. On a 32-vCPU guest the new defaults were not slower than the previous behavior on any tested workload and raised the synchronization benchmark geometric mean from 13.6% to 38.5% of an ordinary VM.
Install a TDX guest scaling profile helper as /usr/lib/dstack/tdx-guest-tune.sh with throughput, default and off profiles. It is kept out of PATH because it may be removed later; the kernel switches remain the stable interface.
Patch 0010 defers the private-to-shared conversion of the default SWIOTLB pool to fs_initcall_sync on TDX guests. A bounce through the pool before that would fail silently: the device reads a buffer it cannot decrypt. Record in SWIOTLB when swiotlb_update_mem_attributes() has run and WARN_ON_ONCE() in swiotlb_tbl_map_single() if the default pool is used before that on a platform with memory encryption, so a future early DMA user is caught instead of corrupting data. The flag is __ro_after_init; it is written from __init code before mark_rodata_ro(). Patches 0001-0010 still apply to pristine 6.18.40 with --fuzz=0, and swiotlb.o and mem_encrypt.o build with W=1 without new warnings.
…st-boot os/kernel: fix quadratic boot time of many-vCPU TDX guests
feat(vmm): clean up all netd interfaces for a VM and retry failed removals
The sanitize-on-attach path issued the Secondary Bus Reset by writing Bridge Control in the upstream bridge sysfs config space and re-probed devices through /sys/bus/pci/drivers_probe. Both files are writable by root only, so the feature could not be enabled in production where dstack-vmm runs as an unprivileged user with no sudo. Switch to the VFIO_DEVICE_PCI_HOT_RESET ioctl, which makes the kernel perform the same bus reset. The ioctl is authorized by device ownership rather than privilege: the caller presents fds for every VFIO group affected by the reset, and the /dev/vfio group nodes are the same ones QEMU opens to attach the GPU, so the VMM user already has access. A single group fd suffices because every sanitized GPU sits alone behind a dedicated PCIe bridge and alone in its IOMMU group. The bridge topology check is kept as defense, and the kernel-reported set of affected devices must all belong to the GPU own group or the launch is aborted. Devices stay bound to vfio-pci across the reset, so the drivers_probe re-probe logic is no longer needed and is removed. Not yet validated on GPU hardware; see plans/2026-08-14-vfio-gpu-hot-reset.md for the pending experiment.
Expose the sanitize path as "dstack-vmm sanitize-gpu <slot>..." so operators can reset GPUs by hand and the pending hardware experiment can exercise exactly the code path used at VM launch, running as the unprivileged VMM user. The subcommand needs no server configuration, only /dev/vfio access, and is handled before config loading like the other special modes.
…octls vmm: sanitize GPUs through vfio-ioctls hot reset
…mages ci: publish release images to GHCR
ci: verify GHCR release images are public
ci: avoid updating shallow KMS release dependencies
The console could search by keyword but not by lifecycle status. The list is paginated server-side, so a client-only filter would only cover the current page and would report the wrong total. Add StatusRequest.status and apply it in list_vms before pagination, using the same running/stopped/exited/stopping/removing values already shown in the Status column. Empty status keeps the previous all-status behavior. The toolbar select resets to page 1, matching page-size changes. Status computation is shared through vm_runtime_status so the filter and the list agree. Verified with `npm run build` in dstack/vmm/ui, `cargo check -p dstack-vmm-rpc`, and prek on the changed files.
prek and cargo fmt --check failed on the four-item use super line introduced while resolving the rebase conflict.
…-filter feat(vmm): add a status filter to the console instance list
The dstack recipes copy source trees from outside BitBake's work directory. Archive-mode rsync attempts to preserve their group ownership, which can fail when the build user cannot assign the source group.\n\nDisable group preservation while retaining the other archive-mode attributes.
…trap-root-keys fix(kms): refuse to auto-bootstrap over existing root keys
…-canonicalization fix(verifier): a DebugSwap guest verifies, and four more ways host bytes reach a measurement unchecked
…device-id fix(attest): derive Nitro Enclave device_id from the EC2 instance id
…-bounds fix(dstack-mr): bound the TDVF parser and the TD HOB witness against attacker bytes
…overflow fix(guest-agent): reject a certificate validity bound it cannot represent
Only `app_id` was length-checked, and both Ethereum backends `padStart` a short hex value to full width before the `readContract` call. A 31-byte compose hash is therefore submitted as - and is indistinguishable from - the whitelisted 32-byte hash whose leading byte is `0x00`. The trusted OS always emits a 32-byte sha256 and either a 20-byte instance id or none, but nothing between the event log and the contract call asserted that. Assert it at `AuthApi::is_app_allowed`, the one point every backend call goes through, and stop `DstackApp.addComposeHash`/`addDevice` from allowlisting `bytes32(0)` - which is exactly what that padding produces - since `_initializeCommon` already declines to seed it.
`GetMeta`, `GetAppEnvEncryptPubKey` and `GetTempCaCert` need no client certificate - `RaTlsClientVerifier::client_auth_mandatory()` is false - and each drives two or three `readContract` calls on the operator's chain RPC. 24 concurrent anonymous `GetMeta` calls cost 24 upstream calls, and because the backend answers `isAllowed: false` when it cannot reach the chain, burning an RPC quota converts directly into denial of key release for every app. `InFlight` collapses requests that are identical *and* overlapping in time into one upstream call; the same burst now costs one. It is not a cache: the entry is retired before the result is published, so a call issued the instant the burst finishes goes upstream again, which is what `repeated_authorization_is_never_served_from_a_decision_cache` pins. It bounds amplification under concurrency, not the serial request rate - that belongs at the listener, which the KMS does not have.
Keys::store wrote the root keys before the certificates, so a crash could leave a root key without the rest. keys_exists() then sent the KMS into onboarding, where every path refused because a root key existed. Write the root keys last and treat both of them existing as the only sign of a completed bootstrap. Anything short of that is overwritten by the next bootstrap.
…otstrap fix(kms): let a bootstrap interrupted by a crash be redone
…idths fix(kms): bound compose_hash and instance_id before the policy backend
Cache the auth API info and the KMS self-authorization for 1s in KmsState. moka coalesces concurrent misses and handles cancellation, and the TTL also bounds the serial rate. App authorization is left uncached.
…nflight fix(kms): bound auth api calls from unauthenticated RPCs
…ficates Only substitute the local os_image_hash when the source attestation carries no config at all (legacy cert format), and warn when it does.
The allow check and gatewayAppId were read at latest in separate calls, so one decision could mix blocks. Pin both to the same block number.
…-pinning fix(kms): read each auth-eth decision from a single block
…age-fallback fix(kms): scope the onboarding os_image_hash fallback to legacy certificates
…-tcb-vacuous docs(kms): note that requireTcbUpToDate does not gate AWS NitroTPM
…py-bound fix(dstack-util): bound host-shared copy by bytes read, not a prior stat
fix(attest): require /dev/nsm before selecting the Nitro Enclave variant
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Sync upstream
Dstack-TEE/dstack@d9de8915a648c889714b187c984059b05debef49into the AlphaCompute fork, starting fromalpha@c12e96adaeea51d1c79608123d41a6f521db46cd. This imports 124 upstream commits while preserving the fork’s independently encrypted environment entries, legacy encrypted-env support, and Yocto loader compatibility.The release workflows retain upstream release-tag checkout, provenance and anonymous-pull verification while publishing gateway, KMS, verifier and local-key-provider images to this repository owner’s GHCR namespace. Kernel-builder release notes match their existing dynamic publish path. A registry contract test executes the owner normalization and validates publish/pull/provenance agreement without publishing. CI branch filters include
alphaso fork PRs receive the upstream validation jobs. Upstream Nitro key release remains explicitly opt-in and gated after attestation verification and external authorization.Validation so far: all 7 release build-orchestration and registry contract tests pass locally with GNU coreutils; Rust workspace formatting passes; actionlint passes for all changed workflows; the fork delta passes whitespace checks. Upstream patch payload whitespace is preserved. The local Rust test build is blocked by the existing Linux-only SEV firmware dependency on macOS; mkosi acceptance also requires GNU tools. Linux-hosted CI is running for the PR. This PR remains pending exact-head hosted results and review.
This change does not deploy an enclave, modify AWS KMS or encrypted volumes, prove production key release, or establish hardware acceptance. Hardware deployment and encrypted-state recovery require separate evidence against the built artifact.