Repository navigation
Conversation
The PCI transport and the vhost-user backend each carried a copy of the code that sends a wake event to a device worker and kicks its notifier. Move the event sender and the notifier from VirtioDevice into a new WorkerCtl, which the device shares with its transport, and keep the only copy of that code there. The following changes extend WorkerCtl to track device resets. VuChannel events from the vhost-user backend now also kick the notifier, so that a running worker picks them up right away. Signed-off-by: Changyuan Lyu <changyuanl@google.com>
A transport asks the worker of a device to reset it with a Reset event, but has no way to learn when the worker has stopped using the queues. The transport therefore reports a reset as completed right away, while the worker might still write to the rings, e.g. into memory the driver has already reused. Number each Reset event, and have the worker send the number back once it has left its event loop, drained in-flight requests, cleaned up the backend and released the guest memory layout. A worker that has not started acknowledges a reset right away. The worker is the only holder of the sending end, so its exit also completes all pending resets. WorkerCtl offers poll_reset() to transports that must not block, and wait_reset() to those that can. WorkerCtl takes a sequence number and sends the event under one lock, so that the worker sees resets in the order of their numbers. It compares the numbers in serial number arithmetic, which stays correct even if the counter wraps around. So that a reset can still complete after an error, the worker no longer exits when its event loop fails. It cleans up the backend, which also covers a partially activated device, and waits for the next start. For the same reason, vhost-vsock now logs errors during reset instead of panicking. The transports still report resets as completed right away. The following changes make them wait for the acknowledgement. Signed-off-by: Changyuan Lyu <changyuanl@google.com>
When the driver writes 0 to device_status, the transport clears the registers and reports the reset as completed right away, while the worker might still be using the queues. The driver then reuses the ring memory, which the worker can still write into. Pci::reset, which runs when the VM reboots, has the same problem. For a reset requested by the driver, keep all registers, including the non-zero device_status, until the worker acknowledges the reset. The driver must poll device_status until it reads 0 before reinitializing the device (VirtIO spec 1.4, Sec 4.1.4.3.2), so the vCPU does not wait. The transport completes the reset once a later register access finds the acknowledgement, and ignores all writes in the meantime. A device that has not started completes its reset right away. Pci::reset waits for the worker instead, for at most 10 seconds. If the worker does not complete the reset in time, the reboot fails with an error rather than hanging on a stuck worker. Queue notifications do not touch the registers, so they skip the new lock. Signed-off-by: Changyuan Lyu <changyuanl@google.com>
To reset a device, the vhost-user frontend disables the rings with SET_VRING_ENABLE and then sends SET_STATUS(0). Per the vhost-user spec, the backend still processes a disabled ring, only without side effects. For example, it still consumes transmitted packets and writes the used ring. The worker could therefore report the reset as completed while the backend still writes guest memory. Stop each ring with GET_VRING_BASE first. Once all rings are stopped, the device is suspended and the backend no longer writes guest memory or sends notifications. Each message waits for its reply, so the rings have stopped by the time the worker acknowledges the reset. Signed-off-by: Changyuan Lyu <changyuanl@google.com>
The vhost-user backend replies to SET_STATUS without DRIVER_OK right after asking the worker to reset, and replies to GET_VRING_BASE without stopping anything. A frontend takes either reply to mean that the rings have stopped, and might then hand the ring memory back to the guest while the worker still uses it. Wait for the worker to complete the reset before replying. Blocking is fine here: in the vhost-user protocol the reply is the only completion signal, and the backend thread serves no one else. GET_VRING_BASE stops the whole device, since one worker serves all queues. Track whether the worker runs separately from the device status, so that a frontend can restart the device after stopping it with GET_VRING_BASE, and can change the memory layout while it is stopped. Signed-off-by: Changyuan Lyu <changyuanl@google.com>
A device reset clears the whole MSI-X table, which masks all vectors and zeroes their addresses and data. The table belongs to the PCI function, though, and the virtio spec only requires a device reset to unmap the configuration and queue events from the vectors. A driver that resets the device and then reuses the vectors it has programmed would receive no more interrupts. Linux reprograms the table after each reset, which hides the problem. Keep what the guest has programmed into the table on a device reset, and only release the irqfds that workers set up. Clear the table on a PCI reset instead. Signed-off-by: Changyuan Lyu <changyuanl@google.com>
A worker that fails while its device is running cleans up and waits for the next start, but the transport cannot tell, and keeps presenting a working device to the driver. Turn the reset acknowledgement channel into a channel for all worker reports, and have the worker report failures there. A failure stands until a reset completes. Reports arrive in order, so a failure reported before the acknowledgement of a reset is gone with that reset, while a later one is not. The following change lets the transports tell the driver. Signed-off-by: Changyuan Lyu <changyuanl@google.com>
When the worker of a running device fails, e.g. because the driver has set up a queue that the device cannot use, the device silently stops serving the queues. Set DEVICE_NEEDS_RESET and send a configuration change notification, which the virtio spec asks of a device that enters an error state that needs a reset (VirtIO spec 1.4, Sec 2.1.2). The PCI transport sets the bit when the driver reads device_status, and the vhost-user backend when the frontend reads the status. Resetting the device clears it. Signed-off-by: Changyuan Lyu <changyuanl@google.com>
Lencerf
force-pushed
the
fix/virtio-reset
branch
from
November 9, 2026 21:01
f2fa169 to
b27a6e1
Compare
Coverage Report for CI Build 42698488512Coverage increased (+1.4%) to 40.834%Details
Uncovered Changes
Coverage Regressions2 previously-covered lines in 2 files lost coverage.
Coverage Stats
💛 - Coveralls |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A virtio transport reports a device reset as completed right away, while the
device worker might still be using the queues, e.g. writing into ring memory
that the driver has already reused. This series makes workers acknowledge each
reset, and the transports wait for that acknowledgement:
device_statusuntil the worker acknowledgesa reset requested by the driver. The driver polls
device_statusuntil itreads 0 (VirtIO spec 1.4, Sec 4.1.4.3.2), so vCPUs never block. Writes to the
common configuration are ignored in the meantime.
Pci::reset, which runs on reboot, waits up to 10 seconds for the worker, andfails the reboot with an error otherwise.
GET_VRING_BASEinstead of onlydisabling them, and the vhost-user backend waits for its worker before
replying to
SET_STATUSorGET_VRING_BASE.function.
DEVICE_NEEDS_RESETand sends a configuration change notification.Every commit passes
cargo +nightly fmt --check,cargo clippy --all-targets -D warningson x86_64-unknown-linux-gnu/musl, aarch64-unknown-linux-gnu andaarch64-apple-darwin, and
cargo test --workspacelocally.