[Japabu]

Problem

The kernel drove QEMU's virtio console itself: a single-port virtconsole that it replayed its records into, with logkeeper's program lines going through it too. That driver needed its own DMA pool, a kernel-driver half of DeviceSpace, and a rule (toyos_pci::console) for which 1af4:1043 function belonged to the kernel. QEMU's virtconsole drops whatever a full host pipe refuses, and the kernel waited on the device from its console path, the panic path included.

What it does now

/system/bin/virtio-console is the virtio console device's driver (virtio 1.2 §5.3), with one process per virtio-serial function. Each process holds its function's claim, reads the name the device gives the function's one port, and hands the port to the service that name belongs to. Its row receives exactly those services:

port name	handed to	what crosses
org.toyos.console (QEMU's stdio)	logkeeper's log-carrier	the log from its first line; DELIVERED counts back
com.redhat.spice.0 (qemu-vdagent)	vdagent's new vdagent port	both directions on one connection (toyos_vdagent::endpoint)

• A port whose service the image does not run ends the driver with exit 0. This is the existing "nothing to serve" rule.
• vdagent is now a boot service. It serves vdagent and holds the compositor's clipboard, and it holds no device.
  • Only virtio-console may receive vdagent (HELD_ALONE), and a row that receives both vdagent and clipboard is refused.
  • So the driver holds the device and the endpoints, and the agent holds the clipboard. No process ever holds both.
  • That is the owner's ruling on items 73/84: distinct device instances, one transport implementation, explicitly granted service endpoints, and no transient cross-role authority.
• The agent's endpoint carries the port on one connection: the host's opens, closes and bytes, and the agent's bytes.
  • The agent answers each close with its own CLOSED. The driver counts the closes not yet answered and drops whatever the agent sends while any is outstanding. So no host session starts with an earlier session's tail, even when the host closes, opens and closes again before the agent reads the first close (Fence, host-tested).
  • The device gives no order between its control and receive queues. The driver tells the agent the host's buffers before the opens and closes read in the same round; that order is the driver's choice, not the device's.
• Backpressure, not drops. A host buffer the agent has no room for stays unposted until it does, and the agent's frames are read only into a free transmit buffer. Following IPC phase 2: a connection carries whole messages #1121's pattern, the server side makes everything a client receives. Here that is no event at all: the connection's own readiness is the doorbell.

The UART goes quiet while the console's carrier carries the log (owner item 73.1):

• Once the host has taken the first bytes, logkeeper calls SYS_CONSOLE_CARRIED on its console and stops writing program lines to it.
• klogd writes console: the log is carried elsewhere from here… on the UART, then advances past what it would have written.
• No second output path is kept.
• The UART takes the log back in any of these cases:
  • The carrier goes: the driver ends or its pipe closes, and klogd says console: the UART carries the log again.
  • The carrier stalls: its channel has taken none of the bytes it is owed for STALL (5 s, measured from its last progress). logkeeper tells the kernel the UART carries the log again and keeps feeding the carrier. Once the channel takes more, the UART goes quiet again. So a wedged driver, or a device that never completes a transmit, cannot leave the operator with neither channel. While it lasts, lines can reach both channels, out of order. That trade sets the short bound: a false stall only duplicates lines, while a missed one leaves both channels silent. Under QEMU a stall is a defect, so the harness fails any capture carrying logkeeper's stall line (CARRIER_STALLED).
  • The machine stops: logkeeper waits up to 2 s for the carrier to have the log up to the flush, then hands it back. The kernel's take of the wire for the stop is the backstop.
  • A fatal path: the log goes back for good, before that path's first record.
  • logkeeper dies: its console is a deferred object whose last handle releases the carrier (on_zero_handles), and a dying process's releases run before its end is published. The single carrier word is kernel::carrier, host-tested.

How panic and early-boot output reach the operator

• Early boot is on the UART, unchanged. Nothing is quiet until the carrier's channel has taken bytes, and the carrier is handed the log from its first line, so the console port shows early boot as well.
  • Where no carrier ever runs, the UART is the console throughout: metal (no virtio-serial), machines that refuse every claim, and aarch64 cargo run today.
• Every fatal entry sets for-good (SeqCst) before its first record, and klogd's discard reads the word after each record, so it never advances past a fatal report.
  • The fatal entries are: the panic handler, halt_all_cpus, panic_registers, double fault, #MC, fatal_exception, Ring 0 #PF, aarch64 exception() and its recursive user fault, FATAL_HALT, and the IOMMU drain's kernel-owned fault.
  • panic_flush then drains the ring to the UART without the carrier.
  • I re-listed every halt_all_cpus/panic_registers caller after the rebase, and main added none. The one that logs before halting, the panel-painter-stalls boot actuator, runs before userland, when no carrier can exist.
• cargo run and the harness hear both, through one rule (build/vmconsole.rs: Uart, merge, UartSocket). The console port is on QEMU's stdout and the UART on a socket the host listens on.
  • Every UART line after the kernel's quiet line is shown, and so is a kernel death before it. UART lines before the quiet line are on the port too, so they go only to the UART's file (/tmp/toyos-uart-early.log for cargo run).
  • One thread reads both channels and drains the port before each UART read. At the port's end it reads the UART to its end too, so the stop's last lines and a fatal report's tail reach both the terminal and the file.
  • cargo run's terminal therefore shows a panic, the stop's last lines and everything after the carrier goes or stalls. Before this round those reached only the UART file.
  • The death vocabulary moved from tests/common/serial.rs to toyos_logstream::death, so the launch reads the harness's one table.

Deleted

• From the kernel: drivers/virtio_console.rs, drivers/virtio.rs, mm/dma.rs (DmaPool/Dma), DeviceSpace, the untranslated facades, toyos_pci::console, the virtio backend of serial.rs, the backend-change replay, and pmm::alloc_contiguous_for_kernel (its last caller was the DMA pool).
  • toyos-dma keeps only the register bound, as toyos-register.
  • The PCI capability walk's self-test moves to drivers/pci.rs (6/6).
• Records: Profile::HeadlessConsoleUntranslated and its test, build/kernelconsole.rs's firmware split, and the issues for the kernel console's wedge and drops.

Decisions

• logkeeper is the console port's service, with no relay process between it and the driver. The driver carries every byte of the log anyway, so a relay would add a process and isolate nothing.
• vdagent runs on every machine. On a machine with no clipboard channel (the T14, the AMD) it waits on its port and never connects to the compositor. The alternative was on-demand start, which the supervisor does not have. Its memory cost is not measured.
• SYS_CONSOLE_CARRIED is 160. Main's last number is SYS_MESSAGE_RECV, 159; SYSCALL_PROFILE_BINS is now 162.
• Every image that boots on a q35 virtio machine carries the carrier's rows, and the harness refuses one that does not, by name.
  • After the rebase that also covers linkcase (new on main) and benchcase, which osbench_capability_groups_exit_0 now boots on q35.
  • On a bench the row starts nothing.
• An earlier round's design does not land. Rows taking turns at a function by exit 0 (Drives::declined, overlapping drives rows, and a shared claim crate) existed only on this branch. It is gone from the branch and its history: the PR is one commit, and against main it changes neither the supervisor nor the manifest.
• Kept from earlier rounds:
  • logkeeper's reader STALLED and MAX_READERS are unchanged; the carrier is its own slot.
  • test-runner keeps reading the kernel console, which is the UART.

Checks

Rebased onto ece8ef4e6 as caf414e4a (one commit):

• The conflicts were main's wlan/wifi rows and the HELD_ALONE list; both sides are kept.
• SYS_CONSOLE_CARRIED stays 160, since main still tops out at 159.
• cargo run -- --ci host: exit 0, 108/108 (log).

Review r2 (d7fd28f00):

• B5: the carrier's header and the STALL rationale are corrected.
• B6: merge reads the UART to its end. The merge host test now writes the UART's last line after the port has ended. A negative control shows the test fails without the drain.
• B7: the console driver waits with no deadline once it has reported the silence, so it no longer spins.
• S1: a capture carrying the stall line is not clean.
• S2: the stall line is said only when the UART actually goes loud.
• S3: a lease is recorded as the residual's proposed exit.
• Checks:
  • the five console rows: exit 0, 5/5;
  • vmconsole host tests: 6/6;
  • logkeeper host tests: 17/17;
  • harness checks: 43/43.
• Evidence: https://github.com/ToyOSOrg/evidence/tree/c9b59e2a1e86d7a7a7d8182363e4e6e7e48f68e1/r5-review-r2
• Host CI was not rerun for this round; its last green run was on a3c1c5c30's tree.

One commit, a3c1c5c30 on origin/main 190f3b164, rebased unchanged onto 9b3711cdb as 2de594d8a; main's nine commits there touch no file this change does. Evidence: round 3 at https://github.com/ToyOSOrg/evidence/tree/96e250ea132d81c06b2c37404219c464edb50011/r3-rebase-rework, review r1 at https://github.com/ToyOSOrg/evidence/tree/7b207c5c9af4d67e1732fa4eeb5690681eed4288/r4-review-r1 (masked).

Review r1 (a3c1c5c30):

• cargo run -- --ci host: exit 0, 108/108. cargo build --tests and the vmconsole host tests also pass after the rebase onto 9b3711cdb.

• Guest rows (exit 0, 5/5):

  • the_opening_exchange_with_qemus_agent_completes, after B1;
  • the_uart_carries_the_log_again_once_the_console_carrier_goes;
  • machine_shutdown_short_stop;
  • machine_shutdown_wire_held;
  • a_drives_row_holds_one_process_per_function.

  These are the rows the merge and logkeeper's change reach.

• cargo run, stdout to a file and the guest powered down over QMP after the UART went quiet. The output carries the stop's UART-only lines, from console: the UART carries the log again to Shutting down.

• New host tests:

  • the Fence's close, open, close case and an unasked CLOSED;
  • Carriage's stall: the bound runs from the last progress, it is said once, it is quiet again on progress, and an idle log is no stall;
  • Uart's rule, whatever the cuts, and a kernel's death before the quiet line;
  • merge's order and file, with the port's unended last line.

• The whole-suite run below predates this round. The round changed the harness's reader to the shared merge, and the five rows above exercise it.

Resolving the rebase:

• Main's keep_reserved and VT-d unclaimable are kept; untranslated is dropped, since its only outside caller was DeviceSpace.
• alloc_contiguous_for_kernel is deleted.
• logkeeper-api moved to FrameRx, and vdagent's IPC phase 2: a connection carries whole messages #1121 SAFETY wording was carried into the moved channel code.
• The record for a socket-export race that main fixed is dropped.

Host and build:

• cargo run -- --ci host:
  • at f3046c768 (the plain rebase): exit 0, 108/108;
  • at 7d9f5d277: 107/108. The red was ServesWithoutService("vdagent"), fixed;
  • at 13af3ee77: exit 0, 108/108.
• cargo run -- --build-only at 13af3ee77: exit 0.

Guest, whole q35 suite (cargo test --test toyos-build) at 5ff35be86: 88 of 89.

• The red, osbench_capability_groups_exit_0, was the harness refusing benchcase for lacking the carrier rows.
• 13af3ee77 adds only those rows, and the row passes there. No other test reads benchcase, so the suite result stands for the head.
• The console-reading rows among the 88:
  • the_uart_carries_the_log_again_once_the_console_carrier_goes;
  • the_opening_exchange_with_qemus_agent_completes, which now also requires the driver's hand-over line before the agent's;
  • the four machine_shutdown_* rows;
  • a_drives_row_holds_one_process_per_function;
  • iommu_virtio_platform;
  • a_link_ends_under_a_held_stream_and_comes_back;
  • session_lifecycle.

Negative control: a checked patch that makes the driver pass none of the host's bytes to the agent. The clipboard exchange stalls (exit 1); the patch was restored in the same run. Earlier rounds' controls still apply: the harness's UART merge, logkeeper's restore, and FATAL_HALT's ordering.

Host tests added in round 3:

• the probe (name on port 1 only, contradictions refused);
• Fence;
• the gates on the agent's port and on the union of vdagent and clipboard.

Metal: no new run is requested.

• The kernel console path this PR changes is shared with metal: the carrier word in klogd, for-good on fatal entries, and serial.rs without its virtio backend. With no carrier it behaves as before.
• That path was run on the T14 at ff2af0a4d (testcases and testcases-watchdog: 255 passed; the 5 reds are known and independent) and at 7280e0970 (pci_capability_walk 1/1).
• The rebase conflicted in none of those files, and the rework touches no kernel code.
• One T14 testcases boot at the head would check the merged result. It is the orchestrator's call.

Temporary gaps and risks

• aarch64 cargo run's terminal shows nothing but a kernel death. Claims are refused there today (issues/an-aarch64-cargo-run-boot-hands-over-no-pci-claim.md), so no carrier runs, and the merge holds back the pre-quiet UART lines the port would otherwise replay. The whole log is in /tmp/toyos-uart-early.log. The harness's virt profiles have no console port and read the PL011 as before.
• When the carrier goes or stalls, the lines it had not delivered reach neither channel. The same holds for the kernel records klogd advanced past before logkeeper gave the UART back, and across logkeeper's own death. /log keeps the program lines.
• Residual: logkeeper itself stuck but alive keeps the UART quiet until the stop or a fatal path. Nothing in the kernel watches whether the holder still reads the ring.
• Both are recorded, each with an exit, in issues/the-console-loses-what-its-carrier-had-not-delivered.md.
• The agent's endpoint has no guest test of its close/reopen path. QEMU's chardev never closed during the suite. Fence is host-tested, the close, open, close case included; vdagent's one CLOSED per close rests on inspection.
• A large clipboard push now reaches the device in 8 KiB frames instead of one element. The vdagent stream is chunk-framed, but no test moves a large clipboard through QEMU (issues/the-shared-clipboard-is-unmeasured-on-linux-and-windows-hosts.md is the nearest record).
• Inference, not measured: the boot before the carrier starts, about 4.6 s into a TCG boot, pays the UART's per-byte price throughout.
• issues/a-forced-reboots-last-userland-lines-miss-the-console.md (main) is about the console at a forced stop. session_lifecycle passed in this suite; this PR does not claim to close it.

🤖 Generated with Claude Code

[Japabu]

Review r1: #1062 at 7280e0970

Read: both commits, the full diff against origin/main (merge base 3a19907ab), every changed source file, and the callers and boundaries that matter here: console handle minting (loader/start.rs), the supervisor's console narrowing (STDIN_RIGHTS), the object lifecycle rules (object/mod.rs), every halt_all_cpus and panic_registers caller, Virtqueue::poll_used, the harness reader, the issue edits and the evidence tree at 2bcd0cc3f. I ran nothing.

Architecture

The direction is right, and it simplifies the system. The kernel loses its only DMA driver, DmaPool/Dma, DeviceSpace, the untranslated facades, toyos_pci::console, the backend switch and the replay rewind. The VM console becomes a drives row like vdagent. The kernel adds back one small thing userland cannot do: tell the kernel's UART writer to step aside. kernel::carrier is a single word with a pure host test. The kernel and userland split their responsibilities sensibly:

• logkeeper decides when the log is carried.
• The kernel enforces the takebacks: the holder's end, a fatal path, the stop.

The new syscall's authority checks out. It needs WRITE on a console. Only the supervisor's console and logkeeper's endowed console have it. Every other program gets STDIN_RIGHTS, spawn mints keep the narrowed rights, and a holder cannot widen them.

Turn-taking by exit 0. I judge this ordinary composition, not a mechanism of weight:

• It adds a declined list per function.
• A row's first turn spends no budget.
• With a single row, an exit 0 ends where it did before.

The probe window, though, gives one process both roles' authority in substance (B3). That needs an owner ruling.

Net size: production and build +2327/−2491 (inline unit tests included), tests +417/−112, issues +28/−252. Nothing dead is left from the deletions. I found no reference to the removed types, modules or issue files outside issues/, and kernel clippy runs with -D warnings.

Findings

B1. BLOCKER: kernel/src/iommu/fault.rs:212, :247: one fatal path logs its first record before for-good.

• The drain logs each fault, then conclude → halt_all_cpus sets on_the_uart_for_good.
• An enumerated function no process claims counts as Owner::Kernel, so its fault halts the machine.
• While a carrier carries the log, klogd on another CPU can advance past those fault lines in discard_pending(true).
• panic_flush then drains from DRAINED, after them, and the carrier halts with the machine.
• So the fault, the one line that explains the halt, reaches neither channel.

This contradicts the PR's claim ("a fatal path … before the fatal path's first record") and the module contract in log/console.rs. Every other halt path I traced sets for-good first: panic!, panic_registers, double fault, machine check, kernel exceptions, aarch64's recursive user fault and FATAL_HALT.

Fix: call on_the_uart_for_good() where the drain decides Owner::Kernel, before its log!. Better still, give the fatal reports one funnel, so a new fatal path cannot miss it.

B2. BLOCKER: kernel/src/object/device.rs:195, object/mod.rs:260: logkeeper's death releases the quiet on the last Arc, not the last handle.

• Console is an immediate row.
• object/mod.rs:7 says userland-visible lifecycle rides handle_count, because "a syscall's Arc can be stranded on a killed thread's kernel stack".
• The owner required that logkeeper's crash cannot leave the UART quiet, and asked for established lifetime mechanisms.

Today I found no path that strands one: console reads park on a static watch, and writes and SYS_CONSOLE_CARRIED never park. The guarantee therefore rests on an invariant written nowhere, contrary to the documented rule.

Fix, either:

• make Console deferred and release the carrier in on_zero_handles, which a dying process's Releases runs before its end is published; or
• state the "no console syscall parks holding its Arc" invariant at the immediate Console row, as the File row does.

B3. BLOCKER (owner decision): images/system.toml:135, system/vdagent/src/main.rs:237: during the probe, one process holds both roles' authority.

• Rows take turns in order, so vdagent is started on the console's function too.
• It then holds the console function's claim, which can drive org.toyos.console and the port's host input, alongside its clipboard endowment.
• Its only guard in that window is its own code: it connects to the clipboard after the verdict.
• The reverse case arises if vdagent ever exits 0 on its own function: virtio-console then holds the clipboard function with log-carrier.

"Each holds only its own authority" (system.toml, the PR, the handoff) is therefore not true as stated.

The overlap is bounded. Both are signed system programs, it lasts until the verdict or PROBE_BOUND, and the current row order has the lower-value device as the overlap.

The cleaner alternative keeps the probe in a process with no role authority. That is a broker row, which costs a process and a hand-over.

Item 73.2 said "don't give both the union", so whether this bounded window is acceptable is the owner's call, not ours. Whatever the ruling, correct the comment to describe what is granted.

B4. BLOCKER: system/logkeeper/src/carrier.rs:17: the lost-lines compromise is recorded only in a doc comment and the PR body.

• When the driver dies, two things reach neither channel:
  • the bytes in [delivered, sent);
  • the kernel records klogd discards between the death and logkeeper's Loud. These include the kernel's own fault record for a crashing driver, the line that explains the event.
• The same holds across logkeeper's death.

I accept the behaviour for landing because /log keeps the lines and the owner asked only that ordinary output returns. Root's "No silent debt" still requires an issues/ record with its evidence, owner and exit. One exit to record: logkeeper knows delivered and holds the replay, so on Loud it could put the undelivered tail on the console.

B5. BLOCKER: build/vmconsole.rs:8, lib/virtio-serial/src/lib.rs:31: QEMU's source is cited as the authority.

• These are new citations of hw/char/virtio-console.c / flush_buf.
• Root forbids directly citing another implementation's source, and clean-room provenance belongs in evidence, not the repository.
• The throttling behaviour is already measured: stall/, two 20 s stalls. Cite the measurement ("QEMU 11.1.1, measured"), as other sites do.
• The chardev/char-stdio.c citation in relay is older text moved from kernelconsole.rs. I don't require changing it here.

B6. BLOCKER: evidence: the guest suite is untested against the rewired harness.
tests/common/qemu.rs changes every q35 virtio boot:

• commands now go to a UART socket;
• the published log is logkeeper's replay through the carrier;
• UART lines before the quiet are no longer published;
• every virtio test image gained two rows and is refused without them.

Only 8 rows ran (guest/guest-batch-7280e0970.log), and every test-runner-driven q35 row depends on this. The cheapest sufficient check is one run of the q35 TCG guest suite at head, with its log in the evidence. The virt profiles and metal-only images (benchcase, latencycase) are untouched by this.

S1. SUGGESTION: system/vdagent/src/main.rs:237, system/virtio-console/src/main.rs:99: duplicated probe loop.
The two identify loops are near-identical, about 30 lines each. Returning the held Driver controls from one identify in toyos-virtio-serial-claim would serve both.

S2. SUGGESTION: tests/common/vmconsole.rs, a_fatal_report_reaches_the_uart_while_a_carrier_carries_the_log: the test's distinct claim is thin.

• Its negative control shows it cannot see the race it was written for: the take-back removed, it stays green 3 of 3.
• Its harness claim (UART lines after the quiet are published) is already held by the restore test.

No guest test is owed for the ordering itself. SeqCst for-good before the commit, with Discard reading the word after its record, is sound by inspection once B1 covers every entry point. A one-funnel structure (B1) is the better guard. Either name a defect only this test catches, or drop the boot.

S3. SUGGESTION: early-boot console cost on QEMU is unmeasured.
The PR's inference is that boot before the carrier now pays the UART for every byte, about 4.6 s into a TCG boot. The measurement rig exists, so one boot-to-ready comparison would turn the inference into a number. Metal is unaffected.

The orchestrator's questions

• SYS_CONSOLE_CARRIED
  • Authority: correct, as above.
  • Release on the driver's death: logkeeper's went sends Loud. The restore test and the gone control establish it.
  • Release on logkeeper's death: through Drop. See B2.
  • The stop's wait: at most 2 s. STOP_WAIT bounds the loop, inside FLUSH_MS 5 s, with the kernel's take_for_the_stop for-good as the backstop. sync() runs before the wait, so a slow sync plus 2 s can push the supervisor past its bound. It answers late and goes on, so this is acceptable.
  • Fatal ordering: B1. No test is owed beyond inspection (S2).
• Lost lines: acceptable given /log, once recorded (B4).
• Kernel deletions: complete. The only residue is the kernelslice issue's mention of the deleted mm::Dma, kept as a design precedent, which is fine. Mmio::size's removal is legitimate: carrying the GICR length avoids an architecture cfg on a generic accessor.
• aarch64 gap: honestly reported. The issue now states the empty terminal and the log's location, names PCI apertures are a declaration the kernel holds, made from each root's _CRS (design; stage A: the declaration and the hybrid IOMMU rule) #916's stage B as what restores it, and its exit condition shows the console.
• Evidence: no identifiers found. Paths are masked, and the GPT GUIDs belong to throwaway QEMU images. The tested revision and exit codes are recorded. --build-only predates the aarch64 fix, but the guest batch at head rebuilt the images, so that build is superseded.
• T14 pci_capability_walk: the orchestrator is judging this. The judge now expects selftest 6/6 and split 13/13, which matches the moved self-test.

SEND BACK

[Japabu]

Review r2: #1062 at ebac17a93

Scope: the r1 blockers and suggestions, plus the r1 delta (7280e0970..ebac17a93, +208/−203 across 16 files). I read the delta, then the fatal entry points in the kernel, the object lifecycle (object/mod.rs, releases.rs, process::teardown, console minting in loader/start.rs), the harness shapes and the two changed guest tests. Evidence read at 81a2e3a72: guest/suite-ebac17a93-green.log, guest/suite-a2a172d47-4-red.log, guest/r1-fixed-rows-2.log, guest/socket-export-hang-base.log. I ran nothing. The worktree is mid-rebase onto e9abdd348, so I reviewed the commit objects, not the checkout.

Architecture

Unchanged from r1, and still sound. The r1 delta is small and stays inside the design:

• Console now uses the established deferred-release mechanism.
• One identify replaces two copies.
• The harness gains one shape fact.
• The kernel's virtio negotiation check, which no longer had a subject, is gone.

Nothing in the delta needs an owner decision beyond B3's ruling, which already sits with the owner.

Previous blockers

B1: not resolved. The IOMMU fault drain is fixed. fault.rs:207-210 sets for-good where it decides Owner::Kernel, before the line, and Owner::Kernel implies function.is_some(), so the line is always written. Every other halt_all_cpus and panic_registers caller I traced also sets it first:

• panic!
• double fault
• #MC
• fatal_exception
• nested NMI
• aarch64's recursive user_fatal
• FATAL_HALT

Two fatal entry points still write their explanatory records before for-good. r1 wrongly said "kernel exceptions" were covered; I missed both.

• kernel/src/arch/x86_64/idt/exceptions.rs:477, :500, :503: a Ring 0 page fault.
  • page_fault_handler logs the SMAP report with its kernel backtrace, #PF UNHANDLED … user=false or #PF PRESENT, and only then calls fatal_exception, which sets for-good.
  • A Ring 0 #PF always halts (:487: "a Ring 0 fault is a kernel bug, and it halts below").
  • While a carrier carries the log, klogd can advance past these lines. The SMAP backtrace in particular is in no later record.
• kernel/src/arch/aarch64/trap.rs:141-157: exception(), which covers an EL1 sync, FIQ or SError.
  • It writes KERNEL PANIC, all 31 registers, the page walk and the kernel backtrace, then panic!. Only then does the panic handler set for-good.
  • The handoff says the driver runs on virt with an SMMUv3 and an ITS, so this is a carried machine.
  • report_panic repeats only the message. The registers and backtrace of the most common aarch64 kernel fault are therefore lost from the UART.

The PR's "every fatal entry now sets it before its first record" is therefore false.

Fix: set for-good first in exception(), and in page_fault_handler once the fault is known to be Ring 0, before the SMAP block. A user #PF PRESENT only ends its process, so the call must not be unconditional. Alternatively, use the single funnel r1 suggested. A T14 boot does not settle this; inspection does.

B2: resolved.

• Console is deferred, and on_zero_handles both flushes the line and releases the carrier. It takes only spin locks and post_wake, which meets ZeroHandles' no-park rule.
• Race with logkeeper's death. process::teardown runs Releases on the last thread out, after "every other thread has left". sys_console_carried never parks, so a carry in flight completes before the hook runs. The hook then releases what it set, and logkeeper's death cannot leave a stale carrier.
• Shared consoles. loader/start.rs mints each child its own ConsoleObject, so no other process's handle keeps logkeeper's console alive.
• One remaining window: see S4.

B3: resolved (the text); the ruling stays with the owner, item 84.

• system.toml:133-141 and the PR now say that a prober holds the foreign function's claim beside its row's port until its verdict or the probe's bound. Today that prober is vdagent, on the console's function.
• That is true. identify takes its 5 s PROBE_BOUND from its own start. Channel::open before it has no waits, and the process exits on either verdict, so the claim goes with it.
• "vdagent … which comes to it first" matches the row order.

B4: resolved.

• issues/the-console-loses-what-its-carrier-had-not-delivered.md names both runs of lost lines and the logkeeper-death case, with an owner and a testable exit.
• Its frontmatter obeys issues/README.md.
• carrier.rs points at the issue by file name.

B5: resolved.

• The hw/char/virtio-console.c/flush_buf citations are gone from build/vmconsole.rs and lib/virtio-serial/src/lib.rs. The stall measurement is cited instead.
• No added line in the PR's whole diff cites QEMU or other implementation source.

B6: partly resolved.

• Suite. guest/suite-ebac17a93-green.log covers 79 of 79 at ebac17a9326… with exit 0, so the guest-suite gap r1 named is closed. The red run at a2a172d47 shows exactly the four reds the PR names.

• console_port shape. The values match each machine:

  • Headless and its VT-d derivatives: true.
  • HeadlessNoIommu: false, because there is no unit.
  • HeadlessUsbSpareAmdVi and …NoRemap, which inherits it: false, because QEMU's AMD-Vi has no SMI filter or remaps nothing, so every claim is refused.
  • Virt, Metal and their derivatives: false.

  The reds at a2a172d47 show why: a machine that can never run the carrier never sees the quiet, so no UART line was published.

• usbd_refused_*. These tests are unchanged, and their claim does not involve the console function. Keeping the UART as their console weakens nothing.

• iommu_virtio_platform. It keeps its argv and hand-over claims, and adds to the no-unit claim: no_unit_is_no_claim now also requires the refusal of virtio-console's claim at 00:05.0. The negotiation count it dropped had lost its subject, since the kernel drives no virtio function. The replacement grep for a kernel VirtIO: PCI line checks a driver that no longer exists (S5). But see B8.

• netstack_holders_exported_to_children. Its independence holds.

  • The base log has no carrier or virtio-console line, consistent with main.
  • It hung silent 965 s there.
  • main's existing record, issues/two-holders-racing-on-an-exported-socket-can-both-stall.md, shows the same stall at main 32046264b (1 of 6) and at fileserver and bcachefs: a served stat, listing and close search no tree, and CRC-32C is the CPU's instruction #1060, which touched no console code.
  • The test passed in the green suite.

  At ebac17a93, though, the branch adds a second record of the same defect. That is B7.

New findings

B7. BLOCKER: issues/socket-export-hangs-at-its-race.md duplicates main's issues/two-holders-racing-on-an-exported-socket-can-both-stall.md.

• They describe one stall of one test, so after the rebase the tracker would hold two records of one defect under two slugs.
• Fold this PR's evidence into the existing record as an added evidence line: base 2a5b446b0, hung 1 of 1; branch, hung 2 of 3; logs in pr-1062. Then delete the new file.
• Point the PR body at the existing slug.
• I understand the implementer is doing this in the rebase. It is not at the reviewed head.

B8. BLOCKER: two comments are false after the r1 delta.

• tests/common/iommu.rs:1-4. The header still says the test checks "whether each virtio function the kernel drives itself negotiated VIRTIO_F_ACCESS_PLATFORM behind QEMU's emulated VT-d unit and none without one". The kernel drives none, and that check was deleted. State what the test checks now: every created function carries iommu_platform=on exactly where a unit is, and a claim is handed over behind the unit and refused without one.
• tests/common/qemu.rs:946-947. virtio's doc still says it includes "the console on virtio-serial". The console's port is now console_port. Drop that clause.

Suggestions

S1: resolved.

• toyos_virtio_serial_claim::identify is a faithful merge of the two loops.
• virtio-console discards the held controls as it did before.
• The callers' doc comments are accurate. I found nothing false.

S2: resolved.

• The boot and its MACHINE_TESTS row are gone.
• The RUST_SKIP comment is corrected.
• No reference to a_fatal_report_reaches_the_uart… or REBOOT_ARMED remains.
• FATAL_HALT_NONCE still has its other users.
• The PR now says honestly that the ordering rests on inspection. Its "every fatal entry" clause is false only until B1 is fixed.

S4. SUGGESTION: kernel/src/syscall/io.rs:266-272: a close racing a carry.

• One logkeeper thread closes its console's last handle while another is between handles.get and carry(true).
• The hook then runs first, and the carry lands afterwards. Nothing would ever release it, and the UART stays quiet until the stop.
• This is the holder's own concurrent misuse, and logkeeper holds its console for life, so I do not block on it.
• A cheap closure: after CARRIER.carry, release again if the object's handle count is zero.

S5. SUGGESTION: tests/common/iommu.rs:103-107. The VirtIO: PCI kernel-line check guards a kernel driver this PR deleted, which inspection establishes. Delete it.

S6. SUGGESTION: tests/common/qemu.rs console_port. It is a second declaration of a fact the shape already implies: virtio is present and the unit admits claims. Deriving it from virtio and iommu would keep a future profile from setting it inconsistently. Optional.

SEND BACK. B1 (two fatal entry points), B7 and B8 remain. All three are small edits, and none needs a new guest run: B1 is settled by inspection under the ordering already accepted, and B7 and B8 are records and comments.

[Japabu]

Review r3: #1062 at 13af3ee77

What I read: both commits (b9cedd8d2 is the rebased console change and 13af3ee77 the rework), the whole diff against merge base 190f3b164, and every changed source file. I also read these callers and boundaries:

• every halt_all_cpus, panic_registers and fatal-exception entry;
• drain_ordered;
• handle retirement and the deferred release (object/handle.rs, object/mod.rs);
• console minting at spawn (loader/start.rs);
• the supervisor's console endowment to logkeeper;
• restart::owed;
• Virtqueue::poll_used;
• vdagent's bridge closed/transmit;
• main's panic_flush;
• the harness reader and the cargo run launch.

I also checked the evidence: suite-5ff35be86.log, guest-osbench-groups.log and mutations/host-bytes-dropped.*. I ran nothing.

Architecture

The rework matches the owner's rulings on items 73 and 84:

• one transport process per virtio-serial function;
• endpoints granted explicitly (log-carrier, vdagent);
• the device and the clipboard in different processes;
• nothing transient between roles.

The kernel's only addition is the carrier word and SYS_CONSOLE_CARRIED. Userland cannot quiet the kernel's own UART writer, so that addition belongs in the kernel. The kernel loses its only DMA driver, DmaPool, DeviceSpace, the backend switch and the replay rewind, and no second output path remains. Making vdagent a device-free boot service is the cost of having no on-demand start, and the PR states that cost.

Size: production +2869/−2802, tests +434/−192, issues +75/−257.

Things I checked that hold:

• Fatal entries. Every fatal entry sets for-good before its first record:

  • the panic handler, halt_all_cpus, panic_registers and FATAL_HALT;
  • double fault, #MC and fatal_exception (Ring 0 or recursive);
  • a Ring 0 #PF before its SMAP block (a user #PF stays unaffected);
  • aarch64 exception() and its recursive user_fatal;
  • the IOMMU drain's Owner::Kernel.

  The two r2 B1 sites are fixed.

• No skipped fatal record. Discard checks the word after it has read each record, and the commit's release pairs with that read. So klogd never advances past a record that follows for-good, whatever the shard timestamps are.

• logkeeper's death. Spawn mints logkeeper a console of its own. Its fd 0 and CONSOLE endowments come from the same parent object, so they are one child object, and no other process holds a handle to it. When logkeeper dies its last handle goes, and on_zero_handles releases the carrier.

• r2 S4 (a close racing a carry). This is closed. If carry reads retired as false, the hook's release CAS comes after the carry in the SeqCst order, and that release then sees the holder.

• Gates. HELD_ALONE gives vdagent to virtio-console alone, and clipboard is held by vdagent alone, so no row can receive both. The tests cover both directions and an unrelated row.

• Hostile device input. Port ids are bounded by decode. Names are bounded, UTF-8 and free of control characters, and a name the buffer cut short is refused. poll_used bounds written and the head, and the probe refuses contradictory messages. Every one of these is host-tested.

• The turn-taking. The exit-0 handoff, Drives::declined and the claim crate existed only in b9cedd8d2. Against main, system/supervisor and lib/manifest are untouched, and restart::owed (exit 0 means nothing to serve) is main's rule, unchanged.

• Evidence. The tree at 5ff35be86 differs from the head only in tests/images/benchcase.toml. The suite shows 88/89, and its one red is the harness refusing benchcase; that row passes at the head. The mutation control is real: dropping the host's bytes makes the clipboard row fail with exit 1. I did not re-run the host CI claims. Their logs exist.

Findings

B1. BLOCKER: lib/vdagent/src/endpoint.rs:52, system/vdagent/src/main.rs:52, system/virtio-console/src/agent.rs:180: the clipboard fence is a boolean, so it fails when two of the host's closes are outstanding.

Scenario: the host closes (C1), opens (O) and sends capabilities (B2), then closes again (C2), all before the agent has answered C1.

1. The driver tells C1, O, B2 and C2. host_closed() sets closing twice, but a bool holds only one of them.
2. vdagent pumps C1, O and B2, then reaches Idle. B2 makes the bridge owe capabilities, a REQUEST and, where the guest owns the clipboard, the guest's clipboard push. flush sends CLOSED followed by those frames.
3. vdagent then reads C2 and sends a second CLOSED.
4. The driver's first CLOSED clears closing, so session 2's frames are Transmitted, after the host closed session 2 and into whatever session it opens next.

The guest's clipboard can therefore reach a session that never asked for it. That falsifies the PR's claim that "a host's next session never starts with the last session's tail".

Fix:

• Fence counts the closes it has told and not seen answered, and drops AGENT_BYTES while that count is above zero.
• vdagent answers every HOST_CLOSED with its own CLOSED: owe_closed becomes a count.

The host test needs one more case: close, open, close, then CLOSED, AGENT_BYTES (expect Drop), CLOSED, AGENT_BYTES (expect Transmit). It is cheap.

B2. BLOCKER: build/qemu.rs:266, :316: cargo run's terminal no longer shows a kernel panic, the stop's last lines, or anything after the carrier goes.

• Main. panic_flush's usual Hold::Held arm drained the report through write_raw's virtio backend, and that backend is the terminal on stdio.
• This branch. The report goes only to the UART, which cargo run writes to /tmp/toyos-uart-early.log. The relay reads only the console port.

Scenario: a developer's cargo run panics. The terminal stops at the last line the carrier delivered, and the report sits in a file the developer is not told about.

The PR's section "How panic and early-boot output reach the operator" holds for the harness, which merges both channels, but not for the launch developers use. Root counts development ergonomics, and this branch's own new comments say that the UART is where the fatal path's report goes.

Fix: give the launch the harness's merge. Put the UART on a socket or pipe, and have vmconsole::relay show:

• the UART lines after CARRIED_ELSEWHERE;
• a kernel death before it;
• nothing else from the UART, since the port carries the rest.

Moving UartConsole's rule into build/vmconsole.rs and using it from both places also removes the harness-only copy. (The empty aarch64 terminal is separate. It is already recorded.)

B3. BLOCKER (record): system/logkeeper/src/carrier.rs:358, :118: a carrier or logkeeper that is stuck but still alive leaves the UART quiet with nothing reaching either channel, and no record says so.

The UART goes quiet at the first delivered byte. It comes back only on the carrier's end, the stop or a fatal path. Two scenarios:

• The driver wedges with its pipe still open: a bug, or a device that never completes a transmit on the console path, which has no TRANSMIT_BOUND. logkeeper's feed then waits unbounded on WRITABLE.
• logkeeper deadlocks with its console still held.

In both, klogd keeps discarding every kernel record. The operator sees console: the log is carried elsewhere and then nothing until a panic or the stop. On main, kernel records never depended on logkeeper's liveness.

Owner item 73.1 asks that "suppression cannot remain silently stuck". The PR covers crashes and deaths, not hangs. Under "No silent debt" this needs at least an issues/ record with evidence, an owner and an exit. Extending the-console-loses-what-its-carrier-had-not-delivered.md would do.

A cheap partial fix, optional: Carriage goes Loud while keeping the carrier, whenever sent > delivered has made no progress for a bound. A loud UART costs duplicate output and drops nothing, so a wedge is only slow. A wedged logkeeper would still need something on the kernel side, such as releasing the carrier when the holder's log cursor stalls. That part can stay in the record.

B4. BLOCKER: images/system.toml:174: the comment misleads. "…and no clipboard, which no other row may receive either" reads as saying no other row receives the clipboard. vdagent's row, ten lines below, receives it. The intended sense is that no other row may receive either endpoint, so say that.

S1. SUGGESTION: the PR body (the orchestrator's) and b9cedd8d2's message.

• The body's "Deleted → Turn-taking" lists Drives::declined, the exit-0 handoff, overlapping rows and the claim crate as deleted. None of them was ever on main, and the net diff touches neither the supervisor nor the manifest. Describe them as an earlier round's design that does not land.
• main keeps a PR's commits individually (tgl-display: the DMC's program loaded, PG3-PG5 down and DC6 allowed (stage 2) #1132's review commit sits on main). So b9cedd8d2's message would put the turn-taking design, which the owner did not approve, into history as if it were current. Consider rewording that message or squashing the two commits.

S2. SUGGESTION: system/virtio-console/src/agent.rs:1-15. The device gives no order between the control queue and the port's receive queue. Host bytes read in the same round as a close and a reopen are told before both of them. So a new session's opening bytes can be fed to the old session and then reset, which loses the reconnect's handshake. That is a functional failure, not a leak. The header already says bytes go "before the controls read beside them". It would be worth also saying that this is the driver's chosen order, not the device's, and keeping "in order" out of the PR body.

S3. SUGGESTION: kernel/src/object/device.rs:184. The explicit fence(SeqCst) is redundant: the carry's SeqCst CAS reads from the hook's release CAS. It does no harm, so delete it or keep it.

Tests

• The new guest row is justified. It is the only check that the kernel, logkeeper and the driver's process restore the UART on a real device removal. It declares q35, and it orders BACK_ON_THE_UART before the supervisor's end line.
• Host tests cover the boundary inputs, Carriage, kernel::carrier, Fence and the gates.
• The gap is B1's case, which a host test settles.
• B2's merge belongs in a host-tested vmconsole unit, as Painter already is. It needs no new guest boot.

SEND BACK. B1 needs a small protocol fix and one host test. B2 needs the launch to hear the UART. B3 needs a record. B4 is one line of comment. None of these needs a new guest suite run; the clipboard row is worth re-running after B1.

[Japabu]

Review r4: #1062 at 2de594d8a

Scope. I read r3's blockers and the delta behind them, 13af3ee77..a3c1c5c30: 12 files, +707/−374. Of that, about 300 lines are moved code:

• death.rs moved into toyos_logstream;
• the harness reader moved into build/vmconsole.rs.

I also read:

• the new and moved files in full;
• the driver's console loop (system/virtio-console/src/console.rs);
• the kernel's sys_console_carried and ConsoleObject::carry;
• the supervisor's console endowments;
• the harness death waits;
• the evidence at 7b207c5c9.

I ran nothing.

History. It is one commit. range-diff shows a3c1c5c30 = 2de594d8a. The nine commits main gained between them touch no file this branch touches, which I checked. The commit message no longer describes the turn-taking design, so r3's S1 is resolved.

Size against main. Production +3679/−3050, tests +300/−304, issues +92/−257.

Architecture

The design is unchanged, and the delta stays inside it.

• logkeeper owns the stall policy. It already owns the quiet decision and knows sent and delivered, so it is the right place.
• The kernel gains nothing for the stall.
• The launch and the harness now share one merge, so the harness-only copy is gone.
• The death table has one home. toyos_logstream already owns is_kernel_line.

Previous blockers

• B1: resolved. Fence counts the closes it has not seen answered, and vdagent answers each close (owe_closed: u32, and flush loops). An unasked CLOSED is refused.
  • I traced r3's scenario against the new code. Both answers are counted down, session 2's frames are dropped, and nothing is transmitted until the second CLOSED.
  • The host test covers it, plus the unasked case.
• B2: resolved. vmconsole::merge is used by cargo run, through both relay and the non-terminal path, and by the harness's read_consoles.
  • I compared it line by line with r3's UartConsole/read_consoles. It polls the same way, drains the port before each UART read, applies the same live rule and publishes the same lines. The only changes are retrying on Interrupted and calling died on a trimmed line.
  • toyos_logstream::death holds the table, and one_vocabulary still guards tests/common/qemu.rs.
  • The cargo run evidence shows the stop's UART-only lines on the terminal. One defect is shared by both callers; see B6.
• B3: resolved in substance. Answers to the questions asked:
  • The 5 s bound. It is labelled as policy, and a short bound is justified because the costs are lopsided. A false stall costs duplicated lines. A missed one costs silence. However, the reason written at the constant is inaccurate; see B5.
  • Flapping. It is bounded by construction. stalled clears only on progress, and the bound restarts from the last progress. So the UART goes loud at most once per STALL with no progress. Each flap costs about four status lines plus the duplicates. It loses nothing beyond the recorded tail.
  • Duplicates. Yes, they can mislead verdicts, but only after a stall. During a stall the UART copies of new lines arrive before the port's backlog, so the merged capture both duplicates and reorders lines. A count check such as tests/toyos.rs:5214 (!= 1) would go red, and :5275 (>= 2 of HOLDING) could go green falsely. In QEMU a stall is itself a defect, so see S1.
  • The watcher's own timeout. The watcher is the failure timeout. Its unbounded wait happens only while nothing is owed, which is idle and not a wait for a completion. Ring writes never block, so say! under the lock cannot deadlock. A panic in the watcher poisons the lock and ends logkeeper, and that releases the carrier. Its being stuck is covered by the recorded residual.
  • The syscall's authority. sys_console_carried requires Rights::WRITE on a ConsoleObject. Only the supervisor's console and the console the supervisor endows to logkeeper (supervisor/src/main.rs:3149) carry WRITE. Every other program gets STDIN_RIGHTS, and rights narrow on spawn. So the gate is a capability, not a name, which is correct, and the supervisor is trusted.
  • The residual is recorded with an owner and an exit.
• B4: resolved. The comment is now true: HELD_ALONE covers both log-carrier and vdagent.

Findings

B5. BLOCKER: system/logkeeper/src/carrier.rs:8-9 and :38-41: the module header contradicts STALL, and the constant's stated reason is false.

• The header. "with no stall bound: a slow channel is not a failed one" now sits directly after the sentence that introduces STALL. What it means is that the carrier is never let go for being slow. Say that. Line 8 is also unwrapped.
• The constant. "past what a channel that takes bytes at all leaves untaken" is false. This branch's own measurement says otherwise: vmconsole.rs:8-10 records a virtserialport holding its transmit for 20 s behind a stopped reader, and the driver's own SILENT is 10 s because "a slow host is not a failed device".
• Fix. State the actual trade in one line: a false stall only duplicates lines on both channels, and a missed one leaves both channels silent.

B6. BLOCKER: build/vmconsole.rs:191-195, :221-231: merge returns at the port's end without reading the UART's remaining bytes. Those bytes reach neither show nor the UART's file.

• The race. When QEMU exits, the two fds become readable together. The port is read first. Its EOF returns from merge, and the UART bytes still in the socket are never read.
• What is lost. The UART-only lines at the end are exactly the stop's last lines, such as Shutting down., and a panic's tail.
• Before this branch. On main, QEMU wrote the UART straight to a file, so that file was complete.
• Consequences.
  • The doc comments at :125-126 and qemu.rs:58-59 claim the file holds "every byte it says". That claim is false.
  • The B2 promise that the terminal shows the stop's last lines holds only when the reader thread wins the race.
  • The machine_shutdown_* rows can flake on a loaded host.
• History. r3's reader had the same defect, and r3 missed it.
• Fix. At the port's end, read the UART socket to its end before returning. QEMU has exited or closed it, so this ends promptly.
• Test. Extend the_merge_shows_the_port_and_the_uarts_own_lines_in_order so that UART bytes are written after the port's last read and before both ends close.

B7. BLOCKER: system/virtio-console/src/console.rs:73: once a transmit has been in flight for SILENT, the driver busy-spins.

• The cause. until(oldest, SILENT) saturates to 0. A Poller timeout of 0 means non-blocking (kernel/src/inbox/mod.rs:399). say_silence is then a no-op, so the loop spins without pause until the host takes the transmit.
• When it happens. This is the case the header calls supported ("never ends the program"): a stopped terminal on cargo run. It also happens when a device never completes a transmit, which is B3's own scenario. Either way, one guest CPU burns for the whole stall.
• History. The code predates this round and was missed in earlier reviews. It falls within B3's subject.
• Fix. Once silent_since is Some, wait with no deadline. The claim's interrupt wakes the loop when the device takes the transmit. Inspection is enough to establish it.

S1. SUGGESTION: the harness could fail a capture that carries logkeeper's stall line.

• Under QEMU the host reads at once, so a stall is a defect.
• After a stall, the duplicates and reordering above can satisfy or break count and order checks without saying why.
• must_be_clean refusing "logkeeper: the console carrier's channel has taken none of" would make the cause the verdict.

S2. SUGGESTION: carrier.rs:273-279: a stall before the first delivered byte prints a misleading line. tick changes nothing in that case, because the UART was never quiet. Yet the watcher still says "the UART carries the log too until it takes more". Say it only when tick answers Some(Tell::Loud).

S3. SUGGESTION: the residual's exit in issues/the-console-loses-what-its-carrier-had-not-delivered.md.

• The gap. The exit proposes releasing the carrier "when the holder's read of the record ring makes no progress", but the kernel keeps no per-reader cursor (sdk/toyos/src/log/mod.rs:11-13).
• An alternative. A lease would fit what exists: the carry expires unless the holder renews it, and logkeeper's main loop renews it as it reads.
• A nit. The **Exit:** line is unwrapped.

Tests and evidence

• The host tests added this round are focused and deterministic. They cover:
  • Fence's close, open, close case;
  • Carriage's stall: the bound runs from the last progress, the stall is said once, progress quiets it again, and an idle log is no stall;
  • Uart across cuts, and a death before the quiet line;
  • the order merge keeps.
• The five guest rows (guest-console-rows-a3c1c5c30-tree.log, exit 0, 5/5) and the cargo run log match their claims.
• No whole-suite rerun is needed for this round.
  • The harness reader is a move that preserves behaviour.
  • The stall logic stays silent on a healthy boot, and the five rows exercise its bookkeeping.
• After B6 and B7:
  • Rerun the five rows: B6 changes the merge, and B7 runs in every q35 boot's driver.
  • The merge host test must cover B6.

SEND BACK. B5 is a comment fix. B6 and B7 are each a few lines.