[Japabu]

Status: draft. One PR for #935 (PCI apertures stage B) and #934 (lifecycle stage 2), on main e3a421563, head da40835f2. It answers review rounds 1 to 5, and wires #1137's revocation to a function's departure. #935 is closed in its favour; its branch wt/toyos-aperture-b (f9e44e0fb) is kept and is an ancestor of this one. Host CI, both image builds and the guest rows are green. The r5b metal boots (699baad1c) judge green on both benches and stand as the metal evidence.

Problem

• The loader's window list (each root bridge's Configuration() plus every unowned GCD MMIO range) counted ECAM and flash among the declared windows, so an aarch64 cargo run handed over no PCI claim. The owner ruled that aperture discovery leaves the loader, AML stays in acpiserver, and a refused or missing declaration fails closed.
• Nothing reached userland when a PCI function arrived or left: the kernel enumerated once, and nothing served native PCIe hot-plug.

What this changes

Root bridges are acpiserver's declaration (from #935):

• acpiserver reads each root's _SEG, bus range and _CRS memory windows, queries its _OSC (host bridge UUID, Support bits 0 and 4, Control bits 0 and 4), and declares them through op::DECLARE on the acpi claim (toyos_abi::acpi::Apertures: up to 16 roots and 64 windows across all of them).
• The kernel holds the declaration once, from the acpi claim's holder, checked record by record and then window by window (whole, disjoint, none over RAM). Any failure is a named refusal that stands as the answer for the boot.
• A PCI claim made before the declaration waits for it, on the claimant's thread, for at most 30 s.
• Every IOMMU domain is built clear of the declared windows and of the backend's reserved ranges.
• The supervisor starts every acpi claim holder before the disk drivers. Every image whose rows claim a PCI function starts acpiserver, and a build test holds that.
• Deleted: loader/src/gcd.rs, loader/src/rootbridge.rs, PciRootBridgeIo, KernelArgs::root_bridge_windows and its count (KernelArgs 2592 → 1560 bytes), and the OVMF Configuration() fixture.

Functions arrive and leave (from #934):

• The device set (toyos_abi::devset, ObjectKind::DeviceSet 21, SYS_DEVICE_SET 166, SYS_DEVICE_SET_READ 167) is a level-triggered snapshot of every function the kernel hands out. It is minted from DEVICE with READ and WAIT only, and the supervisor's drive diffs it.
• Native PCIe hot-plug on a port whose root's _OSC commit granted both bits. It fails closed: acpiserver commits only where the image passes --hot-plug-ports-internal (the QEMU hot-plug image alone), and never on a root with an ExternalFacingPort port or an unreadable _DSD. So no metal slot is armed.
• A slot is armed before the declaration is published, so before any claim runs. It is armed only where the unit blocks every requester on its buses it has not seen, and only once every function behind it at boot is on a domain of its own. The unit's coverage is judged per backend:
  • VT-d: a DRHD on the enumerated segment that includes every PCI device or names a bridge over those buses.
  • AMD-Vi: its programmed ids, with no IVMD keeping one.
  • SMMUv3: the IORT judged by range.
• A slot is never armed over a function a kernel driver drives or the unit cannot isolate. On VT-d, a seated card still reaches its requester's reserved regions.
• One record per requester: a returning device is a new arrival, and every call made through an older arrival's claim answers Gone.
• An arrived function's unassigned BARs get one assignment inside its port's windows and the declared memory the slot was armed under, with no probe. place_unassigned is gone.

Review round 4 (b89897e5e):

• A slot carries the declared windows it was armed under. A card seated between a slot's arming and the declaration's publication reached arrive's unreachable!, a kernel panic decided by when the card went in. The lookup and the unreachable! are gone.
• VT-d coverage counts only DRHDs on toyos_acpi::SEGMENT_GROUP, as RMRR resolution and AMD-Vi already do.
• SMMUv3 coverage is Iort::translates, decided by range.
  • It reads the root complex's mappings: for each mapping holding part of the range, one walk for an overlap and one for a shared stream. alone now judges a stream range for route and translates alike.
  • Before, it asked route for each of up to 65 536 requesters under DECLARING with preemption off, each a walk of every node and its mappings. Now the work under the lock is bounded by the table, not by the bus count.
  • It does not read where the unit sends a stream on. An arrival behind an SMMUv3 is unclaimable anyway, and governing means aborting it.
• Docs and small fixes: program takes the RSDP, the misplaced governs docs are moved, and the track's VT-d text and sys_acpi's comment are corrected.

A departed function's apertures are revoked promptly (699baad1c, corrected in ad70b5bf5, after merging #1137 in 8388ab868):

• kernel, supervisor: a claim's firmware window, granted at its mint by the signed image's row, and every PCI claim's mappings ending with it #1137 revokes what a claim handed out at its last handle and when it moves. That already kept a departed function's addresses from reuse: while its claim is held, the slot enumerates nothing behind it, retire keeps its BARs, and the release revokes before it gives the function up. What a departure lacked was promptness: a mapping still reached the departed slot until the claim was released.
• Where: the device set's reader copies the departed claim's handed-out list in slot::refresh's per-record loop, under REFRESHING, and leaves it on the claim. It revokes once the lock is let go, on its thread, never the slot's interrupt, and with no MACHINE held. The revoke's shootdown waits on other CPUs, and REFRESHING disables preemption.
• Release: the claim keeps the list, so its release still revokes everything before held clears. Revoking twice is idempotent.
• Ordering: the reader, not depart in the interrupt, wakes the departed function's holder, and only after the revoke. A holder woken by its departure finds its apertures gone, while every call on its claim still answers Gone at once.
• with_bound re-checks here() under the claim's lock.
• The check: pciholder keeps a BAR 0 handle from before the pull, and a_function_arrives_and_leaves requires that mapping it again answers Gone. Five runs pass. With the reader's revoke dropped and its wake kept, it answers None and the row fails.

Main's merge (06241e941, f234a58a2, ded3c605c):

• The console: main's userland console (console: the VM's console is a userland virtio-serial driver's, and the UART goes quiet while it carries the log #1062) deletes the kernel console, mm/dma.rs and DeviceSpace. This PR's console refusal behind an armed SMMUv3 went with them, along with untranslated and aborts_undomained.
• Images: the five images main gave a virtio-console row now start acpiserver: launchcase, libccase, panelcase, scalecase and sessioncase. hotplugcase carries its console.
• Boots with no console port: acpicase's acpi holders are jobs that declare nothing, and undeclaredcase has no holder. Every claim on those boots is refused, so a console port could never be carried. Their rows boot Profile::HeadlessNoConsolePort and Profile::HeadlessTwoEduNoConsolePort, where the UART is the console. Main's own profiles with no remapping unit do the same.
• acpi_mediated's HeldFunction arm is removed. It wrote the console function's configuration and expected HeldFunction. On acpicase no function is held, by a claim or by a kernel driver, so the arm had nothing to ask about. The decision itself is host-tested in toyos-userbound. The guest reading of the kernel's held is recorded in issues/no-guest-reads-a-held-functions-configuration-refused-to-the-acpi-holder.md.
• The aarch64 issue is deleted as met: cargo run -- --arch aarch64 now hands over every claim and shows its log on the terminal.

Why the guest rows need QEMU

• a_function_arrives_and_leaves (Profile::HeadlessHotplugPort: q35, a pcie-root-port with an edu behind it from boot, ACPI PCI hot-plug off) reads the boot edu isolated before the slot arms, pulls it over QMP and plugs another. No bench has a hot-plug port the owner lets a driver claim behind.
• a_claim_waits_out_an_undeclared_machine (Profile::HeadlessTwoEduNoConsolePort) boots q35 without acpiserver to judge the refusal.
• The acpicase rows (Profile::HeadlessNoConsolePort) mediate the firmware's own registers, which no host build has.
• The slot logic, the BAR assignment, the device-set decode, the records' refusals, _CRS, _OSC and the IORT's range judgement are host-tested.

Checks

Evidence: ToyOSOrg/evidence pr-934 at 732d80ef8 (review r5's answers), with earlier rounds in combined-r1/ to combined-r6/.

check	revision	result
cargo run -- --ci host	ad70b5bf5, da40835f2	exit 0, 109 green, each
cargo run -- --build-only, x86-64 and aarch64	ad70b5bf5, da40835f2	exit 0, 0, each
a_function_arrives_and_leaves	ad70b5bf5 ×5, da40835f2 ×1	exit 0 each, kept=Some(Gone); without the reader's revoke, kept=None and exit 1
a_drives_row_holds_one_process_per_function, a_claim_waits_out_an_undeclared_machine, acpi_mediated_access, virt_claim_lpi, the_log_lands_on_the_boot_stick, a_link_ends_under_a_held_stream_and_comes_back, usbd_drives_the_spare, the_opening_exchange_with_qemus_agent_completes, bar_map_again	699baad1c	exit 0, 9 passed (combined-r6/)
cargo test -p toyos-acpi --test iort, with 4 negative controls	b89897e5e	exit 0, 33 passed; each control fails (combined-r5/)
r5b metal readbacks (boot 699baad1c), judged	harness ad70b5bf5	T14 4/4, AMD 3/3

• Reused results. The nine rows and the IORT tests are not rerun. Since 699baad1c, the revoke changes only what a departure copies and who wakes its holder, plus with_bound's re-check, and da40835f2 renumbers two syscalls. The hot-plug row covers the departure path.
• The aarch64 launch without a terminal is recorded in issues/a-cargo-run-with-no-terminal-relays-nothing-of-the-console-port.md.

Metal

• r5b at 699baad1c (the merge of kernel, supervisor: a claim's firmware window, granted at its mint by the signed image's row, and every PCI claim's mappings ending with it #1137 and the departure wiring), read back:
  • T14: 4/4 pass (acpi_tables_loaded, claim_reuses_its_remapping_entry, domains_map_clear_of_the_host_bridges, usbd_drives_the_type_c_controller). It declares 3 windows; the first claim waited 70 ms; its three slots stay unarmed because _OSC committed nothing.
  • AMD: 3/3 pass (amdvi_translates_the_boot_controller, amdvi_remaps_the_boot_controller, amdvi_usbd_drives_its_own_entry). It declares its 16-window _CRS; the first claim waited 20 ms.
• r5 at 4424a76f9 read the same: 4/4 and 3/3.
• Why r5b stands: ad70b5bf5 and da40835f2 change no path these rows read, the declaration, the domains or the remapping entries. No bench arms a slot, so a departure is QEMU's alone.

Risks and open points

• acpi_server_death is unregistered (issues/the-acpi-servers-death-has-no-readback-on-the-t14.md). The guest arm on a held function's configuration is gone for the same reason (its new issue). Both follow from boots whose acpi holder is a job that declares nothing. The owner ruled (1–2 Nov) to restore both: the kernel's claim-time wait goes, and the supervisor starts storage and the PCI drivers off the device set once the declaration is held. That is recorded on the lifecycle track and built in its own PR after this one lands.
• Every machine's PCI claims depend on acpiserver. An x86 machine with no acpi row loses them all (its issue).
• The AMD-Vi and SMMUv3 coverage checks are reached by no booted row. The SMMUv3 range check is host-tested against route.
• aarch64 cargo run without a terminal: with stdin on /dev/null and no terminal, the console port relayed nothing after the UART went quiet. On a pseudo-terminal it relays the whole log. I take this to be the launch, not a defect; the cause is not established.
• Kept from PCI root bridges declared by acpiserver, device lifecycle and native hot-plug #934: acpiserver commits before the kernel knows a slot can be armed, the 512-record bound, a surprise removal reached only by host tests, the silent link stall, and Readiness Notifications. All are recorded gaps.

🤖 Generated with Claude Code

[Japabu]

Review of #934, round 3, at 97edf3932

One commit since r2 (e5b34b8a4..97edf3932), +98/−70 across 4 files:

• production code, about +60/−55 in pcidev/slot.rs and pcidev/mod.rs. isolate goes, isolate_behind comes in, arrive is back to one publication under MACHINE, and listed is new;
• issue edits in the two track records.

No test changed. GitHub reports the PR MERGEABLE, with the head at 97edf3932.

Architecture

The r2 simplification is the right shape:

• The isolation sits at the arming, on the declarer's thread.
• arrive touches no unit.
• Domain creation is off REFRESHING.
• The arrival-0 record is gone.

Two things stop it short of the guarantee the track and module header now state:

• The isolation infers "this function is on its own domain" from facts that do not imply it.
• Whether the unit blocks a requester it has never seen is checked for VT-d and not for AMD-Vi or SMMUv3.

Verification of the brief

(1) slot::arm isolates every boot function behind the slot. Partly.

What holds:

• isolate_behind (slot.rs:150-172) runs before arm_one. It runs on the declarer's thread after claim.isa's borrow (syscall/device.rs:121-123), and it drops MACHINE after cloning kernel_driven.
• Domain creation still nests Function::space → DOMAINS → TABLES/UNITS, all sync::Lock, which is bring_up's existing order.
• A kernel-driven function, or one isolates refuses, stops the arming. The refusal is logged as "{port}'s hot-plug slot is not armed: {function} behind it …".
• attach is a stateless entry write plus invalidation on all three units, so a double attach is harmless.
• A released successful claim stays attached to its own domain, because release unmaps and never detaches.

What does not hold:

• The held skip and the made proxy are wrong. See Required 1.

Is a never-seen requester blocked by default on each unit?

• VT-d: yes. identity_root writes root and context entries only for enumerated requesters (vtd/mod.rs:715-721, vtd/table.rs:353-376). A not-present root or context entry blocks the request and reports a fault.
  • A unit left unprogrammed makes remappable answer None (vtd/mod.rs:364-369).
  • arm_one's remapping() then refuses the slot.
• AMD-Vi: only under a programmed unit.
  • The table holds all 64 Ki entries. Dte::blocked is V, TV, Mode 000b with IR/IW clear, so every request is target-aborted (kernel/amdvi/src/dte.rs:80-85, amdvi/mod.rs:443-453).
  • A unit "it cannot use is … left off" (amdvi/mod.rs:11), and its requesters pass untranslated.
  • Interrupt remapping is decided over the ready units alone (:304-310), so remapping() does not refuse such a machine the way VT-d's does. See Required 2.
• SMMUv3: only for a RID the IORT routes through the armed unit.
  • Such a RID is aborted by an invalid STE (C_BAD_STE), by a StreamID past the table (C_BAD_STREAMID), or by GBPA (smmu/mod.rs:5-17).
  • Routes are read only for the functions enumerated at boot (smmu/domain.rs:100-112). So whether a late RID enters the unit at all is never established. See Required 2.

(2) One shared listed check. Yes.

• inventory (mod.rs:569), enumerated (:596) and claim (:984) all go through listed.
• claim(Arrival(0)) answers Absent.
• Readers cannot observe the transient new_record(pci, Arrival(0)) in arrive (slot.rs:550):
  • The record is pushed and its arrival stored (:569) under one hold of MACHINE.
  • inventory and enumerated read identities under MACHINE and only then load arrival.
• The != 0 term is now dead. See Required 3.

(3) The track lines.

• The SMMUv3 line (issues/a-device-function-is-driven-from-its-arrival-to-its-end.md:210-212) is true. arrive no longer calls isolates, so a late arrival is listed. Its claim is then refused in bring_up as NotIsolated, logged with its BDF.
• The departed-BAR line (:200-209) is true: retire returns while held (mod.rs:1543).
• The a-bars-mapping-… edit is true.
• The arming sentence (:142-148) overstates the code, for the reasons in Required 1 and 2.

Removed machinery. It is gone.

• isolate is gone.
• So are arrive's declared parameter and its second MACHINE take.
• So is the kernel-driven refusal at arrival.
• So is any identity of arrival 0. next_arrival starts at 1 (mod.rs:475), and every identity takes it (:701, slot.rs:543).
• No doc link to isolate is left, and REFRESHING's doc is restored.
• The PR body's round line matches the code apart from Required 1 and 2.

Required

1. kernel/src/pcidev/slot.rs:161-169: a boot function behind the slot can be left on the identity domain, and the slot armed over it.
   • Claims run while arm runs.
     • declare publishes the answer and posts DECLARATION (mod.rs:776-786) before arm_slots runs (syscall/device.rs:121-123).
     • Every driver parked in answered() wakes at that post and claims while isolate_behind walks.
   • How bring_up leaves a function. It creates the space at mod.rs:1088 and attaches it only at :1107. Every refusal in between leaves Function::space Some and the requester still on identity, "as it found it":
     • bars;
     • claim_msi;
     • MSI-X/MSI;
     • quiesce.
   • The two paths that miss it:
     • If that claim holds the function when isolate_behind reaches it, held skips it (:161), and the refusal then returns it to identity.
     • If the claim was already refused, made reads false (:164) and attach is skipped (:167).
     • Either way the slot arms. A card reseated at that BDF reaches all of memory from link-up, which is the exact gap r2's Required 1 closed.
   • Evidence: inspection, since no row has a device behind the slot at boot.
   • Fix: make the isolation independent of claims. The simplest is to arm before any claim can see the declaration.
     • Run the arming before the answer is visible to claimants: pass the windows and roots in, then publish and post.
     • held is then always false and no space exists yet, so both the held skip and the made check go.
     • Alternatively, take each function's held while isolating it, as arm_one takes the port, and attach whatever made says. A claim that holds the function at that moment still needs an answer, so the first option is simpler.
2. kernel/src/pcidev/slot.rs:28-31,144-149 and the track's :142-148: "every requester first seen after boot is blocked at the unit" is unconditional, and the code establishes it for VT-d only.
   • AMD-Vi: a requester on the slot's buses blocks only if a programmed unit serves its ID. Nothing checks the slot's bus range against translated. isolate_behind asks only about functions that existed at boot, and an empty slot, the usual case, asks nothing beyond the port.
   • SMMUv3: a late RID that the IORT does not route through the armed unit is never governed by its stream table.
   • Fix: at arm time, ask the unit whether it governs every requester on secondary..=subordinate, and refuse the slot by port otherwise.
     • On AMD-Vi that is the range inside translated.
     • On SMMUv3 it is the IORT mapping of the RID range to the armed unit. Not arming there is also defensible, since every late arrival is refused a claim on SMMUv3 anyway.
   • Minimum: narrow the sentences to what is checked, and record AMD-Vi's unprogrammed units and SMMUv3's unread routes as a gap whose exit precedes arming any metal slot.
   • Evidence: inspection.
3. kernel/src/pcidev/mod.rs:553-559: listed was inserted between inventory's doc comment and inventory.
   • inventory is left undocumented.
   • listed's doc now opens with "Every function in the device set … In rising arrival order", which is false for it.
   • The term identity.arrival.0 != 0, and "a record of no arrival is in it never", describe a state that no longer exists, since no identity carries arrival 0.
   • Fix:
     • Put inventory's doc back on inventory.
     • Reduce listed to its one claim: the record's arrival is the function's live one.
     • Drop the dead term.

Optional

1. slot.rs:119-123: isolate_behind runs before arm_one's refusals that change nothing: the message entry, remapping(), the port's isolates and the held take.
   • A slot refused there has its boot functions moved off identity for nothing, and there is no move back.
   • Under Required 1's first fix, take those checks first.
2. slot.rs:28-31, VT-d: an own domain maps the RMRRs kept for its function (vtd/domain.rs:101). A card reseated at such a BDF reaches those ranges.
   • Either "nothing but what firmware reserved for that requester" is the true sentence, or arming over an RMRR-scoped function is refused.
3. No row runs isolate_behind. Starting the hot-plug row's edu behind the port at boot, before its removal and reinsertion, would show the attach before "slot is armed" in the same boot. The row already reaches arrival, departure and re-arrival, so this costs no extra boot.

Evidence checked

Evidence is in r2-* at 97edf3932 (r2-ci.rev):

• --ci host: 108 steps green, EXIT=0.
• Both image builds: EXIT=0.
• Guests: 6/6, EXIT=0. In the hot-plug row:
  • the slot arms (r2-guests.log:514);
  • the edu arrives as arrival 12 (:551);
  • its claim moves it to domain2 (:556).
• Metal: restaged, not run, which is the orchestrator's.

None of this reaches the findings above. Inspection settles all three.

SEND BACK on Required 1 to 3.

🤖 Generated with Claude Code

[Japabu]

Round 3 at 016d6269e. Evidence: pr-934 at cdc882e93.

1. The slots arm before any claim can see the declaration.
   • pcidev::declare validates the records and windows, arms the slots the roots commit, and only then stores the answer and posts DECLARATION.
   • All of that runs under one DECLARING lock, which withhold takes too, and outside the acpi claim's borrow: sys_acpi makes the declaration after claim.isa returns.
   • So no claim runs on a function behind a slot while it is isolated. isolate_behind's held skip and its made check are deleted, and so are Machine::roots and arm_slots.
2. The slot's whole bus range is checked against the unit before arming (iommu::governs). A slot that fails is refused, said by port.
   • VT-d: a programmed unit must include every PCI device, or its device scope must name a bridge whose sub-hierarchy covers the slot's buses (VT-d §8.3). The sub-hierarchies are now read off each DRHD's scopes.
     • QEMU's DMAR has no INCLUDE_PCI_ALL unit and names the port by a bridge scope.
     • An intermediate run before the scopes were read refused the slot, as it should (README; that log was overwritten).
     • The not-present-entry guarantee is stated at the site.
   • AMD-Vi: every id on the buses must be inside a programmed unit's translated ranges, and no IVMD may keep one of them on identity.
   • SMMUv3: the IORT, read again with the unit's base and RSDP now kept in Live, must route every RID on the buses through the armed unit.
3. inventory's doc is back on inventory. listed states its one claim, and its dead arrival-0 term is gone.

Optional, all taken:

• The checks that change nothing run before the port is taken and any function is moved: the message entry, remapping(), the port's isolation, the coverage, and each function behind it (kernel-driven or unisolatable).
• The VT-d residual is stated in the slot module and the track: a seated card reaches the regions firmware reserved for its requester, since its domain maps them at their own addresses.
• The hot-plug row's machine now has an edu behind the port from boot. The row requires 01:00.0 moves to domain2 before slot is armed (r3-hotplug.log, lines 194 and 198), then pulls the device and plugs another, whose BAR is assigned and whose arrival is later.

Checks at 016d6269e:

• --ci host: exit 0, 108 steps green.
• Both image builds: exit 0.
• Seven QEMU rows passed: the six from round 2 plus acpi_mediated_access.

T14 and AMD metal are staged into the new metal-r4/.

🤖 Generated with Claude Code

[Japabu]

Review of #934, round 4, at 016d6269e

One commit since r3 (97edf3932..016d6269e), +300/−165 across 14 files:

• Kernel: +242/−121. declare absorbs the arming, and DECLARING is new. iommu::governs is new on all three backends, and unarmable is split out of arm_one.
• Tests: +46/−38. The hot-plug row now has an edu behind the port from boot.
• Issues: +12/−6 in the track.

GitHub reports the PR CONFLICTING. A trial merge with origin/main conflicts in pcidev/mod.rs, iommu/mod.rs, smmu/domain.rs, iommu_unit.rs, sdk/abi/src/syscall.rs and two images. It also has modify/delete conflicts on virtio_console.rs, mm/dma.rs and an aarch64 issue. That merge touches the code reviewed here, so it needs a look of its own.

Architecture

Arming before publication is the right fix for r3's Required 1. A claim cannot reach bring_up before the declaration is published, because claim takes held only after answered() (mod.rs:998-1008). Nothing else creates a Function::space: function_space's only callers are bring_up and isolate_behind. So the isolation needs no knowledge of claims, and deleting held/made, Machine::roots and arm_slots removes state rather than moving it. One consequence was missed: the slot's own arrival path still assumes publication came first. See Required 1.

governs is the right question for each unit and refuses where it cannot answer. The VT-d version leaves out the segment that every other DMAR consumer here checks.

Verification of the brief

(1) DECLARING. Holds, apart from one consequence (Required 1) and one comment (Required 4).

How the declaration is serialised:

• declare takes DECLARING and refuses Late while it is held. It then runs records → roots → check, arms, and only then stores the answer and posts DECLARATION. That makes the Late arm's unreachable! sound.
• withhold takes DECLARING too.
  • If it runs first, declare sees an answer and returns Late without arming.
  • If it runs second, it finds the answer held and changes nothing.
• sys_acpi makes the declaration after claim.isa returns.

Lock order:

• The order is DECLARING → MACHINE → Function::space → DOMAINS/TABLES/UNITS (or UNIT) → SLOTS.
• Only declare and withhold take DECLARING.
• withhold's callers hold no lock:
  • answered() runs on a claimant before it takes anything.
  • declarer_gone runs from Claim::drop, through on_zero_handles → Held::release. That path drops outside Held's lock (object/mod.rs:57-60), and the hook runs from Releases::run/drain_zero_handles with no lock held.
• So no lock is held while DECLARING is taken, and nothing under DECLARING waits on a claim.
• Everything under it spins rather than parks (wait_until in pcidev is a spin, and the unit waits are polls), so a sync::Lock with preemption off is sound.
• A withhold on another CPU spins until the arming ends, which is bounded.

The deleted held skip and made check:

• They are unneeded, as shown above.
• port.held's exchange in arm now cannot fail in practice. It is still the right take, since it marks the port as the kernel's.

(2) iommu::governs.

• VT-d:
  • governs holds only after remapping() has refused any machine with a unit left unprogrammed (remappable, settled in r3), and unarmable asks remapping() first.
  • Coverage is a pci_all unit, or a sub-hierarchy whose (secondary, subordinate) contains the whole of buses. This is conservative.
  • hierarchies uses the existing Scopes parser, which checks each scope's length is at least 6 and inside its DRHD (dmar.rs:213-224).
    • path() is (len − 6)/2 pairs, and every field is read through bounded.
    • map_while(Result::ok) stops at the first malformed scope, so a malformed table yields less coverage, never more.
    • resolve walks only enumerated devices.
    • bridged reads config on an enumerated bridge. No arithmetic is done on table values, and nothing can panic.
  • QEMU exercises the bridge-scope arm: pci_all=n, scope pci-bridge 00:03.0 (r3-hotplug.log:95,100).
  • What is missing is the segment check. See Required 2.
• AMD-Vi:
  • translated holds the ranges each unit's IVHD entries declare. It is extended only for a unit that usable_unit accepted (amdvi/mod.rs:216-222), and the table blocks every id not enumerated (build).
  • So "every id on the buses inside translated, none IVMD-kept" is the right condition.
  • I considered IVHD alias entries:
    • An alias source's range enters translated only through another entry.
    • The hardware looks up the DTE of the requester id it receives.
    • Bridge aliasing on a newly seated card stays on the slot's own unenumerated ids, and those are blocked.
    • So I found no escape through aliasing.
• SMMUv3:
  • Route::Translated already guarantees that no other mapping outputs the same StreamID (iort.rs:146-150, alone).
  • The unit is the one armed, matched by the IORT node's base, which program is given (smmu/mod.rs:281-284).
  • So every RID on the buses lands in the armed unit's stream table. An invalid STE, or a StreamID past the table, aborts.
  • A failed re-read or a refused route refuses the slot. See Optional 2 for the cost.

(3) Docs.

• The slot module header and the isolate_behind and unarmable docs are true.
• The PR body's round line is true. It says "VT-d by INCLUDE_PCI_ALL or a bridge scope".
• The track's parenthetical is not true. See Required 3.
• Two new docs repeat r3's misplacement. See Required 3.

Ordering (r3's Optional 1). Right.

• unarmable asks, before the port is taken or any function is moved, every check that changes nothing:
  • the message;
  • remapping();
  • the port's isolates;
  • governs;
  • each function behind the port, kernel-driven or unisolatable.
• What can still fail after the isolation is OwnSpace::create on a later function, claim_msi or the interrupt arm. Each of these leaves functions on their own empty domains, which is stricter and is what their next claim reuses. That is acceptable.

The hot-plug row and isolate_behind. Yes, it exercises it.

• iommu: 01:00.0 moves to domain2 (:194) comes before slot is armed (:198), and both come before the declaration's survey line (:199). The claim's own attach comes after (:204).
• Claims can no longer run before publication, so the first "moves to domain" before ARMED can only be isolate_behind's. Without it, the first such line is the claim's, after ARMED, and the row fails. The test discriminates.
• Evidence: run at 016d6269e, after the commit:
  • r3-ci.rev;
  • --ci host 108 green, EXIT=0;
  • both builds EXIT=0;
  • 7/7 guests, EXIT=0, r3-guests.log:5003, with the row at :442.
• The AMD-Vi and SMMUv3 governs are reached by no row. They rest on inspection, which is adequate for these checks.

Required

1. kernel/src/pcidev/slot.rs:448-451: arrive's unreachable!("pcidev: a slot is armed only once the root bridges' memory is held") is now false. It is reachable from the order in which a card is seated.
   • How it is reached:
     • arm pushes the slot into SLOTS and posts super::changed() (:137-140) while MACHINE.apertures is still Unknown. It is published only when slot::arm returns to declare, after every remaining port has been armed.
     • The device-set reader can run slot::refresh on another CPU in that window.
     • A slot that is occupied and powered with its link up, but lists nothing, goes to Next::Enumerate once §6.6.1's 100 ms has passed. Then arrive reads answer() as None, and the kernel panics.
     • Such a slot is a card seated after boot enumeration and before arming, or one whose record did not fit.
     • It needs the rest of the arming loop to outlast 100 ms. That is improbable on QEMU and plausible with Optional 2's per-RID IORT walk over a wide bus range. This is inference, not measured.
   • Consequence: the panic message states an invariant this round removed, and a seating time, which is an external event, decides whether the kernel panics.
   • Fix: give HotPlug the declared windows arm already receives, and have arrive read them from self. That deletes the MACHINE lookup and the unreachable!.
2. kernel/src/arch/x86_64/vtd/mod.rs:104-108,754-763 and vtd/domain.rs:96-104: governs counts a DRHD of any segment.
   • pci_all is copied from every DRHD, and hierarchies resolves every DRHD's scope paths against the segment-0 devices.
   • Both RMRR resolution (:222-228) and AMD-Vi (amdvi/mod.rs:418) refuse another segment for this reason: "a region on another names devices this kernel cannot tell from its own". The new field's own doc says "on its segment".
   • Consequence: on a machine with a catch-all unit on segment 1 and explicit scopes on segment 0 that miss the port, the slot arms over buses whose late requesters no unit translates. A segment-1 bridge scope whose path happens to match a segment-0 bridge counts as coverage in the same way. Both tables are valid under the spec.
   • Fix: take pci_all and hierarchies only from a DRHD whose segment() is toyos_acpi::SEGMENT_GROUP.
   • Evidence: inspection.
3. Docs.
   • vtd/domain.rs:86 and amdvi/domain.rs:73-75: r3's Required 3 again. governs was inserted between unclaimable's doc and unclaimable. Each governs doc now opens with "Why no claim may have stream…", and unclaimable is undocumented on both backends. Move each governs above or below the pair.
   • issues/a-device-function-is-driven-from-its-arrival-to-its-end.md:146: "(a VT-d unit including every PCI device; …)" names only the catch-all. The code also accepts a bridge scope covering the buses, and QEMU's run, the only one, passes through that arm (pci_all=n, scope pci-bridge 00:03.0). Say "a VT-d unit including every PCI device or naming a bridge over those buses", or the equivalent.
4. kernel/src/syscall/device.rs:96: "The holder's check and the act are one borrow of the row, as a read's are" is now false for DECLARE.
   • Its act runs after the borrow, as the arm's own comment two screens down says.
   • The race this opens is harmless:
     • only the bound pid passes;
     • the declaration is once-only;
     • a concurrent release is ordered by DECLARING.
   • The comment, though, contradicts the code. Narrow it, for example to "…one borrow of the row, but for DECLARE's, made below", or drop the clause.

Optional

1. pcidev/mod.rs:755-785: the windows are checked twice.
   • staged runs aperture::check, and Held::declare runs it again on the same input. It cannot disagree, but the reader has to prove that.
   • Held could take the Declaration that check returns: a Held::hold(declaration) from Unknown alone.
2. smmu/domain.rs:118: governs calls iort.route once per RID, up to 65 536 for a port forwarding buses 1..=0xff.
   • Each call walks every node and runs alone.
   • This all runs under DECLARING, with preemption off, on the declarer's thread.
   • It is bounded but possibly long. That is an inference, not a measurement, and it widens Required 1's window.
   • Deciding coverage on the IORT's ID-mapping ranges would be one walk. Alternatively, measure one wide-range arm on virt and record it.
3. smmu/mod.rs:419: program builds Live with rsdp: 0, and init overwrites it. Passing rsdp_addr to program beside base removes the placeholder.

Verdict

SEND BACK on Required 1 to 4. All four are small and found by inspection. None needs a new row:

• Required 1 deletes code.
• Required 2 is a one-line filter.
• Required 3 and 4 are text.

A compile and --ci host suffice for them. The QEMU hot-plug row is worth a rerun only if Required 1 changes arrive's path, which it does. Rerun that one row.

🤖 Generated with Claude Code

[Japabu]

Review r4, answered at 4424a76f9. Main 058bea2d3 is merged in 06241e941, and every conflict is accounted for in its message.

Required

1. Fixed. HotPlug keeps the declared windows arm is given, and arrive places BARs inside self.declared. The MACHINE lookup and the unreachable! are gone. A claim on an arrival listed before publication still waits in answered(). The hot-plug row passes on the new path.
2. Fixed. VT-d takes pci_all and hierarchies only from a DRHD on toyos_acpi::SEGMENT_GROUP; a unit on another segment counts for neither.
3. Fixed. Both governs docs now sit outside unclaimable's doc and its function. The track names the bridge-scope arm and the segment.
4. Fixed. The comment now reads "…one borrow of the row, as a read's are, but for DECLARE's act, made below."

Optional

• 2, taken. Iort::translates decides a RID range from the root complex's mappings. Per mapping that holds part of the range, it makes one walk for an overlap and one for a shared stream. alone takes a stream range. The work under DECLARING, with preemption off, is now bounded by the table rather than by 65 536 requesters.
  • It is host-tested against route over every range across crafted tables' edges.
  • A read-bound test fails for a per-requester translates: 1.26 G bytes read of a 22.5 KB table.
  • Three mutations each drop one check, and each fails the oracle test.
• 3, taken. program takes rsdp.
• 1, declined this round.

The merge's own consequences (f234a58a2, ded3c605c; evidence combined-r5/):

• Main's harness reads a virtio machine's log only through the userland console's claim. acpicase and undeclaredcase can hold no claim, so they boot Profile::HeadlessNoConsolePort and Profile::HeadlessTwoEduNoConsolePort, following main's own precedent for machines where every claim is refused.
• acpi_mediated's HeldFunction arm on the console function had no held function on that boot, so it is removed and the gap recorded in an issue. Please look at this one specifically.
• The aarch64 issue is deleted as met: the claims are handed over and the terminal shows the log.

Metal

• The r4 boots judge T14 4/4 and AMD 3/3, with the declaration on real firmware.
• The r5 boots are staged and not run.

🤖 Generated with Claude Code

[Japabu]

Review of #934, round 5, at 699baad1c

This round re-reviews everything since r4 (016d6269e). Not counting the two merges, there are six commits, +400/−135:

• the r4 fixes (b89897e5e, +201/−43; 82 of the added lines are host tests);
• the post-merge image and test commits (f234a58a2, ded3c605c, 4424a76f9, cce412876);
• the departure wiring (699baad1c, +46/−26).

Architecture

Placing the departure's revoke on the set's reader fits the design:

• It runs on the thread charged with slot work, never in the ISR, with no MACHINE or slot lock held.
• Taking the list under REFRESHING does guarantee it belongs to the departed claim. A new arrival on the record needs listed == false, held == false and a later refresh's §6.6.1 wait. left is consumed by the loop of the first refresh after depart, and depart zeroes link_since before it swaps arrivals. So no enumeration can come between them. I checked the interleavings of isr/depart against refresh's seen → link_since → loop and found none that breaks this.

What the change does not get right is the ownership of the list. Before this commit, DeviceClaim::on_zero_handles revoked everything in Bound::handed before release cleared held. The rest of the reuse gate follows from that: next() answers Nothing while anything behind the slot is held, and retire keeps the BARs in decoded until the release. So reuse came after revocation by construction. Moving the list out with mem::take takes that guarantee away from the release path (Required 1).

r4's required items: all resolved

1. The HotPlug windows. HotPlug::declared is set from the windows that arm is given. arrive reads MACHINE only for decoded, and the unreachable! is gone (slot.rs:450-462).
2. VT-d segment. pci_all and hierarchies are set only for a DRHD with segment() == SEGMENT_GROUP (vtd/mod.rs:182-187). describe_unit starts both empty, and governs reads nothing else.
3. Docs.
   • Both governs docs now sit apart from the unclaimable pair.
   • The track's VT-d parenthetical names the bridge-scope arm and the segment.
4. device.rs:96. It is now true.
5. SMMUv3 by range.
   • Iort::translates agrees with route as route would answer for each requester:
     • Overlap is detected inside the requested range only, and so is route's.
     • The counted ranges are disjoint, so the sum test is exact and covered cannot overflow.
     • alone over a stream range uses saturating_add for types check does not decode.
     • The one stated difference is that the onward mapping is not read. The doc says so.
   • Under DECLARING the cost is O(held mappings × table), bounded by firmware's table and not by the bus count, as the comment says. The bound test covers the single-mapping case.
   • The host test compares the result with route for every edge range over six crafted tables and two segments, and the corruption sweep calls it.
   • The kernel side maps Err to a refusal.
   • Satisfied. The program(rsdp) placeholder is gone.

The merges

06241e941 (#1062). Checked with --remerge-diff.

• Deleted with main's files: virtio_console.rs, mm/dma.rs, DeviceSpace, untranslated/aborts_undomained (x86 dispatch and SMMU) and IommuError::Undeclared. In those files this branch had only:

  • the SMMUv3 refusal of the console function;
  • DeviceSpace::create's Option;
  • the aborts_undomained stub.

  These existed only for the kernel console. Nothing at the head references any of them (git grep). AMD-Vi's own private untranslated is unrelated and kept.

• ABI: 164/165 at that merge, then 165/166 in 8388ab868 with 168 bins. The assert names the highest number.

• Images: the seven images that have both main's virtio-console row and this branch's acpiserver row keep both. clipboardcase starts acpiserver and vdagent.

8388ab868 (#1137).

• The only conflicts are the ABI and the BAR issue, and main's text is taken there. The dropped branch paragraph is superseded by main's "What is left" list.
• tear_down_departed's Bound { .., } drops handed silently. That is sound only because on_zero_handles revokes first.

f234a58a2. It adds acpiserver to launchcase, libccase, panelcase, scalecase and sessioncase. All five have main's virtio-console drives row, so they need it. every_config_that_claims_a_pci_function_starts_its_declarer holds them.

ded3c605c.

• The two *NoConsolePort profiles differ from their base only in console_port: false. Each is used by exactly the rows whose boots cannot carry a console claim (acpi_mediated_access, acpi_supply_outlives_holder, acpi_press_is_one_sci, a_claim_waits_out_an_undeclared_machine).
• hotplugcase gains the console row and logkeeper's serves = ["log-carrier"].
• Fine.

4424a76f9. Its exit is met as stated, and nothing references the deleted file.

The weakened acpi_mediated arm

Its removal is justified. On acpicase, after #1062, no function is held, so the arm could not run. The new issue has an owner, an exit and the cause, and the owner's 1–2 Nov ruling, staged on the track, restores it. The decision itself stays host-tested in toyos-userbound. Accepted as a recorded temporary loss.

The kept-handle check as proof of revocation

kept=Some(Gone) comes from mapped_at finding mapped taken, and only SharedMemObject::revoke takes it. release_object runs unconditionally in that same function, so seeing Gone establishes that the revoke ran, and inspection covers the unmap. The claim is still held when the holder probes, because it exits only after printing. So the observation is the departure's revoke and not the release's. The mutation, which drops the take, gives kept=None with exit 1, so the control discriminates.

What the check lacks is ordering (Required 2).

Required

1. kernel/src/pcidev/slot.rs:419 (departed.push(super::take(function))): the departure moves the apertures out from under the release path. A concurrent release then lets a new arrival get the departed BAR addresses before they are revoked.
   • How it happens:
     1. Reader A takes handed under REFRESHING and lets the lock go. It is preemptible now and has not yet revoked.
     2. The holder closes its claim while keeping its aperture handle and the mapping.
     3. on_zero_handles → revoke_handed takes an empty list. release clears held (mod.rs:1518), then retire drops the BARs from decoded, and settle_released powers the slot off.
     4. After the card is reseated, any other reader of the set powers the slot on and, 100 ms later, enumerates it. Readers include every SYS_DEVICE_SET_READ caller, the supervisor and any device syscap holder.
     5. arrive's taken skips bar.who == who (slot.rs:452-458), so the new card can be placed at the departed one's addresses. A's revoke has still not run, so the old holder's mapping reaches the new device's registers.
   • Consequence: the isolation that held by construction before this commit now depends on A not being delayed longer than a reseat plus 100 ms. That is improbable, and it is inference, not measured. But the BAR issue's new text claims the opposite unconditionally: "a mapping the departed claim made reaches nothing the next device behind the port is assigned".
   • Fix: clone the Weak list under the lock and leave it on Bound. A departure helper beside take can return Handed(bound.handed.clone()). SharedMemObject::revoke is idempotent (mapped.take(), shm.rs:321-329), so the release still revokes everything before held clears, and the departure only makes the revoke prompt.
   • Text to correct with the fix:
     • The departure motivation states a hazard that did not exist before the change: "…kept its BAR mappings until that claim's release, and the next arrival behind the port may be given the same addresses." It appears in the commit message, and the PR body's paragraph is the orchestrator's to correct. While the claim is held, next() enumerates nothing and retire keeps the BARs, and the release revoked first.
     • The track at issues/a-device-function-is-driven-from-its-arrival-to-its-end.md:207-211 repeats "so the next arrival behind the port may be given the same addresses". It also sits under "Gaps it records:" although it is no longer a gap. Move it out of that list or drop it, since the BAR issue holds it.
2. tests/toyos.rs:5422 with tests/guest/pciholder/src/main.rs:35-48: kept=Some(Gone) asserts an order nothing establishes.
   • The race:
     • pciholder wakes on the function's watch, which depart posts from the slot's ISR (slot.rs:296).
     • The revoke runs later, on the supervisor's device_set_read, which the same ISR wakes through SET_WATCH (changed() at its end).
     • The holder then makes five syscalls and probes. The reader has to wake, refresh and complete a shootdown.
   • The evidence:
     • In the single passing run, the kernel's "left the device set" line, which comes before the revoke, is stamped 5.723 and the holder's line 5.724 (r5-hotplug.log:280-282, 2 CPUs).
     • No repetition was run.
   • Consequence: the row can go red without a defect. That is non-deterministic by inspection, and the margin observed is about 1 ms.
   • Fix, either of:
     • Make it a kernel contract. Post a departed function's watch from the reader after the revoke, rather than from depart in the ISR. Calls still answer Gone at once through arrival, and a holder woken by its departure then finds its apertures revoked. That also fixes the module header's ISR paragraph.
     • Have the guest assert only what is ordered.
   • A host test cannot cover this. After the change, one rerun of the row is enough, since it would then be ordered by construction.
3. kernel/src/pcidev/slot.rs:376-379: two doc summaries are concatenated. "Act on what this slot is owed; answers how long…" is followed by "What the slot is owed, acted on; …". Keep one sentence that covers both the return value and departed.

Optional

1. kernel/src/pcidev/mod.rs:1662: with_bound checks here() before it takes bound.
   • A map_bar that passed the check before depart, and locks after the reader's take, pushes an aperture the departure never sees.
   • Restoring a refused move after the take does the same (restore_handed).
   • Both are revoked at the release, after Required 1's fix by construction, so this is about promptness only. Re-checking here() under the lock closes the first. The left Release/AcqRel pair orders arrival = 0 before the take.
2. Record or settle the aarch64 no-terminal observation. The PR body says that with stdin on /dev/null and no terminal, the console relayed nothing once the UART went quiet, and that the cause is not established. That observation currently lives only in the PR body. Either record it in issues/, or establish that it is QEMU's stdio chardev, for example by doing the same on x86-64.

Verdict

SEND BACK.

• Required 1 is a one-line change in ownership, plus the text that follows from it.
• Required 2 makes the guest row's observation ordered. Rerun a_function_arrives_and_leaves once after it.
• Required 3 is text.

Compile and --ci host cover the rest. The r4 items and both merges are sound.

🤖 Generated with Claude Code

[Japabu]

Review r5, answered at ad70b5bf5; main e3a421563 (#1063) is merged at da40835f2. Evidence: pr-934 732d80ef8, combined-r7/.

Required

1. Fixed. The departure now copies the claim's list (handed_now) and leaves it on Bound, so the release still revokes everything before held clears; revoking twice is idempotent.
   • ad70b5bf5's message corrects 699baad1c's: no reuse hazard existed while a claim is held.
   • The PR body says the departure adds promptness.
   • The track no longer lists the departure among its gaps, and the BAR issue is corrected likewise.
2. Fixed by ordering. A departed function's holder is woken by the set's reader after the revoke, no longer by depart in the ISR. Calls still answer Gone at once.
   • a_function_arrives_and_leaves passed 5 of 5 at ad70b5bf5 and once at da40835f2, kept=Some(Gone) each time.
   • With the revoke dropped and the wake kept, the row reads kept=None and fails (exit 1).
3. Fixed. One doc summary.

Optional

• with_bound: taken. It re-checks here() under the claim's lock.
• The aarch64 launch with no terminal: recorded in issues/a-cargo-run-with-no-terminal-relays-nothing-of-the-console-port.md, with what was seen and that the cause is not established.

Merge with #1063: SYS_TIMING_CONTROL keeps 165; the device set moves to 166 and 167, with 169 bins.

Checks: --ci host 109 green and both builds exit 0, at each of ad70b5bf5 and da40835f2.

Metal: the r5b readbacks (boot 699baad1c) judge T14 4/4 and AMD 3/3.

🤖 Generated with Claude Code