Skip to content

A claimed function interrupts at most once per read of its record - #1193

Merged
Japabu merged 12 commits into
mainfrom
storm-mask
Nov 6, 2026
Merged

Japabu merged 12 commits into
mainfrom
storm-mask

Conversation

@Japabu

@Japabu Japabu commented Nov 5, 2026 •

Copy link
Copy Markdown
Collaborator

On main at 0b2dec6, after #1188 and #1189; the rebase applied every commit unchanged.

Status: round 3 is in review. The kernel is unchanged since the round-2 A/B, in which R1 to R3 held. virtio-sound now counts periods from its used ring alone.

Problem

A function a process drives raises its claim's interrupt on cpu0 for every message, and nothing bounded how often: a hostile or broken holder could make its device flood cpu0 (issues/a-claimed-function-can-storm-its-cpu.md, kept open and narrowed to what remains).

Behavior

A claim's interrupt is masked from the second interrupt the kernel takes before its holder reads the record, until that read. A driver that reads between messages never pays for a mask. One that falls behind is cut off: the first message never masks, the second does, and that handler's drain can leave at most one more request held in the local APIC. So at most 3 land per read (Intel SDM Vol. 3A, 325384-093, §13.8.4). A message sent while masked is refused at the unit on x86-64 and left pending on AArch64. A record's count is of interrupts the kernel took, never of messages sent: the interrupt controller folds a message that arrives while its vector is pending, and the mask drops what lands past it (toyos_abi::pci). Drivers take the record before servicing the device, which toyos::PciDev::irq states.

unit masked published by
VT-d IRTE P=0, FPD=1, one 64-bit write (Rev. 4.1 §9.9; §5.1.4.1 Table 13: fault 22h is qualified, so not recorded) index-selective IEC invalidation (§6.4, §6.5.2.7), which also drains what was already remapped
AMD-Vi IRTE RemapEn=0, SupIOPF=1 (48882 rev 3.11 Table 22; target abort per §2.1.3.1) INVALIDATE_INTERRUPT_TABLE (§2.4.5)
GICv3 ITS the LPI's configuration byte disabled; the LPI stays pending INV + SYNC; release re-enables before DISCARD

Design decisions

  • The mask follows the record, not each side's intent. Each side changes the count first, then takes the claim's gate lock and puts hardware in the state the word owes (masked while the count is at least MASKED_FROM). The side that takes the gate last sees both changes, so a mask that lands after the read that re-armed the claim cannot strand it. record.rs owns the rule; pcidev owns one gate per function. The gate is armed after the count is cleared at mint and disarmed before the message is given back.
  • Each unit's invalidation queue is a leaf lock: VT-d's queue, AMD-Vi's command buffer (SMI filter state split off), the ITS command queue. Each is taken with interrupts masked and holds nothing else. Making the existing IOMMU locks masked would deadlock, because VT-d's map takes TABLES under UNITS and reaches the page allocator. Every invalidation's completion wait now runs with interrupts off, including ordinary map and unmap.
  • Round 2 trigger (coordinator's ruling on R2): in round 1, masking every delivery cost +33 µs per I219 interrupt. Only the trigger changed.

Checks

virtio-sound (review r1 and r2 blockers)

virtio-sound inferred completed periods from the record's count. The count is of interrupts taken, and the local APIC folds a message that arrives while its vector is pending (SDM §13.8.4). An LPI folds while pending on AArch64, and the mask drops what lands past it. So a record of one interrupt over two periods left a notification owed that never landed, and no later record clocked a period again: the mix loop's DLL stopped being corrected. That stall predates this PR.

Per the coordinator's ruling, the used ring alone says which periods played, on every read. The record is a wake and a time. Its newest time anchors the clock where one interrupt was taken and one period read. Any other read anchors nothing, and its periods carry into the DLL's next update, which it already handles. The notification debt and round 2's Notified type are gone.

The anchor is not certain. A period read ahead of its notification can make the next one-interrupt, one-period read anchor early, by at most that read's wake latency: one error sample the DLL absorbs at its bandwidth, not a stall. This gives up #1188's exact handling of read-ahead. That handling rested on a count of messages, which folding breaks: a count cannot tell a folded message from a period read ahead.

a_record_folded_over_two_periods_does_not_stop_the_clock feeds a count of 1 over two periods, then three steady reads, and gets [0, 1, 1, 1]. #1188's accounting, restored as the control, gives [1, 0, 0, 0]. The existing 70-interleaving test passes. Within its two periods none is stamped before its own notification; a third can be, as above. The trade, ruled by the coordinator without the owner, is a permanent stall that exists today against one bounded early anchor in a two-sided race. virtio-sound runs only under QEMU.

Sightings

Risks

  • The narrowed issue records what remains:
    • a holder looping force-and-read still drives one cpu0 interrupt per iteration, and two per read add invalidation waits with interrupts off;
    • refused messages are counted nowhere;
    • the T14 timer row is still owed.

🤖 Generated with Claude Code

@Japabu

Japabu commented Nov 5, 2026

Copy link
Copy Markdown
Collaborator Author

Review of #1193 at 39cbe73 (base kfu-claim-window 83ddb42)

Architectural assessment

The mechanism fits the approved design. Ownership is clean: record.rs owns the rule (the mask follows the count word, not either side's intent), pcidev owns one Masked<Option<Gated>> gate per function, and each backend exposes one set_masked(Line, bool). Splitting each unit's invalidation queue into a leaf Masked lock is the minimum needed for a handler to invalidate, and the PR explains why the existing locks can't simply be masked. Dropping AMD-Vi's if is_armed() in claim is justified: fences_claims already refuses a claim where nothing is armed. Net change is +596/−173, including the 34-line issue deletion. Most of the growth is the three backends' Line types and the queue split, and both are earned.

Deadlock freedom. I traced the handler path on all three backends.

  • Handler path: gate (Masked), then the unit's queue slot (Masked), then a bounded spin with a Tripwire panic.
  • What the leaves take: no queue holder takes anything else. VT-d with_queue and remapping_queues, AMD-Vi submit per slot, and ITS issue each hold only their own slot.
  • Non-masked locks: ITS, REMAP, UNITS/TABLES and function.bound are only ever taken outside the masked leaves. The handler never takes them, so a cpu0 holder interrupted by the claim's handler cannot deadlock it.
  • EOI order: user_dev::took EOIs after pcidev::isr, so the mask is in place before the vector can re-enter.

Race argument. It holds. Each side changes the count with an RMW, then reads pending() under the gate lock. Lock acquire/release orders the Relaxed word, so whichever side takes the lock last sees both changes. The loom model calls the real took_masking/take_opening through #[path] and lets the handler run whether or not the line is masked, which is a superset of the hardware. The device-irq-gate-intent control is a real defect: it applies each side's intent, so a read that opens followed by a late mask strands the claim. It is registered in build/ci.rs and covered by clippy through $CONTROLS. On claim, gate.arm before irq.clear is safe only because bring_up leaves mastering off (quiesce). That is fine as it stands.

Release ordering. It is right. tear_down disables mastering, masks MSI/MSI-X, then drain_messages, gate.disarm and drop(message); tear_down_departed runs disarm before drop(armed). Every Line is used only under the gate lock, so nothing can mask a released entry. On the ITS, re-enable and INV before DISCARD is necessary as well as tidy: INV translates through the event mapping that DISCARD removes.

Citations. I checked them against the extracts.

  • VT-d: §9.9 FPD is "evaluated by hardware irrespective of … P", Table 13's 22h is qualified, and §6.5.2.7 says an IEC invalidation drains already-remapped requests. G at bit 4, IIDX at 47:32 and IM=0 match Figure 6-7 and Table 22.
  • AMD-Vi: 48882 Table 22 says RemapEn=0 means "target aborted", and "SupIOPF is meaningful independent of the value of RemapEn". §2.1.3.1 says target-aborted transactions "should not set any AER bits". All of this matches the code and comments.

Findings

BLOCKER — system/soundserver/virtio-sound/src/lib.rs:336-360 (consumer of sdk/abi/src/pci.rs:207): the record's count no longer means "one per message", and virtio-sound's clock depends on it.

  • Why it breaks: Sound::played assumes the device "owes one [notification] for every element (§2.7.7), so a ring read after a record was taken holds every period that record counts". It tracks unnotified periods whose notification is "still to land". Under this PR a notification sent while the claim is masked is target-aborted on x86-64 (or folded into one pending LPI on AArch64), so it never lands.
  • The failure: take one read that finds two periods in the ring with count == 1. That read leaves unnotified = 1. Every later read then has count == 1, periods == 1, so fresh = 0, clocked = 0, and unnotified stays at 1. Only a read with count ≥ 2 would clear it, and after this PR that happens only when messages were in flight.
  • Consequence: soundserver/src/mix.rs:441 calls dll.update(.., n, 0), and with clocked == 0 Dll::update only accumulates unclocked and returns (mixer/src/dll.rs:52-55). The mix loop's DLL is never corrected again.
  • Scope: HDA and tgl-display are unaffected, because their devices raise one message per status the driver clears after reading. virtio-sound is the consumer that relies on the old count. The PR's claim that all ten drivers follow the contract covers ordering only, not this.
  • Required: migrate virtio-sound's accounting, comments and host tests to "a count is a wake, not a tally", together with the ABI doc change. This touches audio clocking, so if any behaviour change remains, root CLAUDE.md requires the owner's explicit agreement.

BLOCKER — tests/guest/rust-tests/src/bin/i219_msi.rs:39-47: the storm bound's comment is false and the bound is not derived.

  • What's false: "unmasked, every one lands" is wrong. The local APIC holds at most one request per vector in IRR besides the one in service (Intel SDM Vol. 3A, local APIC interrupt acceptance). Unmasked, the claim lands about once per handler completion, not 10,000 times. With the mask, the handler masks and drains before its EOI (user_dev::took EOIs after pcidev::isr). Everything drained coalesces into that one IRR bit, so B lands at most 2.
  • The fix: derive the bound from the SDM and cite it at the constant. STORM_BOUND = 2 replaces "a few dozen … 64" and the handler-latency argument. This also sharpens the negative control: A needs only to exceed 2, which it will by orders of magnitude, so the A/B rule "A > 64" becomes "A > 2".
  • Why it matters: as written, the comment explains the wrong mechanism, and the bound hides a regression in which the mask lets dozens of messages through.

BLOCKER — issues/a-claimed-function-can-storm-its-cpu.md (deleted): the exit is not met, and a residual exposure goes unrecorded.

  • What's missing: the exit asks for a T14 row that shows cpu0's timer still served, and messages "recorded against its claim". Neither has been done: the metal row is staged and not run, and messages refused under FPD/SupIOPF are recorded nowhere.
  • The residual: "at most once per read" is a bound per read, not per unit of time. A holder that loops {force a cause; read the record} on another CPU still drives one cpu0 interrupt per iteration. Each such interrupt now also costs cpu0 an invalidation completion wait with interrupts off. That work is charged to whatever thread cpu0 interrupted, not to the holder.
  • Required: the design was approved, but deleting the record claims the defect is closed. Either keep a narrowed record stating this residual with its owner and exit, or get the owner's ruling that the residual is acceptable and replaces the exit. Wait for the storm row's result before closing either way.

SUGGESTION — kernel/src/pcidev/mod.rs:1052-1053. Clearing the count before gate.arm (or clearing it under the gate) would make the claim path independent of quiesce. Today an early message between arm and clear would leave the claim masked with nothing to read. Mastering is off here, so nothing can currently trigger it.

SUGGESTION — kernel/loom/tests/device_irq_gate.rs:1-18. The header repeats record.rs's new paragraph and the Cargo.toml control comment. One sentence naming the invariant and the control is enough.

Evidence

  • I inspected rather than re-ran: host CI at 39cbe73, loom, and guest runs 42035956491 (x86) and 42035960983 (aarch64 virt_claim_lpi). They fit what they claim.
  • None of them exercises a device that sends several messages per read under a counting consumer, which is why the virtio-sound break got through. The cheapest check is a host test of Sound::played fed count == 1 records while two periods complete.
  • The T14 A/B and the AMD testcases are outside this round.

SEND BACK

Japabu added a commit that referenced this pull request Nov 5, 2026
…count cleared before the gate

- i219_msi's storm bound is 2: the handler masks and drains before its end
  of interrupt, and a local APIC holds one request per vector beside the one
  in service (Intel SDM Vol. 3A §11.8.4). Unmasked, every forced cause lands
  here, as the A arm measured (10,000 of 10,000): the ICR read between forces
  outlasts the handler.
- issues/a-claimed-function-can-storm-its-cpu.md stays open, narrowed to what
  is left: a holder that loops force-and-read paces cpu0 and its
  invalidation waits, a refused message is counted nowhere, and no T14 row
  shows cpu0's timer served under such a holder.
- A claim clears its record before it arms the gate, so the order no longer
  rests on mastering being off.
- The gate's loom header is one sentence.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Japabu

Japabu commented Nov 5, 2026

Copy link
Copy Markdown
Collaborator Author

Review r2 of #1193 at ebee4a6 (changes since 39cbe73)

Architectural assessment

The round changes only what the ruling asked for. took_masking and take_opening both apply one rule, owes_mask() (count ≥ toyos_abi::pci::MASKED_FROM). The gate, the backends and the leaf invalidation queues are untouched. Moving the constant into the ABI is right, because it is now part of what a record means to its reader. Since 39cbe73 the change is +179/−65. Of that, +36 is the restored issue, +32 the new host test and +12 the sighting; the kernel diff is small. Option (b) stays inside virtio-sound. Soundserver maps the count onto a typed Notified at the boundary, which is the right place to do it.

Race argument under the new threshold. It still holds, and the threshold does not matter to it. Every change to the word is followed by the same side taking the gate lock: took then hold, and take then hold. The only exception is clear, which runs while the gate is unarmed (mod.rs:1052, 1539, 1640). Under the lock, the masked state is a pure function of the word as it then reads, so whichever side takes the lock last sets hardware from the final word. I found no other take/took caller.

Loom. model(2) lets the read land anywhere among the two messages, which covers both cases:

  • the read lands between them, so the count goes 1→0→1 and nothing masks;
  • the handler reaches the second message's hold after the read has emptied the word.

The control now computes owes_mask() before the lock, which is a faithful intent defect. It reds model(2) with "masked where the record owes it open" (r2/logs/loom-gate-control-2.log). model(1) correctly passes under the control, so build/ci.rs was right to drop it from Fails. A model(3) would add nothing: the third message only re-asserts the mask under the same lock.

Claim-time clear (r1 suggestion). clear() now runs before gate.arm, and arm starts with masked: false, which matches a zero count. A message that landed between the two would at worst miss a mask. It could not strand the claim, and mastering is off there anyway. Resolved.

r1 blockers.

  1. virtio-sound (option b): resolved as ruled, but see B1 below. I traced played:

    • an AtLeast record clocks fresh, sets unnotified = 0 and marks the next record doubted;
    • a doubted record whose late notification comes with an empty ring returns None and clears doubted.

    The control patch (AtLeast → dropped: false) never sets doubted and leaves the unnotified formula as it was, so it is equivalent to the old accounting. It reds [0, 0, 0].

  2. Storm bound: resolved. I re-derived 3. Handler 1 counts 1 and does not mask, and messages meanwhile collapse into one IRR bit. Handler 2 counts 2 and masks; what the IEC drain remaps collapses into IRR. Handler 3 counts 3 and nothing follows. The A2 control at 10000/10000 shows that unmasked every message lands on the T14.

  3. Issue: resolved. It is kept, narrowed, and has an owner and an exit. One precision point is under suggestions.

SDM citation. The quotation in r2/metal/ab/rules.md ("one in the IRR and one in the ISR … collapsed into the single bit in the IRR") matches the SDM's text for fixed-interrupt acceptance. I did not re-fetch the 093US PDF, so the §13.8.4 number rests on the implementer's recorded check against the hashed PDF. I have no reason to doubt it.

Findings

BLOCKER — sdk/abi/src/pci.rs:210-211, system/soundserver/virtio-sound/src/lib.rs:241, PR description: "a count below MASKED_FROM is every message" is false, and the PR's own citation shows it.

  • Why it is false: §13.8.4 says a vector already pending in IRR absorbs further requests. While cpu0 has interrupts off, two messages collapse into one handler entry and one count. That includes every invalidation wait, which this PR moved under masked interrupts (R3 measured windows of 26–42 µs). A pending LPI folds the same way on AArch64. The count is exact about the kernel's handler entries, not about messages.

  • What breaks: Notified::All(1) for two notifications reproduces r1's stall exactly. unnotified becomes 1, and every later All(1) with one period gives fresh = 0, clocked = 0 and unnotified = 1, so the DLL is never corrected again.

  • Scope: this is pre-existing. Base's "how many messages arrived" carried the same premise. Option (b) even adds a way out: the next AtLeast resets the debt. So this is not a regression of this round. But this round turns the premise into a stated contract and builds Notified::All on it.

  • Required:

    • Make the ABI text and the Notified::All doc true. A count below MASKED_FROM is exact about interrupts the kernel took; the interrupt controller may fold messages before that.
    • Record the remaining virtio-sound stall under issues/ with an owner and an exit, or fix it. Which one is the orchestrator's call, since it touches the (b) ruling's premise.
    • Correct the PR description's matching sentence.

    If it is fixed, the cheapest check is a host test that feeds All(1) while two periods complete.

BLOCKER — false or stale comments.

  • tests/guest/rust-tests/src/bin/i219_msi.rs:153-155 says "the claim's interrupt is masked from the first until this read". Under the new trigger that is false: the first message never masks. The STORM_BOUND comment twelve lines above already states it correctly.
  • system/soundserver/virtio-sound/src/tests.rs:674-675 ("Today's accounting reds this: …") is history that goes stale on landing. The control and its result belong in evidence, where they already are (r2/vsound/control-b.patch, r2/logs/vsound-control-b.log). Delete it.
  • tests.rs:672 is a ~130-column comment line; re-wrap it. The test's doc could be one sentence: a driver that fell behind is clocked again on the second steady read.

SUGGESTION — kernel/src/pcidev/mod.rs:1981. The edited take_record doc line overruns its neighbours and reads awkwardly ("delivered again after a read that took the ones that masked it"). Re-wrap it, or simply say the read re-opens a masked claim.

SUGGESTION — kernel/src/pcidev/record.rs:54. The added blank line detaches READ from the doc comment that describes both halves of the word. Revert that hunk.

SUGGESTION — issues/a-claimed-function-can-storm-its-cpu.md:17-21. "costs cpu0 an invalidation completion wait … at the mask and again at its read … All of it is charged to whatever cpu0 interrupted" overstates. The read's wait runs on the reader's CPU, in the holder's own call. cpu0 bears it only by spinning on the unit's queue lock behind it.

Evidence

  • Loom and its control: under the new trigger, r2/logs/loom-gate-3.log (exit 0) and loom-gate-control-2.log (exit 101, the expected red). Both are adequate.
  • virtio-sound host tests and control: vsound-b.log 11/11, soundserver-b.log 8/8, and vsound-control-b.log red on the new test only. All adequate.
  • Metal r2: the rules were written and amended before the reading; A2 is a real boot of pcidev: one address window per function, every claim address below 4 GiB, and a released function's fault is never the kernel's #1188's kernel. B2 landed 2, then exactly 1 after the read. R2's median is 65.5 → 60.5 µs and R3 holds on both arms. The rules were applied as written.
  • Guest rows at 3dc7e7b: they stand for the head. After that commit the kernel did not change, and the virtio-sound change is covered by host tests.
  • Gap: the published host-CI log is ci-host-4.log at rev 5e60492. Nothing in the bundle shows --ci host at ebee4a6, which adds a public enum and changes soundserver. Publish that log, or reword the PR's claim to the revision the log shows. The merge queue's required host check will also cover it.

SEND BACK

Japabu added a commit that referenced this pull request Nov 5, 2026
… a wake and a time

A record's count is of interrupts the kernel took, not of messages the
device sent: the local APIC folds a message that arrives while its vector is
pending (Intel SDM Vol. 3A, 325384-093, §13.8.4), an LPI folds while
pending, and the claim's mask drops what lands past it. virtio-sound
inferred periods from that count, so a record of one interrupt over two
periods left a notification owed that never landed, and no later record
clocked a period again: the mix loop's DLL stopped being corrected. That
stall predates #1193.

Now the used ring alone says which periods played, on every read, and the
record's newest time anchors the clock only where it can be no other
period's: one interrupt taken and one period read. Any other read anchors
nothing, and its periods carry into the DLL's next update, which it already
does. The notification debt and option (b)'s Notified type are gone.

a_record_folded_over_two_periods_does_not_stop_the_clock feeds a count of 1
over two periods and then three steady reads: [0, 1, 1, 1]. #1188's
accounting, restored as the control, reads [1, 0, 0, 0].

Review r2 of #1193 besides: the ABI says what the count is; i219_msi's storm
comment names the second interrupt as the one that masks; take_record's doc,
record.rs's word constants and the issue's account of who pays a read's
invalidation wait.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Japabu added a commit that referenced this pull request Nov 5, 2026
Seen in #1193's host CI at ce5d846 under host load; the test passes alone
at the same revision. Unrelated to #1193, recorded with its evidence.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Japabu

Japabu commented Nov 5, 2026

Copy link
Copy Markdown
Collaborator Author

Review r3 of #1193 at 2ede9a1 (changes since ebee4a6)

Architectural assessment

This round is the right simplification, and it carries out the ruling. virtio-sound takes its periods only from the used ring. The record is now just a wake-up and a time, and the code anchors only when the count is 1 and the read found exactly one period. Three things are gone: Notified, the unnotified debt and the doubted flag. Soundserver passes record.count straight through. The round is +79/−100 across ten files. Most of the deletion is the accounting state in lib.rs, and the kernel change is limited to two doc lines and one blank line. Dll::update already folds an unanchored read's periods into its next anchored update (unclocked), so after a folded read the prediction counts the right number of grid points. I traced folded 2, then 1, 1, 1: n = 3 at the first anchored update, predicted at t_est + 2·P.

r2 blockers.

  1. The false count contract: resolved in the ABI and the SDK. sdk/abi/src/pci.rs now says the count is of interrupts the kernel took, never of messages sent, and that the controller folds while one is pending. The "below MASKED_FROM is every message" sentence is gone. The PciDev::irq and take_record text matches the trigger. On the code side, the stall is fixed rather than recorded, as ruled.
  2. Stale comments: resolved. i219_msi.rs now says "from the second", the history paragraph and the long line in tests.rs are gone, and the test doc is one sentence.

Suggestions taken. take_record's doc is re-wrapped and true. The blank line in record.rs is reverted. The issue text now places the read's wait in the holder's call. All three are correct.

Host test and control. a_record_folded_over_two_periods_does_not_stop_the_clock reproduces the r2 stall: count 1 over two periods, then steady reads. r3/vsound/control-r3.patch restores #1188's accounting faithfully: fresh = count − unnotified and the debt carried forward. That control reds with [1, 0, 0, 0], the permanent stall, and the head gives [0, 1, 1, 1] (vsound-r3.log, exit 0; control exit 101). The control is adequate.

The 70-interleaving test. It passes and still holds what it can reach. Every order returns both periods exactly once, and the claim is never left unreadable. But see B1: with two periods it cannot reach the case where the new rule mis-anchors, and its doc states that property in general.

The stated remaining risk: real, bounded, and the docs must say so

The risk is not in the PR description. The body says the anchor case is "the one case where it can be no other period's", and that the change touches clocking "only by removing a stall". I traced the case with the test's own Claim model and three periods:

U0 N0@1 Take(1,@1) U1 Read gives mask {p0, p1} and clocked 0. Then N1@2 Take(1,@2) U2 Read gives mask {p2} and clocked 1, stamped @2. That is p1's notification credited to p2. Extending DEVICE to three periods would red the test's own period < at assertion on this order.

Bound. p1's notification t1 comes before the second take. p2's true time comes after its used element, which comes before the second read. So the anchor is early by at most the time from t1 to the second ring read, plus the device's gap between writing the used element and sending the notification. That is the driver's own wake latency for that record, the same size as wake jitter the DLL already absorbs. It produces one error sample at bw = 0.03; it does not stall. Both timings must also go against the driver, and virtio-sound runs only under QEMU, whose clock is not a timing source.

Fix here or record? Neither code change nor an issue is warranted. #1188's debt counter handled read-ahead correctly, but it is exactly what folding breaks. A count cannot tell a folded message from a read-ahead period, and the doubted flag does not help either: a two-period read-ahead followed by an empty read clears it with debt still outstanding. The ruled rule trades a permanent stall for one bounded, early sample, which is the right trade. What must change is the text, which currently claims certainty.

Findings

BLOCKER B1. The new clocking contract claims certainty that the code lacks.

  • system/soundserver/virtio-sound/src/lib.rs:244-246 says Played::clocked is 1 "where the record's newest notification is the time of the one period this read found".
  • lib.rs:344-346 says the newest time clocks a period "only where it can be no other's".
  • tests.rs:588-591 says no period is "stamped earlier than its own notification".

All three are false for the order above. This is the same class of defect as r2's B1. Required:

  • Say the anchor is the newest notification's time. Say that a period read ahead of its notification can make the next one-and-one read anchor early by at most that read's wake latency. One clause at played is enough; Played::clocked then needs only "1 where this read anchors, else 0".
  • Narrow the test doc to what two periods establish: no period is stranded, each returns once, and no stamp precedes its notification within two periods. Alternatively, drop the stamp assertion.
  • Correct the PR body's "can be no other period's" and "only by removing a stall" (orchestrator). The trade also gives up pcidev: one address window per function, every claim address below 4 GiB, and a released function's fault is never the kernel's #1188's correct handling of read-ahead.

BLOCKER B2. issues/the-sysroot-pin-test-reads-a-stat-dirty-file-as-uncommitted-work.md: the diagnosis is not supported by its own evidence.

  • Which move refused. ci-host-6.log:695-701 shows moves 0a69→3a47, 3a47→0a69 and 0a69→3a47, then the refusal "at 3a47 … pins 0a69". That is the fourth move, pin(&c1) at build/sysroot.rs:1959-1960. It is not the deliberate edit at :1962, and the test had written nothing to library/std/src/lib.rs before it.
  • What .M shows. Equal HEAD and index hashes are what every .M line in porcelain v2 shows. The worktree hash is not printed, so the line says nothing about content.
  • Why the stat explanation does not fit. git status refreshes the index and re-hashes stat-dirty entries before reporting. A racily clean file whose content is unchanged is not reported as .M.

So "dirty by its stat alone, git's racy-timestamp case" is unestablished and probably wrong. Its exit, "refreshing the index first", would then fix nothing. Required: record what was observed (the step, the line, the pass when run alone) and leave the cause open, or investigate it. The file is new in this PR, so its truth is this PR's responsibility.

SUGGESTION. sdk/abi/src/pci.rs:225. "When the newest was:" reads as truncated. Use "When the newest was taken:".

Host CI and evidence

  • 108 of 109: acceptable.
    • buildlock: 7d79229de (Two host tests that reddened under concurrency read their own state right #1186) is on main and is not an ancestor of 2ede9a1. The failure is that known concurrency defect, which goes away on rebase. The merge queue's required host check runs on the rebased head.
    • sysroot: independent of this branch, which touches nothing under build/ except the r1 control registration in ci.rs. sysroot-alone.log passes at the same head. It does not block, but its record must be corrected (B2).
  • Image: build-x86-8.log at ce5d846. 2ede9a1 adds only an issue file, so the evidence carries over.
  • Metal and guest rows: the kernel diff since the r2 A/B is doc text and one blank line, so they stand.

SEND BACK

Japabu added a commit that referenced this pull request Nov 5, 2026
…record held to what was seen

- virtio-sound's anchor is the record's newest time where one interrupt was
  taken and one period read. A period read ahead of its notification can
  make the next such read anchor early, by at most that read's wake latency;
  played, Played::clocked and the interleaving test's doc now say so, the
  test's stamp claim narrowed to the two periods it drives.
- The sysroot record states the step that refused, its line, and that the
  test passes alone, and leaves the cause open; renamed to match.
- DeviceIrqRecord::last_nanos: "when the newest was taken".

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Japabu

Japabu commented Nov 5, 2026

Copy link
Copy Markdown
Collaborator Author

Orchestrator verification of r3's fix (2ede9a1..d080680), which changes no code:

  • Every changed line in the three .rs files is a comment or doc line. The rest is the sighting record, renamed and narrowed to what was observed.
  • B1 is resolved: played states the anchor's bound (early by at most one read's wake latency after a read-ahead), Played::clocked and the interleaving test claim no more than they show, and the PR body states the trade honestly.
  • B2 is resolved: the sysroot record keeps only the observed step and line and leaves the cause open.
  • The kernel is unchanged since the round-2 metal A/B, and the virtio-sound tests pass 11/11.

This lands after #1188, once rebased onto it.

Japabu and others added 11 commits November 5, 2026 23:20
A function a process drives could raise its claim's interrupt on cpu0 as
fast as its holder made it send, and nothing bounded how often. Now the
handler masks the claim's message after it counts one, and the holder's
read that takes the count delivers it again: one interrupt per read, with no
rate to choose.

Each side changes the record's count first and then, under the claim's
gate, puts hardware in the state the record owes (masked exactly while a
count waits), read from the word and not from its own change. The side that
takes the gate last sees both changes, so a mask that lands after the read
that re-armed the claim cannot strand it masked. kernel/loom's
device_irq_gate models it; the device-irq-gate-intent control, where each
side applies its own intent, reds both models.

The masks, each published before it returns:
- VT-d: the entry not present with FPD set (Rev. 4.1 §9.9, §5.1.4.1 Table
  13: the blocked message's fault is qualified and not recorded), one
  64-bit write, then an index-selective interrupt-entry-cache invalidation
  (§6.4, §6.5.2.7), which also drains what the unit already remapped.
- AMD-Vi: RemapEn clear and SupIOPF set (48882 rev 3.11 Table 22; target
  abort, §2.1.3.1), then INVALIDATE_INTERRUPT_TABLE (§2.4.5).
- ITS: the LPI's configuration byte disabled, then INV and SYNC; the message
  stays pending until it is enabled again. Release enables it first.

The handler cannot take the existing IOMMU locks: VT-d's map takes TABLES
under UNITS and reaches the page allocator. So each unit's invalidation queue
(VT-d's queue, AMD-Vi's command buffer, the ITS command queue) moves to a
leaf lock of its own, taken with interrupts masked and holding nothing else.
Every invalidation's completion wait now runs with interrupts off.

Drivers must take their record before servicing the device, stated at
toyos::PciDev::irq; all ten already do.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
10,000 RXSEQ causes forced back to back with ITR off and the record left
unread, each read off ICR as asserted: at most 64 messages may land, and
the next forced cause after the read is exactly one. Without the mask every
cause lands, so the kernel change reverted reds it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Clippy refused the tuple as too complex.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…count cleared before the gate

- i219_msi's storm bound is 2: the handler masks and drains before its end
  of interrupt, and a local APIC holds one request per vector beside the one
  in service (Intel SDM Vol. 3A §11.8.4). Unmasked, every forced cause lands
  here, as the A arm measured (10,000 of 10,000): the ICR read between forces
  outlasts the handler.
- issues/a-claimed-function-can-storm-its-cpu.md stays open, narrowed to what
  is left: a holder that loops force-and-read paces cpu0 and its
  invalidation waits, a refused message is counted nowhere, and no T14 row
  shows cpu0's timer served under such a holder.
- A claim clears its record before it arms the gate, so the order no longer
  rests on mastering being off.
- The gate's loom header is one sentence.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Masking on every delivery cost each device interrupt a remapping-entry
write and an invalidation wait, +33 us median on the T14's I219 (A/B r1).
The gate now masks only when a message lands while the record still holds
an unread count, so a driver that reads between messages never pays it,
and a storm between two reads stops at what lands before the mask takes:
the second message masks, and its handler's drain leaves at most one more
request in the local APIC (Intel SDM Vol. 3A, 325384-093, §13.8.4). The
gate, the backends and the loom model's shape are unchanged; the owed state
is `count >= MASKED_FROM`, read by both sides under the gate.

i219_msi's storm bound is 3 accordingly. The citation in the previous
commit's message, §11.8.4, was that section's number in older revisions.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A record below MASKED_FROM counts every message, and one at or above it is a
lower bound; the kernel's gate and a driver that needs the difference read
the same constant.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
On #1188's and #1193's T14 runs the wire sends the burst faster than the
driver queues it, so the phase's tx_full requirement is never met; recorded
under the issue whose exit that premise is.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…he next is not clocked (option b)

With #1193's gate a claim's interrupt masks at the second notification
before a read, and what lands past it is dropped, so a record at or above
toyos_abi::pci::MASKED_FROM is a lower bound. The notifications it holds
still answer the first periods in order, so they clock as before; what it
leaves owed is unknown and is taken as none, and the next record clocks
nothing, since its first notification may be for a period already read.
Today's accounting kept the dropped notifications owed, so no later record
clocked a period again and the mix loop's DLL stopped being corrected.

Cost: one record's anchor after each record that dropped notifications;
its periods carry into the DLL's next update. A wrong anchor needs that race
twice running.

a_record_that_dropped_notifications_does_not_stop_the_clock holds it; today's
accounting reds it, clocking [0, 0, 0].

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… a wake and a time

A record's count is of interrupts the kernel took, not of messages the
device sent: the local APIC folds a message that arrives while its vector is
pending (Intel SDM Vol. 3A, 325384-093, §13.8.4), an LPI folds while
pending, and the claim's mask drops what lands past it. virtio-sound
inferred periods from that count, so a record of one interrupt over two
periods left a notification owed that never landed, and no later record
clocked a period again: the mix loop's DLL stopped being corrected. That
stall predates #1193.

Now the used ring alone says which periods played, on every read, and the
record's newest time anchors the clock only where it can be no other
period's: one interrupt taken and one period read. Any other read anchors
nothing, and its periods carry into the DLL's next update, which it already
does. The notification debt and option (b)'s Notified type are gone.

a_record_folded_over_two_periods_does_not_stop_the_clock feeds a count of 1
over two periods and then three steady reads: [0, 1, 1, 1]. #1188's
accounting, restored as the control, reads [1, 0, 0, 0].

Review r2 of #1193 besides: the ABI says what the count is; i219_msi's storm
comment names the second interrupt as the one that masks; take_record's doc,
record.rs's word constants and the issue's account of who pays a read's
invalidation wait.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Seen in #1193's host CI at ce5d846 under host load; the test passes alone
at the same revision. Unrelated to #1193, recorded with its evidence.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…record held to what was seen

- virtio-sound's anchor is the record's newest time where one interrupt was
  taken and one period read. A period read ahead of its notification can
  make the next such read anchor early, by at most that read's wake latency;
  played, Played::clocked and the interleaving test's doc now say so, the
  test's stamp claim narrowed to the two periods it drives.
- The sysroot record states the step that refused, its line, and that the
  test passes alone, and leaves the cause open; renamed to match.
- DeviceIrqRecord::last_nanos: "when the newest was taken".

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
virt_claim_lpi's claim_bar_fault job reds on main's own 0b2dec6 exactly as
on #1193's rebased head: the read answers Ok(16), not BadAddress. Main's, not
#1193's; recorded with both runs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Japabu

Japabu commented Nov 6, 2026

Copy link
Copy Markdown
Collaborator Author

Orchestrator verification of the rebase onto main (d080680 → 972ba7c): git range-diff shows all 11 commits identical (=), with no conflicts. One added commit is a markdown-only sighting record. At 9b0fe9c: host CI 109/109, the x86-64 image builds, and the x86 guest rows pass (42120668541). The aarch64 virt_claim_lpi red reproduces on main's own 0b2dec6 (run 42124293522), so it is main's and is recorded as issues/a-read-into-a-claims-bar-is-not-refused-on-aarch64.md. The review verdicts and the round-2 metal apply. Landing.

@Japabu
Japabu marked this pull request as ready for review November 6, 2026 08:20
@Japabu
Japabu added this pull request to the merge queue Nov 6, 2026
Merged via the queue into main with commit f9b5d7f Nov 6, 2026
10 checks passed
@Japabu
Japabu deleted the storm-mask branch November 6, 2026 08:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant