Repository navigation
A claimed function interrupts at most once per read of its record - #1193
Conversation
Review of #1193 at 39cbe73 (base
|
…count cleared before the gate - i219_msi's storm bound is 2: the handler masks and drains before its end of interrupt, and a local APIC holds one request per vector beside the one in service (Intel SDM Vol. 3A §11.8.4). Unmasked, every forced cause lands here, as the A arm measured (10,000 of 10,000): the ICR read between forces outlasts the handler. - issues/a-claimed-function-can-storm-its-cpu.md stays open, narrowed to what is left: a holder that loops force-and-read paces cpu0 and its invalidation waits, a refused message is counted nowhere, and no T14 row shows cpu0's timer served under such a holder. - A claim clears its record before it arms the gate, so the order no longer rests on mastering being off. - The gate's loom header is one sentence. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review r2 of #1193 at ebee4a6 (changes since 39cbe73)Architectural assessmentThe round changes only what the ruling asked for. Race argument under the new threshold. It still holds, and the threshold does not matter to it. Every change to the word is followed by the same side taking the gate lock: Loom.
The control now computes Claim-time clear (r1 suggestion). r1 blockers.
SDM citation. The quotation in FindingsBLOCKER —
BLOCKER — false or stale comments.
SUGGESTION — SUGGESTION — SUGGESTION — Evidence
SEND BACK |
… a wake and a time A record's count is of interrupts the kernel took, not of messages the device sent: the local APIC folds a message that arrives while its vector is pending (Intel SDM Vol. 3A, 325384-093, §13.8.4), an LPI folds while pending, and the claim's mask drops what lands past it. virtio-sound inferred periods from that count, so a record of one interrupt over two periods left a notification owed that never landed, and no later record clocked a period again: the mix loop's DLL stopped being corrected. That stall predates #1193. Now the used ring alone says which periods played, on every read, and the record's newest time anchors the clock only where it can be no other period's: one interrupt taken and one period read. Any other read anchors nothing, and its periods carry into the DLL's next update, which it already does. The notification debt and option (b)'s Notified type are gone. a_record_folded_over_two_periods_does_not_stop_the_clock feeds a count of 1 over two periods and then three steady reads: [0, 1, 1, 1]. #1188's accounting, restored as the control, reads [1, 0, 0, 0]. Review r2 of #1193 besides: the ABI says what the count is; i219_msi's storm comment names the second interrupt as the one that masks; take_record's doc, record.rs's word constants and the issue's account of who pays a read's invalidation wait. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review r3 of #1193 at 2ede9a1 (changes since ebee4a6)Architectural assessmentThis round is the right simplification, and it carries out the ruling. r2 blockers.
Suggestions taken. Host test and control. The 70-interleaving test. It passes and still holds what it can reach. Every order returns both periods exactly once, and the claim is never left unreadable. But see B1: with two periods it cannot reach the case where the new rule mis-anchors, and its doc states that property in general. The stated remaining risk: real, bounded, and the docs must say soThe risk is not in the PR description. The body says the anchor case is "the one case where it can be no other period's", and that the change touches clocking "only by removing a stall". I traced the case with the test's own
Bound. p1's notification t1 comes before the second take. p2's true time comes after its used element, which comes before the second read. So the anchor is early by at most the time from t1 to the second ring read, plus the device's gap between writing the used element and sending the notification. That is the driver's own wake latency for that record, the same size as wake jitter the DLL already absorbs. It produces one error sample at Fix here or record? Neither code change nor an issue is warranted. #1188's debt counter handled read-ahead correctly, but it is exactly what folding breaks. A count cannot tell a folded message from a read-ahead period, and the FindingsBLOCKER B1. The new clocking contract claims certainty that the code lacks.
All three are false for the order above. This is the same class of defect as r2's B1. Required:
BLOCKER B2.
So "dirty by its stat alone, git's racy-timestamp case" is unestablished and probably wrong. Its exit, "refreshing the index first", would then fix nothing. Required: record what was observed (the step, the line, the pass when run alone) and leave the cause open, or investigate it. The file is new in this PR, so its truth is this PR's responsibility. SUGGESTION. Host CI and evidence
SEND BACK |
…record held to what was seen - virtio-sound's anchor is the record's newest time where one interrupt was taken and one period read. A period read ahead of its notification can make the next such read anchor early, by at most that read's wake latency; played, Played::clocked and the interleaving test's doc now say so, the test's stamp claim narrowed to the two periods it drives. - The sysroot record states the step that refused, its line, and that the test passes alone, and leaves the cause open; renamed to match. - DeviceIrqRecord::last_nanos: "when the newest was taken". Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Orchestrator verification of r3's fix (2ede9a1..d080680), which changes no code:
This lands after #1188, once rebased onto it. |
A function a process drives could raise its claim's interrupt on cpu0 as fast as its holder made it send, and nothing bounded how often. Now the handler masks the claim's message after it counts one, and the holder's read that takes the count delivers it again: one interrupt per read, with no rate to choose. Each side changes the record's count first and then, under the claim's gate, puts hardware in the state the record owes (masked exactly while a count waits), read from the word and not from its own change. The side that takes the gate last sees both changes, so a mask that lands after the read that re-armed the claim cannot strand it masked. kernel/loom's device_irq_gate models it; the device-irq-gate-intent control, where each side applies its own intent, reds both models. The masks, each published before it returns: - VT-d: the entry not present with FPD set (Rev. 4.1 §9.9, §5.1.4.1 Table 13: the blocked message's fault is qualified and not recorded), one 64-bit write, then an index-selective interrupt-entry-cache invalidation (§6.4, §6.5.2.7), which also drains what the unit already remapped. - AMD-Vi: RemapEn clear and SupIOPF set (48882 rev 3.11 Table 22; target abort, §2.1.3.1), then INVALIDATE_INTERRUPT_TABLE (§2.4.5). - ITS: the LPI's configuration byte disabled, then INV and SYNC; the message stays pending until it is enabled again. Release enables it first. The handler cannot take the existing IOMMU locks: VT-d's map takes TABLES under UNITS and reaches the page allocator. So each unit's invalidation queue (VT-d's queue, AMD-Vi's command buffer, the ITS command queue) moves to a leaf lock of its own, taken with interrupts masked and holding nothing else. Every invalidation's completion wait now runs with interrupts off. Drivers must take their record before servicing the device, stated at toyos::PciDev::irq; all ten already do. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
10,000 RXSEQ causes forced back to back with ITR off and the record left unread, each read off ICR as asserted: at most 64 messages may land, and the next forced cause after the read is exactly one. Without the mask every cause lands, so the kernel change reverted reds it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Clippy refused the tuple as too complex. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…count cleared before the gate - i219_msi's storm bound is 2: the handler masks and drains before its end of interrupt, and a local APIC holds one request per vector beside the one in service (Intel SDM Vol. 3A §11.8.4). Unmasked, every forced cause lands here, as the A arm measured (10,000 of 10,000): the ICR read between forces outlasts the handler. - issues/a-claimed-function-can-storm-its-cpu.md stays open, narrowed to what is left: a holder that loops force-and-read paces cpu0 and its invalidation waits, a refused message is counted nowhere, and no T14 row shows cpu0's timer served under such a holder. - A claim clears its record before it arms the gate, so the order no longer rests on mastering being off. - The gate's loom header is one sentence. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Masking on every delivery cost each device interrupt a remapping-entry write and an invalidation wait, +33 us median on the T14's I219 (A/B r1). The gate now masks only when a message lands while the record still holds an unread count, so a driver that reads between messages never pays it, and a storm between two reads stops at what lands before the mask takes: the second message masks, and its handler's drain leaves at most one more request in the local APIC (Intel SDM Vol. 3A, 325384-093, §13.8.4). The gate, the backends and the loom model's shape are unchanged; the owed state is `count >= MASKED_FROM`, read by both sides under the gate. i219_msi's storm bound is 3 accordingly. The citation in the previous commit's message, §11.8.4, was that section's number in older revisions. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A record below MASKED_FROM counts every message, and one at or above it is a lower bound; the kernel's gate and a driver that needs the difference read the same constant. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…he next is not clocked (option b) With #1193's gate a claim's interrupt masks at the second notification before a read, and what lands past it is dropped, so a record at or above toyos_abi::pci::MASKED_FROM is a lower bound. The notifications it holds still answer the first periods in order, so they clock as before; what it leaves owed is unknown and is taken as none, and the next record clocks nothing, since its first notification may be for a period already read. Today's accounting kept the dropped notifications owed, so no later record clocked a period again and the mix loop's DLL stopped being corrected. Cost: one record's anchor after each record that dropped notifications; its periods carry into the DLL's next update. A wrong anchor needs that race twice running. a_record_that_dropped_notifications_does_not_stop_the_clock holds it; today's accounting reds it, clocking [0, 0, 0]. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… a wake and a time A record's count is of interrupts the kernel took, not of messages the device sent: the local APIC folds a message that arrives while its vector is pending (Intel SDM Vol. 3A, 325384-093, §13.8.4), an LPI folds while pending, and the claim's mask drops what lands past it. virtio-sound inferred periods from that count, so a record of one interrupt over two periods left a notification owed that never landed, and no later record clocked a period again: the mix loop's DLL stopped being corrected. That stall predates #1193. Now the used ring alone says which periods played, on every read, and the record's newest time anchors the clock only where it can be no other period's: one interrupt taken and one period read. Any other read anchors nothing, and its periods carry into the DLL's next update, which it already does. The notification debt and option (b)'s Notified type are gone. a_record_folded_over_two_periods_does_not_stop_the_clock feeds a count of 1 over two periods and then three steady reads: [0, 1, 1, 1]. #1188's accounting, restored as the control, reads [1, 0, 0, 0]. Review r2 of #1193 besides: the ABI says what the count is; i219_msi's storm comment names the second interrupt as the one that masks; take_record's doc, record.rs's word constants and the issue's account of who pays a read's invalidation wait. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…record held to what was seen - virtio-sound's anchor is the record's newest time where one interrupt was taken and one period read. A period read ahead of its notification can make the next such read anchor early, by at most that read's wake latency; played, Played::clocked and the interleaving test's doc now say so, the test's stamp claim narrowed to the two periods it drives. - The sysroot record states the step that refused, its line, and that the test passes alone, and leaves the cause open; renamed to match. - DeviceIrqRecord::last_nanos: "when the newest was taken". Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Orchestrator verification of the rebase onto main (d080680 → 972ba7c): |
On main at 0b2dec6, after #1188 and #1189; the rebase applied every commit unchanged.
Status: round 3 is in review. The kernel is unchanged since the round-2 A/B, in which R1 to R3 held. virtio-sound now counts periods from its used ring alone.
Problem
A function a process drives raises its claim's interrupt on cpu0 for every message, and nothing bounded how often: a hostile or broken holder could make its device flood cpu0 (
issues/a-claimed-function-can-storm-its-cpu.md, kept open and narrowed to what remains).Behavior
A claim's interrupt is masked from the second interrupt the kernel takes before its holder reads the record, until that read. A driver that reads between messages never pays for a mask. One that falls behind is cut off: the first message never masks, the second does, and that handler's drain can leave at most one more request held in the local APIC. So at most 3 land per read (Intel SDM Vol. 3A, 325384-093, §13.8.4). A message sent while masked is refused at the unit on x86-64 and left pending on AArch64. A record's count is of interrupts the kernel took, never of messages sent: the interrupt controller folds a message that arrives while its vector is pending, and the mask drops what lands past it (
toyos_abi::pci). Drivers take the record before servicing the device, whichtoyos::PciDev::irqstates.P=0,FPD=1, one 64-bit write (Rev. 4.1 §9.9; §5.1.4.1 Table 13: fault 22h is qualified, so not recorded)RemapEn=0,SupIOPF=1 (48882 rev 3.11 Table 22; target abort per §2.1.3.1)INVALIDATE_INTERRUPT_TABLE(§2.4.5)INV+SYNC; release re-enables beforeDISCARDDesign decisions
MASKED_FROM). The side that takes the gate last sees both changes, so a mask that lands after the read that re-armed the claim cannot strand it.record.rsowns the rule;pcidevowns one gate per function. The gate is armed after the count is cleared at mint and disarmed before the message is given back.TABLESunderUNITSand reaches the page allocator. Every invalidation's completion wait now runs with interrupts off, including ordinary map and unmap.Checks
device_irq_gate: one message per read never masks. Two messages raced against a read end with hardware where the record owes it. Controldevice-irq-gate-intent(the handler masks on its own view, and the read re-arms by intent) reds the two-message model with "masked where the record owes it open"; it is registered inbuild/ci.rs.git range-diff: all=). Evidence: https://github.com/ToyOSOrg/evidence/tree/05c9ce10c698b6618a8ac9c5f38f41fe1e2451a0/pr-1193/r4cargo run -- --ci hostat 9b0fe9c: 109 steps green, buildlock included. The x86-64 image builds.usbd_drives_the_spare nvme_disk_boots bar_map_againat 9b0fe9c, green (run 42120668541).virt_claim_lpi:claim_lpiitself passes, with 16 messages read as 16 interrupts. The row is red onclaim_bar_fault: a read into the BAR answersOk(16), notBadAddress(run 42120673220). Main's own 0b2dec6 reds the same way (run 42124293522), so this is main's and not this PR's. It is recorded inissues/a-read-into-a-claims-bar-is-not-refused-on-aarch64.md(972ba7c, markdown only).issues/the-sysroot-pin-test-refused-a-move-it-expects-to-make.mdcomes from round 3's host run on pcidev: one address window per function, every claim address below 4 GiB, and a released function's fault is never the kernel's #1188's base.usbd_drives_the_spare nvme_disk_boots bar_map_againgreen (run 42077059466); aarch64virt_claim_lpigreen (run 42077063953). Round 1's rows at f21ae33 were green on both.filesis memory-only.kernel/as pcidev: one address window per function, every claim address below 4 GiB, and a released function's fault is never the kernel's #1188 left it, with the storm test, a real boot of that kernel; B2 is 3dc7e7b. Rows:i219_messages_on_its_msi_vectorandmask_windows, judged under rules written before reading (R1 is B2 1..=3 and A2 > 3; R2 and R3 are unchanged). Evidence: https://github.com/ToyOSOrg/evidence/tree/acac73476da68a5e217cc43859f9d8544b9c203d/pr-1193/r2mask_windowspasses on both arms. Log write: 30.2 → 41.9 µs (×1.38). Herd: 21.2 → 26.1 µs. Round 1's A read 42.3 µs for the same log write, so the spread between arms is of the same order.virtio-sound (review r1 and r2 blockers)
virtio-sound inferred completed periods from the record's count. The count is of interrupts taken, and the local APIC folds a message that arrives while its vector is pending (SDM §13.8.4). An LPI folds while pending on AArch64, and the mask drops what lands past it. So a record of one interrupt over two periods left a notification owed that never landed, and no later record clocked a period again: the mix loop's DLL stopped being corrected. That stall predates this PR.
Per the coordinator's ruling, the used ring alone says which periods played, on every read. The record is a wake and a time. Its newest time anchors the clock where one interrupt was taken and one period read. Any other read anchors nothing, and its periods carry into the DLL's next update, which it already handles. The notification debt and round 2's
Notifiedtype are gone.The anchor is not certain. A period read ahead of its notification can make the next one-interrupt, one-period read anchor early, by at most that read's wake latency: one error sample the DLL absorbs at its bandwidth, not a stall. This gives up #1188's exact handling of read-ahead. That handling rested on a count of messages, which folding breaks: a count cannot tell a folded message from a period read ahead.
a_record_folded_over_two_periods_does_not_stop_the_clockfeeds a count of 1 over two periods, then three steady reads, and gets[0, 1, 1, 1]. #1188's accounting, restored as the control, gives[1, 0, 0, 0]. The existing 70-interleaving test passes. Within its two periods none is stamped before its own notification; a third can be, as above. The trade, ruled by the coordinator without the owner, is a permanent stall that exists today against one bounded early anchor in a two-sided race. virtio-sound runs only under QEMU.Sightings
i219_msi's burst never fills the T14's transmit ring ("the ring full 0 time(s)"). This holds both without the mask (pcidev: one address window per function, every claim address below 4 GiB, and a released function's fault is never the kernel's #1188's run) and with it. It is recorded underissues/the-intel-nics-room-and-wake-are-unread-on-hardware.md.Risks
🤖 Generated with Claude Code