diff --git a/issues/a-two-thread-ping-pong-costs-toyos-up-to-twice-linuxs-on-the-t14.md b/issues/a-two-thread-ping-pong-costs-toyos-up-to-twice-linuxs-on-the-t14.md index dc3a01c71b1..d83a704263b 100644 --- a/issues/a-two-thread-ping-pong-costs-toyos-up-to-twice-linuxs-on-the-t14.md +++ b/issues/a-two-thread-ping-pong-costs-toyos-up-to-twice-linuxs-on-the-t14.md @@ -36,7 +36,7 @@ on its head than its base (`issues/a-64-kib-pipe-write-is-two-percent-below-the-two-mebibyte-ring.md`), so the `pipe` rows' current state against Linux is owed a new reading. The scheduler's pair-wake records bear on both kinds: -`issues/a-wake-whose-waker-parks-next-is-forwarded-to-an-idle-cpu.md` and +`issues/a-wake-held-for-its-posters-park-is-unpriced-on-metal.md` and `issues/an-ordinary-wake-posted-on-its-own-cpu-waits-for-that-cpus-next-pass.md`. **Exit**: `pingpong.*` within Linux's on both benches in a labelled diff --git a/issues/a-wake-held-for-its-posters-park-is-unpriced-on-metal.md b/issues/a-wake-held-for-its-posters-park-is-unpriced-on-metal.md new file mode 100644 index 00000000000..662b57a76f0 --- /dev/null +++ b/issues/a-wake-held-for-its-posters-park-is-unpriced-on-metal.md @@ -0,0 +1,35 @@ +--- +status: open +kind: defect +opened: 2026-10-12 +--- + +# A wake held for its poster's park is unpriced on metal + +A pass's opening holds every task it would forward for queueing behind the +running task, and places them after the pass's disposition +(`CpuSched::place_settled`, `kernel/pure/sched/cpu.rs`). A poster that parks, +stops or exits leaves the CPU to the first held task of each band; one that +yields or keeps the CPU forwards them to idle siblings as before, and so do +the rest of a burst. The simulator's `turn_taking_pair` holds the case: no +turn is forwarded off its CPU. + +What it costs or saves on metal is unread. No row prices the forward against +the local switch. The first A/B that carried the rule, on the T14 with base +`0c7e62ca7` against a head that also carried CPU affinity and the spawn +tie-break, read twelve disk and memfs rows 1–3% slower (fstat, seek, stat, +pread_4k, read_4k_at, write_4k_at, tree.100), four boots a side with one image +each, so a layout difference between the images is not ruled out +([evidence](https://github.com/ToyOSOrg/evidence/tree/6cd2768011f0e6f36c4e2c1ce76f986fc6c2265b/pr-1116/r2)). +An A/B of the rule alone on the T14, base `24a538c88` against `042b2b299`, +reproduced seven disk rows 1.2–2.3% slower and left the ipc rows unchanged. +Timer rows the rule cannot reach (`wake.periodic.idle`, `tail.*`) read slower +by the same amount there and in two A/Bs of other changes, where the base +boots first and the images differ, so these A/Bs carry a head bias of that +size and the rows are not attributed +([evidence](https://github.com/ToyOSOrg/evidence/tree/f47a3a6c8324cc7c4a1f72921a350b2c2034dbec/pr-1111/r5)). + +**Exit**: a T14 and an AMD A/B of the rule alone against its base, in an +order that boots each side first equally often, reads those rows within their +spread or names the cost, and a metal row prices a wake its poster parks +behind against the forward. diff --git a/issues/a-wake-whose-waker-parks-next-is-forwarded-to-an-idle-cpu.md b/issues/a-wake-whose-waker-parks-next-is-forwarded-to-an-idle-cpu.md deleted file mode 100644 index 8ec35dd46d0..00000000000 --- a/issues/a-wake-whose-waker-parks-next-is-forwarded-to-an-idle-cpu.md +++ /dev/null @@ -1,27 +0,0 @@ ---- -status: open -kind: finding -opened: 2026-10-12 ---- - -# A wake whose waker parks next is forwarded to an idle CPU - -`CpuSched::place` (`kernel/pure/sched/cpu.rs`) hands a fair wake that would -queue behind the running fair task to an idle sibling. When the waker is about -to park on that same CPU — a request answered and waited on, a thread started -and then joined — the CPU is free a moment later, and the forward pays an IPI -and a cold cache where a local switch after the park was free. Nothing tells -`place` that the running task is about to park. - -`acpi_mediated` met it on a guest: a thread started from a busy CPU went to the -idle one, so it stages its asker differently since PR #874. On metal the -spawn-then-wait row reads no change past one boot's spread: `spawn_cost`'s -median went from 1,025 to 1,029 µs on the T14 and from 2,054 to 2,086 µs on -the AMD bench, base to head, one boot each, read from the second judgement's -`testcases` readbacks. - -Owned by stage S3 of the scheduler design (#872), whose 1:1 pair rule decides -where a pair's wakes go. - -**Exit**: S3 decides this case, with a simulator scenario of a wake whose waker -parks next and a metal row that prices the forward against the local switch. diff --git a/issues/an-ordinary-wake-posted-on-its-own-cpu-waits-for-that-cpus-next-pass.md b/issues/an-ordinary-wake-posted-on-its-own-cpu-waits-for-that-cpus-next-pass.md index 93222217aaa..a087f8b5ffc 100644 --- a/issues/an-ordinary-wake-posted-on-its-own-cpu-waits-for-that-cpus-next-pass.md +++ b/issues/an-ordinary-wake-posted-on-its-own-cpu-waits-for-that-cpus-next-pass.md @@ -27,8 +27,8 @@ process's fair-share lock, which the storm's other waiters were taking as they re-parked. Owned by stage S3 of the scheduler design (#872), which decides where a -pair's wakes go: a wake whose waker parks next is cheapest left here -(`issues/a-wake-whose-waker-parks-next-is-forwarded-to-an-idle-cpu.md`). +pair's wakes go: a wake whose waker parks next is held and left here +(`issues/a-wake-held-for-its-posters-park-is-unpriced-on-metal.md`). **Exit**: a wake posted on its own CPU by a poster that keeps running reaches a CPU within the bound `wake_onto_busy` holds for other CPUs' wakes, on the diff --git a/kernel/pure/sched/cpu.rs b/kernel/pure/sched/cpu.rs index 9273993293b..6945cbaafb0 100644 --- a/kernel/pure/sched/cpu.rs +++ b/kernel/pure/sched/cpu.rs @@ -295,6 +295,14 @@ pub struct CpuSched { /// In no fair band, as [`Self::dying`]'s tasks are not; excluded from the /// published load, because nothing queues behind work that will never run. stopped: VecDeque>, + /// Tasks the pass's opening (its drain and its deadlines) found behind + /// the running task, held for [`SchedPass`]'s disposition: whether that + /// task leaves the CPU in this pass decides whether they run here or are + /// forwarded. Empty outside a pass; the buffer is kept between passes. + held: Vec>, + /// Set by [`SchedPass::begin`] and cleared by the disposition, so only + /// the pass's opening holds. + holding: bool, /// The task that exited on this CPU, freed by the NEXT pass — a pass /// cannot free the stack it is running on. zombie: Option>, @@ -393,6 +401,8 @@ impl CpuSched { parked: Parked::new(), dying: VecDeque::new(), stopped: VecDeque::new(), + held: Vec::new(), + holding: false, zombie: None, mailbox, steal_probe: MailboxNode::new(), @@ -615,6 +625,8 @@ impl CpuSched { (core::mem::offset_of!(Self, parked), "parked (the park map)"), (core::mem::offset_of!(Self, dying), "dying"), (core::mem::offset_of!(Self, stopped), "stopped"), + (core::mem::offset_of!(Self, held), "held"), + (core::mem::offset_of!(Self, holding), "holding"), (core::mem::offset_of!(Self, zombie), "zombie"), (core::mem::offset_of!(Self, mailbox), "mailbox"), (core::mem::offset_of!(Self, steal_probe), "steal_probe (excluded: a sibling writes it)"), @@ -1013,7 +1025,8 @@ impl CpuSched { /// Wake placement: keep the task local — that is where its cache lines /// are — unless it would queue behind the running task in its own band, RT /// behind RT or fair behind fair, in which case an idle sibling gets it: it - /// displaces nobody there. + /// displaces nobody there. A pass's opening holds such a task until the + /// pass's disposition and then places it with [`Self::place_settled`]. /// /// A fair task that stays and is what the fair band would now run in place /// of the running fair thread takes the CPU [`MIN_SLICE_NS`] after that @@ -1025,6 +1038,34 @@ impl CpuSched { task: ReadyTask, env: Env<'_, H, P>, now: Nanos, + ) { + self.place_behind(task, false, env, now); + } + + /// [`Self::place`] for a task the pass's opening held, once the + /// disposition has left the running task: a task its poster parked or + /// exited behind runs here, where it was posted, unless another task of + /// its band is already queued, an earlier held one included. So a burst + /// posted just before its poster parks keeps one task here and forwards + /// the rest, as it would have behind the poster. + fn place_settled, P: PreemptGuard>( + &mut self, + task: ReadyTask, + env: Env<'_, H, P>, + now: Nanos, + ) { + let queued_ahead = if task.is_rt() { self.rq.has_rt() } else { self.rq.fair_len() > 0 }; + self.place_behind(task, queued_ahead, env, now); + } + + /// [`Self::place`], where `queued_ahead` says a task of `task`'s band is + /// already queued here. + fn place_behind, P: PreemptGuard>( + &mut self, + task: ReadyTask, + queued_ahead: bool, + env: Env<'_, H, P>, + now: Nanos, ) { // **Ahead of the RT forward and of the kill check, because it outranks // both.** This is the one route by which a stopped task can be offered @@ -1049,7 +1090,11 @@ impl CpuSched { && self.rq.running_is_fair() && self.running.as_ref().is_some_and(|r| !r.shared().kill_pending()); let behind_rt = task.is_rt() && self.running.as_ref().is_some_and(|r| r.is_rt()); - if behind_fair || behind_rt { + if behind_fair || behind_rt || queued_ahead { + if self.holding { + self.held.push(task); + return; + } if let Some(dst) = self.forward_target(env) { self.hand_off(task, dst, env, now); return; @@ -1569,6 +1614,8 @@ impl<'c, 'e, H: Hw, P: PreemptGuard> SchedPass<'c, 'e, H, P, Undisposed> { cpu.rq.charge_running(ns); } } + assert!(!cpu.holding && cpu.held.is_empty(), "the previous pass on cpu{} was never disposed", cpu.id.0); + cpu.holding = true; cpu.drain(env, now); cpu.fire_deadlines(env, now); Self { @@ -1579,6 +1626,17 @@ impl<'c, 'e, H: Hw, P: PreemptGuard> SchedPass<'c, 'e, H, P, Undisposed> { } } + /// Place what the opening drain held, against the running task as the + /// disposition has left it. + fn settle(&mut self) { + self.cpu.holding = false; + let mut held = core::mem::take(&mut self.cpu.held); + for task in held.drain(..) { + self.cpu.place_settled(task, self.env, self.now); + } + self.cpu.held = held; + } + pub fn cpu(&self) -> &CpuSched { self.cpu } @@ -1589,12 +1647,16 @@ impl<'c, 'e, H: Hw, P: PreemptGuard> SchedPass<'c, 'e, H, P, Undisposed> { /// The current task keeps its claim on the CPU (subject to the preemption /// decision in `finish`). - pub fn dispose_none(self) -> SchedPass<'c, 'e, H, P, Disposed> { + pub fn dispose_none(mut self) -> SchedPass<'c, 'e, H, P, Disposed> { + self.settle(); self.dispose() } /// Voluntary yield: a fair thread gives up the rest of its request. - pub fn dispose_yield(self) -> SchedPass<'c, 'e, H, P, Disposed> { + pub fn dispose_yield(mut self) -> SchedPass<'c, 'e, H, P, Disposed> { + // Before the yield: the yielder stays runnable here, so what queues + // behind it is forwarded as it would be behind a running task. + self.settle(); if let Some(current) = self.cpu.running.take() { let mut task = current.preempt(self.cpu.id, self.now); if task.shared().kill_pending() { @@ -1613,7 +1675,7 @@ impl<'c, 'e, H: Hw, P: PreemptGuard> SchedPass<'c, 'e, H, P, Undisposed> { /// necessarily runs after the switch completes. The stack-reuse race is /// sequentially impossible, not locked away. pub fn dispose_block( - self, + mut self, ticket: CommittedTicket>, deadline: Option, ) -> SchedPass<'c, 'e, H, P, Disposed> { @@ -1643,6 +1705,7 @@ impl<'c, 'e, H: Hw, P: PreemptGuard> SchedPass<'c, 'e, H, P, Undisposed> { ); self.cpu .trace(self.env, self.now, TraceKind::ParkCommit { task: key, name }); + self.settle(); self.dispose() } @@ -1654,7 +1717,7 @@ impl<'c, 'e, H: Hw, P: PreemptGuard> SchedPass<'c, 'e, H, P, Undisposed> { /// only the caller knows it is standing somewhere it holds nothing, and /// this is where that knowledge is spent. Everything else a stopped task /// meets — a wake, an adopt, a retire — reads the bit in `place`. - pub fn dispose_stop(self) -> SchedPass<'c, 'e, H, P, Disposed> { + pub fn dispose_stop(mut self) -> SchedPass<'c, 'e, H, P, Disposed> { let current = self .cpu .running @@ -1664,12 +1727,13 @@ impl<'c, 'e, H: Hw, P: PreemptGuard> SchedPass<'c, 'e, H, P, Undisposed> { self.cpu.rq.leave_running(task.key(), task.standing_mut()); task.shared().mark_stop(); self.cpu.keep_stopped(task); + self.settle(); self.dispose() } /// The current task exits. Its record survives as the zombie until the /// next pass, which runs on another stack. - pub fn dispose_exit(self) -> SchedPass<'c, 'e, H, P, Disposed> { + pub fn dispose_exit(mut self) -> SchedPass<'c, 'e, H, P, Disposed> { let mut current = self .cpu .running @@ -1682,6 +1746,7 @@ impl<'c, 'e, H: Hw, P: PreemptGuard> SchedPass<'c, 'e, H, P, Undisposed> { self.cpu.dispose_dead(dead, self.env); self.cpu .trace(self.env, self.now, TraceKind::Retire { task: key, name }); + self.settle(); self.dispose() } @@ -4172,6 +4237,103 @@ mod tests { w.abandon(); } + /// The forward test's setup: `sleeper` parked on cpu0, a hog running + /// there, cpu1 asleep. Answers the hog and the sleeper. + fn hog_and_a_parked_sleeper(w: &mut World) -> (TaskKey, Arc>>) { + let (_, sleeper) = w.spawn(C0); + w.run_a_pass(C0); + w.park_running(C0); + let (hog, _) = w.spawn(C0); + w.run_a_pass(C0); + w.run_a_pass(C1); + assert!(w.handles.get(C1).doorbell().sleeping(), "cpu1 has published SLEEPING"); + (hog, sleeper) + } + + /// **A burst its poster parks behind keeps one task where it was posted**: + /// the freed CPU runs the first, and the rest, which would queue behind it, + /// are forwarded to idle siblings as they would have been behind the + /// poster, in the RT band when `rt`. + fn a_burst_its_poster_parks_behind(rt: bool) { + let spawn = |w: &mut World, cpu| if rt { w.spawn_rt(cpu) } else { w.spawn(cpu) }; + let mut w = World::new(2); + let mut sleepers = Vec::new(); + for _ in 0..2 { + let (_, sleeper) = spawn(&mut w, C0); + w.run_a_pass(C0); + w.park_running(C0); + sleepers.push(sleeper); + } + spawn(&mut w, C0); + w.run_a_pass(C0); + w.run_a_pass(C1); + assert!(w.handles.get(C1).doorbell().sleeping(), "cpu1 has published SLEEPING"); + + for sleeper in &sleepers { + wake(&w, C0, sleeper); + } + w.park_running(C0); + let states: Vec<_> = sleepers.iter().map(|s| s.state()).collect(); + w.abandon(); + assert!(states.contains(&TaskState::Running(C0)), "one runs where it was posted: {states:?}"); + assert!(states.contains(&TaskState::InTransit(C1)), "and the other went to the idle sibling: {states:?}"); + } + + #[test] + fn a_fair_burst_its_poster_parks_behind_keeps_one_here() { + a_burst_its_poster_parks_behind(false); + } + + #[test] + fn an_rt_burst_its_poster_parks_behind_keeps_one_here() { + a_burst_its_poster_parks_behind(true); + } + + /// **A wake whose poster parks in the pass that drains it runs where it was + /// posted**: the CPU is free once the poster has gone, so a forward would + /// buy an IPI and a cold cache for nothing. + #[test] + fn a_wake_its_poster_parks_behind_runs_where_it_was_posted() { + let mut w = World::new(2); + let (_, sleeper) = hog_and_a_parked_sleeper(&mut w); + wake(&w, C0, &sleeper); + w.park_running(C0); + assert_eq!(w.cpus[0].running().map(|t| t.key()), Some(sleeper.key()), "the wake ran here"); + assert!(!w.hw.state().kicks.contains(&C1), "and the sibling was not woken for it"); + w.abandon(); + } + + /// **The same for a poster that exits**: a thread started and joined, the + /// join's wake drained in the pass the thread ends in. + #[test] + fn a_wake_its_poster_exits_behind_runs_where_it_was_posted() { + let mut w = World::new(2); + let (_, sleeper) = hog_and_a_parked_sleeper(&mut w); + wake(&w, C0, &sleeper); + { + let (cpus, env) = w.split(); + let _ = SchedPass::begin(&mut cpus[0], env, Nanos(NOW.0 + 1)).dispose_exit().finish(); + } + assert_eq!(w.cpus[0].running().map(|t| t.key()), Some(sleeper.key()), "the wake ran here"); + assert!(!w.hw.state().kicks.contains(&C1), "and the sibling was not woken for it"); + w.abandon(); + } + + /// **A poster that yields stays runnable**, so what queues behind it is + /// forwarded as it is behind a poster that keeps the CPU. + #[test] + fn a_wake_its_poster_yields_behind_is_still_forwarded() { + let mut w = World::new(2); + let (_, sleeper) = hog_and_a_parked_sleeper(&mut w); + wake(&w, C0, &sleeper); + { + let (cpus, env) = w.split(); + let _ = SchedPass::begin(&mut cpus[0], env, Nanos(NOW.0 + 1)).dispose_yield().finish(); + } + assert_eq!(sleeper.state(), TaskState::InTransit(C1), "the wake was forwarded"); + w.abandon(); + } + /// **One forward per idle sibling**: wakes drained in one pass are handed /// to a sibling only while it has no kick coming, so a burst does not pile /// onto the first sleeper — the rest queue here. diff --git a/kernel/sim/src/scenarios.rs b/kernel/sim/src/scenarios.rs index cbb72691644..5809bca75fb 100644 --- a/kernel/sim/src/scenarios.rs +++ b/kernel/sim/src/scenarios.rs @@ -868,6 +868,7 @@ pub fn all() -> Vec { wakeup_storm(2, 16), busy_home(), sleep_spin_siblings(), + turn_taking_pair(), ] } @@ -1162,6 +1163,37 @@ pub fn busy_home() -> Scenario { /// How many times [`busy_home`]'s sleeper wakes in one run. pub const BUSY_HOME_ROUNDS: usize = 8; +/// **Two threads of one process taking turns on one CPU beside an idle one**: +/// each posts the other's wake and parks next, so neither ever queues behind +/// the other. +/// +/// Both start on cpu0 and the balance path is off, so a migration is a +/// forward: one that hands a wake to cpu1 while its poster is parking in the +/// pass that drained it. +pub fn turn_taking_pair() -> Scenario { + let turn = MS / 10; + let wake = |queue| Op::Wake { queue, all: false, boost: None }; + let block = |queue| Op::Block { queue, deadline: None }; + scenario( + "turn_taking_pair", + 2, + vec![queue(WaitClass::Io), queue(WaitClass::Io)], + vec![process( + "pair", + vec![0, 1], + vec![ + Script::looping(vec![Op::Run(turn), wake(1), block(0)], TURN_TAKING_ROUNDS), + Script::looping(vec![block(1), Op::Run(turn), wake(0)], TURN_TAKING_ROUNDS), + ], + )], + ) + .with_placement(PlacementShape::AllOn(0)) + .with_balance(Balance::None) +} + +/// How many turns each of [`turn_taking_pair`]'s threads takes. +pub const TURN_TAKING_ROUNDS: usize = 8; + /// **Two threads of one process on one CPU: one that sleeps a microsecond /// between millisecond bursts, and one that never sleeps.** /// diff --git a/kernel/sim/tests/policy.rs b/kernel/sim/tests/policy.rs index 85d1f435b82..16a45db80d7 100644 --- a/kernel/sim/tests/policy.rs +++ b/kernel/sim/tests/policy.rs @@ -470,6 +470,21 @@ fn a_wake_to_a_busy_cpu_beside_an_idle_one_runs_within_two_hops() { ); } +/// **A pair that takes turns on one CPU stays there**: a wake whose poster +/// parks in the pass that drains it runs where it was posted, so an idle CPU +/// beside the pair is never handed a turn. +/// +/// `scenarios::turn_taking_pair` starts both threads on cpu0 with the balance +/// path off, so the forward is the only thing that could move either. +#[test] +fn a_turn_taking_pair_is_never_forwarded_off_its_cpu() { + const SEEDS: u64 = 40; + let scenario = scenarios::turn_taking_pair(); + sweep(&scenario, SEEDS, |outcome| { + assert_eq!(outcome.migrations, 0, "a turn was handed to the idle CPU: {}", outcome.report()); + }); +} + /// **A wake due ahead of the running task waits out [`MIN_SLICE_NS`] and a /// hop, not a request** — on one CPU, against one rival process, every wake /// but one per run.