From 1233d1881e415beb7968680dc2dcdabb898823dc Mon Sep 17 00:00:00 2001 From: japabu Date: Wed, 28 Oct 2026 19:54:21 +0100 Subject: [PATCH 1/5] sched: a wake its poster parks or exits behind runs where it was posted SchedPass::begin drains the mailbox before the pass knows its disposition, so a fair wake posted on the poster's own CPU was judged behind the running task in the very pass that parks or ends that task, and was forwarded to an idle CPU: an IPI and a cold cache where a local switch a moment later was free. osbench's same-CPU pingpong rows held their placement in none of 53 T14 boots, which is what this forward predicts; a thread started and then joined meets it at the join. The opening drain now holds what it finds behind the running task, and each disposition places it against the CPU as it leaves it: after a park, a stop or an exit it runs here; before a yield or with the task kept it is forwarded as before. Only the opening drain holds, so the CPU's own later drains are unchanged. WakeCause gains `parker`, the CPU whose task posted the wake and parks next in the same call. The home CPU hands such a fair wake to that CPU inside one LLC, where the poster's park runs it and the pair's later wakes are local. No caller sets it yet: it is the placement input the whole-message IPC's combined call drives in its phase 2. The simulator's turn_taking_pair holds the first half across seeds: both threads start on cpu0 beside an idle cpu1 with balancing off, and no turn is forwarded. Co-Authored-By: Claude Opus 5.5 --- ...-parks-next-is-forwarded-to-an-idle-cpu.md | 8 +- kernel/pure/sched/cpu.rs | 192 +++++++++++++++++- kernel/pure/sched/task.rs | 10 + kernel/sim/src/scenarios.rs | 32 +++ kernel/sim/tests/policy.rs | 15 ++ 5 files changed, 248 insertions(+), 9 deletions(-) diff --git a/issues/a-wake-whose-waker-parks-next-is-forwarded-to-an-idle-cpu.md b/issues/a-wake-whose-waker-parks-next-is-forwarded-to-an-idle-cpu.md index 8ec35dd46d0..2647f57c314 100644 --- a/issues/a-wake-whose-waker-parks-next-is-forwarded-to-an-idle-cpu.md +++ b/issues/a-wake-whose-waker-parks-next-is-forwarded-to-an-idle-cpu.md @@ -1,5 +1,5 @@ --- -status: open +status: assigned kind: finding opened: 2026-10-12 --- @@ -23,5 +23,11 @@ the AMD bench, base to head, one boot each, read from the second judgement's Owned by stage S3 of the scheduler design (#872), whose 1:1 pair rule decides where a pair's wakes go. +Branch `sched-place` decides it: the opening drain of a pass holds a wake +found behind the running task until the disposition, so a poster that parks +or exits leaves it to run here, and one that yields or keeps the CPU still +forwards it (`CpuSched::place`; the simulator's `turn_taking_pair`). The +metal price is what remains. + **Exit**: S3 decides this case, with a simulator scenario of a wake whose waker parks next and a metal row that prices the forward against the local switch. diff --git a/kernel/pure/sched/cpu.rs b/kernel/pure/sched/cpu.rs index 9273993293b..fa298b44e58 100644 --- a/kernel/pure/sched/cpu.rs +++ b/kernel/pure/sched/cpu.rs @@ -295,6 +295,14 @@ pub struct CpuSched { /// In no fair band, as [`Self::dying`]'s tasks are not; excluded from the /// published load, because nothing queues behind work that will never run. stopped: VecDeque>, + /// Wakes and adopts the opening drain found behind the running task, + /// held for [`SchedPass`]'s disposition: whether that task leaves the CPU + /// in this pass decides whether they run here or are forwarded. Empty + /// outside a pass. + held: Vec>, + /// Set by [`SchedPass::begin`] and cleared by the disposition, so only + /// the opening drain holds. + holding: bool, /// The task that exited on this CPU, freed by the NEXT pass — a pass /// cannot free the stack it is running on. zombie: Option>, @@ -393,6 +401,8 @@ impl CpuSched { parked: Parked::new(), dying: VecDeque::new(), stopped: VecDeque::new(), + held: Vec::new(), + holding: false, zombie: None, mailbox, steal_probe: MailboxNode::new(), @@ -1013,7 +1023,9 @@ impl CpuSched { /// Wake placement: keep the task local — that is where its cache lines /// are — unless it would queue behind the running task in its own band, RT /// behind RT or fair behind fair, in which case an idle sibling gets it: it - /// displaces nobody there. + /// displaces nobody there. In a pass's opening drain the running task is + /// the one the disposition leaves, so a wake its poster parks or exits + /// behind runs here, where it was posted. /// /// A fair task that stays and is what the fair band would now run in place /// of the running fair thread takes the CPU [`MIN_SLICE_NS`] after that @@ -1050,6 +1062,10 @@ impl CpuSched { && self.running.as_ref().is_some_and(|r| !r.shared().kill_pending()); let behind_rt = task.is_rt() && self.running.as_ref().is_some_and(|r| r.is_rt()); if behind_fair || behind_rt { + if self.holding { + self.held.push(task); + return; + } if let Some(dst) = self.forward_target(env) { self.hand_off(task, dst, env, now); return; @@ -1102,7 +1118,28 @@ impl CpuSched { }; let task = entry.task.wake(self.id, cause, entry.class, now); self.trace(env, now, TraceKind::Wake { task: key, name: task.ext().name() }); - self.place(task, env, now); + match cause.parker.filter(|&parker| self.joins_its_parker(&task, parker, env)) { + Some(parker) => self.hand_off(task, parker, env, now), + None => self.place(task, env, now), + } + } + + /// Whether a wake whose poster parks next on `parker` goes there: the CPU + /// it frees runs the woken task with no IPI behind it, and every wake of + /// the pair after this one is local. Inside one LLC, where the move costs + /// a refill from the shared cache; a fair task, so the RT band's own + /// placement is unchanged; and never a stopped or killed task, which + /// [`Self::place`] keeps here. + fn joins_its_parker, P: PreemptGuard>( + &self, + task: &ReadyTask, + parker: CpuId, + env: Env<'_, H, P>, + ) -> bool { + env.cpus.topology().level(self.id, parker).is_some_and(|shared| shared <= Level::Llc) + && !task.is_rt() + && !task.shared().stop_pending() + && !task.shared().kill_pending() } fn handle_adopt, P: PreemptGuard>( @@ -1569,6 +1606,7 @@ impl<'c, 'e, H: Hw, P: PreemptGuard> SchedPass<'c, 'e, H, P, Undisposed> { cpu.rq.charge_running(ns); } } + cpu.holding = true; cpu.drain(env, now); cpu.fire_deadlines(env, now); Self { @@ -1579,6 +1617,15 @@ impl<'c, 'e, H: Hw, P: PreemptGuard> SchedPass<'c, 'e, H, P, Undisposed> { } } + /// Place what the opening drain held, against the running task as the + /// disposition has left it. + fn settle(&mut self) { + self.cpu.holding = false; + for task in core::mem::take(&mut self.cpu.held) { + self.cpu.place(task, self.env, self.now); + } + } + pub fn cpu(&self) -> &CpuSched { self.cpu } @@ -1589,12 +1636,16 @@ impl<'c, 'e, H: Hw, P: PreemptGuard> SchedPass<'c, 'e, H, P, Undisposed> { /// The current task keeps its claim on the CPU (subject to the preemption /// decision in `finish`). - pub fn dispose_none(self) -> SchedPass<'c, 'e, H, P, Disposed> { + pub fn dispose_none(mut self) -> SchedPass<'c, 'e, H, P, Disposed> { + self.settle(); self.dispose() } /// Voluntary yield: a fair thread gives up the rest of its request. - pub fn dispose_yield(self) -> SchedPass<'c, 'e, H, P, Disposed> { + pub fn dispose_yield(mut self) -> SchedPass<'c, 'e, H, P, Disposed> { + // Before the yield: the yielder stays runnable here, so what queues + // behind it is forwarded as it would be behind a running task. + self.settle(); if let Some(current) = self.cpu.running.take() { let mut task = current.preempt(self.cpu.id, self.now); if task.shared().kill_pending() { @@ -1613,7 +1664,7 @@ impl<'c, 'e, H: Hw, P: PreemptGuard> SchedPass<'c, 'e, H, P, Undisposed> { /// necessarily runs after the switch completes. The stack-reuse race is /// sequentially impossible, not locked away. pub fn dispose_block( - self, + mut self, ticket: CommittedTicket>, deadline: Option, ) -> SchedPass<'c, 'e, H, P, Disposed> { @@ -1643,6 +1694,7 @@ impl<'c, 'e, H: Hw, P: PreemptGuard> SchedPass<'c, 'e, H, P, Undisposed> { ); self.cpu .trace(self.env, self.now, TraceKind::ParkCommit { task: key, name }); + self.settle(); self.dispose() } @@ -1654,7 +1706,7 @@ impl<'c, 'e, H: Hw, P: PreemptGuard> SchedPass<'c, 'e, H, P, Undisposed> { /// only the caller knows it is standing somewhere it holds nothing, and /// this is where that knowledge is spent. Everything else a stopped task /// meets — a wake, an adopt, a retire — reads the bit in `place`. - pub fn dispose_stop(self) -> SchedPass<'c, 'e, H, P, Disposed> { + pub fn dispose_stop(mut self) -> SchedPass<'c, 'e, H, P, Disposed> { let current = self .cpu .running @@ -1664,12 +1716,13 @@ impl<'c, 'e, H: Hw, P: PreemptGuard> SchedPass<'c, 'e, H, P, Undisposed> { self.cpu.rq.leave_running(task.key(), task.standing_mut()); task.shared().mark_stop(); self.cpu.keep_stopped(task); + self.settle(); self.dispose() } /// The current task exits. Its record survives as the zombie until the /// next pass, which runs on another stack. - pub fn dispose_exit(self) -> SchedPass<'c, 'e, H, P, Disposed> { + pub fn dispose_exit(mut self) -> SchedPass<'c, 'e, H, P, Disposed> { let mut current = self .cpu .running @@ -1682,6 +1735,7 @@ impl<'c, 'e, H: Hw, P: PreemptGuard> SchedPass<'c, 'e, H, P, Undisposed> { self.cpu.dispose_dead(dead, self.env); self.cpu .trace(self.env, self.now, TraceKind::Retire { task: key, name }); + self.settle(); self.dispose() } @@ -2934,7 +2988,10 @@ mod tests { shared: &Arc>>, reason: WakeReason, ) { - let cause = WakeCause::new(reason); + self.post_claimed_cause(cpu, shared, WakeCause::new(reason)); + } + + fn post_claimed_cause(&self, cpu: CpuId, shared: &Arc>>, cause: WakeCause) { let slot = shared .wake_node() .claim() @@ -4172,6 +4229,125 @@ mod tests { w.abandon(); } + /// The forward test's setup: `sleeper` parked on cpu0, a hog running + /// there, cpu1 asleep. Answers the hog and the sleeper. + fn hog_and_a_parked_sleeper(w: &mut World) -> (TaskKey, Arc>>) { + let (_, sleeper) = w.spawn(C0); + w.run_a_pass(C0); + w.park_running(C0); + let (hog, _) = w.spawn(C0); + w.run_a_pass(C0); + w.run_a_pass(C1); + assert!(w.handles.get(C1).doorbell().sleeping(), "cpu1 has published SLEEPING"); + (hog, sleeper) + } + + /// **A wake whose poster parks in the pass that drains it runs where it was + /// posted**: the CPU is free once the poster has gone, so a forward would + /// buy an IPI and a cold cache for nothing. + #[test] + fn a_wake_its_poster_parks_behind_runs_where_it_was_posted() { + let mut w = World::new(2); + let (_, sleeper) = hog_and_a_parked_sleeper(&mut w); + wake(&w, C0, &sleeper); + w.park_running(C0); + assert_eq!(w.cpus[0].running().map(|t| t.key()), Some(sleeper.key()), "the wake ran here"); + assert!(!w.hw.state().kicks.contains(&C1), "and the sibling was not woken for it"); + w.abandon(); + } + + /// **The same for a poster that exits**: a thread started and joined, the + /// join's wake drained in the pass the thread ends in. + #[test] + fn a_wake_its_poster_exits_behind_runs_where_it_was_posted() { + let mut w = World::new(2); + let (_, sleeper) = hog_and_a_parked_sleeper(&mut w); + wake(&w, C0, &sleeper); + { + let (cpus, env) = w.split(); + let _ = SchedPass::begin(&mut cpus[0], env, Nanos(NOW.0 + 1)).dispose_exit().finish(); + } + assert_eq!(w.cpus[0].running().map(|t| t.key()), Some(sleeper.key()), "the wake ran here"); + assert!(!w.hw.state().kicks.contains(&C1), "and the sibling was not woken for it"); + w.abandon(); + } + + /// **A poster that yields stays runnable**, so what queues behind it is + /// forwarded as it is behind a poster that keeps the CPU. + #[test] + fn a_wake_its_poster_yields_behind_is_still_forwarded() { + let mut w = World::new(2); + let (_, sleeper) = hog_and_a_parked_sleeper(&mut w); + wake(&w, C0, &sleeper); + { + let (cpus, env) = w.split(); + let _ = SchedPass::begin(&mut cpus[0], env, Nanos(NOW.0 + 1)).dispose_yield().finish(); + } + assert_eq!(sleeper.state(), TaskState::InTransit(C1), "the wake was forwarded"); + w.abandon(); + } + + /// A task parked on `home` and a hog running on `parker`, which is about + /// to post the parked task's wake and park. Answers the parked task. + fn parked_on_and_a_parker(w: &mut World, home: CpuId, parker: CpuId) -> Arc>> { + let (_, sleeper) = w.spawn(home); + w.run_a_pass(home); + w.park_running(home); + let _ = w.spawn(parker); + w.run_a_pass(parker); + sleeper + } + + fn wake_from_a_parker(w: &World, home: CpuId, parker: CpuId, shared: &Arc>>) { + assert_eq!(shared.claim_wake(), Claim::Parked(home), "the task is parked there"); + w.post_claimed_cause(home, shared, WakeCause::new(WakeReason::Woken).parks_next(parker)); + } + + /// **A wake whose poster parks next goes to the poster's CPU inside its + /// LLC**, and runs there in the poster's park: the pair's next wakes are + /// local, with no IPI behind them. + #[test] + fn a_wake_whose_poster_parks_next_runs_on_the_posters_cpu() { + let (parker, home) = (CpuId(0), CpuId(2)); + let mut w = World::on(two_llcs()); + let sleeper = parked_on_and_a_parker(&mut w, home, parker); + wake_from_a_parker(&w, home, parker, &sleeper); + w.run_a_pass_at(home, Nanos(NOW.0 + 1)); + assert_eq!(sleeper.state(), TaskState::InTransit(parker), "cpu2 shares cpu0's LLC: handed to the parker"); + w.park_running(parker); + assert_eq!(w.cpus[0].running().map(|t| t.key()), Some(sleeper.key()), "and run in the parker's park"); + w.abandon(); + } + + /// **Across LLCs the wake stays home**: the move would refill the task + /// from memory rather than a shared cache. + #[test] + fn a_wake_whose_poster_parks_next_in_another_llc_stays_home() { + let (parker, home) = (CpuId(0), CpuId(1)); + let mut w = World::on(two_llcs()); + let sleeper = parked_on_and_a_parker(&mut w, home, parker); + wake_from_a_parker(&w, home, parker, &sleeper); + w.run_a_pass_at(home, Nanos(NOW.0 + 1)); + assert_eq!(w.cpus[1].running().map(|t| t.key()), Some(sleeper.key()), "cpu1 is in the other LLC: run at home"); + w.abandon(); + } + + /// **A real-time wake keeps the RT band's placement**, whoever posted it. + #[test] + fn a_real_time_wake_whose_poster_parks_next_stays_home() { + let (parker, home) = (CpuId(0), CpuId(2)); + let mut w = World::on(two_llcs()); + let (_, sleeper) = w.spawn_rt(home); + w.run_a_pass(home); + w.park_running(home); + let _ = w.spawn(parker); + w.run_a_pass(parker); + wake_from_a_parker(&w, home, parker, &sleeper); + w.run_a_pass_at(home, Nanos(NOW.0 + 1)); + assert_eq!(w.cpus[2].running().map(|t| t.key()), Some(sleeper.key()), "the RT task runs at home"); + w.abandon(); + } + /// **One forward per idle sibling**: wakes drained in one pass are handed /// to a sibling only while it has no kick coming, so a burst does not pile /// onto the first sleeper — the rest queue here. diff --git a/kernel/pure/sched/task.rs b/kernel/pure/sched/task.rs index 862e17ff634..72610b60e47 100644 --- a/kernel/pure/sched/task.rs +++ b/kernel/pure/sched/task.rs @@ -58,6 +58,9 @@ pub struct Standing { pub struct WakeCause { pub reason: WakeReason, pub boost: Option, + /// The CPU whose running task posted this wake and parks next, inside the + /// same call: the CPU it frees is where the woken task runs. + pub parker: Option, } impl WakeCause { @@ -65,6 +68,7 @@ impl WakeCause { Self { reason, boost: None, + parker: None, } } @@ -72,9 +76,15 @@ impl WakeCause { Self { reason, boost: Some(BoostWindow { until }), + parker: None, } } + /// This wake, posted by the task running on `cpu`, which parks next. + pub fn parks_next(self, cpu: CpuId) -> Self { + Self { parker: Some(cpu), ..self } + } + /// RT and boost wakes must preempt the target promptly; ordinary wakes /// ride the target's next safe point. pub fn urgency(&self) -> crate::sched::mailbox::Urgency { diff --git a/kernel/sim/src/scenarios.rs b/kernel/sim/src/scenarios.rs index cbb72691644..5809bca75fb 100644 --- a/kernel/sim/src/scenarios.rs +++ b/kernel/sim/src/scenarios.rs @@ -868,6 +868,7 @@ pub fn all() -> Vec { wakeup_storm(2, 16), busy_home(), sleep_spin_siblings(), + turn_taking_pair(), ] } @@ -1162,6 +1163,37 @@ pub fn busy_home() -> Scenario { /// How many times [`busy_home`]'s sleeper wakes in one run. pub const BUSY_HOME_ROUNDS: usize = 8; +/// **Two threads of one process taking turns on one CPU beside an idle one**: +/// each posts the other's wake and parks next, so neither ever queues behind +/// the other. +/// +/// Both start on cpu0 and the balance path is off, so a migration is a +/// forward: one that hands a wake to cpu1 while its poster is parking in the +/// pass that drained it. +pub fn turn_taking_pair() -> Scenario { + let turn = MS / 10; + let wake = |queue| Op::Wake { queue, all: false, boost: None }; + let block = |queue| Op::Block { queue, deadline: None }; + scenario( + "turn_taking_pair", + 2, + vec![queue(WaitClass::Io), queue(WaitClass::Io)], + vec![process( + "pair", + vec![0, 1], + vec![ + Script::looping(vec![Op::Run(turn), wake(1), block(0)], TURN_TAKING_ROUNDS), + Script::looping(vec![block(1), Op::Run(turn), wake(0)], TURN_TAKING_ROUNDS), + ], + )], + ) + .with_placement(PlacementShape::AllOn(0)) + .with_balance(Balance::None) +} + +/// How many turns each of [`turn_taking_pair`]'s threads takes. +pub const TURN_TAKING_ROUNDS: usize = 8; + /// **Two threads of one process on one CPU: one that sleeps a microsecond /// between millisecond bursts, and one that never sleeps.** /// diff --git a/kernel/sim/tests/policy.rs b/kernel/sim/tests/policy.rs index 85d1f435b82..16a45db80d7 100644 --- a/kernel/sim/tests/policy.rs +++ b/kernel/sim/tests/policy.rs @@ -470,6 +470,21 @@ fn a_wake_to_a_busy_cpu_beside_an_idle_one_runs_within_two_hops() { ); } +/// **A pair that takes turns on one CPU stays there**: a wake whose poster +/// parks in the pass that drains it runs where it was posted, so an idle CPU +/// beside the pair is never handed a turn. +/// +/// `scenarios::turn_taking_pair` starts both threads on cpu0 with the balance +/// path off, so the forward is the only thing that could move either. +#[test] +fn a_turn_taking_pair_is_never_forwarded_off_its_cpu() { + const SEEDS: u64 = 40; + let scenario = scenarios::turn_taking_pair(); + sweep(&scenario, SEEDS, |outcome| { + assert_eq!(outcome.migrations, 0, "a turn was handed to the idle CPU: {}", outcome.report()); + }); +} + /// **A wake due ahead of the running task waits out [`MIN_SLICE_NS`] and a /// hop, not a request** — on one CPU, against one rival process, every wake /// but one per run. From ff6bb90c4afb2ee3bf9cec5b0bdaccc3502f8752 Mon Sep 17 00:00:00 2001 From: japabu Date: Thu, 29 Oct 2026 00:22:39 +0100 Subject: [PATCH 2/5] sched: the tripwire names the held wakes and the holding flag b24b37a56 added CpuSched::held and ::holding without entries in tripwire_field, so a shadow mismatch inside either was reported as the nearest field below it. Co-Authored-By: Claude Opus 5.5 --- kernel/pure/sched/cpu.rs | 2 ++ 1 file changed, 2 insertions(+) diff --git a/kernel/pure/sched/cpu.rs b/kernel/pure/sched/cpu.rs index fa298b44e58..8d97303e8dd 100644 --- a/kernel/pure/sched/cpu.rs +++ b/kernel/pure/sched/cpu.rs @@ -625,6 +625,8 @@ impl CpuSched { (core::mem::offset_of!(Self, parked), "parked (the park map)"), (core::mem::offset_of!(Self, dying), "dying"), (core::mem::offset_of!(Self, stopped), "stopped"), + (core::mem::offset_of!(Self, held), "held"), + (core::mem::offset_of!(Self, holding), "holding"), (core::mem::offset_of!(Self, zombie), "zombie"), (core::mem::offset_of!(Self, mailbox), "mailbox"), (core::mem::offset_of!(Self, steal_probe), "steal_probe (excluded: a sibling writes it)"), From e6403828f2f3dfc395871943b68313c3fe96b73d Mon Sep 17 00:00:00 2001 From: japabu Date: Sun, 1 Nov 2026 02:48:24 +0100 Subject: [PATCH 3/5] sched: drop the parks-next hand-off WakeCause::parks_next(cpu) named the CPU of a poster that parks next, and the home CPU handed such a fair wake there inside one LLC. No caller ever set it. The IPC call meant to set it is recommended for dropping, and the waker-LLC wake affinity that follows this PR carries the waker on every wake instead, so the field, the hand-off and their three tests go. Step 1 stays: a wake the opening drain finds behind a poster that parks, stops or exits still runs where it was posted. Co-Authored-By: Claude Opus 5.5 --- kernel/pure/sched/cpu.rs | 89 +-------------------------------------- kernel/pure/sched/task.rs | 10 ----- 2 files changed, 2 insertions(+), 97 deletions(-) diff --git a/kernel/pure/sched/cpu.rs b/kernel/pure/sched/cpu.rs index 8d97303e8dd..b8aee2c78e9 100644 --- a/kernel/pure/sched/cpu.rs +++ b/kernel/pure/sched/cpu.rs @@ -1120,28 +1120,7 @@ impl CpuSched { }; let task = entry.task.wake(self.id, cause, entry.class, now); self.trace(env, now, TraceKind::Wake { task: key, name: task.ext().name() }); - match cause.parker.filter(|&parker| self.joins_its_parker(&task, parker, env)) { - Some(parker) => self.hand_off(task, parker, env, now), - None => self.place(task, env, now), - } - } - - /// Whether a wake whose poster parks next on `parker` goes there: the CPU - /// it frees runs the woken task with no IPI behind it, and every wake of - /// the pair after this one is local. Inside one LLC, where the move costs - /// a refill from the shared cache; a fair task, so the RT band's own - /// placement is unchanged; and never a stopped or killed task, which - /// [`Self::place`] keeps here. - fn joins_its_parker, P: PreemptGuard>( - &self, - task: &ReadyTask, - parker: CpuId, - env: Env<'_, H, P>, - ) -> bool { - env.cpus.topology().level(self.id, parker).is_some_and(|shared| shared <= Level::Llc) - && !task.is_rt() - && !task.shared().stop_pending() - && !task.shared().kill_pending() + self.place(task, env, now); } fn handle_adopt, P: PreemptGuard>( @@ -2990,10 +2969,7 @@ mod tests { shared: &Arc>>, reason: WakeReason, ) { - self.post_claimed_cause(cpu, shared, WakeCause::new(reason)); - } - - fn post_claimed_cause(&self, cpu: CpuId, shared: &Arc>>, cause: WakeCause) { + let cause = WakeCause::new(reason); let slot = shared .wake_node() .claim() @@ -4289,67 +4265,6 @@ mod tests { w.abandon(); } - /// A task parked on `home` and a hog running on `parker`, which is about - /// to post the parked task's wake and park. Answers the parked task. - fn parked_on_and_a_parker(w: &mut World, home: CpuId, parker: CpuId) -> Arc>> { - let (_, sleeper) = w.spawn(home); - w.run_a_pass(home); - w.park_running(home); - let _ = w.spawn(parker); - w.run_a_pass(parker); - sleeper - } - - fn wake_from_a_parker(w: &World, home: CpuId, parker: CpuId, shared: &Arc>>) { - assert_eq!(shared.claim_wake(), Claim::Parked(home), "the task is parked there"); - w.post_claimed_cause(home, shared, WakeCause::new(WakeReason::Woken).parks_next(parker)); - } - - /// **A wake whose poster parks next goes to the poster's CPU inside its - /// LLC**, and runs there in the poster's park: the pair's next wakes are - /// local, with no IPI behind them. - #[test] - fn a_wake_whose_poster_parks_next_runs_on_the_posters_cpu() { - let (parker, home) = (CpuId(0), CpuId(2)); - let mut w = World::on(two_llcs()); - let sleeper = parked_on_and_a_parker(&mut w, home, parker); - wake_from_a_parker(&w, home, parker, &sleeper); - w.run_a_pass_at(home, Nanos(NOW.0 + 1)); - assert_eq!(sleeper.state(), TaskState::InTransit(parker), "cpu2 shares cpu0's LLC: handed to the parker"); - w.park_running(parker); - assert_eq!(w.cpus[0].running().map(|t| t.key()), Some(sleeper.key()), "and run in the parker's park"); - w.abandon(); - } - - /// **Across LLCs the wake stays home**: the move would refill the task - /// from memory rather than a shared cache. - #[test] - fn a_wake_whose_poster_parks_next_in_another_llc_stays_home() { - let (parker, home) = (CpuId(0), CpuId(1)); - let mut w = World::on(two_llcs()); - let sleeper = parked_on_and_a_parker(&mut w, home, parker); - wake_from_a_parker(&w, home, parker, &sleeper); - w.run_a_pass_at(home, Nanos(NOW.0 + 1)); - assert_eq!(w.cpus[1].running().map(|t| t.key()), Some(sleeper.key()), "cpu1 is in the other LLC: run at home"); - w.abandon(); - } - - /// **A real-time wake keeps the RT band's placement**, whoever posted it. - #[test] - fn a_real_time_wake_whose_poster_parks_next_stays_home() { - let (parker, home) = (CpuId(0), CpuId(2)); - let mut w = World::on(two_llcs()); - let (_, sleeper) = w.spawn_rt(home); - w.run_a_pass(home); - w.park_running(home); - let _ = w.spawn(parker); - w.run_a_pass(parker); - wake_from_a_parker(&w, home, parker, &sleeper); - w.run_a_pass_at(home, Nanos(NOW.0 + 1)); - assert_eq!(w.cpus[2].running().map(|t| t.key()), Some(sleeper.key()), "the RT task runs at home"); - w.abandon(); - } - /// **One forward per idle sibling**: wakes drained in one pass are handed /// to a sibling only while it has no kick coming, so a burst does not pile /// onto the first sleeper — the rest queue here. diff --git a/kernel/pure/sched/task.rs b/kernel/pure/sched/task.rs index 72610b60e47..862e17ff634 100644 --- a/kernel/pure/sched/task.rs +++ b/kernel/pure/sched/task.rs @@ -58,9 +58,6 @@ pub struct Standing { pub struct WakeCause { pub reason: WakeReason, pub boost: Option, - /// The CPU whose running task posted this wake and parks next, inside the - /// same call: the CPU it frees is where the woken task runs. - pub parker: Option, } impl WakeCause { @@ -68,7 +65,6 @@ impl WakeCause { Self { reason, boost: None, - parker: None, } } @@ -76,15 +72,9 @@ impl WakeCause { Self { reason, boost: Some(BoostWindow { until }), - parker: None, } } - /// This wake, posted by the task running on `cpu`, which parks next. - pub fn parks_next(self, cpu: CpuId) -> Self { - Self { parker: Some(cpu), ..self } - } - /// RT and boost wakes must preempt the target promptly; ordinary wakes /// ride the target's next safe point. pub fn urgency(&self) -> crate::sched::mailbox::Urgency { From 5802d4d02b2f3915c3fadc70ffe63c6501036a54 Mon Sep 17 00:00:00 2001 From: japabu Date: Sun, 1 Nov 2026 10:01:58 +0100 Subject: [PATCH 4/5] review r1 of #1111: a held burst keeps one task, and the hold keeps its buffer The pass's opening held every wake it found behind the running task, and the disposition placed each against the CPU it left. When the poster parked, every held wake saw an idle CPU and stayed, so a burst its poster parked behind queued whole on the poster's CPU. Before the hold each had gone to its own idle sibling. For RT wakes nothing could move the second, because only fair tasks are stolen: the soundserver waking two boosted clients and parking ran them in turn on one CPU. place_settled now treats a held task as behind when its band already has a queued task, an earlier held one included. The first held task of each band takes the freed CPU and the rest are forwarded as before. Two host tests, fair and RT, hold it; dropping the queued-ahead check reds both and nothing else. settle took the held vector by value, so every hold allocated and freed under the kernel's global heap lock, twice per request/reply round trip. It now drains the vector and puts the buffer back. begin asserts that the previous pass was disposed, since an undisposed pass would now lose held tasks rather than skip a pick. The issue record said the branch decided the case and stayed assigned. It is renamed for what remains: no metal row prices the hold, and the T14 rows that read 1-3% slower in the first A/B, whose head also carried affinity and the spawn tie-break, are not attributed. Co-Authored-By: Claude Opus 5.5 --- ...sts-toyos-up-to-twice-linuxs-on-the-t14.md | 2 +- ...r-its-posters-park-is-unpriced-on-metal.md | 28 ++++++ ...-parks-next-is-forwarded-to-an-idle-cpu.md | 33 ------- ...s-own-cpu-waits-for-that-cpus-next-pass.md | 4 +- kernel/pure/sched/cpu.rs | 91 ++++++++++++++++--- 5 files changed, 111 insertions(+), 47 deletions(-) create mode 100644 issues/a-wake-held-for-its-posters-park-is-unpriced-on-metal.md delete mode 100644 issues/a-wake-whose-waker-parks-next-is-forwarded-to-an-idle-cpu.md diff --git a/issues/a-two-thread-ping-pong-costs-toyos-up-to-twice-linuxs-on-the-t14.md b/issues/a-two-thread-ping-pong-costs-toyos-up-to-twice-linuxs-on-the-t14.md index dc3a01c71b1..d83a704263b 100644 --- a/issues/a-two-thread-ping-pong-costs-toyos-up-to-twice-linuxs-on-the-t14.md +++ b/issues/a-two-thread-ping-pong-costs-toyos-up-to-twice-linuxs-on-the-t14.md @@ -36,7 +36,7 @@ on its head than its base (`issues/a-64-kib-pipe-write-is-two-percent-below-the-two-mebibyte-ring.md`), so the `pipe` rows' current state against Linux is owed a new reading. The scheduler's pair-wake records bear on both kinds: -`issues/a-wake-whose-waker-parks-next-is-forwarded-to-an-idle-cpu.md` and +`issues/a-wake-held-for-its-posters-park-is-unpriced-on-metal.md` and `issues/an-ordinary-wake-posted-on-its-own-cpu-waits-for-that-cpus-next-pass.md`. **Exit**: `pingpong.*` within Linux's on both benches in a labelled diff --git a/issues/a-wake-held-for-its-posters-park-is-unpriced-on-metal.md b/issues/a-wake-held-for-its-posters-park-is-unpriced-on-metal.md new file mode 100644 index 00000000000..74ac1b3d51c --- /dev/null +++ b/issues/a-wake-held-for-its-posters-park-is-unpriced-on-metal.md @@ -0,0 +1,28 @@ +--- +status: open +kind: defect +opened: 2026-10-12 +--- + +# A wake held for its poster's park is unpriced on metal + +A pass's opening holds every task it would forward for queueing behind the +running task, and places them after the pass's disposition +(`CpuSched::place_settled`, `kernel/pure/sched/cpu.rs`). A poster that parks, +stops or exits leaves the CPU to the first held task of each band; one that +yields or keeps the CPU forwards them to idle siblings as before, and so do +the rest of a burst. The simulator's `turn_taking_pair` holds the case: no +turn is forwarded off its CPU. + +What it costs or saves on metal is unread. No row prices the forward against +the local switch. The first A/B that carried the rule, on the T14 with base +`0c7e62ca7` against a head that also carried CPU affinity and the spawn +tie-break, read twelve disk and memfs rows 1–3% slower (fstat, seek, stat, +pread_4k, read_4k_at, write_4k_at, tree.100), four boots a side with one image +each, so a layout difference between the images is not ruled out +([evidence](https://github.com/ToyOSOrg/evidence/tree/6cd2768011f0e6f36c4e2c1ce76f986fc6c2265b/pr-1116/r2)). +Nothing attributes them to this rule. + +**Exit**: a T14 and an AMD A/B of the rule alone against its base reads those +rows within their spread or names the cost, and a metal row prices a wake its +poster parks behind against the forward. diff --git a/issues/a-wake-whose-waker-parks-next-is-forwarded-to-an-idle-cpu.md b/issues/a-wake-whose-waker-parks-next-is-forwarded-to-an-idle-cpu.md deleted file mode 100644 index 2647f57c314..00000000000 --- a/issues/a-wake-whose-waker-parks-next-is-forwarded-to-an-idle-cpu.md +++ /dev/null @@ -1,33 +0,0 @@ ---- -status: assigned -kind: finding -opened: 2026-10-12 ---- - -# A wake whose waker parks next is forwarded to an idle CPU - -`CpuSched::place` (`kernel/pure/sched/cpu.rs`) hands a fair wake that would -queue behind the running fair task to an idle sibling. When the waker is about -to park on that same CPU — a request answered and waited on, a thread started -and then joined — the CPU is free a moment later, and the forward pays an IPI -and a cold cache where a local switch after the park was free. Nothing tells -`place` that the running task is about to park. - -`acpi_mediated` met it on a guest: a thread started from a busy CPU went to the -idle one, so it stages its asker differently since PR #874. On metal the -spawn-then-wait row reads no change past one boot's spread: `spawn_cost`'s -median went from 1,025 to 1,029 µs on the T14 and from 2,054 to 2,086 µs on -the AMD bench, base to head, one boot each, read from the second judgement's -`testcases` readbacks. - -Owned by stage S3 of the scheduler design (#872), whose 1:1 pair rule decides -where a pair's wakes go. - -Branch `sched-place` decides it: the opening drain of a pass holds a wake -found behind the running task until the disposition, so a poster that parks -or exits leaves it to run here, and one that yields or keeps the CPU still -forwards it (`CpuSched::place`; the simulator's `turn_taking_pair`). The -metal price is what remains. - -**Exit**: S3 decides this case, with a simulator scenario of a wake whose waker -parks next and a metal row that prices the forward against the local switch. diff --git a/issues/an-ordinary-wake-posted-on-its-own-cpu-waits-for-that-cpus-next-pass.md b/issues/an-ordinary-wake-posted-on-its-own-cpu-waits-for-that-cpus-next-pass.md index 93222217aaa..a087f8b5ffc 100644 --- a/issues/an-ordinary-wake-posted-on-its-own-cpu-waits-for-that-cpus-next-pass.md +++ b/issues/an-ordinary-wake-posted-on-its-own-cpu-waits-for-that-cpus-next-pass.md @@ -27,8 +27,8 @@ process's fair-share lock, which the storm's other waiters were taking as they re-parked. Owned by stage S3 of the scheduler design (#872), which decides where a -pair's wakes go: a wake whose waker parks next is cheapest left here -(`issues/a-wake-whose-waker-parks-next-is-forwarded-to-an-idle-cpu.md`). +pair's wakes go: a wake whose waker parks next is held and left here +(`issues/a-wake-held-for-its-posters-park-is-unpriced-on-metal.md`). **Exit**: a wake posted on its own CPU by a poster that keeps running reaches a CPU within the bound `wake_onto_busy` holds for other CPUs' wakes, on the diff --git a/kernel/pure/sched/cpu.rs b/kernel/pure/sched/cpu.rs index b8aee2c78e9..6945cbaafb0 100644 --- a/kernel/pure/sched/cpu.rs +++ b/kernel/pure/sched/cpu.rs @@ -295,13 +295,13 @@ pub struct CpuSched { /// In no fair band, as [`Self::dying`]'s tasks are not; excluded from the /// published load, because nothing queues behind work that will never run. stopped: VecDeque>, - /// Wakes and adopts the opening drain found behind the running task, - /// held for [`SchedPass`]'s disposition: whether that task leaves the CPU - /// in this pass decides whether they run here or are forwarded. Empty - /// outside a pass. + /// Tasks the pass's opening (its drain and its deadlines) found behind + /// the running task, held for [`SchedPass`]'s disposition: whether that + /// task leaves the CPU in this pass decides whether they run here or are + /// forwarded. Empty outside a pass; the buffer is kept between passes. held: Vec>, /// Set by [`SchedPass::begin`] and cleared by the disposition, so only - /// the opening drain holds. + /// the pass's opening holds. holding: bool, /// The task that exited on this CPU, freed by the NEXT pass — a pass /// cannot free the stack it is running on. @@ -1025,9 +1025,8 @@ impl CpuSched { /// Wake placement: keep the task local — that is where its cache lines /// are — unless it would queue behind the running task in its own band, RT /// behind RT or fair behind fair, in which case an idle sibling gets it: it - /// displaces nobody there. In a pass's opening drain the running task is - /// the one the disposition leaves, so a wake its poster parks or exits - /// behind runs here, where it was posted. + /// displaces nobody there. A pass's opening holds such a task until the + /// pass's disposition and then places it with [`Self::place_settled`]. /// /// A fair task that stays and is what the fair band would now run in place /// of the running fair thread takes the CPU [`MIN_SLICE_NS`] after that @@ -1039,6 +1038,34 @@ impl CpuSched { task: ReadyTask, env: Env<'_, H, P>, now: Nanos, + ) { + self.place_behind(task, false, env, now); + } + + /// [`Self::place`] for a task the pass's opening held, once the + /// disposition has left the running task: a task its poster parked or + /// exited behind runs here, where it was posted, unless another task of + /// its band is already queued, an earlier held one included. So a burst + /// posted just before its poster parks keeps one task here and forwards + /// the rest, as it would have behind the poster. + fn place_settled, P: PreemptGuard>( + &mut self, + task: ReadyTask, + env: Env<'_, H, P>, + now: Nanos, + ) { + let queued_ahead = if task.is_rt() { self.rq.has_rt() } else { self.rq.fair_len() > 0 }; + self.place_behind(task, queued_ahead, env, now); + } + + /// [`Self::place`], where `queued_ahead` says a task of `task`'s band is + /// already queued here. + fn place_behind, P: PreemptGuard>( + &mut self, + task: ReadyTask, + queued_ahead: bool, + env: Env<'_, H, P>, + now: Nanos, ) { // **Ahead of the RT forward and of the kill check, because it outranks // both.** This is the one route by which a stopped task can be offered @@ -1063,7 +1090,7 @@ impl CpuSched { && self.rq.running_is_fair() && self.running.as_ref().is_some_and(|r| !r.shared().kill_pending()); let behind_rt = task.is_rt() && self.running.as_ref().is_some_and(|r| r.is_rt()); - if behind_fair || behind_rt { + if behind_fair || behind_rt || queued_ahead { if self.holding { self.held.push(task); return; @@ -1587,6 +1614,7 @@ impl<'c, 'e, H: Hw, P: PreemptGuard> SchedPass<'c, 'e, H, P, Undisposed> { cpu.rq.charge_running(ns); } } + assert!(!cpu.holding && cpu.held.is_empty(), "the previous pass on cpu{} was never disposed", cpu.id.0); cpu.holding = true; cpu.drain(env, now); cpu.fire_deadlines(env, now); @@ -1602,9 +1630,11 @@ impl<'c, 'e, H: Hw, P: PreemptGuard> SchedPass<'c, 'e, H, P, Undisposed> { /// disposition has left it. fn settle(&mut self) { self.cpu.holding = false; - for task in core::mem::take(&mut self.cpu.held) { - self.cpu.place(task, self.env, self.now); + let mut held = core::mem::take(&mut self.cpu.held); + for task in held.drain(..) { + self.cpu.place_settled(task, self.env, self.now); } + self.cpu.held = held; } pub fn cpu(&self) -> &CpuSched { @@ -4220,6 +4250,45 @@ mod tests { (hog, sleeper) } + /// **A burst its poster parks behind keeps one task where it was posted**: + /// the freed CPU runs the first, and the rest, which would queue behind it, + /// are forwarded to idle siblings as they would have been behind the + /// poster, in the RT band when `rt`. + fn a_burst_its_poster_parks_behind(rt: bool) { + let spawn = |w: &mut World, cpu| if rt { w.spawn_rt(cpu) } else { w.spawn(cpu) }; + let mut w = World::new(2); + let mut sleepers = Vec::new(); + for _ in 0..2 { + let (_, sleeper) = spawn(&mut w, C0); + w.run_a_pass(C0); + w.park_running(C0); + sleepers.push(sleeper); + } + spawn(&mut w, C0); + w.run_a_pass(C0); + w.run_a_pass(C1); + assert!(w.handles.get(C1).doorbell().sleeping(), "cpu1 has published SLEEPING"); + + for sleeper in &sleepers { + wake(&w, C0, sleeper); + } + w.park_running(C0); + let states: Vec<_> = sleepers.iter().map(|s| s.state()).collect(); + w.abandon(); + assert!(states.contains(&TaskState::Running(C0)), "one runs where it was posted: {states:?}"); + assert!(states.contains(&TaskState::InTransit(C1)), "and the other went to the idle sibling: {states:?}"); + } + + #[test] + fn a_fair_burst_its_poster_parks_behind_keeps_one_here() { + a_burst_its_poster_parks_behind(false); + } + + #[test] + fn an_rt_burst_its_poster_parks_behind_keeps_one_here() { + a_burst_its_poster_parks_behind(true); + } + /// **A wake whose poster parks in the pass that drains it runs where it was /// posted**: the CPU is free once the poster has gone, so a forward would /// buy an IPI and a cold cache for nothing. From 1b9e7c57d472acef9ae6d50a74a232e7ffd43f0d Mon Sep 17 00:00:00 2001 From: japabu Date: Sun, 1 Nov 2026 15:12:01 +0100 Subject: [PATCH 5/5] issues: the T14 A/B of the held-wake rule alone, read The A/B of #1111 alone on the T14 (base 24a538c88 against 042b2b299, 4+4 boots) reproduced seven disk rows 1.2-2.3% slower and left ipc unchanged. Timer rows the hold cannot reach read slower by the same amount there and in the AMD A/Bs of #1129, each with the base booting first and a different image per side, so the rows are not attributed. The record says so and asks for an order-balanced A/B. Co-Authored-By: Claude Opus 5.5 --- ...d-for-its-posters-park-is-unpriced-on-metal.md | 15 +++++++++++---- 1 file changed, 11 insertions(+), 4 deletions(-) diff --git a/issues/a-wake-held-for-its-posters-park-is-unpriced-on-metal.md b/issues/a-wake-held-for-its-posters-park-is-unpriced-on-metal.md index 74ac1b3d51c..662b57a76f0 100644 --- a/issues/a-wake-held-for-its-posters-park-is-unpriced-on-metal.md +++ b/issues/a-wake-held-for-its-posters-park-is-unpriced-on-metal.md @@ -21,8 +21,15 @@ tie-break, read twelve disk and memfs rows 1–3% slower (fstat, seek, stat, pread_4k, read_4k_at, write_4k_at, tree.100), four boots a side with one image each, so a layout difference between the images is not ruled out ([evidence](https://github.com/ToyOSOrg/evidence/tree/6cd2768011f0e6f36c4e2c1ce76f986fc6c2265b/pr-1116/r2)). -Nothing attributes them to this rule. +An A/B of the rule alone on the T14, base `24a538c88` against `042b2b299`, +reproduced seven disk rows 1.2–2.3% slower and left the ipc rows unchanged. +Timer rows the rule cannot reach (`wake.periodic.idle`, `tail.*`) read slower +by the same amount there and in two A/Bs of other changes, where the base +boots first and the images differ, so these A/Bs carry a head bias of that +size and the rows are not attributed +([evidence](https://github.com/ToyOSOrg/evidence/tree/f47a3a6c8324cc7c4a1f72921a350b2c2034dbec/pr-1111/r5)). -**Exit**: a T14 and an AMD A/B of the rule alone against its base reads those -rows within their spread or names the cost, and a metal row prices a wake its -poster parks behind against the forward. +**Exit**: a T14 and an AMD A/B of the rule alone against its base, in an +order that boots each side first equally often, reads those rows within their +spread or names the cost, and a metal row prices a wake its poster parks +behind against the forward.