sched: a wake its poster parks or exits behind runs where it was posted

issues/a-two-thread-ping-pong-costs-toyos-up-to-twice-linuxs-on-the-t14.md

@@ -36,7 +36,7 @@ on its head than its base
 (`issues/a-64-kib-pipe-write-is-two-percent-below-the-two-mebibyte-ring.md`),
 so the `pipe` rows' current state against Linux is owed a new reading.
 The scheduler's pair-wake records bear on both kinds:
-`issues/a-wake-whose-waker-parks-next-is-forwarded-to-an-idle-cpu.md` and
+`issues/a-wake-held-for-its-posters-park-is-unpriced-on-metal.md` and
 `issues/an-ordinary-wake-posted-on-its-own-cpu-waits-for-that-cpus-next-pass.md`.
 
 **Exit**: `pingpong.*` within Linux's on both benches in a labelled

issues/a-wake-held-for-its-posters-park-is-unpriced-on-metal.md

@@ -0,0 +1,35 @@
+---
+status: open
+kind: defect
+opened: 2026-10-12
+---
+
+# A wake held for its poster's park is unpriced on metal
+
+A pass's opening holds every task it would forward for queueing behind the
+running task, and places them after the pass's disposition
+(`CpuSched::place_settled`, `kernel/pure/sched/cpu.rs`). A poster that parks,
+stops or exits leaves the CPU to the first held task of each band; one that
+yields or keeps the CPU forwards them to idle siblings as before, and so do
+the rest of a burst. The simulator's `turn_taking_pair` holds the case: no
+turn is forwarded off its CPU.
+
+What it costs or saves on metal is unread. No row prices the forward against
+the local switch. The first A/B that carried the rule, on the T14 with base
+`0c7e62ca7` against a head that also carried CPU affinity and the spawn
+tie-break, read twelve disk and memfs rows 1–3% slower (fstat, seek, stat,
+pread_4k, read_4k_at, write_4k_at, tree.100), four boots a side with one image
+each, so a layout difference between the images is not ruled out
+([evidence](https://github.com/ToyOSOrg/evidence/tree/6cd2768011f0e6f36c4e2c1ce76f986fc6c2265b/pr-1116/r2)).
+An A/B of the rule alone on the T14, base `24a538c88` against `042b2b299`,
+reproduced seven disk rows 1.2–2.3% slower and left the ipc rows unchanged.
+Timer rows the rule cannot reach (`wake.periodic.idle`, `tail.*`) read slower
+by the same amount there and in two A/Bs of other changes, where the base
+boots first and the images differ, so these A/Bs carry a head bias of that
+size and the rows are not attributed
+([evidence](https://github.com/ToyOSOrg/evidence/tree/f47a3a6c8324cc7c4a1f72921a350b2c2034dbec/pr-1111/r5)).
+
+**Exit**: a T14 and an AMD A/B of the rule alone against its base, in an
+order that boots each side first equally often, reads those rows within their
+spread or names the cost, and a metal row prices a wake its poster parks
+behind against the forward.

issues/an-ordinary-wake-posted-on-its-own-cpu-waits-for-that-cpus-next-pass.md

@@ -27,8 +27,8 @@ process's fair-share lock, which the storm's other waiters were taking as
 they re-parked.
 
 Owned by stage S3 of the scheduler design (#872), which decides where a
-pair's wakes go: a wake whose waker parks next is cheapest left here
-(`issues/a-wake-whose-waker-parks-next-is-forwarded-to-an-idle-cpu.md`).
+pair's wakes go: a wake whose waker parks next is held and left here
+(`issues/a-wake-held-for-its-posters-park-is-unpriced-on-metal.md`).
 
 **Exit**: a wake posted on its own CPU by a poster that keeps running reaches
 a CPU within the bound `wake_onto_busy` holds for other CPUs' wakes, on the

kernel/pure/sched/cpu.rs

@@ -295,6 +295,14 @@ pub struct CpuSched<X: SchedPayload> {
     /// In no fair band, as [`Self::dying`]'s tasks are not; excluded from the
     /// published load, because nothing queues behind work that will never run.
     stopped: VecDeque<ReadyTask<X>>,
+    /// Tasks the pass's opening (its drain and its deadlines) found behind
+    /// the running task, held for [`SchedPass`]'s disposition: whether that
+    /// task leaves the CPU in this pass decides whether they run here or are
+    /// forwarded. Empty outside a pass; the buffer is kept between passes.
+    held: Vec<ReadyTask<X>>,
+    /// Set by [`SchedPass::begin`] and cleared by the disposition, so only
+    /// the pass's opening holds.
+    holding: bool,
     /// The task that exited on this CPU, freed by the NEXT pass — a pass
     /// cannot free the stack it is running on.
     zombie: Option<DeadTask<X>>,
@@ -393,6 +401,8 @@ impl<X: SchedPayload> CpuSched<X> {
             parked: Parked::new(),
             dying: VecDeque::new(),
             stopped: VecDeque::new(),
+            held: Vec::new(),
+            holding: false,
             zombie: None,
             mailbox,
             steal_probe: MailboxNode::new(),
@@ -615,6 +625,8 @@ impl<X: SchedPayload> CpuSched<X> {
                 (core::mem::offset_of!(Self, parked), "parked (the park map)"),
                 (core::mem::offset_of!(Self, dying), "dying"),
                 (core::mem::offset_of!(Self, stopped), "stopped"),
+                (core::mem::offset_of!(Self, held), "held"),
+                (core::mem::offset_of!(Self, holding), "holding"),
                 (core::mem::offset_of!(Self, zombie), "zombie"),
                 (core::mem::offset_of!(Self, mailbox), "mailbox"),
                 (core::mem::offset_of!(Self, steal_probe), "steal_probe (excluded: a sibling writes it)"),
@@ -1013,7 +1025,8 @@ impl<X: SchedPayload> CpuSched<X> {
     /// Wake placement: keep the task local — that is where its cache lines
     /// are — unless it would queue behind the running task in its own band, RT
     /// behind RT or fair behind fair, in which case an idle sibling gets it: it
-    /// displaces nobody there.
+    /// displaces nobody there. A pass's opening holds such a task until the
+    /// pass's disposition and then places it with [`Self::place_settled`].
     ///
     /// A fair task that stays and is what the fair band would now run in place
     /// of the running fair thread takes the CPU [`MIN_SLICE_NS`] after that
@@ -1025,6 +1038,34 @@ impl<X: SchedPayload> CpuSched<X> {
         task: ReadyTask<X>,
         env: Env<'_, H, P>,
         now: Nanos,
+    ) {
+        self.place_behind(task, false, env, now);
+    }
+
+    /// [`Self::place`] for a task the pass's opening held, once the
+    /// disposition has left the running task: a task its poster parked or
+    /// exited behind runs here, where it was posted, unless another task of
+    /// its band is already queued, an earlier held one included. So a burst
+    /// posted just before its poster parks keeps one task here and forwards
+    /// the rest, as it would have behind the poster.
+    fn place_settled<H: Hw<Payload = X>, P: PreemptGuard>(
+        &mut self,
+        task: ReadyTask<X>,
+        env: Env<'_, H, P>,
+        now: Nanos,
+    ) {
+        let queued_ahead = if task.is_rt() { self.rq.has_rt() } else { self.rq.fair_len() > 0 };
+        self.place_behind(task, queued_ahead, env, now);
+    }
+
+    /// [`Self::place`], where `queued_ahead` says a task of `task`'s band is
+    /// already queued here.
+    fn place_behind<H: Hw<Payload = X>, P: PreemptGuard>(
+        &mut self,
+        task: ReadyTask<X>,
+        queued_ahead: bool,
+        env: Env<'_, H, P>,
+        now: Nanos,
     ) {
         // **Ahead of the RT forward and of the kill check, because it outranks
         // both.** This is the one route by which a stopped task can be offered
@@ -1049,7 +1090,11 @@ impl<X: SchedPayload> CpuSched<X> {
             && self.rq.running_is_fair()
             && self.running.as_ref().is_some_and(|r| !r.shared().kill_pending());
         let behind_rt = task.is_rt() && self.running.as_ref().is_some_and(|r| r.is_rt());
-        if behind_fair || behind_rt {
+        if behind_fair || behind_rt || queued_ahead {
+            if self.holding {
+                self.held.push(task);
+                return;
+            }
             if let Some(dst) = self.forward_target(env) {
                 self.hand_off(task, dst, env, now);
                 return;
@@ -1569,6 +1614,8 @@ impl<'c, 'e, H: Hw, P: PreemptGuard> SchedPass<'c, 'e, H, P, Undisposed> {
                 cpu.rq.charge_running(ns);
             }
         }
+        assert!(!cpu.holding && cpu.held.is_empty(), "the previous pass on cpu{} was never disposed", cpu.id.0);
+        cpu.holding = true;
         cpu.drain(env, now);
         cpu.fire_deadlines(env, now);
         Self {
@@ -1579,6 +1626,17 @@ impl<'c, 'e, H: Hw, P: PreemptGuard> SchedPass<'c, 'e, H, P, Undisposed> {
         }
     }
 
+    /// Place what the opening drain held, against the running task as the
+    /// disposition has left it.
+    fn settle(&mut self) {
+        self.cpu.holding = false;
+        let mut held = core::mem::take(&mut self.cpu.held);
+        for task in held.drain(..) {
+            self.cpu.place_settled(task, self.env, self.now);
+        }
+        self.cpu.held = held;
+    }
+
     pub fn cpu(&self) -> &CpuSched<H::Payload> {
         self.cpu
     }
@@ -1589,12 +1647,16 @@ impl<'c, 'e, H: Hw, P: PreemptGuard> SchedPass<'c, 'e, H, P, Undisposed> {
 
     /// The current task keeps its claim on the CPU (subject to the preemption
     /// decision in `finish`).
-    pub fn dispose_none(self) -> SchedPass<'c, 'e, H, P, Disposed> {
+    pub fn dispose_none(mut self) -> SchedPass<'c, 'e, H, P, Disposed> {
+        self.settle();
         self.dispose()
     }
 
     /// Voluntary yield: a fair thread gives up the rest of its request.
-    pub fn dispose_yield(self) -> SchedPass<'c, 'e, H, P, Disposed> {
+    pub fn dispose_yield(mut self) -> SchedPass<'c, 'e, H, P, Disposed> {
+        // Before the yield: the yielder stays runnable here, so what queues
+        // behind it is forwarded as it would be behind a running task.
+        self.settle();
         if let Some(current) = self.cpu.running.take() {
             let mut task = current.preempt(self.cpu.id, self.now);
             if task.shared().kill_pending() {
@@ -1613,7 +1675,7 @@ impl<'c, 'e, H: Hw, P: PreemptGuard> SchedPass<'c, 'e, H, P, Undisposed> {
     /// necessarily runs after the switch completes. The stack-reuse race is
     /// sequentially impossible, not locked away.
     pub fn dispose_block(
-        self,
+        mut self,
         ticket: CommittedTicket<Msg<H::Payload>>,
         deadline: Option<Nanos>,
     ) -> SchedPass<'c, 'e, H, P, Disposed> {
@@ -1643,6 +1705,7 @@ impl<'c, 'e, H: Hw, P: PreemptGuard> SchedPass<'c, 'e, H, P, Undisposed> {
         );
         self.cpu
             .trace(self.env, self.now, TraceKind::ParkCommit { task: key, name });
+        self.settle();
         self.dispose()
     }
 
@@ -1654,7 +1717,7 @@ impl<'c, 'e, H: Hw, P: PreemptGuard> SchedPass<'c, 'e, H, P, Undisposed> {
     /// only the caller knows it is standing somewhere it holds nothing, and
     /// this is where that knowledge is spent. Everything else a stopped task
     /// meets — a wake, an adopt, a retire — reads the bit in `place`.
-    pub fn dispose_stop(self) -> SchedPass<'c, 'e, H, P, Disposed> {
+    pub fn dispose_stop(mut self) -> SchedPass<'c, 'e, H, P, Disposed> {
         let current = self
             .cpu
             .running
@@ -1664,12 +1727,13 @@ impl<'c, 'e, H: Hw, P: PreemptGuard> SchedPass<'c, 'e, H, P, Undisposed> {
         self.cpu.rq.leave_running(task.key(), task.standing_mut());
         task.shared().mark_stop();
         self.cpu.keep_stopped(task);
+        self.settle();
         self.dispose()
     }
 
     /// The current task exits. Its record survives as the zombie until the
     /// next pass, which runs on another stack.
-    pub fn dispose_exit(self) -> SchedPass<'c, 'e, H, P, Disposed> {
+    pub fn dispose_exit(mut self) -> SchedPass<'c, 'e, H, P, Disposed> {
         let mut current = self
             .cpu
             .running
@@ -1682,6 +1746,7 @@ impl<'c, 'e, H: Hw, P: PreemptGuard> SchedPass<'c, 'e, H, P, Undisposed> {
         self.cpu.dispose_dead(dead, self.env);
         self.cpu
             .trace(self.env, self.now, TraceKind::Retire { task: key, name });
+        self.settle();
         self.dispose()
     }
 
@@ -4172,6 +4237,103 @@ mod tests {
         w.abandon();
     }
 
+    /// The forward test's setup: `sleeper` parked on cpu0, a hog running
+    /// there, cpu1 asleep. Answers the hog and the sleeper.
+    fn hog_and_a_parked_sleeper(w: &mut World) -> (TaskKey, Arc<TaskShared<Msg<TestPayload>>>) {
+        let (_, sleeper) = w.spawn(C0);
+        w.run_a_pass(C0);
+        w.park_running(C0);
+        let (hog, _) = w.spawn(C0);
+        w.run_a_pass(C0);
+        w.run_a_pass(C1);
+        assert!(w.handles.get(C1).doorbell().sleeping(), "cpu1 has published SLEEPING");
+        (hog, sleeper)
+    }
+
+    /// **A burst its poster parks behind keeps one task where it was posted**:
+    /// the freed CPU runs the first, and the rest, which would queue behind it,
+    /// are forwarded to idle siblings as they would have been behind the
+    /// poster, in the RT band when `rt`.
+    fn a_burst_its_poster_parks_behind(rt: bool) {
+        let spawn = |w: &mut World, cpu| if rt { w.spawn_rt(cpu) } else { w.spawn(cpu) };
+        let mut w = World::new(2);
+        let mut sleepers = Vec::new();
+        for _ in 0..2 {
+            let (_, sleeper) = spawn(&mut w, C0);
+            w.run_a_pass(C0);
+            w.park_running(C0);
+            sleepers.push(sleeper);
+        }
+        spawn(&mut w, C0);
+        w.run_a_pass(C0);
+        w.run_a_pass(C1);
+        assert!(w.handles.get(C1).doorbell().sleeping(), "cpu1 has published SLEEPING");
+
+        for sleeper in &sleepers {
+            wake(&w, C0, sleeper);
+        }
+        w.park_running(C0);
+        let states: Vec<_> = sleepers.iter().map(|s| s.state()).collect();
+        w.abandon();
+        assert!(states.contains(&TaskState::Running(C0)), "one runs where it was posted: {states:?}");
+        assert!(states.contains(&TaskState::InTransit(C1)), "and the other went to the idle sibling: {states:?}");
+    }
+
+    #[test]
+    fn a_fair_burst_its_poster_parks_behind_keeps_one_here() {
+        a_burst_its_poster_parks_behind(false);
+    }
+
+    #[test]
+    fn an_rt_burst_its_poster_parks_behind_keeps_one_here() {
+        a_burst_its_poster_parks_behind(true);
+    }
+
+    /// **A wake whose poster parks in the pass that drains it runs where it was
+    /// posted**: the CPU is free once the poster has gone, so a forward would
+    /// buy an IPI and a cold cache for nothing.
+    #[test]
+    fn a_wake_its_poster_parks_behind_runs_where_it_was_posted() {
+        let mut w = World::new(2);
+        let (_, sleeper) = hog_and_a_parked_sleeper(&mut w);
+        wake(&w, C0, &sleeper);
+        w.park_running(C0);
+        assert_eq!(w.cpus[0].running().map(|t| t.key()), Some(sleeper.key()), "the wake ran here");
+        assert!(!w.hw.state().kicks.contains(&C1), "and the sibling was not woken for it");
+        w.abandon();
+    }
+
+    /// **The same for a poster that exits**: a thread started and joined, the
+    /// join's wake drained in the pass the thread ends in.
+    #[test]
+    fn a_wake_its_poster_exits_behind_runs_where_it_was_posted() {
+        let mut w = World::new(2);
+        let (_, sleeper) = hog_and_a_parked_sleeper(&mut w);
+        wake(&w, C0, &sleeper);
+        {
+            let (cpus, env) = w.split();
+            let _ = SchedPass::begin(&mut cpus[0], env, Nanos(NOW.0 + 1)).dispose_exit().finish();
+        }
+        assert_eq!(w.cpus[0].running().map(|t| t.key()), Some(sleeper.key()), "the wake ran here");
+        assert!(!w.hw.state().kicks.contains(&C1), "and the sibling was not woken for it");
+        w.abandon();
+    }
+
+    /// **A poster that yields stays runnable**, so what queues behind it is
+    /// forwarded as it is behind a poster that keeps the CPU.
+    #[test]
+    fn a_wake_its_poster_yields_behind_is_still_forwarded() {
+        let mut w = World::new(2);
+        let (_, sleeper) = hog_and_a_parked_sleeper(&mut w);
+        wake(&w, C0, &sleeper);
+        {
+            let (cpus, env) = w.split();
+            let _ = SchedPass::begin(&mut cpus[0], env, Nanos(NOW.0 + 1)).dispose_yield().finish();
+        }
+        assert_eq!(sleeper.state(), TaskState::InTransit(C1), "the wake was forwarded");
+        w.abandon();
+    }
+
     /// **One forward per idle sibling**: wakes drained in one pass are handed
     /// to a sibling only while it has no kick coming, so a burst does not pile
     /// onto the first sleeper — the rest queue here.