Repository navigation
fanout: thin jobs, one injector, a heartbeat, and a futex parker - #4
Conversation
`submit` took the sleeper word exclusively on every task, with an RMW, and
that word is shared by every worker parking and waking. So the wake
bookkeeping serialised the whole pool, and the cost grew with the number of
workers rather than with the work.
Measured, 100,000 trivial tasks, one producer, median of 5:
workers before after
1 110 ns 113 ns
2 113 ns 110 ns
4 766 ns 309 ns
8 1380 ns 1511 ns
When nobody is asleep - the throughput case - the bitmap is zero and a load
answers the question without taking the line. The load stays `SeqCst` for the
same reason the RMW was: it has to be ordered against publishing the task, or
a worker that sets its bit and then checks its intake could be missed.
Four workers is 2.5x. Eight is unchanged, so there is a second bottleneck and
this is not it. It is not in `submit`: submitting into idle intakes is 2-3 ns
and does not degrade with worker count at all. It is on the worker side, and
the candidate is `steal_once`'s exhaustive sweep - every idle worker probes
every victim, so eight workers with trivial tasks can put seven sweepers on
seven queues at once, each probe a contended RMW. The sweep is exhaustive for
a good reason, stated where it is written, but nothing bounds how many workers
sweep at the same time. Tokio caps its searchers at half.
cargo test 1 + 22 + 3 passed
A worker parked the moment its queue ran dry. Parking is cheap for the parker
and expensive for whoever wakes it, because a wake is a mutex and a condvar
notify, so every later submit paid one.
Counted at eight workers over 100,000 tasks, before this: 0.95 parks a task,
6.81 steal attempts a task with 0.01 hits, and the producer's submit loop was
100% of wall time at 1,402 ns a submit against 196 ns at one worker. The
workers were not slow, they were starved, and the starvation was caused by the
wakes their own parking forced the producer to perform.
Sixty-four empty rounds before announcing sleep. Any work found resets the
count. Swept at 16, 64, 256 and 1024: 445, 353, 372, 365 ns a task.
cargo test 1 + 22 + 3 passed
The previous commit's spin was not a spin: it re-ran the whole search - lock
this worker's intake, probe every other worker's queue - up to 64 times per dry
spell. An empty round is not free to anybody else, so repeating it immediately
starved the producer it was meant to keep feeding.
That it appeared to work was an accident. The measurement was taken with
instrumentation counters in the loop, and the contended fetch_adds were acting
as the backoff the code did not have. Without them the same code measures
884 to 1,141 ns a task, not 372.
Backing off with nothing shared, 100,000 tasks over 8 workers:
backoff 16 64 256 spins
317 311 313 ns a task
Flat, which is what says the waiting is the point rather than its length.
workers before after tokio
1 110 107 177
2 113 167 210
4 766 264 263
8 1380 307 278
`Tuning` makes both values a parameter of `Pool::with_tuning`. They are policy,
not mechanism: the right value depends on how fast work arrives and what else
is on the machine, and the default is what was measured here, on one machine,
which is not a claim about any other.
cargo test 1 + 22 + 3 passed
`unpark` locked a mutex and notified a condvar every time, and `park` locked
to check for a permit. With the pool waking a sleeper per submit under load,
that mutex was on the hot path.
The permit is an `AtomicBool` and the waiters are an `AtomicUsize`, so an
unpark with nobody on the condvar is a store and a load, and a park with a
permit already waiting is one swap. The mutex and condvar are entered only when
a thread genuinely sleeps.
Measured, 100,000 tasks over 8 workers, median of 5:
first poll 67.5 -> 57.3 ms wall, 1,745,514 tasks/s, 515 ms cpu
yielding 55.4 -> 46.9 ms wall, 2,131,075 tasks/s, 422 ms cpu
cargo test 1 + 22 + 3 passed
Three changes that only make sense together, and they are a work in progress: against the branch they sit on this is currently *slower*, and the numbers are in the commit that follows. **Jobs instead of boxed closures.** `Box<dyn FnOnce()>` is a fat pointer, so a queue of them is a queue of two words that must be moved and dropped as a unit, and a caller whose work is already on the heap has to allocate anyway. `Job` is a thin pointer plus the monomorphised function that runs it. `submit_fn` allocates once, `submit_job` not at all. `submit` still takes a `Task` and now costs two allocations, which is the price of not breaking it. **One injector, not an intake per worker.** The per-worker `Mutex<Vec<Task>>` was the pool's central mistake: nothing but its owner could reach it, so an idle worker could not find work sitting one worker over, and the rule every work-stealing scheduler relies on did not hold. Six scheduling policies were tried on top of it and all six lost. A lock-free queue every worker drains makes the rule true. **A heartbeat.** Each worker now runs out of a private `VecDeque` that costs no atomic to touch at all, and promotes half of it into its stealable deque every 64 jobs, and only when the sleeper bitmap says somebody is waiting to receive it. Sharing is paid for on a timer rather than on every task. This is what `forte`, `chili` and Spice do, and it is why forte spends 71 ms of CPU on a workload that costs this pool 392. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Added: thin jobs, one injector, a heartbeatFour more commits, and the last three are one architectural change rather than three tunings.
One lock-free injector in place of the per-worker A heartbeat. Each worker now runs out of a private MeasuredBoth revisions rebuilt and alternated in one sitting, three pairs, 100,000 tasks and 8 workers,
So it is worth about 12% to a runtime built on this pool and costs Gates
|
A condvar wake is a mutex acquisition, a signal and a release, and this pool wakes a worker on its hot path. `atomic-wait` is one atomic word per worker: `futex` on Linux, `WaitOnAddress` on Windows, `__ulock_wait` on macOS. A wake with nobody asleep is a swap and a branch with no system call at all. Measured on 100,000 tasks and one worker, where parking is most of what the pool does: 13.6 ms to 9.5 ms wall. On eight workers it is a wash, because at 64 spin rounds the workers rarely park; what it buys there is the freedom to spin less, and that is still to be earned. The 100 ms park timeout goes. It was a backstop from a time when nothing woke a parked worker and the timeout *was* how stealable work got discovered. `Pool` sets a worker's sleeper bit before its last look for work, so a submit either publishes before that look or reads a bitmap with the bit already set, and `shut_down` unparks every worker; the backstop guarded nothing and cost a wakeup per worker per 100 ms on a pool with nothing to do. `StdHost::timeouts` is deprecated and returns zero; `StdHost::spurious` counts what is left. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The single largest change in this pool, and it is a constant.
A worker that finds no work looks again, and looking means a pop off the
injector and a probe of every other worker's deque. Eight idle workers doing
that in a tight loop take the very cache lines the producer is trying to fill,
so the pool starves itself: 100,000 tasks that one worker finishes in 12.4 ms
took eight workers 49.2 ms. Adding workers made it four times slower.
Swept, with wake latency measured separately on an idle pool over 2,000 samples
because throughput and latency pull opposite ways here:
spins wall cpu latency median p99
64 49.2 ms 440 ms 3917 ns 10333 ns
256 36.0 ms 325 ms 2500 ns 7958 ns
1024 30.3 ms 277 ms 2375 ns 6000 ns
4096 24.4 ms 237 ms 3375 ns 14209 ns
16384 15.6 ms 213 ms 21000 ns 43375 ns
1024 is the last value better than its predecessor on **every** axis, so it is
the default: 1.6x the throughput of 64, 1.6x less CPU, and lower latency at the
median and the 99th percentile both. Past it the trade is real and the caller
should make it: 16384 is three times the throughput and five times the wake
latency, right for a batch pool and wrong for anything waiting on a reply.
`Tuning::injector_batch` joins the other knobs. Swept at 8, 32 and 128 it moved
nothing (49.2, 48.3, 50.3 ms), so it is exposed rather than tuned.
Two things this is *not*, both measured rather than assumed. It is not the
benchmark's own counter: 100,000 increments of one shared `AtomicUsize` cost
0.2 ms on one thread and 2.8 ms on eight, about 5% of the run. And it is not the
producer: one thread submits 100,000 jobs in 1.8 to 2.3 ms.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The backoff was the whole thingTwo more commits, and the second is the largest change in this pool by a wide margin. It is a constant. The measurement that found it
What was left is that a worker finding no work looks again, and looking means a pop off the injector and a probe of every other worker's deque. Eight idle workers doing that in a tight loop take the very cache lines the producer is trying to fill. The sweepThroughput and latency pull opposite ways here, so wake latency is measured separately on an idle pool, 2,000 samples, in
1024 is the last value better than its predecessor on every axis, so it is the default: 1.6x the throughput of 64, 1.6x less CPU, and lower latency at both the median and the 99th percentile. Past it the trade is real and belongs to the caller, which is why it stays a AlsoA futex parker.
Where the pool now sits100,000 tasks, 8 workers, median of 5, arms interleaved, submitted from outside: Gates33 tests, What is leftCPU. 261 ms against forte's 199 on closures, and 273 against 80 when a runtime sits on top. The workers still spin where forte's sleep, and every value of |
The pool kept a bitmap of which workers were asleep, so a submit could skip the unpark when nobody was on the other end. That bitmap was a single cache line taking a `SeqCst` read-modify-write from every worker that parked and every submit that looked, and it cost more than the wake it saved: parking sooner made the pool burn *more* CPU, 818 ms against 445, because the bookkeeping outweighed the sleep. `forte`'s `Semaphore` (`src/latch.rs`, MIT OR Apache-2.0, the same licence as this crate) is the fix, and the reason it works is one observation worth writing down: **both sides swap the same word, and modification order on a single atomic is total even under `Relaxed`.** So there is no store-load race to close and no `SeqCst` to pay for closing it. The bitmap needed `SeqCst` precisely because it was a different word from the queue. `StdHost` is now three states per worker on its own padded word: LOCKED, ASLEEP, SIGNAL. `unpark` is one `Relaxed` swap and calls the kernel only when somebody is genuinely on the address, so `Pool::push` unparks on every submit without asking who is asleep, and `nudge` picks a worker round-robin instead of reading a shared bitmap. A signal to a running worker leaves a permit its next park consumes, so a wrong guess costs one wasted pass around that worker's loop. What survives of the bitmap is `Pool::asleep`, a `Relaxed` counter touched only when a worker actually parks, which the backoff makes rare. It exists so the heartbeat can skip sharing when there is nobody to share with, which is the one thing the bitmap was genuinely needed for. Wake latency on an idle pool is unchanged over 2,000 samples: 2,458 ns median and 6,291 ns at the 99th percentile, against 2,375 and 6,000 before. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Three self-inflicted CI failures, all from the last few commits. `Job::from_raw` was `const`, and a function pointer in a `const fn` is 1.61 while this crate holds a 1.60 floor. It does not need to be const. `Job::from_boxed`'s inner `execute` had an `unsafe` block inside an `unsafe fn`, which on this edition is already an unsafe block, so it was flagged as unnecessary. The SAFETY comment stays; only the redundant block goes. `an_idle_pool_stays_idle` called `StdHost::timeouts`, which the previous commit deprecated, and CI compiles with `-D warnings`. It counts **spurious wakes** now, which is a stronger version of the same test rather than a weaker one: parking has no timeout left to expire, so a regression to polling has to show up as the platform returning from a wait with nothing to show for it, and the same bound catches it. `examples/floor.rs` prints the same thing. And `cargo fmt`, which had never been run against any of this. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
pathscale
changed the title
**A `Job` now owns its work.** `from_boxed` handed a `Box` to a raw pointer and only `execute` took it back, so a job that was never run leaked the closure and everything it captured: a pool dropped with work queued, jobs accepted after shutdown, a worker unwinding with a private queue. `Box<dyn FnOnce()>`, which this replaced, released them for free. The callback takes an `Act` now, `Run` or `Drop`, which keeps a `Job` two words rather than adding a second function pointer to a type the queues store by value. Two tests: a job dropped unrun releases its capture exactly once and does not run, and a pool dropped with sixteen queued jobs releases all sixteen. **Consuming a wake permit acquires it.** The fast path swapped `Relaxed` and returned. Total modification order on that word proves which transition happened first and nothing about what the unparker wrote before it, so a worker could take the shutdown notification and still read `running` as `true`, park again, and never be told twice. The swap is `Acquire`, and both places that clear the word are now compare-exchanges: a plain store would erase a *newer* permit that arrived in between, which is a lost wakeup. **A submit wakes a worker that is actually asleep.** Waking only the hinted worker leaves an available worker parked while a busy one takes the notification, and with no park timeout nothing corrects that until the busy worker finishes. The bitmap is back, but read on submit and written only on an actual park, which the backoff makes rare, rather than a read-modify-write per submit. It is `SeqCst` on both sides and that is load-bearing: a worker sets its bit then looks at the injector, a submit publishes then reads the bitmap, and only a total order across both makes one see the other. It also removes the `next_hint` read-modify-write the review flagged separately. **The heartbeat now fires with something to share.** With `injector_batch = 32` and `promote_every = 64`, every heartbeat landed on the last job of the second batch, when the private queue was empty. The mechanism ran forever and shared nothing. Sharing happens at the refill, which is the one moment a worker is known to hold more than it needs, and the countdown is a decrement rather than a `u64` remainder per job. **And shutdown is checked on the spin path.** Found by trying to prove the idle test had teeth: a pool with a large `rounds_before_park` hung on `shut_down`, because `is_running` was only reached after the spin budget ran out. Same class as the review's `injector_batch = 0` finding. Also: `with_tuning` rejects zero workers, zero `promote_every` and zero `injector_batch`; the `host` feature documents that it needs 1.65 on Linux, Android and FreeBSD, where `atomic-wait` pulls `libc`, against the crate's 1.60. **The idle test measures CPU.** Counting spurious wakes was not the claim: a worker that spins without ever reaching the platform wait burns a core while that counter stays at zero. `getrusage` cannot be fooled that way, and the test is verified to fail on a pool that never parks, at 7,999.8 ms of CPU per second. No regression: 100,000 tasks on 8 workers, nagoya on this pool measured 26.4 and 24.3 ms against 29.6 before, at a machine load of 25. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The review said work behind a long job could be unreachable. It is, and the test it asked for reproduces it: **0 of 31 short jobs ran** while one worker held a long one. Everything a worker takes off the injector goes into a private queue nothing else can see, and a worker that starts a long job never pops again, so it never reaches the point where it would share. Two changes, and the second is the one that matters. **Sharing at a refill no longer asks whether anybody is asleep.** The check ran before the other worker had parked, found nobody, shared nothing, and there was no second chance. That took it to 16 of 31: half a batch published, half stranded. **The batch is 8 rather than 32.** That is the actual bound on how much one worker can hide, and the sweep already said the size buys no throughput: 49.2, 48.3 and 50.3 ms at 8, 32 and 128. Re-measured at the current defaults, batch 8 against batch 32: 26.9, 25.1, 25.8 ms against 26.5, 29.6, 25.3, with CPU 242, 226, 233 against 237, 267, 228. Free. **30 of 31 short jobs now run**, and the one left is the job the worker is actually inside, which nothing short of preemption reaches. The test asserts the bound rather than perfection, and says why in the code. The private queue itself is worth keeping, which was worth checking rather than assuming: refilling straight into the stealable deque instead measured 30.9, 31.9, 31.3 ms against 26.5, 29.6, 25.3, and 278 to 288 ms of CPU against 228 to 267. About 18%. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…d free
Every argument for batching here was made on a fixture that cannot show its
cost. 100,000 identical trivial tasks: with every job the same length, nothing
can queue behind anything, and the size swept flat at 8, 32 and 128 (49.2, 48.3,
50.3 ms). So batching was chosen for a saving that fixture could measure against
a cost it could not.
`nagoya`'s new `burst` example measures a burst of mixed durations, 20,000 jobs
with one in twenty about a hundred times longer, and reports the time from a job
being submitted to it starting. Makespan is flat across every batch size and
latency is monotonic in it:
batch makespan p50 p99 max
1 5.5 ms 4333 ns 25458 ns 32959 ns
2 5.6 ms 5291 ns 28209 ns 39375 ns
4 5.7 ms 7583 ns 29834 ns 44042 ns
8 5.2 ms 8417 ns 30417 ns 45250 ns
32 5.8 ms 13875 ns 122083 ns 196083 ns
A batch is a job's exposure: to whatever is in front of it in that batch, and to
whatever its worker takes private and then sits on while running something long.
Buying nothing and costing that, it is one.
At one, against the same burst: forte 6.1 ms makespan, 3583 ns p50, 58584 p99;
tokio 8.6 ms, 14000, 54500; rayon 10.5 ms, 667, 16541. This pool wins makespan,
the 99th percentile and the maximum, and loses the median only to rayon, which
does not batch either.
Re-checked against the old throughput fixture so this is not a trade: 28.8 ms at
batch 1 against 30.3 at 8 and 26.3 at 32, which is noise.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Three benchmarks, and the answer is not where I assumed.
100,000 tasks, 8 workers, median of 5, interleaved, null-calibrated
nagoya 167.8 ms 595,927 tasks/s
tokio 30.8 ms 3,242,336 tasks/s
Splitting the arms puts the executor almost nowhere: the pool alone, with a
trivial boxed closure and no future, waker or executor at all, is 154.9 ms
against nagoya's 166.8. Everything on top of ps-st3 costs 7%.
A worker sweep says the rest is not a per-task cost but a scaling collapse:
workers pool tokio
1 110 ns 185 ns <- the pool is 1.7x faster
2 113 ns 190 ns <- 1.7x faster
4 766 ns 218 ns
8 1380 ns 247 ns
And submitting with nobody running is 2-3 ns at every worker count, so the
producer is not it either. It is worker-side contention, half of which is
fixed in pathscale/ps-st3#4.
The first harness spun the main thread waiting for completion, which takes a
core from eight workers while tokio's arm parks inside block_on. Fixed before
any of the above was reported; it moved 161 ms to 168, so it changed nothing
but it was measuring the harness.
The design has been argued twice in conversation and written down neither time. This is the argument: where this crate sits against tokio and rayon, which of them it could actually displace, the five layers that would make `fanout` feel like a runtime, and the one boundary that matters - `Host` is already the whole OS seam, so the first four keep the no_std core and only an I/O driver ends it. It also records the name, and why the two obvious ones were rejected: `osaka` is taken by an abandoned async runtime, which is the one collision worth avoiding, and `tokyo` is a typosquat of the project whose queue design this builds on. Nothing is built. The note says so.
Re-verified 9 September, and the headline understates it
This branch is what closes nagoya's wall-clock gap to tokio. The runtime that sits on this pool was recorded on 8 September at 2x tokio's wall and 7x its CPU. Re-measured today against this branch, The null arm ran 5-6%, so read it accordingly: the yielding shape is a real 16% win, repeated in both runs. First poll is parity to modestly ahead, 20% in one run and 6% in the other, and the run-to-run spread swallows it, so it is not a 20% result. Load average was 4.5 to 6.0 on 16 cores, all of it desktop applications, so the absolute figures are soft and only the interleaved comparison is worth quoting. CPU is the gap that remains, ~3.2x on first poll and ~1.6x on yielding, consistent and far outside any floor. That is the exhaustive steal sweep this PR's own body identifies and defers, and it is the next change rather than a defect in this one. One correction to the body above"clippy, fmt clean" is true of what CI runs, which is a bare |
Blocking regression:
|
Reverts 509fc9f, which removed batched injector takes on the grounds that "batching only looked free". Measured against `examples/floor.rs`, on 16 cores with 400,000 items in 4,096-item tasks, batching is worth 3.6x: with 509fc9f 56.9 / 57.2 / 57.8 ms reverted 14.9 / 15.9 / 15.9 ms The null arm separates by 1% and the sequential arm is unchanged across every run, so this is the pool and not the machine. **Nothing was lost by putting it back.** The case 509fc9f was written for is 100,000 trivial tasks from one producer, and that case is measured through nagoya, which stays ahead of tokio on wall clock either way: shape reverted with 509fc9f first poll 25.7 vs tokio 29.4 24.5 vs 29.4 yield 4x 44.6 vs tokio 49.4 40.8 vs 47.3 So the commit cost 3.6x on the workload `par_for` actually produces and bought nothing an application can see. A micro benchmark improvement that regresses the macro one is not an improvement. This does not close the whole gap: `main` runs this workload at 9.5 ms, so 15.9 is still 1.7x off. The rest arrives with 22b5975, the single-injector shape, and is a separate question from this revert.
Bisected. Two commits, and the worst is the last one.Full curve, Commits 10 and 11 were clawing it back, 37 to 23 to 15. Commit 12 undid all of that and went past where it started. What I have done: reverted 12
3.6x. Null arm separates by 1%, sequential arm unchanged throughout, load 4.5 to 6.3 on 16 cores. The case that commit was written for does not lose anything. Its target is 100,000 trivial tasks from one producer, which is measured through nagoya, and nagoya stays ahead of tokio on wall clock either way: Its own message says batching "only looked free". On the workload What is still open: commit 5
Two ways to read it, and I do not have the evidence to pick:
Worth settling before this merges, because nagoya's |
Every wake went through the one injector every worker contends on, plus a wake for a worker that in the common case was already awake, and the job then came back on whichever worker won the race rather than the one holding its cache lines. There was no other route: the private `mine` deque lives on the run loop's stack, so nothing outside `run` can reach it. That is the whole cost for a task that wakes itself. `yield_now` does exactly that, so a task yielding in a retry loop paid a round trip through the shared queue per iteration, which is the shape a contended storage engine has. `submit_local` gives each worker an inbox it drains before its sharing deque. A `SegQueue` rather than that deque, because `run` moves the deque out and owns it for the whole loop, while this has to be reachable while that is in flight. The trade is explicit: a job in an inbox is not stealable until its worker promotes it, so this is only for work that belongs on this worker. A sleeping worker is still woken, so it stays correct when called from another thread, and the sleep path checks the inbox alongside the injector under the same publish-then-read-the-bitmap ordering that already protects the injector.
Running an inbox job directly looks cheaper and costs far more: nothing outside the worker can reach a job in the inbox, so a worker that keeps waking its own tasks keeps all of them and the pool stops balancing. Measured on a read-only workload with eight client tasks over sixteen workers, that halved throughput and made it bimodal, tasks pinning to whichever worker first woke them while other workers idled. Through `mine` they are ordinary local work: still LIFO, so a task that just yielded runs next and keeps its cache lines, but now subject to the promotion that makes local work stealable on the heartbeat.
A self-wake almost always wants to run next, and saying so through the inbox costs a SegQueue push and pop, an MPMC structure priced for many producers. The slot is one uncontended swap, which is what tokio's LIFO slot is for. It holds one job; a second self-wake displaces the first into the inbox behind it, so nothing is lost and the slot never becomes a queue. Runs from it are capped at 32 consecutive, or two tasks ping-ponging through the slot would starve everything else this worker holds, including work it has already promised to share. The sleep path checks it alongside the inbox and the injector. Written without a let chain: this crate supports Rust 1.60.
Two changes to what a worker does when it is not running its own work. `injector_batch` back to 1. It was 8, from a sweep over 100,000 short independent tasks that found 8, 32 and 128 within noise. That sweep is still true and no longer decisive: `submit_local` changed what a batch costs, because a job that wakes itself now stays on the worker that ran it. A worker taking 8 long-lived tasks off the injector no longer merely holds them to its next heartbeat, it keeps them. Measured on 16 client tasks over 16 workers, a few workers held everything and the rest idled: 8.9M ops/s against tokio's 21.6M, and 15.2M at a batch of 1. `searching` caps concurrent hunters at half the pool, which is tokio's rule and for tokio's reason: a steal probe is a compare-exchange against the victim's deque, so a worker looking for work slows the worker that has it, and that cost lands inside the busy worker's own pops where no profile attributes it to stealing. Together with the inbox and the LIFO slot, the update-heavy workloads go from 41% behind tokio to 6% ahead, and read-latest from 174% to 245% ahead. A residual remains: read-only still lands occasionally in a slow mode, 2.7M against 16.2M on an otherwise identical run, which looks like the initial spread of tasks off the injector and is not addressed here.
…leeps The sleep path is a Dekker pair: a worker stores its sleeping bit and then reads its slot and inbox, and submit_local stores the job and then reads the bit. One side sees the other only if both pairs are separated by a full barrier. The worker's fetch_or is SeqCst and is one; on the submit side the job went into a spin::Mutex, whose release is a store-release and is not, so both could miss. Both did. Caught in a profile of a slow run with all sixteen workers parked in __ulock_wait while the run still had work outstanding, at a sixth of normal throughput. This does not close the mode entirely: two runs in ten still land there, so at least one more lost-wakeup path remains.
Whether submit_local keeps work on the calling worker is now a Tuning field rather than a decision the pool makes for everyone. Measured at eight threads against a tokio-driven build of the same storage engine: workload local injector 50% update +6.4% -40.2% read-modify-write +2.0% -38.4% 95% read / 5% update -19.2% +74.4% 95% read / 5% insert +137.1% +321.6% Contended work wants the handoff kept warm: its wakes are a chain and the successor wants the rows the releasing worker just touched. Sparse work wants the opposite, because a rare wake queued behind a busy worker waits while other workers sleep. `true` is the default on worst case, one workload 19% behind against two at about 40% the other way. Three static rules were tried and each reproduced one column exactly rather than splitting them: gating on whether the wake was a task waking itself, gating on whether any worker was idle, and letting a thief take the slot on second sight. The distinguishing signal is contention, which the pool cannot see and the caller can.
The scheduler experiments in this branch produced a trade surface, not a winner. Two settings each win a workload shape decisively and lose the other, measured on YCSB at eight threads against a tokio-driven build of the same storage engine: shape locality spread 50% update +6.4% -40.2% read-modify-write +2.0% -38.4% 95% read / 5% update -19.2% +74.4% 95% read / 5% insert +137.1% +321.6% So they are constructors with the numbers on them: Tuning::locality for work whose wakes are a chain, Tuning::spread for work whose wakes are independent, Tuning::throughput for a firehose from outside the pool. locality stays the default on worst case. throughput carries the warning the measurements earned: a large injector batch combined with local wakes makes a worker keep every long-lived task it takes, and a read-only run went 15.2M to 8.9M ops/s against tokio's 21.6M.
Adding `local_wakes` broke four downstream benchmarks that built `Tuning`
from a struct literal, and the runtime backend work has more fields
coming. `#[non_exhaustive]` makes the next one a non-event.
That alone leaves legitimate callers with no way to build one, and the
functional-update form does not rescue them: `Tuning { injector_batch: 4,
..Tuning::default() }` is E0639 from another crate, the same as a bare
literal. So the presets grow `with_*` methods, and a caller starts from
`locality`, `spread` or `throughput` and overrides what it needs.
The fields stay public to read, because callers derive values from the
defaults rather than restating them.
Minor rather than patch: Tuning is now non_exhaustive, so a downstream struct literal no longer compiles, and functional-update syntax does not rescue it either. That is a breaking change and 0.x treats a minor bump as the incompatible one.
1 participant
submittook the sleeper word exclusively, with an RMW, on every task. That word is shared by every worker parking and waking, so the wake bookkeeping serialised the pool and the cost grew with worker count rather than with work.One line: read first, and only RMW if somebody is actually parked. In the throughput case the bitmap is zero and a load answers the question without taking the line. The load stays
SeqCstfor exactly the reason the RMW was — it must be ordered against publishing the task, or a worker that sets its bit and then checks its intake could be missed.What this does not fix, and where it is
Eight workers is unchanged, so there is a second bottleneck and this is not it.
It is not in
submit. Submitting into idle intakes with nobody running is 2–3 ns and does not degrade with worker count:So the collapse is on the worker side — draining, stealing, waking. The candidate is
steal_once's exhaustive sweep: every idle worker probes every victim, so eight workers running trivial tasks can put seven sweepers onto seven queues simultaneously, each probe a contended RMW on a victim's head. The sweep is exhaustive for a sound reason, argued where it is written — a bounded number of draws does not bound discovery, and a miss means parking with work available. But nothing bounds how many workers sweep at once, and that is the part tokio caps, at half its workers.That is a separate change with a real design question in it, so it is not in this PR.
Where these numbers come from
A scratch runtime (
~/code/nagoya) built onfanoutwas 5x slower than tokio at eight workers. Splitting the arms showed the executor on top added 7% and the pool underneath was the rest, which is what pointed here. At one and two workers this pool is 1.7x faster than tokio; the problem is entirely how it scales.