Timers, and a file abstraction that works without an operating system by pathscale · Pull Request #1 · pathscale/nagoya

pathscale · 2026-09-09T03:58:54Z

Two things WorkTable needs before it can stop naming tokio, and one correction.

Measured by pathscale/perf-benchmarks#2.

Timers

sleep and timeout, plus the registry and driver behind them. 12 of WorkTable's 57 production tokio uses are time::sleep and time::timeout, and nothing in the ecosystem provides them without an operating system.

Not a timed park on Host. StdHost parks on atomic_wait::wait, which has no timeout, so the trait method would have nothing to call. Buying one means trading a futex for a condition variable on the parking path, a cost every pool would pay whether or not it used a timer. So the registry sits beside the pool, waking goes through the ordinary Waker and back through the ordinary submit path, and the pool learns nothing about time.

Both gates measured

Unused costs nothing. submit_only, medians of 9 interleaved:

tasks/s	before	this	ratio
executor.spawn	54,367,534	54,725,791	1.007x
pool.submit_fn (identical code in both arms)	46,591,983	47,890,817	1.028x

The floor is 2.8%, measured on paths byte-identical between the arms. executor.spawn moved 0.7%.

Used is competitive. Against tokio's wheel, lower is better: sleep 1ms p50 0.25x, sleep 10ms p50 0.76x, 1000 concurrent p50 ~1.0x, arm-and-cancel 0.95x.

No total regression. The nagoya-using acceptance suite: layout-iter 0.996x and source-to-memory 0.985x against a null of 0.989x and 0.995x.

The benchmark found two real defects in the first version: a slot allocated for timers that timeout never arms, and a past-deadline sleep arming anyway and waiting a full driver round trip to learn what the clock could have said on the spot. That one took the concurrent p99 from 9167 us to 3665.

Files

The crate documentation says there will be no I/O driver and that still holds: an epoll or io_uring reactor is the part that needs an operating system. A file abstraction is not a reactor. It is traits plus one implementation for hosts that have std, the same shape Host already has for parking.

It exists because the operations fit in no existing trait, so every storage crate reinvents them. futures-io covers reading, writing and seeking one file; nothing covers sync_all, set_len, metadata, or opening one. WorkTable had these in a private module and data_bucket could not name them at all, so two halves of one storage engine disagreed about what a file was.

The correction worth reading

The first version supertraited futures-io. That does not work and looks like it does. Every one of that crate's traits sits behind its own std feature. Turn it off and it compiles to nothing: no AsyncRead, no AsyncWrite, no AsyncSeek. They take std::io::Error and IoSlice, so there was nowhere else for them to go.

Checking it the obvious way gives the wrong answer, which is how it got past me. The crate still builds for aarch64-unknown-none; it is the exports that vanish, so a probe that only compiles it reports success. Anything written on those three is a std trait wearing a portable name, including data_bucket's AsyncFile.

So Read, Write and Seek are declared here with a local Error and a local SeekFrom. Under std the ecosystems still meet, through Compat for any futures-io type and HostFile for a plain std::fs::File.

The guard is mod portable: a File implemented over a byte array, touching no std, calling every method including the provided ones. Compiling the crate is not evidence; implementing the trait is. Confirmed separately by an external #![no_std] crate for aarch64-unknown-none that does the same against default-features = false.

The host implementations are async fn rather than a computed Result handed to an async move block. The second form runs the syscall when the method is called, so a future built and dropped unpolled would still have written.

Verified

Bare metal aarch64-unknown-none builds, clippy clean in both configs, 28 tests.

🤖 Generated with Claude Code

The obvious design is a timed park on `Host`, so an idle pool wakes exactly when the next timer is due. It cannot be done: `StdHost` parks on `atomic_wait::wait`, which has no timeout, so the trait method would have nothing to call. Buying one means trading a futex for a condition variable on the parking path, a cost every pool would pay whether or not it used a timer. So the registry sits beside the pool. It holds deadlines, something drives it, and waking goes through the ordinary `Waker` back through the ordinary submit path. The pool learns nothing about time. With `std` the driver is a thread, started the first time anything sleeps and never before. Without `std` there is no thread and no clock, so the consumer supplies both: `set_clock` once, then `poll` from a tick. That split is why the registry and the driver are separate things in the file. Cancellation disarms rather than unregisters. Removing a heap entry costs a linear scan under the lock; leaving it costs one wake of nothing, and it is gone at its deadline anyway. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Two costs the benchmark found, both paid by timers that never needed to exist. A `Sleep` allocated its slot in `sleep()`. But `timeout` polls its inner future first and returns without touching the sleep whenever that future is ready, which in production is nearly every call, so the allocation was charged to timers that were never armed. Deferring it to the first poll that finds the deadline still ahead took arm-and-cancel from 65 ns to parity with tokio's 27. A `Sleep` whose deadline had already passed still armed, then waited for the driver to tell it what the clock could have said on the spot. Awaiting a batch of timers in order is exactly that case, and it cost a round trip each: 5495 us of overshoot on a 10 ms deadline against tokio's 3844. Checking the clock first inverts it to 3374 against 4903. `now_ns` installed the default clock ahead of reading it, putting a second `OnceLock` on the path of every sleep to answer a question the first call in the process had already settled. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

The crate documentation says there will be no I/O driver, and that still holds: an epoll or io_uring reactor is the part that needs an operating system. A file abstraction is not a reactor. It is a trait plus one implementation for hosts that have `std`, the same shape `Host` already has for parking. It exists because the operations fit in no existing trait, so every storage crate reinvents them. `futures-io` covers reading, writing and seeking one file and nothing covers `sync_all`, `set_len`, `metadata`, or opening one. WorkTable had these in a private module and `data_bucket` could not name them at all, so two halves of one storage engine disagreed about what a file was. **The module needs `std`, and the reason is not the one I expected.** `futures-io` puts every one of its traits behind its own `std` feature: turn it off and the crate compiles to nothing. The traits take `std::io::Error` and `IoSlice`, so there was nowhere else for them to go. This is easy to check the wrong way, because the crate still *builds* for a bare-metal target and only its exports vanish, so a probe that compiles it reports success. Any `File` trait written on those three is a `std` trait wearing a portable name, and that includes `data_bucket`'s `AsyncFile`. A file trait that works without an operating system needs its own read, write and seek over an error type that is not `std::io::Error`. That is deliberate design and it is not done here. What is here is the shared home, so the two crates stop disagreeing while it gets done. `AllowStdIo` rather than `async-fs`, because `async-fs` keeps a user-space write buffer and flushes it on drop, best effort and silently; its own documentation says errors detected on closing are ignored. A storage engine that drops a handle and reads the file back loses its last writes and is told nothing. Bare metal still builds, clippy is clean in both configs, 25 tests. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

The first version supertraited `futures-io`, which does not work and looks like it does. **Every one of that crate's traits sits behind its own `std` feature.** Turn it off and it compiles to nothing: no `AsyncRead`, no `AsyncWrite`, no `AsyncSeek`. They take `std::io::Error` and `IoSlice`, so there was nowhere else for them to go. Checking it the obvious way gives the wrong answer, which is how it got past me. The crate still *builds* for `aarch64-unknown-none`; it is the exports that vanish, so a probe that only compiles it reports success. Anything written on those three is a `std` trait wearing a portable name, and that includes `data_bucket`'s `AsyncFile`. So `Read`, `Write` and `Seek` are declared here, the same shape with two differences that matter: the error is a local `Error`, and the origin is a local `SeekFrom`. Under `std` the two ecosystems still meet, through `Compat` for any `futures-io` type and `HostFile` for a plain `std::fs::File`. The guard is `mod portable`: a `File` implemented over a byte array, touching no `std` and calling every method including the provided ones. Compiling the crate for a bare-metal target is not evidence, as above; implementing the trait is. Confirmed separately by an external `#![no_std]` crate for `aarch64-unknown-none` that does the same and links only `default-features = false`. The host implementations are `async fn` rather than a computed `Result` handed to an `async move` block. The second form runs the syscall when the method is called, so a future built and dropped unpolled would still have written. Bare metal builds, clippy is clean in both configs, 28 tests. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

The module was written as though the consumer had no operating system. Nothing we ship is like that: it is Linux, macOS and Windows, and `no_std` here means not linking `std`, not running without a platform. The framing invented constraints that do not exist and made the reason for the traits harder to see. The reason is unchanged and does not need bare metal to justify it: `futures-io` puts every one of its traits behind its own `std` feature, so with that off it exports nothing, and anything written on top of them is a `std` trait wearing a portable name.

`[patch.crates-io]` named `/Users/revenge/code/ps-st3`, so this branch resolved on that machine and failed everywhere else: a reviewer, another clone, or CI got a resolution error rather than a build. The same pin as a git branch reaches the same commit and travels. The comment also claimed the patch only affected benchmark accuracy; it is a compile dependency, because `Act`, `Job`, `submit_fn` and `submit_job` are not in the published 0.5.0.

pathscale · 2026-09-09T08:42:25Z

QA, 9 September

cargo test                        14 + 12 + 2 passed
cargo check --no-default-features clean

Six commits, linear, no merge commits.

One fix, and it was a real blocker

This branch committed [patch.crates-io] ps-st3 = { path = "/Users/revenge/code/ps-st3" }, an absolute path on one machine. It built there and nowhere else: any other clone, any reviewer, any CI runner got a resolution failure rather than a build. 402d4dc pins the same commit by git branch instead, and the lock now resolves to 169865de, the head of pathscale/ps-st3#4.

The comment above it also claimed the patch only affected the accuracy of the benchmark numbers. That is wrong and worth correcting, because it reads as though the patch is droppable at a cost: Act, Job, Pool::submit_fn and Pool::submit_job do not exist in the published ps-st3 0.5.0, so this crate does not compile against it at all.

Merge order, which is not visible from the PRs

This cannot be published until ps-st3 0.5.1 is. And DataBucket now depends on nagoya, so nothing downstream can land before this one ships a version:

ps-st3 #4  ->  publish 0.5.1  ->  nagoya #1  ->  publish 0.1.0  ->  DataBucket #75  ->  publish  ->  WorkTable #102

Delete the [patch.crates-io] block and drop the version requirement to 0.5.1 once that publish happens.

Not in this PR

feat/join-scope-hooks carries one commit, e176030, whose own message says it is pending a design call. It stays parked rather than getting folded in here, since collapsing branches should not decide an open design question.

`tokio::sync` is waker-based and runtime-agnostic, so a crate could sit on this runtime's file layer, lock with tokio's `RwLock`, and pass its tests. WorkTable did exactly that. It is still the wrong dependency, and the cost is not visible from the code: `cargo tree --no-default-features -e normal -i tokio` on WorkTable showed tokio linked with every feature off, because the uses were never gated. A crate whose point is needing no runtime cannot reach into one for a lock, and `cargo check --no-default-features` passing says only that the crate's own source is clean, not its closure. So: `Notify`, `Semaphore`, `RwLock` and the owned guards, which is what was actually used and nothing more. Written down rather than discovered later: these are not fair. A woken waiter races arrivals, so a writer can be overtaken, where `tokio::sync::RwLock` is write-preferring. Nothing depends on that today. cargo test --lib 19 passed cargo check --no-default-features clean cargo clippy --all-targets clean

`Debug` and `Default`, owned try-locks, and `Notified::enable`. The last is the interesting one: tokio needs it because a `Notified` must be linked into the waiter list before the caller reads state, or a notification in between is lost. Here both notify paths are already durable across that window, a permit for `notify_one` and a generation snapshot for `notify_waiters`, so the call is a check rather than a registration and the race it guards cannot happen.

`Executor` takes a pool whose threads the caller provides, and that stays the rule: it is what lets this crate run without an operating system. It pushes a real problem onto some callers though. A storage engine's vacuum sweep and persistence writer are intrinsic to the engine, not to its user, and no caller should have to hand over a thread for work it does not know exists. WorkTable's answer was `tokio::spawn`, which works because tokio has an implicit global runtime to find, and that one call put the whole of tokio into a no_std dependency graph. This is the explicit version of what was being got implicitly. It owns threads, needs `std`, and says so in its name and its gate. Without `std` it does not exist, and an engine that wanted a background worker has to say what it does instead of pretending the problem is absent. cargo test --lib 20 passed cargo check --no-default-features clean, module absent cargo clippy --all-targets clean

A consumer holding one in a `#[derive(Debug)]` struct could not derive. Hand-written rather than derived so the impl does not demand `T: Debug`: the task's output type is the thing that consumer cannot name yet.

WorkTable uses one as a one-shot signal: `add_permits` says an event happened, the waiter takes exactly one and `forget`s it, so the count tracks events rather than returning to a fixed number of slots. Neither half existed here. `add_permits` wakes one waiter per permit rather than one and letting it cascade, because a woken task takes a single permit and the rest would sleep through permits already available. Debug for both, for the same reason JoinHandle needed it.

A consumer with intrinsic background work had no way to run it without either demanding a thread from its own caller or reaching for tokio's implicit global. It reached for the global, which is how a whole runtime entered a no_std dependency graph unremarked. The fix cannot be "each crate keeps a OnceLock<Runtime>": two such crates in one binary start two pools, and neither can steal the other's idle threads. One pool per process is the answer to that. This is the same shape as the global it replaces, with the difference that matters: it is behind `std`, so a --no-default-features build cannot reach it and cannot silently acquire threads. Sized from available_parallelism, which is what tokio::spawn was already giving these callers, so the swap stays measurable.

A benchmark that spawns n client tasks and times them has to release them together, or the first task's warm-up is measured against the last's steady state. That is the whole reason the consumer was still taking tokio::sync. Generation-counted, so a task slow to be scheduled after a release cannot rejoin the next rendezvous and hang, and it registers with the notify before re-reading the generation because notify_waiters retains no permit. n of 0 or 1 never blocks, which is what a single-threaded run wants without a special case at the call site. wait() returns true for exactly one waiter per rendezvous, so a caller can elect a task for once-per-round work without a second primitive.

A wake happening inside a pool worker now goes to that worker's inbox rather than the shared injector: no contention on the queue every worker drains, no wake for a thread that is already running, and the task comes back where its cache lines are. `yield_now` is the case that pays most, because it wakes itself, so a task yielding in a retry loop was making a round trip through the shared queue every iteration. The current worker is tracked as a raw `Pool` pointer, not an id, so a process with two pools cannot hand one pool's job to the other's worker; it is compared and never dereferenced. Off a worker thread, and without `std`, this is the old path, which is what a wake from a timer or an application thread wants: no locality to keep, and the injector is where anyone can find the job.

… nothing

The patch block was a compile dependency, not a preference: Act, Job, submit_fn and submit_job do not exist in ps-st3 0.5.0. They exist in 0.6.0, so the requirement becomes a caret and the block goes. A consumer of this crate could never see that patch anyway, since [patch] only takes effect from a workspace root, which is what made the block a problem rather than a convenience. Tuning is non_exhaustive from 0.6, so the two examples that built one as a struct literal now build it from a preset and the with_* methods. Cargo.lock was tracked and gitignored at the same time, so the ignore never took effect. Untracked: this is a library, and a committed lock file hides fixes published in the crates we control. Adds the release workflow this repository did not have. It watches master rather than main, because that is this repository's default branch.

target-ab is an A/B benchmark build tree, 106 MiB of it, and nothing ignored it. cargo packages untracked files that git does not ignore, so the crate went out as 559 files and 104 MiB against a 10 MiB registry limit, and the upload failed as an HTTP/2 stream error rather than as anything that named the size.

target-ab is an A/B benchmark target directory and 532 of its files were tracked, so .gitignore could not touch them and cargo packaged every one. The crate went to crates.io as 559 files and 104.4 MiB against a 10 MiB limit, and the registry refused it as an HTTP/2 stream error that named nothing. The blobs stay in history; this only stops them shipping.

meh and others added 4 commits September 9, 2026 09:16

pathscale changed the title ~~Add sleep and timeout, driven beside the pool~~ Timers, and a file abstraction that works without an operating system Sep 9, 2026

meh added 3 commits September 9, 2026 14:53

Give the iterative split its real reason, not a bare-metal one

b17fe1c

pathscale force-pushed the feat/timers branch from 3a2d8a4 to b48816a Compare September 9, 2026 11:05

meh and others added 11 commits September 9, 2026 18:07

Let a JoinHandle be printed

8e4ac5f

A consumer holding one in a `#[derive(Debug)]` struct could not derive. Hand-written rather than derived so the impl does not demand `T: Debug`: the task's output type is the thing that consumer cannot name yet.

Make that a plain comment: a doc comment on thread_local! attaches to…

1dfd742

… nothing

pathscale merged commit c308d1f into master Sep 9, 2026

pathscale deleted the feat/timers branch September 9, 2026 20:54

Navigation Menu

Repository navigation

Timers, and a file abstraction that works without an operating system - #1

Timers, and a file abstraction that works without an operating system#1
pathscale merged 19 commits into
masterfrom
feat/timers

pathscale commented Sep 9, 2026 •

edited

Loading

Uh oh!

pathscale commented Sep 9, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

pathscale commented Sep 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Timers

Both gates measured

Files

The correction worth reading

Verified

Uh oh!

pathscale commented Sep 9, 2026

QA, 9 September

One fix, and it was a real blocker

Merge order, which is not visible from the PRs

Not in this PR

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

pathscale commented Sep 9, 2026 •

edited

Loading