Repository navigation
Serve on every shard: one reactor and SO_REUSEPORT listener each - #58
Merged
Merged
Conversation
The server ran on one thread whatever WS_SHARDS said, and warned about it at every boot, because a nagoya listener could not be shared between reactors. Consumers set WS_SHARDS=1 to hide the warning. nagoya 0.1.13 adds bind_shared (SO_REUSEPORT). listen now starts a thread per extra shard, each with its own local reactor and its own listener on the one port, so a connection is accepted, registered and served by the same reactor and never crosses a thread. The first bind picks the address and the others take it exactly, which matters when the port was zero. The server and the connection table are prepared once and shared, so the toolbox reaches a connection on any shard. The first shard owns the stop, whether a signal or the caller's, and cancels the rest; listen returns once all have joined. With one shard nothing changes. tests/sharded_listen.rs drives three shards over TCP: every client completes a request, and one stop ends them all. Release as 3.3.1, needing nagoya ^0.1.13.
pathscale
pushed a commit
to pathscale/api.support.cafe
that referenced
this pull request
Sep 23, 2026
It hid endpoint-libs' warning that it served one thread whatever was asked. endpoint-libs 3.3.1 (pathscale/endpoint-libs#58, on nagoya 0.1.13) serves every shard on its own reactor with an SO_REUSEPORT listener, so the default, the CPU count, is what runs.
pathscale
pushed a commit
to pathscale/api.support.cafe
that referenced
this pull request
Sep 24, 2026
It hid endpoint-libs' warning that it served one thread whatever was asked. endpoint-libs 3.3.1 (pathscale/endpoint-libs#58, on nagoya 0.1.13) serves every shard on its own reactor with an SO_REUSEPORT listener, so the default, the CPU count, is what runs.
pathscale
pushed a commit
to pathscale/api.support.cafe
that referenced
this pull request
Sep 24, 2026
It hid endpoint-libs' warning that it served one thread whatever was asked. endpoint-libs 3.3.1 (pathscale/endpoint-libs#58, on nagoya 0.1.13) serves every shard on its own reactor with an SO_REUSEPORT listener, so the default, the CPU count, is what runs.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes the root cause behind
WS_SHARDS=1in consumers' fly.toml. The server ran on one thread whateverWS_SHARDSsaid, and warned at every boot that a nagoya listener couldn't be shared between reactors.Depends on pathscale/nagoya#16 (0.1.13,
bind_shared/SO_REUSEPORT).listen_on: with more than one shard, it binds the first listener shared, then starts aws-shard-Nthread for each extra shard. Each thread has its ownReactor::localand its ownSO_REUSEPORTlistener on the concrete address, so a connection is accepted, registered and served on one reactor and never crosses a thread.preparesets up the server (Arc<Self>) and the connection table once, shared by every shard, so the toolbox reaches a connection whichever shard holds it.serve_shardis the old accept loop.Shutdown.listenreturns after all shards have joined.TcpListener::bind_shared/bind_any_shared. The docs onlistenand the accept loop are updated.bind_sharedrequires it.Tested: new
tests/sharded_listen.rs, aWS_SHARDS=3server over TCP. 12 clients each completed an echo request, served by the first shard andws-shard-2(macOS doesn't balance; Linux does), and one stop ended all shards withlisten_untilreturning Ok. Full suite: 151 pass. Clippy is clean with--all-features. Built against the local nagoya branch.Once both are released, consumers can drop
WS_SHARDSfrom fly.toml.