Repository navigation
perf(reactor): bound epoll/kqueue rounds with the drain-chunk cadence - #378
Merged
Merged
Conversation
The io_uring dispatch loop drains foreign-thread posts (cwist_reactor_post) every CWIST_REACTOR_DRAIN_CHUNK callbacks so a completion's tail latency stops scaling with how many other connections happened to be ready in the same wake. The epoll path -- now the default backend under seccomp/no-uring (RX-epoll fast path) -- drained posts only once at the top of each round and then ran the full epoll_wait array (up to 1024 events) with nothing in between, so a post arriving mid-round waited behind every remaining event's handler. Same shape on the kqueue branch. Port the same cadence to both loops: count every dispatched callback and call reactor_drain_posts() every N events, guarded by i + 1 < n so the last event is never followed by a redundant drain (mirrors the io_uring head != tail guard). CWIST_REACTOR_DRAIN_CHUNK=0 restores the legacy single-drain-at-top behavior; the knob default (64, or 8 under CWIST_PROFILE) is unchanged and now applies on every backend. A/B (tests/bench_cooperative_queuing_epoll, BATCH=512 x 80us fake handlers, post lands mid-batch; baseline = HEAD reactor.c, 5 interleaved rounds x 10 runs, plus a local re-run): epoll backend, chunk=0 (legacy): mean 41.6-42.2 ms, max 46.6 ms epoll backend, chunk=8 (default): mean 0.43-0.49 ms (~90-99x) No regression: io_uring-backend bench unchanged (loop untouched), and cwist_bench + wrk -t8 -c256 on the forced-epoll plain request path shows identical throughput/latency across legs (that workload posts nothing foreign-thread, and the added check short-circuits on drain_chunk == 0). Adds the epoll-backend twin bench (bench_cooperative_queuing_epoll, -CWIST_BENCH_EPOLL pins CWIST_REACTOR_BACKEND=epoll), the io_uring twin now pins its backend symmetrically, and a path-filtered .github/workflows/perf-epoll-drain.yml running the env-only A/B (chunk=0 vs 8, 5 interleaved rounds) on push to dev. Docs: CWIST_REACTOR_DRAIN_CHUNK and the cooperative-queuing page now state the cadence applies on all backends in both en/ko.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
CWIST_REACTOR_DRAIN_CHUNK) from the io_uring dispatch loop to the epoll and kqueue loops incwist_reactor_run().cwist_reactor_post) only once at the top of each round, then ran the fullepoll_waitarray (up to 1024 events) with no drain point in between. A background/foreign-thread completion arriving mid-round waited behind every remaining event's handler, so its tail latency scaled with the whole same-wake batch — the exact P99.999 mechanism issue [BUG][Plan/CWIST 3.5] Enhance bad P99.999 latency #25 fixed for io_uring, un-fixed here.A/B result (meaningful ✅)
Deterministic harness
bench_cooperative_queuing_epoll(512 pipes readable in one round × 80 µs fake handler each, one foreign-thread post mid-batch; baseline binary built fromHEAD, 5 interleaved rounds × 10 runs, re-verified locally):CWIST_REACTOR_DRAIN_CHUNK=8(profile default)≈ 90–99× tail improvement, matching the theoretical bound (chunk × handler cost).
chunk=0reproduces baseline exactly (legacy opt-out preserved).No regressions: io_uring-backend bench unchanged (that loop is untouched);
cwist_bench+wrk -t8 -c256on the forced-epoll plain request path shows identical throughput/latency across legs (no foreign-thread posts in that workload, and the added check short-circuits ondrain_chunk == 0).Changes
src/sys/io/reactor.c: epoll + kqueue loops drain posts every N dispatched callbacks (i + 1 < nguard mirrors the io_uringhead != tailguard);reactor_drain_chunk()hoisted out of the__linux__guard.tests/bench_cooperative_queuing.c+Makefile: newbench_cooperative_queuing_epolltarget (-DCWIST_BENCH_EPOLLpinsCWIST_REACTOR_BACKEND=epoll); the io_uring twin now pins its backend symmetrically..github/workflows/perf-epoll-drain.yml: path-filtered A/B scenario — same binary,CWIST_REACTOR_DRAIN_CHUNK=0vs8, 5 interleaved rounds on push to dev (+ io_uring no-regression leg), artifacts uploaded.CWIST_REACTOR_DRAIN_CHUNKand the cooperative-queuing page now state the cadence applies on all three backends.Test plan
make bench_cooperative_queuing_epollA/B rounds (numbers above)test_reactor_drain_chunk,test_reactor_wake,test_reactor_timer(default + forced epoll),test_rx_epoll,test_http_fairness,test_async_defer— passscripts/ci/check_test_wiring.py,scripts/ci/check_env_knobs.py— passperf-epoll-drain.ymlworkflow runs on this PR's merge to dev (path-filtered to the workflow file itself)