Skip to content

perf(reactor): bound epoll/kqueue rounds with the drain-chunk cadence - #378

Merged
gg582 merged 1 commit into
devfrom
perf/epoll-drain-chunk
Oct 15, 2026
Merged

gg582 merged 1 commit into
devfrom
perf/epoll-drain-chunk

Conversation

@gg582

@gg582 gg582 commented Oct 15, 2026

Copy link
Copy Markdown
Member

Summary

  • What: ports the existing cooperative-queuing drain cadence (CWIST_REACTOR_DRAIN_CHUNK) from the io_uring dispatch loop to the epoll and kqueue loops in cwist_reactor_run().
  • Why: the epoll loop (now the default backend under seccomp/no-uring via the RX-epoll fast path) drained foreign-thread posts (cwist_reactor_post) only once at the top of each round, then ran the full epoll_wait array (up to 1024 events) with no drain point in between. A background/foreign-thread completion arriving mid-round waited behind every remaining event's handler, so its tail latency scaled with the whole same-wake batch — the exact P99.999 mechanism issue [BUG][Plan/CWIST 3.5] Enhance bad P99.999 latency #25 fixed for io_uring, un-fixed here.

A/B result (meaningful ✅)

Deterministic harness bench_cooperative_queuing_epoll (512 pipes readable in one round × 80 µs fake handler each, one foreign-thread post mid-batch; baseline binary built from HEAD, 5 interleaved rounds × 10 runs, re-verified locally):

epoll backend post-to-callback latency
baseline (HEAD, no mid-round drain) mean 41.6–42.2 ms, max 46.6 ms
CWIST_REACTOR_DRAIN_CHUNK=8 (profile default) mean 0.43–0.49 ms

≈ 90–99× tail improvement, matching the theoretical bound (chunk × handler cost). chunk=0 reproduces baseline exactly (legacy opt-out preserved).

No regressions: io_uring-backend bench unchanged (that loop is untouched); cwist_bench + wrk -t8 -c256 on the forced-epoll plain request path shows identical throughput/latency across legs (no foreign-thread posts in that workload, and the added check short-circuits on drain_chunk == 0).

Changes

  • src/sys/io/reactor.c: epoll + kqueue loops drain posts every N dispatched callbacks (i + 1 < n guard mirrors the io_uring head != tail guard); reactor_drain_chunk() hoisted out of the __linux__ guard.
  • tests/bench_cooperative_queuing.c + Makefile: new bench_cooperative_queuing_epoll target (-DCWIST_BENCH_EPOLL pins CWIST_REACTOR_BACKEND=epoll); the io_uring twin now pins its backend symmetrically.
  • .github/workflows/perf-epoll-drain.yml: path-filtered A/B scenario — same binary, CWIST_REACTOR_DRAIN_CHUNK=0 vs 8, 5 interleaved rounds on push to dev (+ io_uring no-regression leg), artifacts uploaded.
  • Docs (en + ko): CWIST_REACTOR_DRAIN_CHUNK and the cooperative-queuing page now state the cadence applies on all three backends.

Test plan

  • make bench_cooperative_queuing_epoll A/B rounds (numbers above)
  • test_reactor_drain_chunk, test_reactor_wake, test_reactor_timer (default + forced epoll), test_rx_epoll, test_http_fairness, test_async_defer — pass
  • scripts/ci/check_test_wiring.py, scripts/ci/check_env_knobs.py — pass
  • New perf-epoll-drain.yml workflow runs on this PR's merge to dev (path-filtered to the workflow file itself)

The io_uring dispatch loop drains foreign-thread posts (cwist_reactor_post)
every CWIST_REACTOR_DRAIN_CHUNK callbacks so a completion's tail latency
stops scaling with how many other connections happened to be ready in the
same wake. The epoll path -- now the default backend under seccomp/no-uring
(RX-epoll fast path) -- drained posts only once at the top of each round
and then ran the full epoll_wait array (up to 1024 events) with nothing in
between, so a post arriving mid-round waited behind every remaining event's
handler. Same shape on the kqueue branch.

Port the same cadence to both loops: count every dispatched callback and
call reactor_drain_posts() every N events, guarded by i + 1 < n so the
last event is never followed by a redundant drain (mirrors the io_uring
head != tail guard). CWIST_REACTOR_DRAIN_CHUNK=0 restores the legacy
single-drain-at-top behavior; the knob default (64, or 8 under
CWIST_PROFILE) is unchanged and now applies on every backend.

A/B (tests/bench_cooperative_queuing_epoll, BATCH=512 x 80us fake
handlers, post lands mid-batch; baseline = HEAD reactor.c, 5 interleaved
rounds x 10 runs, plus a local re-run):

  epoll backend, chunk=0 (legacy): mean 41.6-42.2 ms, max 46.6 ms
  epoll backend, chunk=8 (default): mean 0.43-0.49 ms  (~90-99x)

No regression: io_uring-backend bench unchanged (loop untouched), and
cwist_bench + wrk -t8 -c256 on the forced-epoll plain request path shows
identical throughput/latency across legs (that workload posts nothing
foreign-thread, and the added check short-circuits on drain_chunk == 0).

Adds the epoll-backend twin bench (bench_cooperative_queuing_epoll,
-CWIST_BENCH_EPOLL pins CWIST_REACTOR_BACKEND=epoll), the io_uring twin
now pins its backend symmetrically, and a path-filtered
.github/workflows/perf-epoll-drain.yml running the env-only A/B
(chunk=0 vs 8, 5 interleaved rounds) on push to dev. Docs:
CWIST_REACTOR_DRAIN_CHUNK and the cooperative-queuing page now state the
cadence applies on all backends in both en/ko.
@gg582
gg582 merged commit b5ac881 into dev Oct 15, 2026
23 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant