Skip to content

epoll: the 25 ms sweep timer is armed on every loop, whatever the timeouts are (Refs #1194) - #1197

Merged
FumingPower3925 merged 5 commits into
mainfrom
epoll-1194-fine-sweep-timer
Nov 1, 2026
Merged

FumingPower3925 merged 5 commits into
mainfrom
epoll-1194-fine-sweep-timer

Conversation

@FumingPower3925

@FumingPower3925 FumingPower3925 commented Oct 31, 2026 •

Copy link
Copy Markdown
Contributor

Refs #1194

With ReadHeaderTimeout off (-1) the epoll engine had no sweep timer, so IdleTimeout, ReadTimeout and WriteTimeout (and the closing drain, the HTTP/2 deadlines and a detached connection's idle deadline, all of which ride the sweep) were enforced seconds late. This PR arms the 25 ms timerfd on every loop, whatever the timeouts are, and stops its tick from restarting the idle backoff of the wait (a cost the straightforward fix would otherwise add; see "Cost").

Root cause (confirmed in the code)

  • loop.go created the timerfd only if l.cfg.ReadHeaderTimeout > 0, and the sweep gate was 0x1F only for ReadHeaderTimeout > 0 or a detached conn, 0x3FF otherwise.
  • With the listener open an idle loop's epoll_wait settles at 4 ms (adaptiveTimeoutMs: base 1 ms x 4), so 1024 iterations are a sweep every 4 to 5 s. That is the 3.2 to 5.8 s of the issue. Only with the listener closed (a pause, a drain) is the wait 500 ms and a sweep ~8 minutes apart: that is the "minutes late" case; a quiet loop with its listener open was 4 to 5 s late.
  • Adaptive follows epoll (its primary is the epoll engine) and shows the same numbers; io_uring has its own sweep timer and is untouched.

Measurements (docker arm64, --cpus 2, seccomp unconfined, memlock 8 MiB, golang:1.27.3, one container at a time, 20 runs per cell, 4 engines in parallel per cell, each run a fresh 2-worker engine)

Closing time in ms after the due point (idle: from the end of the response; read: from the last byte of a stalled upload; write: from the request of a 32 MiB response to a peer that reads nothing, observed with OnDisconnect):

timeout ReadHeaderTimeout main a4b10df, min / median / max this PR, min / median / max
IdleTimeout 1s -1 (off) 4412 / 4733 / 5043 1000 / 1008 / 1022
IdleTimeout 1s default (10 s) 1000 / 1010 / 1024 1001 / 1011 / 1024
IdleTimeout 5s -1 (off) 9176 / 9399 / 9766 5000 / 5012 / 5025
IdleTimeout 5s default (10 s) 5000 / 5006 / 5022 5001 / 5011 / 5023
ReadTimeout 1s -1 (off) 4841 / 4962 / 5095 1000 / 1010 / 1024
ReadTimeout 1s default (10 s) 1000 / 1013 / 1023 1000 / 1007 / 1023
ReadTimeout 5s -1 (off) 5001 / 9670 / 10327 5001 / 5012 / 5024
ReadTimeout 5s default (10 s) 5000 / 5011 / 5025 5000 / 5014 / 5023
WriteTimeout 1s -1 (off) 3990 / 4177 / 4357 2010 / 2046 / 2074
WriteTimeout 1s default (10 s) 2005 / 2028 / 2069 2007 / 2046 / 2070
WriteTimeout 5s -1 (off) 9368 / 9883 / 14211 10021 / 10041 / 10074
WriteTimeout 5s default (10 s) 10011 / 10048 / 10070 10026 / 10045 / 10074

The write rows close at two WriteTimeouts (the stall rule: the bytes the peer's kernel acknowledged in the first window pass it); that is the documented rule, only its lateness changes. Two rows of the first measurement run were dropped by a stale-port bug of the measuring hook (a negative time); the table above is the final run, 20 valid samples per cell. The measuring test is not in the PR.

Adaptive (internal/adaptive, IdleTimeout 1 s, Protocol Auto, 12 runs): main, ReadHeaderTimeout off 4884 to 5099 ms, default 1002 to 1023 ms; this PR, off 1002 to 1023 ms, default 1004 to 1021 ms.

Decision, with the numbers

Options: (1) timerfd on every loop; (2) only when one of Idle/Read/Write/ReadHeader is set; (3) a sweep period that follows the nearest pending deadline; plus (4) a coarser fine cadence (100 ms).

  • (2) is (1) for every real configuration: the defaults are 10 s / 60 s / 60 s / 600 s, so the loop is the same for anyone who has not set all four to -1, and with all four at -1 the sweep still carries the closing drain (5 s floor), the HTTP/2 deadlines and a detached conn's idle deadline set by middleware. So the predicate buys nothing and leaves a gap. Chosen: (1). The default configuration already arms this timer, so only servers with ReadHeaderTimeout: -1 change.
  • (3) needs the minimum of every conn's deadlines kept up to date on the request path, or a walk of the conns to find it. The sweep walk it would save costs 0.7 us for 256 conns, 19 to 21 us for 4096 and 65 to 77 us for 10,000 (BenchmarkSweep1194, hot cache, 0 allocs), that is 2.6 to 3.1 ms/s = 0.26 to 0.31 % of a core per loop at 40 sweeps/s with 10k idle conns. Not worth a hot-path structure.
  • (4) 100 ms: measured with the same binary and sweepFloor = 100 ms, an idle loop still wakes 466 to 484 times a second per loop (vs 665 to 690 at 25 ms), because the cost is not the timer's own tick but the backoff it restarts (below). It would also loosen the "about 30 ms" the default configuration has always had for ReadHeaderTimeout, for ~30 % fewer wakeups. Kept 25 ms, bound stated in the docs: IdleTimeout/ReadTimeout/WriteTimeout are enforced within about 30 ms of their time on epoll (measured lateness 0 to 25 ms for idle and read, 1 s and 5 s).

Cost (idle loop; /proc/self/task deltas over 15 to 20 s after 3 s of settle; 2 workers; HTTP/1, 10k idle keep-alive conns or none)

Context switches per second of each loop thread (the two highest-CPU threads that are the loops; "process" is all threads, the Go runtime's included):

build ReadHeaderTimeout 0 conns: loop threads / process 10k conns: loop threads / process
main -1 (no timerfd) 200 to 216 / 1,600 to 1,740 198 to 214 / 1,600 to 1,740
main default (timerfd) 652 to 681 / 4,590 to 4,630 668 to 684 / 8,770 to 9,550
timerfd only (first version of this PR) -1 664 to 679 / 4,550 to 4,730 664 to 688 / 9,000 to 9,310
this PR -1 216 to 235 / 1,900 to 2,290 236 to 239 / 6,150 to 6,460
this PR default 218 to 237 / 1,860 to 2,270 227 to 257 / 5,220 to 6,860
  • The straightforward fix (the timerfd alone) puts an RHT-off loop on the default configuration's 665 to 690 wakeups/s per loop: 3.3 times what it had. The 40/s per worker estimated in the iouring: the header deadline of a silent connection is a kernel timer armed at the accept; document the bound (Refs #1189) #1193 review missed that a tick is an event of epoll_wait and counted as activity, so it resets consecutiveEmpty and the wait goes back to 1 ms x 10 then 2 ms for the next ~17 rounds every 25 ms.
  • The second change makes a round whose only event is the timerfd's tick an idle round (timerOnly, 4 lines in the run loop): loop wakeups 216 to 257/s with the timer, within about 15 % of a loop without one, and the default configuration drops from ~675 to ~230/s per loop as a side effect. Closing times are unchanged (table above, "this PR" column is with it).
  • CPU: the process total (10 ms tick) is 26 to 42 ms/s in every build and arm and does not separate (one run of 127 ms/s was a host spike); each loop thread is 7 to 12 ms/s in all of them. The wakeup count is the figure that moves.
  • Sweep cost at 10k idle conns: see (3) above, 0.3 % of a core per loop, and that is for servers with ReadHeaderTimeout: -1 only; the default configuration already pays it.
  • The 5.4 to 6.9 k/s of process context switches at 10k conns with the timer (vs 1.6 k/s without) is in the runtime's own threads, not the loops (the loop threads stay at 240/s); the default configuration on main has 8.8 to 9.6 k/s there, so this PR lowers it. I did not chase it further (it scales with the loops' wakeups and the sweep's length, not with the requests).

Hot path

No per-request code changed: the diff in the run loop is the gate (a constant 0x1F for 0x3FF/0x1F by config), the epoll_wait cap and one bool per batch. BenchmarkKeepAlive1194 (8 keep-alive conns, 11 allocs/op on the client side, 834 B/op, identical on every build), test binaries built once, order main / this PR / main twin / this-PR twin x6, -test.cpu 2 -test.benchtime 3s:

ReadHeaderTimeout main median this PR median delta A/A main twin A/A this-PR twin
-1 18.7 us 18.9 us +0.8 % (pooled 12 runs each) -0.9 % +7.2 %
default 19.2 us 18.4 us -3.9 % -1.4 % +2.7 %

The host was noisy (single runs between 10.9 and 26 us), so the result is "inside the A/A spread", and the structural argument above is the actual evidence: no allocation and no syscall per request was added.

Tests

  • Commit 1 (RED on main, GREEN here): TestIdleTimeoutIsOnTimeWithoutReadHeaderTimeout1194, TestReadTimeoutIsOnTimeWithoutReadHeaderTimeout1194, TestWriteTimeoutIsOnTimeWithoutReadHeaderTimeout1194 (window [timeout - 50 ms, timeout + 200 ms]; write [1 s - 50 ms, 2.8 s + 200 ms]; measured jitter on the fixed build 0 to 25 ms, 74 ms at most with 4 engines in parallel; on main: 4.7 to 5.1 s, 4.2 to 5.1 s, 4.7 to 4.9 s), TestEveryLoopHasASweepTimer1194 (RHT off, all four off, default) and TestSweepTickAloneIsAnIdleRound1194 (the idle counter at a tick, 355 to 367 here and under 20 before). They take 1.0 + 1.0 + 2.0 + 0.05 + 1.5 s = 5.7 s per run and live in the epoll package, which the root -race step (540 s timeout, ~362 s used) runs; the stepped-clock tests that stay out of the race steps are 80 s, these are not.
  • Commit 2: the fix, the docs, and the tests that used ReadHeaderTimeout: -1 to mean "no timerfd" (TestQueuedZeroTimeoutRoundsDoNotStretchTheIdleWait885, TestResumeAcceptReachesALoopThatCannotPark886, TestBeginPauseAcceptReachesAnIdleLoop886, TestDetachedIdleDeadlineIsReapedPromptlyUnderLoad885) now disarm the timerfd by hand (TimerfdSettime zero), so they still test what they were written for; TestTimeoutSentinelReachesEngine no longer reads the sentinel through the epoll_wait cap (it checks the config fields, as before).

Verification run (docker arm64, --cpus 2, memlock 8 MiB)

  • New tests, -race -count=30 on the final commit: all pass; idle 1.001 to 1.023 s, read 1.014 to 1.022 s, write 2.114 to 2.179 s (the first -race -count=20 run, on the commit before the write window was widened, had write at 2.058 to 2.167 s and one close at 1.242 s: the peer's kernel acknowledged less than the stall rule's floor in the first window, so the rule closed at one WriteTimeout. That is why the write window is [1 s - 50 ms, 2.3 s + 200 ms]; main closes at 4.2 to 5.1 s).
  • RED on main (new tests on main's code): idle 4.5 to 5.0 s, read 4.2 to 5.1 s, write 4.7 to 4.9 s, TestEveryLoopHasASweepTimer1194 (RHT off, all off) and TestSweepTickAloneIsAnIdleRound1194 fail.
  • -race -count=1: internal/engine/epoll ok 489 s, internal/adaptive ok 168 s, root package ok 64 s, middleware/sse ok 216 to 218 s (2 of 2 on this PR, 2 of 2 on main).
  • -tags celeris_monoskew, CELERIS_MONO_SKEW=+3600s and -3600s: the whole epoll package passes in both (TestEngineTimeoutsIgnoreAStepOfTheWallClock1178, the monotest scenarios, 75.1 s and 77.3 s, included), and internal/mono passes in both.
  • One flake to look at: TestSlowReadingClientKeepsStream888/epoll/async (middleware/sse) failed once ("the stream of a client that kept reading (75776 bytes taken) was ended before 5s", after 2.3 s) in the first full-package -race run of this PR, a package run of 211 s on a loaded host. It passed 8 of 8 on this PR and 8 of 8 on main in isolation, and 2 of 2 full-package runs on this PR afterwards. I could not reproduce it. This PR changes the idle wait of every epoll loop (the timer tick no longer restarts the backoff), so the reviewer should look at the async path of that test with that in mind. A search of the repository's issues finds no earlier report of this test flaking.

Docs

One paragraph on the sweep, under ReadTimeout (epoll within about 30 ms for ReadTimeout and IdleTimeout, io_uring about 0.8 s apart on an idle worker, 0.83 s measured, about 1 s with ReadHeaderTimeout off; WriteTimeout is judged in windows that end at a sweep, see its own paragraph); IdleTimeout points to it; the detached-stream rule says 25 ms on epoll; the WriteTimeout sentence "the times hold with ReadHeaderTimeout set" is gone.

Interaction with #1193

#1193 is merged. Main is merged into this branch (no textual conflict); the semantic one is resolved in config.go: its sentence "With ReadHeaderTimeout off epoll has no such timer..." and the qualifier "when ReadHeaderTimeout is set (the default)" are gone, the two near-identical sweep paragraphs are one (under ReadTimeout), and there is one io_uring figure (the 0.8 s / 1 s of #1193, checked against worker.go's sweep comments).

Not in this PR

  • io_uring's sweep (about 0.8 to 1 s apart on an idle worker) is unchanged; the docs say so.
  • The docs site (goceleris/docs) describes ReadHeaderTimeout/idle timeouts without this bound.

Release note

epoll: with ReadHeaderTimeout disabled (-1), IdleTimeout, ReadTimeout and WriteTimeout (and the HTTP/2 deadlines, the closing drain and a detached connection's idle deadline) were checked only every few seconds: a 1 s IdleTimeout closed an idle connection at 3 to 6 s, and while the listener was closed (PauseAccept, a drain) minutes late. Every epoll worker now arms its 25 ms sweep timer whatever the timeouts are, so they are enforced within about 30 ms (1.00 to 1.03 s measured; WriteTimeout keeps its documented stall-rule windows). The timer's tick no longer restarts the adaptive wait, so an idle epoll worker with the default configuration wakes about 230 times a second instead of about 675. A server with ReadHeaderTimeout: -1 and many idle connections now pays what the default configuration always did (about 0.3% of a core per worker for the sweep at 10,000 connections, and the default configuration's context-switch rate). A worker whose sweep timer cannot be created logs it once and sweeps from its gate alone (about 130 ms apart with the listener open).

Review follow-up (r1, APPROVE-WITH-FIXES, no blockers)

Main (f067838) is merged in; two further commits (the merge and 49e12b3):

  • The comment on the epoll_wait ceiling was wrong: with base 1 ms the backoff tops out at 4 ms, so the ceiling never binds and does not keep the bound if the timerfd cannot be created (the loop then sweeps from the gate, ~32 x 4 ms = ~130 ms apart with the listener open, ~16 s with none). The comment says so, and a failed timerfd creation, settime or epoll_ctl is logged once per loop (TestSweepTimerFailureIsLogged1194, through a timerfdCreate seam).
  • Docs as above (WriteTimeout excluded from "about 30 ms"; one paragraph; one io_uring figure; 25 ms in the detached-stream rule). The stale "every 1024 iterations" comment at the gate is rewritten.
  • The write test closed at up to 2.141 s under -race against a 2.5 s bound; its window is now [1 s - 50 ms, 2.8 s + 200 ms] (main closes at 4.2 s and later, so it is still RED there).
  • The sse small-events-no-header-timer cell is kept: it now guards the ReadHeaderTimeout -1 configuration that used to have no sweep timer, and its comment says so.
  • Re-run on the new head, docker arm64 --cpus 2: the 5 original tests RED on main f067838 (idle 4.69 s, read 4.65 s, write 4.67 s, the two structural tests), and the 6 tests GREEN with -race -count=20 (120 of 120; idle 1.005 to 1.016 s, read 1.006 to 1.017 s, write 2.090 to 2.141 s); epoll package -race ok 492 s, adaptive ok 168 s; CELERIS_MONO_SKEW +3600 s and -3600 s with -tags celeris_monoskew: the epoll package (457 s, the 1178 scenarios included) and internal/mono pass; golangci-lint: 0 issues.

CI

CI of the first head (a6f4ca2): all checks pass. The epoll package in the root -race step took 393.3 s there; against its own base a4b10df (388.6 s) that is +4.7 s, the new tests; the +29 s I first compared (e449c2c, 364.5 s) was noise between main runs (362 to 389 s). The new head's CI is below.

New head 49e12b3: all CI checks pass (31 pass, 2 skipping: CodeRabbit and the arm64 macro benchmark). The epoll package in the root -race step took 398.6 s (a6f4ca2: 393.3 s; base a4b10df: 388.6 s; budget 540 s).

… close on time with ReadHeaderTimeout off, and that the sweep timer ticks do not restart the idle backoff (Refs #1194)

RED on main: with ReadHeaderTimeout -1 the engine's 25 ms timerfd does not
exist, and a 1 s IdleTimeout closes an idle keep-alive connection at 3.2 to
5.8 s (docker arm64, 2 CPUs), a stalled upload at ReadTimeout + 4 s, a
non-reading peer at 4 to 5 s instead of 2 s.
…eouts are; its tick does not restart the idle backoff (Refs #1194)

The timerfd existed only with ReadHeaderTimeout set. With it off, every
deadline that rides the sweep (IdleTimeout, ReadTimeout, WriteTimeout, the
closing drain, the HTTP/2 deadlines, a detached connection's idle deadline)
was seen once per 1024 returns of epoll_wait, 4 to 5 s on an idle loop with
its listener open. The timerfd, the sweep gate (every 32 iterations) and the
epoll_wait cap (the sweep period) no longer depend on the configuration: a
loop sweeps like the default configuration's.

A tick of the timerfd as the only event of a round is an idle round for the
adaptive backoff of the wait. Counted as activity it restarted the backoff
every 25 ms and kept an idle loop at 675 wakeups a second (215 without the
timer); with this it settles at the 4 ms wait it has without one.

IdleTimeout, ReadTimeout and WriteTimeout document the bound; the tests that
used ReadHeaderTimeout -1 as 'no timerfd' disarm the timerfd by hand.
@coderabbitai

coderabbitai Bot commented Oct 31, 2026 •

Copy link
Copy Markdown

Review in Change Stack →

Caution

Review failed

The pull request is closed.

  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@codecov

codecov Bot commented Oct 31, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 82.60870% with 4 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
internal/engine/epoll/loop.go 82.60% 4 Missing ⚠️

📢 Thoughts on this report? Let us know!

…cs say what bounds the sweep; the write test's window is 3 s (Refs #1194)

Review of the PR: the ceiling of the epoll_wait wait does not keep the bound
without the timerfd (the backoff tops out at 4 ms, so it never binds); the
loop then sweeps from the gate alone, about 130 ms apart with its listener
open. The failure of the timerfd's creation, settime or epoll_ctl is logged
once per loop, with a test through a seam.

config.go: one paragraph on the sweep (ReadTimeout's, merged with the one of
the IdleTimeout doc), one io_uring figure, the detached-stream rule says 25 ms
on epoll, WriteTimeout is excluded from 'about 30 ms'. loop.go: the stale
'every 1024 iterations' comment.
@FumingPower3925
FumingPower3925 marked this pull request as ready for review November 1, 2026 01:49
@FumingPower3925
FumingPower3925 merged commit 64fe9e5 into main Nov 1, 2026
33 of 34 checks passed
@FumingPower3925
FumingPower3925 deleted the epoll-1194-fine-sweep-timer branch November 1, 2026 01:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant