Repository navigation
driver/memcached: apply Config.Timeout as the per-operation deadline (v1.6.0, Refs #1218) - #1222
Merged
Merged
Conversation
|
Caution Review failedThe pull request is closed.
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. CodeRabbit Verify
|
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
… value and a reusable timer, no context); wake a direct read on the caller's cancel (Refs #1218)
…(the CI guard step names every !race test of a package) (Refs #1218)
…ellable context costs 2 allocations, not 5); tests for the health-check ping after a limited command, the limit on a reused conn, and the cancellable-context budget (Refs #1218)
…I runs it without -race); the -race run of the comparison test keeps only the limit-vs-off case (Refs #1218)
FumingPower3925
force-pushed
the
fix/memcached-op-timeout-1218
branch
from
November 2, 2026 22:39
5125ce6 to
ee7e5a1
Compare
FumingPower3925
marked this pull request as ready for review
November 2, 2026 23:19
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
driver/memcached: apply Config.Timeout as the per-operation deadline. Refs #1218 (found by the review of #1217; pre-existing).
The problem
Config.Timeout, documented as the per-operation deadline with a 3 s default, was never applied (only forwarded byClusterConfig). A command on a server that accepts TCP and never answers it blocked the caller until its context ended: for ever undercontext.Background(). #1217 bounded the binary handshake byDialTimeout; nothing bounded the commands.The change
Client.run(the single choke point of every command: storage, get/multi-get, delete, arithmetic, touch, flush, stats, version, text and binary) runs the command under the per-operation timeout (a read/write deadline on the conn, plus the caller's context, which wakes a blocked direct read througharmWake/disarmWake).Client.Ping(it has its own acquire) does the same. The timeout isConfig.Timeout, 3 s when it is zero (defaultTimeout, the documented default), none when it is negative.context.WithTimeoutper command (+4 allocs, ~300 B, and CI's strict alloc guard step failed: GET_small 7 > 4, SET_small 8 > 4, GetMulti_10 30 > 26). The limit is now a value:opLimitcomputesmin(caller deadline, now+Timeout)once; the conn carriesopTimeout/opDeadlinefor the command. A direct conn passes it to its socket deadline (directDeadline, the sameSetDeadlinepair a caller's deadline already paid); a loop conn arms one reusable per-conntime.Timeronly when it actually parks (the spin window is untouched) and selects on it next toctx.Done()and the reply. Alloc counts are back to main's: 3 / 4 / 26 / 0 (TESTING_STRICT_ALLOC_BUDGETS=1 go test ./driver/memcached -run '^TestAllocBudgets$', plain, not-race, macOS and Linux);TestOperationLimitDoesNotAllocate1218pins that a client with the limit allocates exactly what one without it does (direct, event loop, net.Conn loop).Config.Timeout). A caller's cancellation is returned as it is, on the direct path too: a cancellable context arms the wake of the blocked read (armWake, driver/memcached: bound the binary handshake by DialTimeout (v1.6.0, Refs #1213) #1217), so a cancel does not wait forTimeoutand comes back ascontext.Canceled(wrapped around the socket's i/o timeout); a caller's deadline that fires on the socket comes back matchingcontext.DeadlineExceeded. A context that cannot be cancelled (Background) costs nothing; a cancellable one (WithCancel,WithTimeout, an HTTP/2 stream's) costs a direct conncontext.AfterFunc's two allocations (3 -> 5 allocs/op); the loop paths cost nothing. See the r3 follow-up. A multi-key get,StatsandFlushare one operation each under oneTimeout.DialTimeout, driver/memcached: bound the binary handshake by DialTimeout (v1.6.0, Refs #1213) #1217); the timeout starts when the conn is acquired. Documented.celeris-memcached: operation timed out after 300ms: context deadline exceeded(on the direct path the socket's owni/o timeoutis appended).opTimeoutErrormatcheserrors.Is(err, context.DeadlineExceeded), which is also what makesrun/Pingdrop the conn from the pool: it may still receive a stale reply. Only a deadline error is relabelled (opError): an application answer that lands as the timer fires (ErrCacheMiss, a server error) is never reported as a timeout.execTextDirect/execBinaryDirectset the socket deadline from the context), the unix event-loop path and the net.Conn loop (wait/waitTransportselect on the context).Config.Timeout,WithTimeout,ClusterConfig.Timeout.testHookDefaultOpTimeout(nil in production, like driver/redis: bound the connection handshake by DialTimeout (v1.6.0, Refs #1205) #1209's) lowers the default for the tests;testHookWakeDelay(nil in production) delays the wake callback of a direct read.Gaps in the other drivers (the issue asked to check)
Config.ReadTimeout/WriteTimeout(defaults of 3 s declared inoptions.go) are documented as "unused currently" and are never applied either (WithReadTimeout,ClusterConfig,SentinelConfigonly forward them). So a redis command on a silent server has the same gap: it waits for the caller's context. Not changed here (honest doc, a separate PR if wanted).statement_timeoutcan be passed through the DSN as a server GUC, but it only limits statements the server executes; it does nothing for a server that does not answer. The caller's context is the only bound (a deadline wakes the cancel path, driver/postgres: bound the startup by ConnectTimeout and end it with the caller's context (v1.6.0, Refs #1212) #1216 covers the startup). Same gap.RED / GREEN
RED,
a7f52cfalone (macOS arm64,go test ./driver/memcached -run 1218 -v): 36 x "the operation is still blocked after 2s: Config.Timeout is not applied" (text|binary x direct|eventloop|netconn x Get|Ping|Version x background|later-deadline); the two guards pass on main by design. GREEN: all pass.Mutants (each alone on
38b1d26,go test -run 1218 -count=3, macOS)opErrorrelabels any erroropErrordoes not wrap the caller's context errorTestOperationLimitDoesNotAllocate1218Dropped from the earlier table: M3 "always derive a context" (equivalent: the caller's deadline ended the derived context first, so nothing observable but allocations) and M5 "socket-deadline clause dropped" (probabilistic; the clause is now exercised deterministically by the
opErrorunit test).R6of the review (no discard onDeadlineExceededinrun) is equivalent: every timed-out path has already closed the conn andreleasediscards closed conns.Checks
-race) strict alloc guards, which the first version never ran:TESTING_STRICT_ALLOC_BUDGETS=1 go test ./driver/memcached -run '^TestAllocBudgets$'GET_small 3 (budget 4), SET_small 4 (4), GetMulti_10 26 (26), PoolAcquireRelease 0 (0): ok on macOS and on Linux docker--cpus 2. The redis and postgres guards, also run by that CI step, ok on macOS.-race -run 1218 -count=20Linux docker ok (200 s); wholedriver/memcached-race -count=1Linux ok; macOS-race -count=10ok.GOOS=windows go vet ./driver/memcached: clean.golangci-lintv2.15.0GOOS=linux: only the pre-existingerrcheckfinding inclose_after_teardown_859_linux_test.go.Release note
driver/memcached:
Config.Timeout(WithTimeout, default 3 s) is now applied as the per-operation deadline, as documented: a command (andPing,Version,Stats, ...) that a server does not answer within the timeout fails with an error matchingcontext.DeadlineExceededand its conn is dropped from the pool, instead of blocking until the caller's context ends (for ever undercontext.Background()). A caller's earlier deadline still wins; a later one (or none) is capped byTimeout. Set a largerTimeoutfor operations that legitimately take longer than 3 s, or a negative one to turn the bound off. Waiting for a pool slot and dialling are not covered (DialTimeout).Review r3 follow-up (commits
231f3c1,8339b61)AfterFuncstop func) now lives on the conn (armWake/disarmWake) and the callback is built once per conn, so a command pays onlycontext.AfterFunc's own two allocations: 3 -> 5 on the direct path, 3 -> 3 on the loop paths.beginOp/endOpno longer return or build a closure. The cost is stated in theConfig.Timeoutbehaviour above, and budgeted:TestAllocBudgets/GET_small_cancellable(budget 5), which CI runs without-racein its alloc-guard step, so noci.ymlentry changes (the step's tally stays 15).TestOperationLimitDoesNotAllocate1218keeps the limit-vs-off comparison, valid withBackgroundand under-race(the cancellable case is too noisy under-raceto assert there: it failed 1 run in 20 on Linux with 4 and 6 allocs, which is why it lives in the plain guard). The qualification "no allocation" means: with a context that cannot be cancelled. A freshWithCancelcontext per command adds the context's ownchildrenmap entry on top (AfterFunc registers in it); the guard reuses one context.endOpclearing the limit.TestHealthCheckPingAfterALimitedCommand1218: a command underTimeout40 ms, 80 ms later a health-styleconn.Pingon the same conn (5 times, three paths). Mutant "endOpdoes not clear the limit" fails it (the check would close every idle conn).TestLimitIsArmedAgainOnAReusedConn1218: the first command parks and is answered after 40 ms, the second on the same conn stalls and must end at aboutTimeout(150 ms), on the three paths. Mutant "noResetof the timer on reuse" fails it (the second command ran unbounded).231f3c1: N1 (endOp not clearing) -> HealthCheckPing test; N13 (no Reset) -> ReusedConn test; the wake callback built per command ->TestAllocBudgets/GET_small_cancellable(6 > 5); no wake -> the cancellation test.8339b61: plainTESTING_STRICT_ALLOC_BUDGETS=1 go test ./driver/memcached -run '^TestAllocBudgets$'3 / 4 / 26 / 5 (cancellable) / 0, macOS and Linux docker--cpus 2;-race -run '1218|Allocate|HealthCheck|ReusedConn' -count=20Linux ok (216 s); wholedriver/memcached-race -count=1Linux ok; the CI alloc-guard discovery logic replicated locally: ok (14!racefiles, 11 entries);GOOS=windows go vetclean; golangci-lint only the pre-existingerrcheckfinding.Review r4 follow-up (commits
492ad7c,ee7e5a1)disarmWake's stop failed (theAfterFunccallback had started but not yet taken the lock), the next command'sarmWakereset the state, and the late callback then set a deadline of "now" on the shared socket: a live, unrelated command failed with a falseoperation timed out after 3sand a healthy conn was closed. Fix, no allocation per command: each conn has a capacity-1 channel (wakeRan, made once with the callback) that the callback signals when it has run, anddisarmWakewaits on it whenever stopping the wake fails (the callback has been started on its own goroutine; stop is called once per arm). SoarmWakenever resets the state while a callback of the previous command is in flight. A single "stale" flag was not used: a late callback and the next command's own cancel can reach the lock in either order, so a flag could swallow a genuine later cancel. Other paths: a context that is already done at arm time starts the callback at once (same path, covered by the test); the handshake'sarmWake/disarmWakeare fixed by the same change; the loop paths never arm the wake (beginOpand the handshake gate onuseDirect), their per-conn timer relies onResetplus Go 1.27's synchronous timer channels; everyarmWakehas exactly onedisarmWake(run/endOp,Ping, the handshake).TestLateWakeDoesNotFailTheNextCommand1218(text and binary).testHookWakeDelay(nil in production) holds the wake callback 100 ms; the test cancels command 1's context, disarms (stop fails), arms command 2 on a live context, waits for the held callback, and pings. RED on492ad7c(tests + hook, old logic), macOS and Linux docker, every run:a live command on the conn after a cancelled one: write tcp ...: i/o timeout(text and binary, 3/3 runs on Linux). GREEN onee7e5a1: passes.TestCancelOnAReusedDirectConn1218(text: after a first command; binary: after the handshake). A cancel on a reused direct conn returnscontext.Canceledat about the cancel (150 ms), not at the 30 sTimeout. Mutant "armWakeno longer resetswakeFired/wakeDoneper command": KILLED, text and binary ("still blocked after 3s").MaxOpen1, command 1 cancelled by a goroutine racing its completion, command 2 on a live context; 8 s on macOS:492ad7c18 failures of 105129,ee7e5a10 of 107643.TESTING_STRICT_ALLOC_BUDGETS=1 go test ./driver/memcached -run '^TestAllocBudgets$'(macOS and Linux docker--cpus 2): GET_small 3 (budget 4), SET_small 4 (4), GetMulti_10 26 (26), GET_small_cancellable 5 (5), PoolAcquireRelease 0 (0). Both new tests are inop_timeout_1218_test.go(no build tag), soci.ymlis untouched.ee7e5a1, Linux dockergolang:1.27.3--cpus 2:-race -run 1218 -count=20ok (234 s); wholedriver/memcached-race -count=1ok;GOOS=windows go vet ./driver/memcachedclean.Alloc numbers at a glance (direct path, fake server,
AllocsPerRun)context.Background()(HTTP/1 handlers get one)TestAllocBudgetsGET_small 3, SET_small 4, GetMulti_10 26, andTestOperationLimitDoesNotAllocate1218)WithCancel,WithTimeout, HTTP/2 stream)TestAllocBudgets/GET_small_cancellable, budget 5; first version of the wake: 8)Test-tag note
op_alloc_1218_test.gocarries no//go:build !racetag on purpose: CI's alloc-guard step requires every!racetest of a package to be named in itsci.ymlentry (it failed once for that reason), and the file's limit-vs-off comparison holds under-racetoo. The strict, CI-enforced budgets (Background and cancellable) live inTestAllocBudgets, an existing!racetest that the step already names, soci.ymlis untouched and its tally stays 15.