Skip to content

driver/memcached: apply Config.Timeout as the per-operation deadline (v1.6.0, Refs #1218) - #1222

Merged
FumingPower3925 merged 11 commits into
mainfrom
fix/memcached-op-timeout-1218
Nov 2, 2026
Merged

FumingPower3925 merged 11 commits into
mainfrom
fix/memcached-op-timeout-1218

Conversation

@FumingPower3925

@FumingPower3925 FumingPower3925 commented Nov 1, 2026 •

Copy link
Copy Markdown
Contributor

driver/memcached: apply Config.Timeout as the per-operation deadline. Refs #1218 (found by the review of #1217; pre-existing).

The problem

Config.Timeout, documented as the per-operation deadline with a 3 s default, was never applied (only forwarded by ClusterConfig). A command on a server that accepts TCP and never answers it blocked the caller until its context ended: for ever under context.Background(). #1217 bounded the binary handshake by DialTimeout; nothing bounded the commands.

The change

  • Client.run (the single choke point of every command: storage, get/multi-get, delete, arithmetic, touch, flush, stats, version, text and binary) runs the command under the per-operation timeout (a read/write deadline on the conn, plus the caller's context, which wakes a blocked direct read through armWake/disarmWake). Client.Ping (it has its own acquire) does the same. The timeout is Config.Timeout, 3 s when it is zero (defaultTimeout, the documented default), none when it is negative.
  • No context per command (r2 BLOCKER): the first version built a context.WithTimeout per command (+4 allocs, ~300 B, and CI's strict alloc guard step failed: GET_small 7 > 4, SET_small 8 > 4, GetMulti_10 30 > 26). The limit is now a value: opLimit computes min(caller deadline, now+Timeout) once; the conn carries opTimeout/opDeadline for the command. A direct conn passes it to its socket deadline (directDeadline, the same SetDeadline pair a caller's deadline already paid); a loop conn arms one reusable per-conn time.Timer only when it actually parks (the spin window is untouched) and selects on it next to ctx.Done() and the reply. Alloc counts are back to main's: 3 / 4 / 26 / 0 (TESTING_STRICT_ALLOC_BUDGETS=1 go test ./driver/memcached -run '^TestAllocBudgets$', plain, not -race, macOS and Linux); TestOperationLimitDoesNotAllocate1218 pins that a client with the limit allocates exactly what one without it does (direct, event loop, net.Conn loop).
  • Interaction with the caller's context: the command ends at the caller's deadline or after the timeout, whichever is first. A caller's deadline that is not later than the timeout wins untouched, and its error is the caller's. A caller's deadline later than the timeout (or none) is capped by it (the old doc said "used when the caller's context carries no deadline"; the issue asked for "no earlier one", so the timeout is a cap; documented on Config.Timeout). A caller's cancellation is returned as it is, on the direct path too: a cancellable context arms the wake of the blocked read (armWake, driver/memcached: bound the binary handshake by DialTimeout (v1.6.0, Refs #1213) #1217), so a cancel does not wait for Timeout and comes back as context.Canceled (wrapped around the socket's i/o timeout); a caller's deadline that fires on the socket comes back matching context.DeadlineExceeded. A context that cannot be cancelled (Background) costs nothing; a cancellable one (WithCancel, WithTimeout, an HTTP/2 stream's) costs a direct conn context.AfterFunc's two allocations (3 -> 5 allocs/op); the loop paths cost nothing. See the r3 follow-up. A multi-key get, Stats and Flush are one operation each under one Timeout.
  • Not covered: waiting for a pool slot and dialling a new conn (the latter has DialTimeout, driver/memcached: bound the binary handshake by DialTimeout (v1.6.0, Refs #1213) #1217); the timeout starts when the conn is acquired. Documented.
  • Error: celeris-memcached: operation timed out after 300ms: context deadline exceeded (on the direct path the socket's own i/o timeout is appended). opTimeoutError matches errors.Is(err, context.DeadlineExceeded), which is also what makes run/Ping drop the conn from the pool: it may still receive a stale reply. Only a deadline error is relabelled (opError): an application answer that lands as the timer fires (ErrCacheMiss, a server error) is never reported as a timeout.
  • Works on the direct path (the one the pool uses without an engine: execTextDirect/execBinaryDirect set the socket deadline from the context), the unix event-loop path and the net.Conn loop (wait/waitTransport select on the context).
  • Docs: Config.Timeout, WithTimeout, ClusterConfig.Timeout. testHookDefaultOpTimeout (nil in production, like driver/redis: bound the connection handshake by DialTimeout (v1.6.0, Refs #1205) #1209's) lowers the default for the tests; testHookWakeDelay (nil in production) delays the wake callback of a direct read.

Gaps in the other drivers (the issue asked to check)

  • redis: Config.ReadTimeout / WriteTimeout (defaults of 3 s declared in options.go) are documented as "unused currently" and are never applied either (WithReadTimeout, ClusterConfig, SentinelConfig only forward them). So a redis command on a silent server has the same gap: it waits for the caller's context. Not changed here (honest doc, a separate PR if wanted).
  • postgres: there is no client-side per-statement timeout. statement_timeout can be passed through the DSN as a server GUC, but it only limits statements the server executes; it does nothing for a server that does not answer. The caller's context is the only bound (a deadline wakes the cancel path, driver/postgres: bound the startup by ConnectTimeout and end it with the caller's context (v1.6.0, Refs #1212) #1216 covers the startup). Same gap.

RED / GREEN

RED, a7f52cf alone (macOS arm64, go test ./driver/memcached -run 1218 -v): 36 x "the operation is still blocked after 2s: Config.Timeout is not applied" (text|binary x direct|eventloop|netconn x Get|Ping|Version x background|later-deadline); the two guards pass on main by design. GREEN: all pass.

Mutants (each alone on 38b1d26, go test -run 1218 -count=3, macOS)

mutant tests that fail
M1 loop timer never fires TestOperationIsBoundedByTimeout, ...ByTheDefaultTimeout
M2 direct deadline ignores the per-operation one TestOperationIsBoundedByTimeout, ...ByTheDefaultTimeout
M3 no wake of a direct read on the caller's cancel TestOperationKeepsTheCallersCancellation (direct)
M4 opError relabels any error TestOpErrorRelabelsOnlyDeadlineErrors
M5 opError does not wrap the caller's context error TestOpErrorRelabelsOnlyDeadlineErrors, TestOperationKeepsTheCallersCancellation
M6 the caller's earlier deadline is not respected TestOperationKeepsTheCallersShorterDeadline
M7 a timed-out loop conn is not closed 7 tests (pool left with the poisoned conn, later ops fail)
M8 the default (3 s) is never applied TestOperationIsBoundedByTheDefaultTimeout
(earlier version) context built per command the repo's strict alloc guard, now also TestOperationLimitDoesNotAllocate1218

Dropped from the earlier table: M3 "always derive a context" (equivalent: the caller's deadline ended the derived context first, so nothing observable but allocations) and M5 "socket-deadline clause dropped" (probabilistic; the clause is now exercised deterministically by the opError unit test). R6 of the review (no discard on DeadlineExceeded in run) is equivalent: every timed-out path has already closed the conn and release discards closed conns.

Checks

  • Plain (not -race) strict alloc guards, which the first version never ran: TESTING_STRICT_ALLOC_BUDGETS=1 go test ./driver/memcached -run '^TestAllocBudgets$' GET_small 3 (budget 4), SET_small 4 (4), GetMulti_10 26 (26), PoolAcquireRelease 0 (0): ok on macOS and on Linux docker --cpus 2. The redis and postgres guards, also run by that CI step, ok on macOS.
  • -race -run 1218 -count=20 Linux docker ok (200 s); whole driver/memcached -race -count=1 Linux ok; macOS -race -count=10 ok.
  • GOOS=windows go vet ./driver/memcached: clean. golangci-lint v2.15.0 GOOS=linux: only the pre-existing errcheck finding in close_after_teardown_859_linux_test.go.

Release note

driver/memcached: Config.Timeout (WithTimeout, default 3 s) is now applied as the per-operation deadline, as documented: a command (and Ping, Version, Stats, ...) that a server does not answer within the timeout fails with an error matching context.DeadlineExceeded and its conn is dropped from the pool, instead of blocking until the caller's context ends (for ever under context.Background()). A caller's earlier deadline still wins; a later one (or none) is capped by Timeout. Set a larger Timeout for operations that legitimately take longer than 3 s, or a negative one to turn the bound off. Waiting for a pool slot and dialling are not covered (DialTimeout).

Review r3 follow-up (commits 231f3c1, 8339b61)

  • MAJOR (fixed): the direct-path wake cost 5 extra allocations for a cancellable context (3 -> 8). The state of the wake (mutex, fired/done flags, the callback, the socket copy, the AfterFunc stop func) now lives on the conn (armWake/disarmWake) and the callback is built once per conn, so a command pays only context.AfterFunc's own two allocations: 3 -> 5 on the direct path, 3 -> 3 on the loop paths. beginOp/endOp no longer return or build a closure. The cost is stated in the Config.Timeout behaviour above, and budgeted: TestAllocBudgets/GET_small_cancellable (budget 5), which CI runs without -race in its alloc-guard step, so no ci.yml entry changes (the step's tally stays 15). TestOperationLimitDoesNotAllocate1218 keeps the limit-vs-off comparison, valid with Background and under -race (the cancellable case is too noisy under -race to assert there: it failed 1 run in 20 on Linux with 4 and 6 allocs, which is why it lives in the plain guard). The qualification "no allocation" means: with a context that cannot be cancelled. A fresh WithCancel context per command adds the context's own children map entry on top (AfterFunc registers in it); the guard reuses one context.
  • MINOR (test): endOp clearing the limit. TestHealthCheckPingAfterALimitedCommand1218: a command under Timeout 40 ms, 80 ms later a health-style conn.Ping on the same conn (5 times, three paths). Mutant "endOp does not clear the limit" fails it (the check would close every idle conn).
  • MINOR (test): the limit on a reused conn. TestLimitIsArmedAgainOnAReusedConn1218: the first command parks and is answered after 40 ms, the second on the same conn stalls and must end at about Timeout (150 ms), on the three paths. Mutant "no Reset of the timer on reuse" fails it (the second command ran unbounded).
  • Mutants on 231f3c1: N1 (endOp not clearing) -> HealthCheckPing test; N13 (no Reset) -> ReusedConn test; the wake callback built per command -> TestAllocBudgets/GET_small_cancellable (6 > 5); no wake -> the cancellation test.
  • Checks on 8339b61: plain TESTING_STRICT_ALLOC_BUDGETS=1 go test ./driver/memcached -run '^TestAllocBudgets$' 3 / 4 / 26 / 5 (cancellable) / 0, macOS and Linux docker --cpus 2; -race -run '1218|Allocate|HealthCheck|ReusedConn' -count=20 Linux ok (216 s); whole driver/memcached -race -count=1 Linux ok; the CI alloc-guard discovery logic replicated locally: ok (14 !race files, 11 entries); GOOS=windows go vet clean; golangci-lint only the pre-existing errcheck finding.

Review r4 follow-up (commits 492ad7c, ee7e5a1)

  • MAJOR (fixed): a late wake of a cancelled command hit the NEXT command on the same direct conn. When a caller cancelled just as a command finished, disarmWake's stop failed (the AfterFunc callback had started but not yet taken the lock), the next command's armWake reset the state, and the late callback then set a deadline of "now" on the shared socket: a live, unrelated command failed with a false operation timed out after 3s and a healthy conn was closed. Fix, no allocation per command: each conn has a capacity-1 channel (wakeRan, made once with the callback) that the callback signals when it has run, and disarmWake waits on it whenever stopping the wake fails (the callback has been started on its own goroutine; stop is called once per arm). So armWake never resets the state while a callback of the previous command is in flight. A single "stale" flag was not used: a late callback and the next command's own cancel can reach the lock in either order, so a flag could swallow a genuine later cancel. Other paths: a context that is already done at arm time starts the callback at once (same path, covered by the test); the handshake's armWake/disarmWake are fixed by the same change; the loop paths never arm the wake (beginOp and the handshake gate on useDirect), their per-conn timer relies on Reset plus Go 1.27's synchronous timer channels; every armWake has exactly one disarmWake (run/endOp, Ping, the handshake).
  • Test, failing first: TestLateWakeDoesNotFailTheNextCommand1218 (text and binary). testHookWakeDelay (nil in production) holds the wake callback 100 ms; the test cancels command 1's context, disarms (stop fails), arms command 2 on a live context, waits for the held callback, and pings. RED on 492ad7c (tests + hook, old logic), macOS and Linux docker, every run: a live command on the conn after a cancelled one: write tcp ...: i/o timeout (text and binary, 3/3 runs on Linux). GREEN on ee7e5a1: passes.
  • Test (MINOR gap): TestCancelOnAReusedDirectConn1218 (text: after a first command; binary: after the handshake). A cancel on a reused direct conn returns context.Canceled at about the cancel (150 ms), not at the 30 s Timeout. Mutant "armWake no longer resets wakeFired/wakeDone per command": KILLED, text and binary ("still blocked after 3s").
  • Supporting probe (not committed): a direct client, MaxOpen 1, command 1 cancelled by a goroutine racing its completion, command 2 on a live context; 8 s on macOS: 492ad7c 18 failures of 105129, ee7e5a1 0 of 107643.
  • Alloc numbers unchanged, plain TESTING_STRICT_ALLOC_BUDGETS=1 go test ./driver/memcached -run '^TestAllocBudgets$' (macOS and Linux docker --cpus 2): GET_small 3 (budget 4), SET_small 4 (4), GetMulti_10 26 (26), GET_small_cancellable 5 (5), PoolAcquireRelease 0 (0). Both new tests are in op_timeout_1218_test.go (no build tag), so ci.yml is untouched.
  • Checks on ee7e5a1, Linux docker golang:1.27.3 --cpus 2: -race -run 1218 -count=20 ok (234 s); whole driver/memcached -race -count=1 ok; GOOS=windows go vet ./driver/memcached clean.

Alloc numbers at a glance (direct path, fake server, AllocsPerRun)

context main this PR
context.Background() (HTTP/1 handlers get one) 3 allocs/op 3 (guarded: TestAllocBudgets GET_small 3, SET_small 4, GetMulti_10 26, and TestOperationLimitDoesNotAllocate1218)
cancellable (WithCancel, WithTimeout, HTTP/2 stream) 3 5 (guarded: TestAllocBudgets/GET_small_cancellable, budget 5; first version of the wake: 8)
loop paths (event loop, net.Conn loop), either context 3 3

Test-tag note

op_alloc_1218_test.go carries no //go:build !race tag on purpose: CI's alloc-guard step requires every !race test of a package to be named in its ci.yml entry (it failed once for that reason), and the file's limit-vs-off comparison holds under -race too. The strict, CI-enforced budgets (Background and cancellable) live in TestAllocBudgets, an existing !race test that the step already names, so ci.yml is untouched and its tally stays 15.

@FumingPower3925 FumingPower3925 added this to the v1.6.0 milestone Nov 1, 2026
@coderabbitai

coderabbitai Bot commented Nov 1, 2026 •

Copy link
Copy Markdown

Review in Change Stack →

Caution

Review failed

The pull request is closed.

  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

CodeRabbit Verify

  • Run CodeRabbit Verify to test the changes in this pull request.

@codecov

codecov Bot commented Nov 1, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 96.87500% with 3 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
driver/memcached/conn.go 95.83% 3 Missing ⚠️

📢 Thoughts on this report? Let us know!

… value and a reusable timer, no context); wake a direct read on the caller's cancel (Refs #1218)
…(the CI guard step names every !race test of a package) (Refs #1218)
…ellable context costs 2 allocations, not 5); tests for the health-check ping after a limited command, the limit on a reused conn, and the cancellable-context budget (Refs #1218)
…I runs it without -race); the -race run of the comparison test keeps only the limit-vs-off case (Refs #1218)
…nd for a cancel on a reused direct conn; a nil-in-production hook delays the wake callback (the late-wake test fails here) (Refs #1218)
…ed, so it cannot run during the next command on the conn and give it a false timeout (Refs #1218)
@FumingPower3925
FumingPower3925 force-pushed the fix/memcached-op-timeout-1218 branch from 5125ce6 to ee7e5a1 Compare November 2, 2026 22:39
@FumingPower3925
FumingPower3925 marked this pull request as ready for review November 2, 2026 23:19
@FumingPower3925
FumingPower3925 merged commit 62e7262 into main Nov 2, 2026
56 checks passed
@FumingPower3925
FumingPower3925 deleted the fix/memcached-op-timeout-1218 branch November 2, 2026 23:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant