Repository navigation
driver/redis: sharded PubSub reroutes share one CLUSTER SLOTS read (v1.6.0, Refs #1203) - #1211
Merged
FumingPower3925 merged 7 commits intoNov 1, 2026
Conversation
…Refs #1203) One slot move that hits 200 sharded PubSubs sends 200 CLUSTER SLOTS to the same node, and 100 PubSubs in a lasting outage (50 ms doubling backoff) send several hundred in 1 s. The first and third tests fail on main; the second (a reroute reads a table newer than its trigger) passes on main and guards the fix.
…efs #1203) Every sharded reroute attempt ran its own CLUSTER SLOTS on the probe node: one SETSLOT that hit N PubSubs cost N, and each backoff step of an outage N again. refreshTopologySince(ctx, since) uses the latest topology refresh if it started at or after since, running or done, and otherwise starts one (check and registration are one step under refMu, so callers that arrive together start one). The reconnect loop passes the moment its reason to look arose (the trigger for the first attempt, the end of the failed attempt for the others), so a reroute never uses a table read before its own trigger. A refresh cut short by its own caller's context is not shared; a joiner reads again. Measured on the fake cluster (200 PubSubs, one SETSLOT): 200 CLUSTER SLOTS -> 1; 100 PubSubs in a 0.8 s outage at a 10-80 ms backoff: 1132 -> 12. Plain singleflight (join only a running read) gives 86 and about 1277 (1.5 s run); a clock window gives one and 15 but reuses reads older than the trigger. The route interface's pick gains the since argument; the sentinel route ignores it. Regular (non-sharded) PubSub reroutes read no topology, so are unchanged.
|
Caution Review failedThe pull request is closed.
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. |
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
…elled leader, a dead probe node (Refs #1203) - a topology read that is still RUNNING when the move lands began before the PubSub's trigger (its table was taken before the move, the fake holds the reply): the reroute must not join it. A variant applying the trigger rule to finished reads only passes the earlier staleness test; - the caller whose read others joined is cancelled: the joiner reads again instead of returning the other caller's context error (a variant without the retry passed every test); - a probe node that is down and not yet marked failing must not fail the read when other nodes are alive (fails on the head: the read returns the dial error, and every PubSub that shared it loses a backoff step).
…wn; close its done on a panic (Refs #1203) Review r6 of #1211: - F3: readTopology asked one node, and a shared read now fails for every PubSub that joined it, where on main the first failed read marked the node failing and the later PubSubs skipped it. It now asks up to three nodes (the next not tried, not marked failing first) while the caller's context is alive. Also helps the command path's refreshes. - F5: runRefresh defers close(done): a panic in the read no longer leaves joiners waiting on a refresh that never ends. - F4: the comment of refreshTopologySince states that a read that started at the same clock tick as since is shared, and why that is harmless.
…logy-refresh-coalesce-1203
…amp (Refs #1203) The "a read that started after my trigger" rule compared monotonic timestamps. On Windows the clock is too coarse to order two instants a few microseconds apart: a read that started in the same tick as the trigger compared equal or inverted, and the reroute used a table read before the move (the required Portable fallback (windows-latest) job failed TestShardedRerouteDoesNotReuseA RefreshOlderThanItsTrigger1203 and ...DoesNotJoinARunningRefreshOlderThanItsTrigger1203). Reads now carry a sequence number, assigned under refMu when they are registered (topoRefresh.seq, ClusterClient.refSeq). A caller takes a topoMark (the number of reads started so far) when its reason to look arises, and may share a read only if its sequence number is greater: strictly later in the order of registration. No clock is involved. pubsubRoute gains topoMark(); pick takes a topoMark instead of a time. The "same tick" paragraph of the doc comment goes. TestRefreshTopologySinceOrdersByRegistrationNotByTime1203 runs 300 rounds of back-to-back reads and marks (the calls a coarse clock would tie) and checks that a mark taken after a read never shares it and one taken before always does.
FumingPower3925
marked this pull request as ready for review
November 1, 2026 13:48
FumingPower3925
deleted the
fix/redis-pubsub-topology-refresh-coalesce-1203
branch
November 1, 2026 13:48
This was referenced Nov 1, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
driver/redis: sharded PubSub reroutes share one CLUSTER SLOTS read. Refs #1203 (found by the review of #1154).
The problem
clusterPubSubRoute.pickranrefreshTopology(CLUSTER SLOTSon the probe node, thenapplyTopology) on every sharded reroute attempt. OneSETSLOTthat hits N sharded PubSubs sent NCLUSTER SLOTSto the same node, and each backoff step of an outage sent N again (the review's estimate: 1000 PubSubs at the 5 s cap, 200 per second).Design: share a read that started after the trigger
ClusterClient.refreshTopologySince(ctx, since)uses the latest topology refresh if it started at or aftersince(running or done), and otherwise starts one. The check and the registration of the new refresh are one step underrefMu, so callers that arrive together start one. The reconnect loop passessince= the moment the reason for the attempt arose: the loop's start for the first attempt (the trigger), the end of the failed attempt for the later ones. So a reroute never uses a table read before its own trigger, and one read serves every PubSub whose trigger it follows. A refresh cut short by its own caller's context (Closeof that PubSub) is not shared: the joiners read again. A refresh that failed for a node's reason (the probe node is down) is shared like a table.Why not the two options in the issue (measured on the fake cluster,
TestShardedReroute*1203, the variants were temporary switches inrefreshTopologySince):(Three runs per variant; the variants were temporary switches in
refreshTopologySince, the table is their output on the fake cluster. In an earlier 1.5 s run at a 10-80 ms backoff: main 2001, singleflight 1277, 100 ms window 15, since about 21.)sincekeeps the rule that makes each attempt worth its read ("a table newer than what made me look") and so the count follows the backoff steps (about one read per window in which no PubSub's trigger is newer than the last read), not the PubSub count. At the 5 s cap with 1000 PubSubs that is about one read per few seconds in place of 200 per second.The two other things the issue asked to check
pubsubNodepicks by address and reads no topology, so it never ranCLUSTER SLOTS; unchanged.doWithRetryCmdrefreshes on every-MOVED(N concurrent commands, N reads), the pipeline once per batch,refreshLoopevery 60 s. This PR does not coalesce those (a command that was just redirected needs a table read after the redirect, the same freshness rule, but it is a separate hot path with its own semantics: proposed as a follow-up issue). What it does: plainrefreshTopologynow records itself as the latest refresh, so a PubSub may reuse a command-path or 60 s-loop read that started after its trigger.RED / GREEN
RED,
8cd5a87alone (main's code,go test ./driver/redis -run 1203 -v):TestShardedRerouteDoesNotReuseARefreshOlderThanItsTrigger1203passes on main by design: it guards the freshness rule of the fix (a refresh done just before the move must not be reused; fails with a clock window).GREEN,
20d166c: 200 PubSubs, one move: 1 CLUSTER SLOTS (main 200); 100 PubSubs, 1 s of an outage (backoff 50 ms doubling): 3-4 (main 396).go test ./driver/redis -race -run 1203 -count=20 -v, Linux (golang:1.27.3,--cpus 2): 20/20 pass; the move test always 1 CLUSTER SLOTS, the outage test 3 or 4. (The first version of this count=20 run found a data race in my test: it replacedps.backoffof a live PubSub to shorten the backoff. The test now uses the real backoff and a 1 s window, no production knob.) Also ok on macOS (-race -count=20, 22 s).driver/redispackage, Linux:-race -count=1ok (168 s), plain-count=1ok (171 s); macOS-raceok.-raceon the whole package (macOS, 41 s): driver/redis: bound the connection handshake by DialTimeout (v1.6.0, Refs #1205) #1209'sDeadlineExceeded-wrapping handshake error goes through the refresh path; thecutflag keys on the leader'sctx.Err(), not the error type, so a node's handshake timeout is shared like any node failure.golangci-lintv2.15.0GOOS=linux: nothing in the files of this PR (the twoerrcheckfindings inclose_after_teardown_859_linux_test.go:135andpubsub_rig_engines_887_linux_test.go:77are on main too);GOOS=windows go vet: clean.Behaviour notes
pubsubRoute.pickgained asince time.Timeargument (the sentinel route ignores it);ClusterClientgetsrefMu/refLast;api/is untouched.openPubSub) passestime.Now(): it needs a read of its own, so it does not join an earlier one (a refresh that started before the call is not newer than it), but concurrent calls and later PubSub reroutes may share the read it starts.DialTimeoutbound).Release note
A cluster shard PubSub reroute no longer sends its own
CLUSTER SLOTS: PubSubs hit together by a slot move, or retrying the same outage, share one topology read that started after their trigger. With 200 sharded PubSubs on one node, one slot move costs oneCLUSTER SLOTSinstead of 200, and an outage costs about one per backoff window instead of one per PubSub per attempt.Review r6 (APPROVE-WITH-FIXES) and what changed
TestShardedRerouteDoesNotJoinARunningRefreshOlderThanItsTrigger1203holds oneCLUSTER SLOTSreply after its table snapshot (a 3-linerpSlotsHoldhook in the fake'swriteSlots), moves the slot, lets the PubSub pick, releases the reply: the old owner must get noSSUBSCRIBE. Mutant "since enforced only for finished reads" now fails it (before, it passed all three tests); plain singleflight and the 100 ms window fail it too.TestRefreshJoinerReadsAgainWhenTheReadItJoinedWasCancelled1203: the owner of the read is cancelled, the joiner gets nil after exactly one extra read. Mutant "drop thecutretry" now fails it.readTopologyasks up to three nodes (next untried, not marked failing first) while the caller's context is alive, instead of one.TestTopologyReadTriesTheNextNodeWhenTheProbeIsDown1203(RED on the previous head: the read returned the dial error of the dead probe). Also helps every otherrefreshTopologycaller (command path). The wording above is corrected.refreshTopologySincestates that a read that started at the same clock tick assinceis shared, and why it is harmless (the reroute loop takessincea whole jittered backoff, at least 40 ms, before it asks; a wrong share costs one backoff step and the next attempt'ssinceis newer). Kept the non-strict comparison: a strict one stops same-tick callers from coalescing on a coarse clock.defer close(f.done).doWithRetryand the pipeline refresh once per-MOVEDwith no sharing (a larger herd on a hotter path). It is a separate v1.6.0 issue withrefreshTopologySinceas its primitive.-race -run 1203 -count=20ok on macOS (36 s) and Linux docker--cpus 2(36 s); wholedriver/redispackage-raceon Linux docker ok (169 s);golangci-lint(GOOS=linux) clean for the touched files (the twoerrcheckfindings in untouched*_linux_test.gofiles are on main too). The new tests take 0.4 s, 0.05 s and 0.00 s.Windows fix (required check "Portable fallback (windows-latest)" failed on the previous head)
Two of the new tests failed there (
...DoesNotReuseARefreshOlderThanItsTrigger1203: "the old owner got 1 SSUBSCRIBE after the move";...DoesNotJoinARunningRefreshOlderThanItsTrigger1203). Cause: the "read started after my trigger" rule compared monotonic timestamps, and the Windows clock is too coarse to order two instants microseconds apart (review r6 F4 was right that this is not harmless: it let a reroute use a table read before the move). The rule is now a logical sequence: reads get a sequence number assigned underrefMuwhen they are registered (topoRefresh.seq); a caller takes atopoMark(the number of reads started so far) when its reason to look arises, and may share a read only if its sequence is strictly greater. There is no clock in the rule at all, so it is the same on any clock (no need to simulate a 16 ms clock: nothing reads one). The "same tick" paragraph of the doc comment is gone;pubsubRoutegainstopoMark()andpicktakes atopoMark. New testTestRefreshTopologySinceOrdersByRegistrationNotByTime1203: 300 rounds of back-to-back reads and marks (the calls a coarse clock ties) checking that a mark taken after a read never shares it and one taken before always does. Existing tests kept as they were (they use no clock-dependent sleeps; the 400 ms/1 s waits only give the loops time to run).-race -run 1203 -count=20ok on macOS (41 s) and Linux docker (42 s); whole package-raceon Linux ok (177 s);GOOS=windows go vetclean.