Skip to content

driver/redis: sharded PubSub reroutes share one CLUSTER SLOTS read (v1.6.0, Refs #1203) - #1211

Merged
FumingPower3925 merged 7 commits into
mainfrom
fix/redis-pubsub-topology-refresh-coalesce-1203
Nov 1, 2026
Merged

FumingPower3925 merged 7 commits into
mainfrom
fix/redis-pubsub-topology-refresh-coalesce-1203

Conversation

@FumingPower3925

@FumingPower3925 FumingPower3925 commented Nov 1, 2026 •

Copy link
Copy Markdown
Contributor

driver/redis: sharded PubSub reroutes share one CLUSTER SLOTS read. Refs #1203 (found by the review of #1154).

The problem

clusterPubSubRoute.pick ran refreshTopology (CLUSTER SLOTS on the probe node, then applyTopology) on every sharded reroute attempt. One SETSLOT that hits N sharded PubSubs sent N CLUSTER SLOTS to the same node, and each backoff step of an outage sent N again (the review's estimate: 1000 PubSubs at the 5 s cap, 200 per second).

Design: share a read that started after the trigger

ClusterClient.refreshTopologySince(ctx, since) uses the latest topology refresh if it started at or after since (running or done), and otherwise starts one. The check and the registration of the new refresh are one step under refMu, so callers that arrive together start one. The reconnect loop passes since = the moment the reason for the attempt arose: the loop's start for the first attempt (the trigger), the end of the failed attempt for the later ones. So a reroute never uses a table read before its own trigger, and one read serves every PubSub whose trigger it follows. A refresh cut short by its own caller's context (Close of that PubSub) is not shared: the joiners read again. A refresh that failed for a node's reason (the probe node is down) is shared like a table.

Why not the two options in the issue (measured on the fake cluster, TestShardedReroute*1203, the variants were temporary switches in refreshTopologySince):

variant one move, 200 PubSubs outage, 100 PubSubs, 1 s of the 50 ms-doubling backoff "reads a table newer than its trigger" test
main 200 396 pass
singleflight (join a running read only) 56-87 about 280 pass
100 ms window ("refreshed within X") 1 5 fail (the reroute reuses a read older than the move and dials the old owner, which answers -MOVED)
1 s window 1 0 fail (it never reads the table during the outage: 0 reads in 1 s)
since (this PR) 1 3-4 pass

(Three runs per variant; the variants were temporary switches in refreshTopologySince, the table is their output on the fake cluster. In an earlier 1.5 s run at a 10-80 ms backoff: main 2001, singleflight 1277, 100 ms window 15, since about 21.)

  • Plain singleflight collapses little: the backoff has 20 % jitter, so the 200 first attempts arrive over about 20 ms and most find no read running.
  • A clock window collapses the count but not the staleness: a read shortly before the move names the old owner, and a window hands it to the PubSub the move hit. The 1 s window's 0 is not a win, it is an outage during which no PubSub learns anything new.
  • since keeps the rule that makes each attempt worth its read ("a table newer than what made me look") and so the count follows the backoff steps (about one read per window in which no PubSub's trigger is newer than the last read), not the PubSub count. At the 5 s cap with 1000 PubSubs that is about one read per few seconds in place of 200 per second.

The two other things the issue asked to check

  • Non-sharded PubSub reroute: pubsubNode picks by address and reads no topology, so it never ran CLUSTER SLOTS; unchanged.
  • The cluster client's own refresh dedup: there is none. doWithRetryCmd refreshes on every -MOVED (N concurrent commands, N reads), the pipeline once per batch, refreshLoop every 60 s. This PR does not coalesce those (a command that was just redirected needs a table read after the redirect, the same freshness rule, but it is a separate hot path with its own semantics: proposed as a follow-up issue). What it does: plain refreshTopology now records itself as the latest refresh, so a PubSub may reuse a command-path or 60 s-loop read that started after its trigger.

RED / GREEN

RED, 8cd5a87 alone (main's code, go test ./driver/redis -run 1203 -v):

    pubsub_refresh_coalesce_1203_test.go:46: 200 sharded PubSubs, one slot move: 200 CLUSTER SLOTS
    pubsub_refresh_coalesce_1203_test.go:48: 200 CLUSTER SLOTS for one slot move that hit 200 PubSubs, want about one (at most 5)
--- FAIL: TestShardedRerouteOfManyPubSubsRefreshesTheTopologyOnce1203 (0.12s)
--- PASS: TestShardedRerouteDoesNotReuseARefreshOlderThanItsTrigger1203 (0.06s)
    pubsub_refresh_coalesce_1203_test.go:120: 100 PubSubs, 1 s of an outage (backoff 50, 100, 200, 400 ms ...): 396 CLUSTER SLOTS
    pubsub_refresh_coalesce_1203_test.go:122: 396 CLUSTER SLOTS in 1 s for 100 PubSubs, want at most 20: the attempts are not shared
--- FAIL: TestShardedRerouteAttemptsOfAnOutageShareTheirRefreshes1203 (1.03s)
FAIL
FAIL	github.com/goceleris/celeris/driver/redis	1.381s
FAIL

TestShardedRerouteDoesNotReuseARefreshOlderThanItsTrigger1203 passes on main by design: it guards the freshness rule of the fix (a refresh done just before the move must not be reused; fails with a clock window).

GREEN, 20d166c: 200 PubSubs, one move: 1 CLUSTER SLOTS (main 200); 100 PubSubs, 1 s of an outage (backoff 50 ms doubling): 3-4 (main 396).

pubsub_refresh_coalesce_1203_test.go:46: 200 sharded PubSubs, one slot move: 1 CLUSTER SLOTS
--- PASS: TestShardedRerouteOfManyPubSubsRefreshesTheTopologyOnce1203 (0.10s)
--- PASS: TestShardedRerouteDoesNotReuseARefreshOlderThanItsTrigger1203 (0.06s)
    pubsub_refresh_coalesce_1203_test.go:120: 100 PubSubs, 1 s of an outage (backoff 50, 100, 200, 400 ms ...): 4 CLUSTER SLOTS
--- PASS: TestShardedRerouteAttemptsOfAnOutageShareTheirRefreshes1203 (1.02s)
ok  	github.com/goceleris/celeris/driver/redis	1.379s

Behaviour notes

  • Unexported API change only: pubsubRoute.pick gained a since time.Time argument (the sentinel route ignores it); ClusterClient gets refMu/refLast; api/ is untouched.
  • The constructors' fallback (openPubSub) passes time.Now(): it needs a read of its own, so it does not join an earlier one (a refresh that started before the call is not newer than it), but concurrent calls and later PubSub reroutes may share the read it starts.
  • A PubSub may now get a table from a read another goroutine started. Same freshness as its own would have had (started after its trigger), same bound (a joiner waits for that read or its own context, the read has its leader's DialTimeout bound).
  • Failure sharing: a failed read is shared by every PubSub whose trigger precedes it, which on main was not the case for a probe node that is down but not yet marked failing (the first failed read marks it and later PubSubs skip it). Review r6 measured it: head 5/30 slow failover runs at 172-183 ms against main 5/30 at 130-154 ms, one backoff step lost for the herd. Fixed below (the read asks the next node).
  • Not changed: each reroute attempt still dials, replays and backs off as before.

Release note

A cluster shard PubSub reroute no longer sends its own CLUSTER SLOTS: PubSubs hit together by a slot move, or retrying the same outage, share one topology read that started after their trigger. With 200 sharded PubSubs on one node, one slot move costs one CLUSTER SLOTS instead of 200, and an outage costs about one per backoff window instead of one per PubSub per attempt.

Review r6 (APPROVE-WITH-FIXES) and what changed

  • F1 (MAJOR, test gap, done): TestShardedRerouteDoesNotJoinARunningRefreshOlderThanItsTrigger1203 holds one CLUSTER SLOTS reply after its table snapshot (a 3-line rpSlotsHold hook in the fake's writeSlots), moves the slot, lets the PubSub pick, releases the reply: the old owner must get no SSUBSCRIBE. Mutant "since enforced only for finished reads" now fails it (before, it passed all three tests); plain singleflight and the 100 ms window fail it too.
  • F2 (done): TestRefreshJoinerReadsAgainWhenTheReadItJoinedWasCancelled1203: the owner of the read is cancelled, the joiner gets nil after exactly one extra read. Mutant "drop the cut retry" now fails it.
  • F3 (done, in this PR): readTopology asks up to three nodes (next untried, not marked failing first) while the caller's context is alive, instead of one. TestTopologyReadTriesTheNextNodeWhenTheProbeIsDown1203 (RED on the previous head: the read returned the dial error of the dead probe). Also helps every other refreshTopology caller (command path). The wording above is corrected.
  • F4 (documented): refreshTopologySince states that a read that started at the same clock tick as since is shared, and why it is harmless (the reroute loop takes since a whole jittered backoff, at least 40 ms, before it asks; a wrong share costs one backoff step and the next attempt's since is newer). Kept the non-strict comparison: a strict one stops same-tick callers from coalescing on a coarse clock.
  • F5 (done): defer close(f.done).
  • Next step, not in this PR: doWithRetry and the pipeline refresh once per -MOVED with no sharing (a larger herd on a hotter path). It is a separate v1.6.0 issue with refreshTopologySince as its primitive.
  • Evidence: -race -run 1203 -count=20 ok on macOS (36 s) and Linux docker --cpus 2 (36 s); whole driver/redis package -race on Linux docker ok (169 s); golangci-lint (GOOS=linux) clean for the touched files (the two errcheck findings in untouched *_linux_test.go files are on main too). The new tests take 0.4 s, 0.05 s and 0.00 s.

Windows fix (required check "Portable fallback (windows-latest)" failed on the previous head)

Two of the new tests failed there (...DoesNotReuseARefreshOlderThanItsTrigger1203: "the old owner got 1 SSUBSCRIBE after the move"; ...DoesNotJoinARunningRefreshOlderThanItsTrigger1203). Cause: the "read started after my trigger" rule compared monotonic timestamps, and the Windows clock is too coarse to order two instants microseconds apart (review r6 F4 was right that this is not harmless: it let a reroute use a table read before the move). The rule is now a logical sequence: reads get a sequence number assigned under refMu when they are registered (topoRefresh.seq); a caller takes a topoMark (the number of reads started so far) when its reason to look arises, and may share a read only if its sequence is strictly greater. There is no clock in the rule at all, so it is the same on any clock (no need to simulate a 16 ms clock: nothing reads one). The "same tick" paragraph of the doc comment is gone; pubsubRoute gains topoMark() and pick takes a topoMark. New test TestRefreshTopologySinceOrdersByRegistrationNotByTime1203: 300 rounds of back-to-back reads and marks (the calls a coarse clock ties) checking that a mark taken after a read never shares it and one taken before always does. Existing tests kept as they were (they use no clock-dependent sleeps; the 400 ms/1 s waits only give the loops time to run). -race -run 1203 -count=20 ok on macOS (41 s) and Linux docker (42 s); whole package -race on Linux ok (177 s); GOOS=windows go vet clean.

…Refs #1203)

One slot move that hits 200 sharded PubSubs sends 200 CLUSTER SLOTS to the same
node, and 100 PubSubs in a lasting outage (50 ms doubling backoff) send several
hundred in 1 s. The first and third tests fail on main; the second (a reroute
reads a table newer than its trigger) passes on main and guards the fix.
…efs #1203)

Every sharded reroute attempt ran its own CLUSTER SLOTS on the probe node: one
SETSLOT that hit N PubSubs cost N, and each backoff step of an outage N again.

refreshTopologySince(ctx, since) uses the latest topology refresh if it started
at or after since, running or done, and otherwise starts one (check and
registration are one step under refMu, so callers that arrive together start
one). The reconnect loop passes the moment its reason to look arose (the
trigger for the first attempt, the end of the failed attempt for the others), so
a reroute never uses a table read before its own trigger. A refresh cut short by
its own caller's context is not shared; a joiner reads again.

Measured on the fake cluster (200 PubSubs, one SETSLOT): 200 CLUSTER SLOTS ->
1; 100 PubSubs in a 0.8 s outage at a 10-80 ms backoff: 1132 -> 12. Plain
singleflight (join only a running read) gives 86 and about 1277 (1.5 s run); a
clock window gives one and 15 but reuses reads older than the trigger.

The route interface's pick gains the since argument; the sentinel route ignores
it. Regular (non-sharded) PubSub reroutes read no topology, so are unchanged.
@FumingPower3925 FumingPower3925 added this to the v1.6.0 milestone Nov 1, 2026
@coderabbitai

coderabbitai Bot commented Nov 1, 2026 •

Copy link
Copy Markdown

Review in Change Stack →

Caution

Review failed

The pull request is closed.

  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@codecov

codecov Bot commented Nov 1, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 91.78082% with 6 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
driver/redis/cluster.go 90.62% 6 Missing ⚠️

📢 Thoughts on this report? Let us know!

…elled leader, a dead probe node (Refs #1203)

- a topology read that is still RUNNING when the move lands began before the
  PubSub's trigger (its table was taken before the move, the fake holds the
  reply): the reroute must not join it. A variant applying the trigger rule to
  finished reads only passes the earlier staleness test;
- the caller whose read others joined is cancelled: the joiner reads again
  instead of returning the other caller's context error (a variant without the
  retry passed every test);
- a probe node that is down and not yet marked failing must not fail the read
  when other nodes are alive (fails on the head: the read returns the dial
  error, and every PubSub that shared it loses a backoff step).
…wn; close its done on a panic (Refs #1203)

Review r6 of #1211:
- F3: readTopology asked one node, and a shared read now fails for every PubSub
  that joined it, where on main the first failed read marked the node failing
  and the later PubSubs skipped it. It now asks up to three nodes (the next not
  tried, not marked failing first) while the caller's context is alive. Also
  helps the command path's refreshes.
- F5: runRefresh defers close(done): a panic in the read no longer leaves
  joiners waiting on a refresh that never ends.
- F4: the comment of refreshTopologySince states that a read that started at the
  same clock tick as since is shared, and why that is harmless.
…amp (Refs #1203)

The "a read that started after my trigger" rule compared monotonic timestamps.
On Windows the clock is too coarse to order two instants a few microseconds
apart: a read that started in the same tick as the trigger compared equal or
inverted, and the reroute used a table read before the move (the required
Portable fallback (windows-latest) job failed TestShardedRerouteDoesNotReuseA
RefreshOlderThanItsTrigger1203 and ...DoesNotJoinARunningRefreshOlderThanItsTrigger1203).

Reads now carry a sequence number, assigned under refMu when they are registered
(topoRefresh.seq, ClusterClient.refSeq). A caller takes a topoMark (the number of
reads started so far) when its reason to look arises, and may share a read only
if its sequence number is greater: strictly later in the order of registration.
No clock is involved. pubsubRoute gains topoMark(); pick takes a topoMark instead
of a time. The "same tick" paragraph of the doc comment goes.

TestRefreshTopologySinceOrdersByRegistrationNotByTime1203 runs 300 rounds of
back-to-back reads and marks (the calls a coarse clock would tie) and checks
that a mark taken after a read never shares it and one taken before always does.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant