Skip to content

driver/redis: commands that meet the same MOVED or lost primary share one topology read (v1.6.0, Refs #1214) - #1215

Merged
FumingPower3925 merged 5 commits into
mainfrom
fix/redis-cluster-moved-refresh-herd-1214
Nov 1, 2026
Merged

FumingPower3925 merged 5 commits into
mainfrom
fix/redis-cluster-moved-refresh-herd-1214

Conversation

@FumingPower3925

@FumingPower3925 FumingPower3925 commented Nov 1, 2026 •

Copy link
Copy Markdown
Contributor

driver/redis: commands that meet the same MOVED, or the same lost primary, share one topology read. Refs #1214 (found by the review of #1211, #1203).

The problem

doWithRetryCmd ran refreshTopology on every -MOVED, and on the primary-network-error path; the pipeline ran one per batch. When a hot slot moves, every in-flight command on it gets its own -MOVED and sends its own CLUSTER SLOTS to the probe node, and each applyTopology takes c.mu to rebuild 16384 slots. Until the probe node reflects the move the refreshed table is still stale, so the storm repeats per command.

Design, and where it departs from the issue's text (please judge this one)

The issue says to use refreshTopologySince (#1211) with a mark taken when the -MOVED reply is received. I did that first. It does not work on this path, measured with the tests of this PR (macOS, one run each):

variant 200 GETs, fresh table 200 GETs, probe node still stale 50 pipelines 100 GETs, primary lost
main 200 200 50 100
plain topoMark() at the reply (mutant M8) 192 196 50 99
event marks (this PR) 2-6 1-6 1-3 1-3

Why: in #1211 the PubSub loop takes its mark at the top of the loop, before a jittered backoff of at least 40 ms, so all marks exist before anyone registers a read and they share one. The command path has no such gap: each command takes its mark as its reply arrives and registers its read right after, so by the time the next reply is read refSeq already includes the previous command's read, and a mark is never below the latest registration. Nothing is shared.

So the mark is taken by evidence, still on the sequence rule and with no clock (eventMark in cluster.go, all under refMu):

  • A -MOVED slot addr is an event keyed on the slot (m<slot>, target addr); a lost primary is an event keyed on its address (l<addr>). The first command that sees an event takes the mark now (refSeq). Another command that sees the same event (same key, same target) while it is unresolved takes the same mark.
  • Why that is sound: a read registered after the first reply was sent after the move happened, so for the same move it is as fresh for the second command as a read registered after its own reply.
  • An event is resolved when a read registered after its mark has ended without being cut short by its caller: refDone, the highest sequence number of such a read (set in runRefresh under refMu), is above the event's mark. Any read counts, however long ago it ended, so an old resolved event does not re-open while a newer read runs. The next reply for that key is then a new event with a new mark. A reply for the same slot naming another target replaces the event (the slot moved again) and takes a new mark.
  • The one hole: a slot that moves away and back to the same node between a reply and the end of the first read registered after it. The command that gets the second reply is then given a read that may predate its move, finds the old table, and its next -MOVED (the event is resolved by then) takes a new mark. Cost: one extra -MOVED round. (Wider than first stated: see the r7 follow-up below.)
  • The pipeline takes the latest of the marks of all the -MOVED in its batch (one batch with two moved slots must not join a read that predates the second move); the mark is taken after every reply of the round was received, which is later than receipt, i.e. the safe side.
  • The redirect itself is unchanged: the command goes to the address of its -MOVED, whatever the refreshed table says (the stale-probe case below checks every GET lands on the new owner).
  • Map bound: at most one event per slot plus one per lost node; past 1024 entries the resolved ones are dropped.

The issue's last item (readTopology falling back to another node when the probe is down) was done in #1211 (TestTopologyReadTriesTheNextNodeWhenTheProbeIsDown1203), so nothing to do here.

Commits: 94cb585 the tests (RED on main), ab652b3 the fix.

RED / GREEN

RED, 94cb585 alone (macOS arm64, go test ./driver/redis -run 1214 -v; main's code):

200 GETs on a moved slot (fresh-table): 200 CLUSTER SLOTS ... want at most 20
200 GETs on a moved slot (probe-node-still-stale): 200 CLUSTER SLOTS ... want at most 20
--- FAIL: TestMovedHerdRefreshesTheTopologyAFewTimes1214 (fresh-table, probe-node-still-stale)
50 pipelines on a moved slot: 50 CLUSTER SLOTS ... want at most 10
--- FAIL: TestMovedHerdOfPipelinesRefreshesTheTopologyAFewTimes1214
100 GETs on a slot whose primary died: 100 CLUSTER SLOTS ... want at most 20
--- FAIL: TestPrimaryLostHerdRefreshesTheTopologyAFewTimes1214
--- PASS: TestMovedCommandDoesNotAdoptARunningReadOlderThanItsMoved1214 (moved, primary-lost)
--- PASS: TestMovedEventIsResolvedByTheReadThatFollowsIt1214
--- PASS: TestMovedToAnotherNodeIsANewEvent1214
--- PASS: TestPipelineOfTwoMovesNeedsAReadAfterBoth1214 (old-event-first, old-event-last)

The last four pass on main by design: main reads the table for every reply, so they guard the freshness rules of the fix (a command never adopts a read older than its reply; an event is resolved; another target is a new event; a batch needs a read after both moves). They fail on the mutants below.

GREEN, ab652b3: all pass. Counts over -race -run 1214 -count=20:

macOS arm64 Linux golang:1.27.3 --cpus 2
200 GETs, fresh table 1-5 2-6
200 GETs, probe node still stale 1-4 1-6
50 pipelines 1-3 1-3
100 GETs, primary lost 1-2 1-3

Bounds in the tests: 20 for the GET herds and the lost primary, 10 for the pipelines (main: 200 / 200 / 50 / 100). Each new test takes under 0.05 s (the whole 1214 set 0.3 s). No sleep and no clock decides a result: the held reads use the rpSlotsHold hook of #1211, the herds are released together.

Mutants (each applied alone to ab652b3's cluster.go, go test -run 1214 -count=3, macOS)

mutant tests that fail
M1 -MOVED site back to refreshTopology TestMovedHerdRefreshes... (both)
M2 network-error site back to refreshTopology TestPrimaryLostHerdRefreshes...
M3 pipeline site back to refreshTopology TestMovedHerdOfPipelinesRefreshes...
M4 ignore the mark (plain singleflight) TestMovedCommandDoesNotAdopt... (moved, primary-lost), TestMovedEventIsResolved..., TestMovedToAnotherNodeIsANewEvent, TestPipelineOfTwoMoves... (both), the four herd tests
M5 an event is never resolved TestMovedEventIsResolvedByTheReadThatFollowsIt
M6 pipeline: mark of the first -MOVED only TestPipelineOfTwoMovesNeedsAReadAfterBoth (old-event-first)
M6b pipeline: mark of the last -MOVED only TestPipelineOfTwoMovesNeedsAReadAfterBoth (old-event-last)
M7 event keyed on the slot, target not compared TestMovedToAnotherNodeIsANewEvent
M8 plain topoMark() at the reply (the issue's literal design) the four herd tests

Checks

  • Whole driver/redis package, -race -count=1, Linux docker golang:1.27.3 --cpus 2: ok (176 s). That run was on 75ac220; git diff 75ac220 ab652b3 -- driver/ is only the movedHerdBound constant (30 to 20) and a comment.
  • GOOS=windows go vet ./driver/redis: clean. golangci-lint v2.15.0 GOOS=linux: nothing in the files of this PR (two errcheck findings in untouched *_linux_test.go files are on main too).
  • Not run locally: the windows-latest and macos-latest Portable fallback jobs (they run driver/redis). The herd tests log the counts they saw; if CI shows a count above about 10 on a runner, the bound needs a look.
  • The fake cluster gained a GET handler (-MOVED for a slot it does not own) and getCmds/getMoved counters; test files only.

Behaviour notes

Release note

A Redis Cluster client no longer sends one CLUSTER SLOTS per -MOVED reply. Commands (and pipeline batches) that hit the same slot move, or lose the same primary, share one topology read that started after the first of their replies: 200 concurrent GETs on a moved slot cost 2-6 CLUSTER SLOTS instead of 200, and each command is still redirected to the address in its own -MOVED.

Review r7 follow-up (commit c9f5a31)

  • MAJOR, allocation in the pipeline scan (fixed): refreshForMoved declared var rerr before the errs[i] == nil check, so it escaped (errors.As) and allocated for every command of every pipeline. The nil check now comes first. TestRefreshForMovedAllocatesNothingWhenNothingFailed1214 (AllocsPerRun over 100 nil errors = 0; fails on the old declaration order). A/A twin (scratch testing.Benchmark, 4 runs each, macOS, same fake 3-node cluster): single Get main 2 allocs/op, PR 2 allocs/op (about 20 us/op both); 100-GET pipeline main 524 allocs/op, PR 524 allocs/op (about 1.3 ms/op both; the first PR run, 576, is the conn dial warm-up).
  • MINOR-1, sticky resolution (fixed): refDone is the highest sequence number of a read that ended without being cut short (set in runRefresh under refMu); an event is resolved when refDone > mark, however long ago, so an old resolved event no longer re-opens whenever a newer read runs. The reviewer's probe is now TestMovedEventStaysResolvedWhenALaterReadIsRunning1214 (the old "latest read only" rule waits on the held read; mutant fails it). Herd counts unchanged.
  • MINOR-2, map pruning (fixed): slot events are a fixed [16384]topoEvent array (O(1), no scan, no pruning); lost-primary events are a map keyed by node address, pruned past 256 entries (nodes are few).
  • MINOR-3, tests (added): TestTwoSlotsMovedToTheSameNodeDoNotShareARead1214 (mutant "key on the target only" fails it; it survived every test before) and TestLostPrimaryDoesNotTakeAReadHeldForAMoved1214 (mutant "a lost primary takes mark 0 / shares any read" fails it, with two older tests).
  • NIT, exact herd test (added): TestMovedHerdSharesTheReadInFlightExactly1214 holds the first read after its table is taken, waits until all 200 commands have their -MOVED and sit in refreshTopologySince (goroutine stacks), releases, and expects exactly 1 CLUSTER SLOTS; it does not depend on the speed of the machine. The plain-topoMark mutant fails it.
  • NIT, cut-short read counts as resolution: documented in eventMarkLocked (a cut-short read does not set refDone; counting it would only cost an extra read). The mutant survives all tests on purpose; not asserted.
  • NIT, test header: describes the event marks now.
  • Not changed (filed separately by the coordinator): an older read ending after a newer one overwrites the table.

Mutants on c9f5a31 (-run 1214 -count=3, macOS): resolution by the latest read only fails TestMovedEventStaysResolvedWhenALaterReadIsRunning; key on target only fails TestTwoSlotsMovedToTheSameNode...; lost mark 0 fails TestLostPrimaryDoesNotTakeAReadHeld... (+2); rerr before the nil check fails TestRefreshForMovedAllocatesNothing...; no read ever resolves fails TestMovedEventIsResolved... and ...StaysResolved...; plain topoMark fails the three herd tests including the exact one; ignore-the-mark fails 10 tests; cut-short counts as resolution survives (see above).

Checks on c9f5a31: -race -run 1214 -count=20 Linux docker --cpus 2 ok; counts (CLUSTER SLOTS) fresh 2-6, probe-stale 1-6, pipelines 1-4, lost 1-2 (bounds 20/10); macOS ok. Whole driver/redis -race -count=1 Linux ok (176 s). GOOS=windows go vet clean; golangci-lint (own cache) only the two pre-existing errcheck findings.

Review r7 delta follow-up (commit 920672d, on top of a merge of origin/main)

  • MINOR, idle cost of the event table (fixed): the 16384-entry table of slot events was inline in every ClusterClient (it doubled the client and the GC scans it). It is now a pointer allocated by the first -MOVED (under refMu), so a client that never meets one pays 8 bytes. TestEventTableIsAllocatedByTheFirstMoved1214 checks that unsafe.Sizeof(ClusterClient{}) is the two slot tables plus at most 2 KiB, that a client with no -MOVED holds no table, and that the first -MOVED allocates it. Mutant "allocate it in NewClusterClient" fails it.
  • NIT, max in refDone (guarded): TestRefDoneNeverGoesDown1214 holds an older read, ends a newer one, releases the older, and expects refDone unchanged. Mutant "refDone = f.seq" fails it.
  • Design bullet corrected above (the earlier text described the "latest read only" rule that the r7 follow-up replaced).
  • Checks on 920672d: -race -run 1214 -count=20 macOS ok; Linux docker and whole-package results below.
  • Linux docker --cpus 2: -race -run 1214 -count=20 ok (CLUSTER SLOTS: fresh 1-6, probe-stale 2-6, pipelines 1-3, lost 1-2); whole driver/redis -race -count=1 ok (176 s). GOOS=windows go vet clean; golangci-lint (own cache) only the two pre-existing errcheck findings.

@FumingPower3925 FumingPower3925 added this to the v1.6.0 milestone Nov 1, 2026
@coderabbitai

coderabbitai Bot commented Nov 1, 2026 •

Copy link
Copy Markdown

Review in Change Stack →

Caution

Review failed

The pull request is closed.

  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

…n the pipeline MOVED scan, tests for the event rules (Refs #1214)
@codecov

codecov Bot commented Nov 1, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 93.75000% with 3 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
driver/redis/cluster.go 93.75% 3 Missing ⚠️

📢 Thoughts on this report? Let us know!

@FumingPower3925
FumingPower3925 marked this pull request as ready for review November 1, 2026 16:17
@FumingPower3925
FumingPower3925 merged commit 6087075 into main Nov 1, 2026
56 checks passed
@FumingPower3925
FumingPower3925 deleted the fix/redis-cluster-moved-refresh-herd-1214 branch November 1, 2026 16:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant