Repository navigation
driver/redis: commands that meet the same MOVED or lost primary share one topology read (v1.6.0, Refs #1214) - #1215
Merged
Conversation
… one topology read (Refs #1214)
|
Caution Review failedThe pull request is closed.
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. |
…n the pipeline MOVED scan, tests for the event rules (Refs #1214)
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
…ed-refresh-herd-1214
…ests for the table and for refDone only rising (Refs #1214)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
driver/redis: commands that meet the same MOVED, or the same lost primary, share one topology read. Refs #1214 (found by the review of #1211, #1203).
The problem
doWithRetryCmdranrefreshTopologyon every-MOVED, and on the primary-network-error path; the pipeline ran one per batch. When a hot slot moves, every in-flight command on it gets its own-MOVEDand sends its ownCLUSTER SLOTSto the probe node, and eachapplyTopologytakesc.muto rebuild 16384 slots. Until the probe node reflects the move the refreshed table is still stale, so the storm repeats per command.Design, and where it departs from the issue's text (please judge this one)
The issue says to use
refreshTopologySince(#1211) with a mark taken when the-MOVEDreply is received. I did that first. It does not work on this path, measured with the tests of this PR (macOS, one run each):topoMark()at the reply (mutant M8)Why: in #1211 the PubSub loop takes its mark at the top of the loop, before a jittered backoff of at least 40 ms, so all marks exist before anyone registers a read and they share one. The command path has no such gap: each command takes its mark as its reply arrives and registers its read right after, so by the time the next reply is read
refSeqalready includes the previous command's read, and a mark is never below the latest registration. Nothing is shared.So the mark is taken by evidence, still on the sequence rule and with no clock (
eventMarkincluster.go, all underrefMu):-MOVED slot addris an event keyed on the slot (m<slot>, targetaddr); a lost primary is an event keyed on its address (l<addr>). The first command that sees an event takes the mark now (refSeq). Another command that sees the same event (same key, same target) while it is unresolved takes the same mark.refDone, the highest sequence number of such a read (set inrunRefreshunderrefMu), is above the event's mark. Any read counts, however long ago it ended, so an old resolved event does not re-open while a newer read runs. The next reply for that key is then a new event with a new mark. A reply for the same slot naming another target replaces the event (the slot moved again) and takes a new mark.-MOVED(the event is resolved by then) takes a new mark. Cost: one extra-MOVEDround. (Wider than first stated: see the r7 follow-up below.)-MOVEDin its batch (one batch with two moved slots must not join a read that predates the second move); the mark is taken after every reply of the round was received, which is later than receipt, i.e. the safe side.-MOVED, whatever the refreshed table says (the stale-probe case below checks every GET lands on the new owner).The issue's last item (
readTopologyfalling back to another node when the probe is down) was done in #1211 (TestTopologyReadTriesTheNextNodeWhenTheProbeIsDown1203), so nothing to do here.Commits:
94cb585the tests (RED on main),ab652b3the fix.RED / GREEN
RED,
94cb585alone (macOS arm64,go test ./driver/redis -run 1214 -v; main's code):The last four pass on main by design: main reads the table for every reply, so they guard the freshness rules of the fix (a command never adopts a read older than its reply; an event is resolved; another target is a new event; a batch needs a read after both moves). They fail on the mutants below.
GREEN,
ab652b3: all pass. Counts over-race -run 1214 -count=20:golang:1.27.3--cpus 2Bounds in the tests: 20 for the GET herds and the lost primary, 10 for the pipelines (main: 200 / 200 / 50 / 100). Each new test takes under 0.05 s (the whole 1214 set 0.3 s). No sleep and no clock decides a result: the held reads use the
rpSlotsHoldhook of #1211, the herds are released together.Mutants (each applied alone to
ab652b3'scluster.go,go test -run 1214 -count=3, macOS)-MOVEDsite back torefreshTopologyrefreshTopologyrefreshTopology-MOVEDonly-MOVEDonlytopoMark()at the reply (the issue's literal design)Checks
driver/redispackage,-race -count=1, Linux dockergolang:1.27.3--cpus 2: ok (176 s). That run was on75ac220;git diff 75ac220 ab652b3 -- driver/is only themovedHerdBoundconstant (30 to 20) and a comment.GOOS=windows go vet ./driver/redis: clean.golangci-lintv2.15.0GOOS=linux: nothing in the files of this PR (twoerrcheckfindings in untouched*_linux_test.gofiles are on main too).driver/redis). The herd tests log the counts they saw; if CI shows a count above about 10 on a runner, the bound needs a look.GEThandler (-MOVEDfor a slot it does not own) andgetCmds/getMovedcounters; test files only.Behaviour notes
ClusterClientgetsrefEvents.readTopologyalready asks up to three nodes (driver/redis: sharded PubSub reroutes share one CLUSTER SLOTS read (v1.6.0, Refs #1203) #1211).rctx, as before.Release note
A Redis Cluster client no longer sends one
CLUSTER SLOTSper-MOVEDreply. Commands (and pipeline batches) that hit the same slot move, or lose the same primary, share one topology read that started after the first of their replies: 200 concurrent GETs on a moved slot cost 2-6CLUSTER SLOTSinstead of 200, and each command is still redirected to the address in its own-MOVED.Review r7 follow-up (commit
c9f5a31)refreshForMoveddeclaredvar rerrbefore theerrs[i] == nilcheck, so it escaped (errors.As) and allocated for every command of every pipeline. The nil check now comes first.TestRefreshForMovedAllocatesNothingWhenNothingFailed1214(AllocsPerRunover 100 nil errors = 0; fails on the old declaration order). A/A twin (scratchtesting.Benchmark, 4 runs each, macOS, same fake 3-node cluster): singleGetmain 2 allocs/op, PR 2 allocs/op (about 20 us/op both); 100-GET pipeline main 524 allocs/op, PR 524 allocs/op (about 1.3 ms/op both; the first PR run, 576, is the conn dial warm-up).refDoneis the highest sequence number of a read that ended without being cut short (set inrunRefreshunderrefMu); an event is resolved whenrefDone > mark, however long ago, so an old resolved event no longer re-opens whenever a newer read runs. The reviewer's probe is nowTestMovedEventStaysResolvedWhenALaterReadIsRunning1214(the old "latest read only" rule waits on the held read; mutant fails it). Herd counts unchanged.[16384]topoEventarray (O(1), no scan, no pruning); lost-primary events are a map keyed by node address, pruned past 256 entries (nodes are few).TestTwoSlotsMovedToTheSameNodeDoNotShareARead1214(mutant "key on the target only" fails it; it survived every test before) andTestLostPrimaryDoesNotTakeAReadHeldForAMoved1214(mutant "a lost primary takes mark 0 / shares any read" fails it, with two older tests).TestMovedHerdSharesTheReadInFlightExactly1214holds the first read after its table is taken, waits until all 200 commands have their-MOVEDand sit inrefreshTopologySince(goroutine stacks), releases, and expects exactly 1CLUSTER SLOTS; it does not depend on the speed of the machine. The plain-topoMarkmutant fails it.eventMarkLocked(a cut-short read does not setrefDone; counting it would only cost an extra read). The mutant survives all tests on purpose; not asserted.Mutants on
c9f5a31(-run 1214 -count=3, macOS): resolution by the latest read only fails TestMovedEventStaysResolvedWhenALaterReadIsRunning; key on target only fails TestTwoSlotsMovedToTheSameNode...; lost mark 0 fails TestLostPrimaryDoesNotTakeAReadHeld... (+2);rerrbefore the nil check fails TestRefreshForMovedAllocatesNothing...; no read ever resolves fails TestMovedEventIsResolved... and ...StaysResolved...; plaintopoMarkfails the three herd tests including the exact one; ignore-the-mark fails 10 tests; cut-short counts as resolution survives (see above).Checks on
c9f5a31:-race -run 1214 -count=20Linux docker--cpus 2ok; counts (CLUSTER SLOTS) fresh 2-6, probe-stale 1-6, pipelines 1-4, lost 1-2 (bounds 20/10); macOS ok. Wholedriver/redis-race -count=1Linux ok (176 s).GOOS=windows go vetclean; golangci-lint (own cache) only the two pre-existingerrcheckfindings.Review r7 delta follow-up (commit
920672d, on top of a merge oforigin/main)ClusterClient(it doubled the client and the GC scans it). It is now a pointer allocated by the first-MOVED(underrefMu), so a client that never meets one pays 8 bytes.TestEventTableIsAllocatedByTheFirstMoved1214checks thatunsafe.Sizeof(ClusterClient{})is the two slot tables plus at most 2 KiB, that a client with no-MOVEDholds no table, and that the first-MOVEDallocates it. Mutant "allocate it inNewClusterClient" fails it.maxinrefDone(guarded):TestRefDoneNeverGoesDown1214holds an older read, ends a newer one, releases the older, and expectsrefDoneunchanged. Mutant "refDone = f.seq" fails it.920672d:-race -run 1214 -count=20macOS ok; Linux docker and whole-package results below.--cpus 2:-race -run 1214 -count=20ok (CLUSTER SLOTS: fresh 1-6, probe-stale 2-6, pipelines 1-3, lost 1-2); wholedriver/redis-race -count=1ok (176 s).GOOS=windows go vetclean; golangci-lint (own cache) only the two pre-existingerrcheckfindings.