Repository navigation
driver/redis: a cluster PubSub replays SSUBSCRIBE per slot and refuses a cross-slot shard call (v1.6.0, Refs #1204) - #1210
Merged
Conversation
…(Refs #1204) The replay after a reroute sent one SSUBSCRIBE for every channel (the base) or only the served slot's channels (#1154), and a single SSubscribe call across slots was sent, answered -CROSSSLOT by the cluster and the reply dropped. These tests fail on main: one SSUBSCRIBE per slot after a move, a refused cross-slot call (ErrCrossSlot), and the -MOVED of another node's slot leaving the served slot up. The fake cluster records the SSUBSCRIBE commands it gets.
…d call (Refs #1204) A cluster refuses a SSUBSCRIBE or SUNSUBSCRIBE whose channels span slots (-CROSSSLOT, checked on Redis 8.0.2), and pubsub mode drops that reply. - The reconnect replay sends one SSUBSCRIBE per slot instead of one for all channels (the base) or only the served slot's channels (#1154): the served slot first, then the others by slot, the channels of each sorted. A channel of another slot is subscribed again when the new node owns its slot, and its -MOVED is ignored when it does not (unchanged). A PubSub that is not routed by slots (a plain Client) still sends one command. - PubSub.SSubscribe on the PubSub of a ClusterClient, and ClusterClient.SSubscribe, return ErrCrossSlot for a call whose channels span slots, before anything is sent or tracked. - PubSub.SUnsubscribe on such a PubSub sends one SUNSUBSCRIBE per slot. Docs of ClusterClient.SSubscribe, PubSub.SSubscribe/SUnsubscribe and ErrCrossSlot updated.
|
Caution Review failedThe pull request is closed.
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. |
This was referenced Nov 1, 2026
Merged
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
…ce probes of #1204 (Refs #1204) Review r5 of #1210: - a held channel whose slot loses its owner (cluster-require-full-coverage no) is answered -CLUSTERDOWN Hash slot not served at every replay, which is a reroute trigger: the PubSub reroutes for ever (fails on the current head: 3 more conns in 1.2 s; main is stable). The fake cluster can leave a slot unbound (unbind) and answers -CLUSTERDOWN for it. - the replay sends another node's slot only to its owner: the test that expected the -MOVED round trip now expects the served slot's command alone. - a refused cross-slot constructor call dials no conn (the up-front check is pinned). - redis_cluster conformance: two slots of one node survive a replay (CLIENT KILL), the served slot moving to the node of the second slot, ErrCrossSlot for a cross-slot SSubscribe, SUnsubscribe across slots. - stale "documented undefined behaviour" wording of the two-slot tests.
…it (Refs #1204) Review r5 of #1210 (F1, regression): the per-slot replay brought back the slots a PubSub does not serve at every reroute. One that has no owner (DELSLOTS with cluster-require-full-coverage no) is answered -CLUSTERDOWN Hash slot not served, which onControlError takes for a reroute trigger (the reply carries no slot number), so the replay of every reroute started the next one, for ever (real Redis 8.0.2: attempt 6 -> 9 over 10 s; main stable). The replay now sends the served slot always and any other slot only if the reroute's own topology (it has just refreshed it) says the node it landed on owns it (pubsubRoute.owns). A slot of another node would only draw -MOVED (and was ignored), a slot with no owner draws the -CLUSTERDOWN. The -CLUSTERDOWN trigger itself is unchanged: the reply cannot say which slot it is for, and one for the served slot is a real reason to look again. Also: the stale comment of onShardUnsubscribed (a channel of another slot is replayed, not dropped) and the SSubscribe doc.
…ILL TYPE pubsub (Refs #1204)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
driver/redis: a cluster PubSub replays SSUBSCRIBE per slot and refuses a cross-slot shard call. Refs #1204 (found by the review of #1154).
What main does today, measured on real Redis
The issue was filed against the head the review ran on. The first head of #1154 sent every shard channel in one
SSUBSCRIBEafter a reroute; the merged #1154 replays only the served slot's channels (servedSlotLocked, round-1 fix of the same finding), so "both channels silently dead" no longer happens through the replay on main (it still happens through a cross-slot constructor call, P1b below). What is left, probed on real Redis 8.0.2 (3 masters,golang:1.27.3,--cpus 2, probe codeTestReal*in the details block at the end; the same cases are now committed asredis_clusterconformance tests, see below):SSubscribe(a)thenps.SSubscribe(b), a and b on two masters, SPUBLISH x3cc.SSubscribe(a, b), two slots in one callErrCrossSlot, no PubSubredis-cli:SSUBSCRIBE {a}x {b}y,SUNSUBSCRIBE {b}y {t}z,SUNSUBSCRIBE {b}y {c}z-CROSSSLOTfor all threeSo the leftover bug on main is P2: the replay drops a channel of another slot that the new node does serve (and delivered before the reroute), plus the silent drop of the
-CROSSSLOTa cross-slot call gets. The server refusesSUNSUBSCRIBEacross slots too.The change
Commits:
16ef152the tests (RED on main),0961511the fix.reconnectLoopbuilds its commands fromshardReplayGroupsLocked: on a PubSub routed by slots (aClusterClient's) oneSSUBSCRIBEper slot, the served slot first, the others by slot number, the channels of each sorted (deterministic). A node that does not own a slot answers its command-MOVED, which is ignored for a slot the PubSub does not serve (unchanged,onControlError). A PubSub with no slot routing (a plainClient, a standalone server has no slots) still sends one command. Same-slot channels still go in one command, soTestClusterRerouteAnswersEveryChannelOfAMoveWithOneLoop887(two channels of{foo}, one SSUBSCRIBE) needed no change and passes as is; the new tests assert the per-slot commands.state.go), so there is no place to hand a server-CROSSSLOTback to its caller, and the PubSub has no error channel. The surfacing is therefore up front, on the return path every control call already has:PubSub.SSubscribeon aClusterClient's PubSub, andClusterClient.SSubscribe, return the existingErrCrossSlot(now documented for this use too) for a call whose channels span slots, before anything is sent or tracked (a refused call changes nothing;ClusterClient.SSubscribereturns no PubSub). Channels of different slots can still be added by separate calls.PubSub.SUnsubscribeacross slots does not fail: unsubscribing is the caller's clear intent, so it is sent as oneSUNSUBSCRIBEper slot (controlGroups).-CROSSSLOTcan therefore not come from a slot-routed PubSub any more; one that reaches a PubSub with no slot routing (a plainClientpointed at a cluster node) is still dropped, as every non-MOVED/CLUSTERDOWN error reply in pubsub mode is. That is the one case left, deliberately: a standalone Redis accepts a multi-slotSSUBSCRIBEand the client cannot tell.ClusterClient.SSubscribe(the old "mixing slots is undefined", "until the next reroute ... leaves it out" paragraph now states the exact behaviour),PubSub.SSubscribe,PubSub.SUnsubscribe,ErrCrossSlot.RED / GREEN
RED,
16ef152alone (main's code,go test ./driver/redis -run 1204 -v):(
TestStandalonePubSubSSubscribeAcrossSlotsIsNotRefused1204passes on main by design: it guards that a plainClientPubSub is not restricted.)GREEN: all five pass in 1.2 s.
go test ./driver/redis -race -run 1204 -count=20: ok (macOS, 21 s).driver/redispackage:-race -count=1macOS ok (37 s); Linux (golang:1.27.3,--cpus 2)-race -count=1ok (167 s) and plain-count=1ok (172 s).golangci-lintv2.15.0GOOS=linux: nothing in the files of this PR (the twoerrcheckfindings inclose_after_teardown_859_linux_test.go:135andpubsub_rig_engines_887_linux_test.go:77are on main too).GOOS=windows go vet: clean.Behaviour notes
cc.SSubscribe(ctx, "{a}x", "{b}y")andps.SSubscribe(ctx, "{a}x", "{b}y")on a cluster PubSub now returnErrCrossSlot. Before, they returned nil (the first with a PubSub) and delivered nothing (P1b). A caller that passed several channels of one slot (hash tags) is unaffected.ctlMu, as before one command: more writes after a reroute for a PubSub with channels in many slots; one for the usual one-slot PubSub.Review r5 (REQUEST-CHANGES) and what changed
DELSLOTSwithcluster-require-full-coverage no) is answered-CLUSTERDOWN Hash slot not served, whichonControlErrortakes for a reroute trigger (the reply carries no slot), so the replay of each reroute started the next one for ever (reviewer's real-Redis probe: attempt 6 -> 9 over 10 s, main stable). Fix: the replay sends the served slot always, and any other slot only to a node that owns it by the topology the reroute has just refreshed (pubsubRoute.owns). That also drops the-MOVEDround trip for another node's slot. I chose this over slot-gating-CLUSTERDOWNbecause the reply cannot say which slot it answers (a gate would have to ignore it for a PubSub with several slots, and a served slot that really is unbound is a reason to look again), and over ignoring it altogether for the same reason. The trigger itself is unchanged. Regression tests:TestClusterReplayOfAnUnboundSlotDoesNotRerouteForEver1204(fake cluster, RED on the previous head: 2 more conns in 1.2 s; the fake gainedunbindand answers-CLUSTERDOWNfor an unowned slot),TestClusterReplayDoesNotSendAnotherNodesSlot1204. The reviewer's real probeTestRvZZP5Unboundon the new head (Redis 8.0.2, 3 masters):attempt 1 -> 1, reconnecting=false, served slot delivered 40/40 (previous head: 6 -> 9).redis-cluster(7.4.8, 8.0.6, 8.6.2) runs-tags redis_cluster ./test/conformance/redis/...; the cases of the probe are now there (pubsub_reroute_cluster_r3_test.go):TestClusterSecondSlotOfTheSameNodeSurvivesAReplay(P2, conn killed withCLIENT KILL TYPE pubsub),TestClusterServedSlotMovesToTheNodeOfTheSecondSlot(P1-move),TestClusterSSubscribeAcrossSlotsReturnsErrCrossSlot,TestClusterSUnsubscribeAcrossSlotsRemovesBoth. Run against a local 3-master Redis 8.0.2 cluster (no replicas;golang:1.27.3,--cpus 2): on main the four fail for the intended reasons (P1-move: b not delivered after 30 s; cross-slot call returns a live PubSub; SUnsubscribe leaves both channels delivering; the P2 one fails earlier on main in my setup), on this head they pass (1.0 s, 0.8 s, 1.0 s, 2.1 s). Three older tests of that file (TestClusterForeignSUnsubscribeKeepsTheSubscriptionConn,...MoveOfASecondSlotOnTheSameNodeKeepsTheSubscriptionConn) fail in my local setup on main as well ("the owner has 0 connections holding a shard subscription"); they are green in CI, so I take it to be my setup and did not chase it. The stale "documented undefined behaviour" wording ofTestClusterSecondSlotOnOnePubSubKeepsTheFirstDeliveringand of its fake twin is fixed.ClusterClient.SSubscribefails it; before it passed).onShardUnsubscribedcomment.-MOVEDwas ignored, without the round trip).-race -run 1204 -count=20ok on macOS (58 s) and on Linux docker (58 s); wholedriver/redispackage-raceon Linux docker ok (170 s);golangci-lint(GOOS=linux, also with-tags redis_clusterfor the conformance directory) clean for the touched files.Release note
A
ClusterClientshard PubSub whose channels span slots now replays oneSSUBSCRIBEper slot after a reroute, so a channel of another slot that the new node owns stays subscribed (before, the replay left it out); a slot the new node does not own is not sent.ClusterClient.SSubscribeandPubSub.SSubscribenow returnErrCrossSlotfor one call whose channels span slots, instead of sending a command the cluster refuses and silently dropping its error.cc.SSubscribe(ctx, a, b)across slots now returns a nil*PubSubandErrCrossSlot; before, it returned a live PubSub that delivered nothing, so a caller that ignored the error and used the PubSub will now panic on the nil. No call site in the repository mixes slots in one call.PubSub.SUnsubscribeacross slots sends oneSUNSUBSCRIBEper slot (the cluster refuses it as one).The real-Redis probe used in the review (the cases are committed as conformance tests in the redis-cluster CI job; this is the original probe)
real.sh(run ingolang:1.27.3with--cpus 2,REAL_CLUSTER=127.0.0.1:7000,127.0.0.1:7001,127.0.0.1:7002;/srcis this branch,/srcmaina checkout of 7e73a14,zz_real_probe_test.gocopied into both):driver/redis/zz_real_probe_test.go:Release note addition: a channel whose slot later becomes owned by the PubSub's node is replayed again only at the next reroute (documented).