Skip to content

driver/redis: a cluster PubSub replays SSUBSCRIBE per slot and refuses a cross-slot shard call (v1.6.0, Refs #1204) - #1210

Merged
FumingPower3925 merged 7 commits into
mainfrom
fix/redis-pubsub-replay-by-slot-1204
Nov 1, 2026
Merged

FumingPower3925 merged 7 commits into
mainfrom
fix/redis-pubsub-replay-by-slot-1204

Conversation

@FumingPower3925

@FumingPower3925 FumingPower3925 commented Nov 1, 2026 •

Copy link
Copy Markdown
Contributor

driver/redis: a cluster PubSub replays SSUBSCRIBE per slot and refuses a cross-slot shard call. Refs #1204 (found by the review of #1154).

What main does today, measured on real Redis

The issue was filed against the head the review ran on. The first head of #1154 sent every shard channel in one SSUBSCRIBE after a reroute; the merged #1154 replays only the served slot's channels (servedSlotLocked, round-1 fix of the same finding), so "both channels silently dead" no longer happens through the replay on main (it still happens through a cross-slot constructor call, P1b below). What is left, probed on real Redis 8.0.2 (3 masters, golang:1.27.3, --cpus 2, probe code TestReal* in the details block at the end; the same cases are now committed as redis_cluster conformance tests, see below):

probe main (7e73a14) this PR
P1 SSubscribe(a) then ps.SSubscribe(b), a and b on two masters, SPUBLISH x3 a=1,1,1 b=0,0,0 a=1,1,1 b=0,0,0 (one conn, one node: b is answered -MOVED, unchanged and documented)
P1b cc.SSubscribe(a, b), two slots in one call returns a PubSub, nil error, both channels dead: a=0,0,0 b=0,0,0 (still on main: the "both dead" of the issue survives through this path) ErrCrossSlot, no PubSub
P2 a and b on the same master, delivered; conn killed (reconnect + replay); SPUBLISH again before kill a=1,1,1 b=1,1,1; after kill a=1,1,1 b=0,0,0 after kill a=1,1,1 b=1,1,1
raw redis-cli: SSUBSCRIBE {a}x {b}y, SUNSUBSCRIBE {b}y {t}z, SUNSUBSCRIBE {b}y {c}z -CROSSSLOT for all three (same server)

So the leftover bug on main is P2: the replay drops a channel of another slot that the new node does serve (and delivered before the reroute), plus the silent drop of the -CROSSSLOT a cross-slot call gets. The server refuses SUNSUBSCRIBE across slots too.

The change

Commits: 16ef152 the tests (RED on main), 0961511 the fix.

  • Replay grouped by slot. reconnectLoop builds its commands from shardReplayGroupsLocked: on a PubSub routed by slots (a ClusterClient's) one SSUBSCRIBE per slot, the served slot first, the others by slot number, the channels of each sorted (deterministic). A node that does not own a slot answers its command -MOVED, which is ignored for a slot the PubSub does not serve (unchanged, onControlError). A PubSub with no slot routing (a plain Client, a standalone server has no slots) still sends one command. Same-slot channels still go in one command, so TestClusterRerouteAnswersEveryChannelOfAMoveWithOneLoop887 (two channels of {foo}, one SSUBSCRIBE) needed no change and passes as is; the new tests assert the per-slot commands.
  • CROSSSLOT surfaced as an error return. Pubsub mode does not match error replies to commands (state.go), so there is no place to hand a server -CROSSSLOT back to its caller, and the PubSub has no error channel. The surfacing is therefore up front, on the return path every control call already has: PubSub.SSubscribe on a ClusterClient's PubSub, and ClusterClient.SSubscribe, return the existing ErrCrossSlot (now documented for this use too) for a call whose channels span slots, before anything is sent or tracked (a refused call changes nothing; ClusterClient.SSubscribe returns no PubSub). Channels of different slots can still be added by separate calls. PubSub.SUnsubscribe across slots does not fail: unsubscribing is the caller's clear intent, so it is sent as one SUNSUBSCRIBE per slot (controlGroups).
  • A server -CROSSSLOT can therefore not come from a slot-routed PubSub any more; one that reaches a PubSub with no slot routing (a plain Client pointed at a cluster node) is still dropped, as every non-MOVED/CLUSTERDOWN error reply in pubsub mode is. That is the one case left, deliberately: a standalone Redis accepts a multi-slot SSUBSCRIBE and the client cannot tell.
  • Docs updated: ClusterClient.SSubscribe (the old "mixing slots is undefined", "until the next reroute ... leaves it out" paragraph now states the exact behaviour), PubSub.SSubscribe, PubSub.SUnsubscribe, ErrCrossSlot.

RED / GREEN

RED, 16ef152 alone (main's code, go test ./driver/redis -run 1204 -v):

--- FAIL: TestClusterReplayGroupsShardChannelsBySlot1204 (2.06s)
--- FAIL: TestClusterReplayOfAnotherNodesSlotLeavesTheServedSlotUp1204 (2.07s)
--- FAIL: TestClusterSSubscribeAcrossSlotsIsCrossSlot1204 (0.01s)
    --- FAIL: TestClusterSSubscribeAcrossSlotsIsCrossSlot1204/constructor (0.00s)
    --- FAIL: TestClusterSSubscribeAcrossSlotsIsCrossSlot1204/control_call (0.01s)
--- FAIL: TestClusterSUnsubscribeAcrossSlotsSendsOneCommandPerSlot1204 (2.01s)
--- PASS: TestStandalonePubSubSSubscribeAcrossSlotsIsNotRefused1204 (0.01s)
FAIL
FAIL	github.com/goceleris/celeris/driver/redis	6.227s
FAIL

(TestStandalonePubSubSSubscribeAcrossSlotsIsNotRefused1204 passes on main by design: it guards that a plain Client PubSub is not restricted.)

GREEN: all five pass in 1.2 s.

  • go test ./driver/redis -race -run 1204 -count=20: ok (macOS, 21 s).
  • Whole driver/redis package: -race -count=1 macOS ok (37 s); Linux (golang:1.27.3, --cpus 2) -race -count=1 ok (167 s) and plain -count=1 ok (172 s).
  • golangci-lint v2.15.0 GOOS=linux: nothing in the files of this PR (the two errcheck findings in close_after_teardown_859_linux_test.go:135 and pubsub_rig_engines_887_linux_test.go:77 are on main too). GOOS=windows go vet: clean.

Behaviour notes

  • New error: cc.SSubscribe(ctx, "{a}x", "{b}y") and ps.SSubscribe(ctx, "{a}x", "{b}y") on a cluster PubSub now return ErrCrossSlot. Before, they returned nil (the first with a PubSub) and delivered nothing (P1b). A caller that passed several channels of one slot (hash tags) is unaffected.
  • A replay of N slots is N commands, in one pass under ctlMu, as before one command: more writes after a reroute for a PubSub with channels in many slots; one for the usual one-slot PubSub.
  • What the PubSub serves is unchanged: one conn on one node, route follows the first channel's slot, a move of another slot does not reroute it. What changes is that the other slots' channels are re-subscribed at every replay where the node owns them (before: only until the next reroute).

Review r5 (REQUEST-CHANGES) and what changed

  • F1 (MAJOR, regression, fixed): the per-slot replay brought back, at every reroute, the slots the PubSub does not serve. One with no owner (DELSLOTS with cluster-require-full-coverage no) is answered -CLUSTERDOWN Hash slot not served, which onControlError takes for a reroute trigger (the reply carries no slot), so the replay of each reroute started the next one for ever (reviewer's real-Redis probe: attempt 6 -> 9 over 10 s, main stable). Fix: the replay sends the served slot always, and any other slot only to a node that owns it by the topology the reroute has just refreshed (pubsubRoute.owns). That also drops the -MOVED round trip for another node's slot. I chose this over slot-gating -CLUSTERDOWN because the reply cannot say which slot it answers (a gate would have to ignore it for a PubSub with several slots, and a served slot that really is unbound is a reason to look again), and over ignoring it altogether for the same reason. The trigger itself is unchanged. Regression tests: TestClusterReplayOfAnUnboundSlotDoesNotRerouteForEver1204 (fake cluster, RED on the previous head: 2 more conns in 1.2 s; the fake gained unbind and answers -CLUSTERDOWN for an unowned slot), TestClusterReplayDoesNotSendAnotherNodesSlot1204. The reviewer's real probe TestRvZZP5Unbound on the new head (Redis 8.0.2, 3 masters): attempt 1 -> 1, reconnecting=false, served slot delivered 40/40 (previous head: 6 -> 9).
  • F2 (conformance, done): the CI job redis-cluster (7.4.8, 8.0.6, 8.6.2) runs -tags redis_cluster ./test/conformance/redis/...; the cases of the probe are now there (pubsub_reroute_cluster_r3_test.go): TestClusterSecondSlotOfTheSameNodeSurvivesAReplay (P2, conn killed with CLIENT KILL TYPE pubsub), TestClusterServedSlotMovesToTheNodeOfTheSecondSlot (P1-move), TestClusterSSubscribeAcrossSlotsReturnsErrCrossSlot, TestClusterSUnsubscribeAcrossSlotsRemovesBoth. Run against a local 3-master Redis 8.0.2 cluster (no replicas; golang:1.27.3, --cpus 2): on main the four fail for the intended reasons (P1-move: b not delivered after 30 s; cross-slot call returns a live PubSub; SUnsubscribe leaves both channels delivering; the P2 one fails earlier on main in my setup), on this head they pass (1.0 s, 0.8 s, 1.0 s, 2.1 s). Three older tests of that file (TestClusterForeignSUnsubscribeKeepsTheSubscriptionConn, ...MoveOfASecondSlotOnTheSameNodeKeepsTheSubscriptionConn) fail in my local setup on main as well ("the owner has 0 connections holding a shard subscription"); they are green in CI, so I take it to be my setup and did not chase it. The stale "documented undefined behaviour" wording of TestClusterSecondSlotOnOnePubSubKeepsTheFirstDelivering and of its fake twin is fixed.
  • F4 (done): the constructor test now asserts that no node accepted a conn for a refused cross-slot call (disabling the up-front check in ClusterClient.SSubscribe fails it; before it passed).
  • F5 (done): the stale onShardUnsubscribed comment.
  • Behaviour note (new): a channel of another slot is now sent at a reroute only if the node owns it by the refreshed topology; it is left out otherwise, with no error (the same as when its -MOVED was ignored, without the round trip).
  • Evidence for this round: -race -run 1204 -count=20 ok on macOS (58 s) and on Linux docker (58 s); whole driver/redis package -race on Linux docker ok (170 s); golangci-lint (GOOS=linux, also with -tags redis_cluster for the conformance directory) clean for the touched files.

Release note

A ClusterClient shard PubSub whose channels span slots now replays one SSUBSCRIBE per slot after a reroute, so a channel of another slot that the new node owns stays subscribed (before, the replay left it out); a slot the new node does not own is not sent. ClusterClient.SSubscribe and PubSub.SSubscribe now return ErrCrossSlot for one call whose channels span slots, instead of sending a command the cluster refuses and silently dropping its error. cc.SSubscribe(ctx, a, b) across slots now returns a nil *PubSub and ErrCrossSlot; before, it returned a live PubSub that delivered nothing, so a caller that ignored the error and used the PubSub will now panic on the nil. No call site in the repository mixes slots in one call. PubSub.SUnsubscribe across slots sends one SUNSUBSCRIBE per slot (the cluster refuses it as one).

The real-Redis probe used in the review (the cases are committed as conformance tests in the redis-cluster CI job; this is the original probe)

real.sh (run in golang:1.27.3 with --cpus 2, REAL_CLUSTER=127.0.0.1:7000,127.0.0.1:7001,127.0.0.1:7002; /src is this branch, /srcmain a checkout of 7e73a14, zz_real_probe_test.go copied into both):

set -e
export DEBIAN_FRONTEND=noninteractive
apt-get update -qq >/dev/null && apt-get install -y -qq redis-server redis-tools >/dev/null
redis-server --version
mkdir -p /r
for p in 7000 7001 7002; do mkdir -p /r/$p; (cd /r/$p && redis-server --port $p --cluster-enabled yes --cluster-config-file /r/$p/n.conf --daemonize yes --save "" --bind 127.0.0.1 --logfile /r/$p/log >/dev/null); done
sleep 1
yes yes | redis-cli --cluster create 127.0.0.1:7000 127.0.0.1:7001 127.0.0.1:7002 --cluster-replicas 0 >/dev/null
sleep 3
redis-cli -p 7000 cluster info | head -3
echo "--- raw replies"
redis-cli -p 7002 SSUBSCRIBE '{a}x' '{b}y' 2>&1 | head -3 &
sleep 1; wait || true
redis-cli -p 7000 SUNSUBSCRIBE '{b}y' '{t}z' 2>&1 | head -3
redis-cli -p 7000 SUNSUBSCRIBE '{b}y' '{c}z' 2>&1 | head -3
echo "=== MAIN"
cd /srcmain && go test ./driver/redis -run 'TestReal' -count=1 -v -timeout 120s 2>&1 | grep -E "probe|P1|P2|err=|^(---|ok|FAIL)"
echo "=== FIX"
cd /src && go test ./driver/redis -run 'TestReal' -count=1 -v -timeout 120s 2>&1 | grep -E "probe|P1|P2|err=|^(---|ok|FAIL)"

driver/redis/zz_real_probe_test.go:

package redis

import (
	"context"
	"fmt"
	"os"
	"strings"
	"sync"
	"testing"
	"time"
)

type realCol struct {
	mu sync.Mutex
	m  map[string]int
}

func realCollect(ps *PubSub) *realCol {
	c := &realCol{m: map[string]int{}}
	go func() {
		for msg := range ps.Channel() {
			c.mu.Lock()
			c.m[msg.Channel+"/"+msg.Payload]++
			c.mu.Unlock()
		}
	}()
	return c
}
func (c *realCol) get(k string) int { c.mu.Lock(); defer c.mu.Unlock(); return c.m[k] }

func realCC(t *testing.T) *ClusterClient {
	addr := os.Getenv("REAL_CLUSTER")
	if addr == "" {
		t.Skip("REAL_CLUSTER unset")
	}
	cc, err := NewClusterClient(ClusterConfig{Addrs: strings.Split(addr, ",")})
	if err != nil {
		t.Fatal(err)
	}
	t.Cleanup(func() { _ = cc.Close() })
	return cc
}

func realPub(cc *ClusterClient, chs ...string) string {
	ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
	defer cancel()
	var out []string
	for _, p := range []string{"m1", "m2", "m3"} {
		for _, ch := range chs {
			_, _ = cc.SPublish(ctx, ch, p)
		}
		time.Sleep(150 * time.Millisecond)
	}
	_ = out
	return ""
}

func realCounts(rc *realCol, ch string) string {
	return fmt.Sprintf("%d,%d,%d", rc.get(ch+"/m1"), rc.get(ch+"/m2"), rc.get(ch+"/m3"))
}

// P1: two slots on two masters, one PubSub (the reviewer's TestReviewRealMixedSlot).
func TestRealMixedSlotTwoNodes(t *testing.T) {
	cc := realCC(t)
	ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
	defer cancel()
	a, b := "{a}x", "{b}y"
	ps, err := cc.SSubscribe(ctx, a)
	if err != nil {
		t.Fatal(err)
	}
	defer ps.Close()
	rc := realCollect(ps)
	time.Sleep(300 * time.Millisecond)
	err = ps.SSubscribe(ctx, b)
	t.Logf("SSubscribe(b) err=%v", err)
	time.Sleep(3 * time.Second)
	realPub(cc, a, b)
	time.Sleep(500 * time.Millisecond)
	t.Logf("P1 a=%s b=%s", realCounts(rc, a), realCounts(rc, b))
}

// P1b: one call with two slots.
func TestRealOneCallTwoSlots(t *testing.T) {
	cc := realCC(t)
	ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
	defer cancel()
	a, b := "{a}x", "{b}y"
	ps, err := cc.SSubscribe(ctx, a, b)
	t.Logf("SSubscribe(a,b) err=%v ps=%v", err, ps != nil)
	if ps == nil {
		return
	}
	defer ps.Close()
	rc := realCollect(ps)
	time.Sleep(2 * time.Second)
	realPub(cc, a, b)
	time.Sleep(500 * time.Millisecond)
	t.Logf("P1b a=%s b=%s", realCounts(rc, a), realCounts(rc, b))
}

// P2: two slots on the SAME master, one PubSub; then the conn is killed (reconnect + replay).
func TestRealSameNodeTwoSlotsReplay(t *testing.T) {
	cc := realCC(t)
	ctx, cancel := context.WithTimeout(context.Background(), 60*time.Second)
	defer cancel()
	a := "{b}y"
	na := cc.route(Slot(a))
	var b string
	for i := 0; ; i++ {
		b = fmt.Sprintf("{t%d}z", i)
		if Slot(b) != Slot(a) && cc.route(Slot(b)) == na {
			break
		}
	}
	ps, err := cc.SSubscribe(ctx, a)
	if err != nil {
		t.Fatal(err)
	}
	defer ps.Close()
	rc := realCollect(ps)
	time.Sleep(300 * time.Millisecond)
	t.Logf("SSubscribe(b) err=%v (a slot %d, b slot %d, same node %s)", ps.SSubscribe(ctx, b), Slot(a), Slot(b), na.addr)
	time.Sleep(500 * time.Millisecond)
	realPub(cc, a, b)
	time.Sleep(300 * time.Millisecond)
	t.Logf("P2 before kill a=%s b=%s", realCounts(rc, a), realCounts(rc, b))
	// kill the pubsub conns of that node
	ps.mu.Lock()
	conn := ps.conn
	ps.mu.Unlock()
	_ = conn.closeWithErr(fmt.Errorf("probe kill"))
	time.Sleep(3 * time.Second)
	for _, p := range []string{"n1", "n2", "n3"} {
		_, _ = cc.SPublish(ctx, a, p)
		_, _ = cc.SPublish(ctx, b, p)
		time.Sleep(150 * time.Millisecond)
	}
	time.Sleep(500 * time.Millisecond)
	t.Logf("P2 after kill  a=%s b=%s", fmt.Sprintf("%d,%d,%d", rc.get(a+"/n1"), rc.get(a+"/n2"), rc.get(a+"/n3")), fmt.Sprintf("%d,%d,%d", rc.get(b+"/n1"), rc.get(b+"/n2"), rc.get(b+"/n3")))
}

Release note addition: a channel whose slot later becomes owned by the PubSub's node is replayed again only at the next reroute (documented).

…(Refs #1204)

The replay after a reroute sent one SSUBSCRIBE for every channel (the base) or
only the served slot's channels (#1154), and a single SSubscribe call across
slots was sent, answered -CROSSSLOT by the cluster and the reply dropped.
These tests fail on main: one SSUBSCRIBE per slot after a move, a refused
cross-slot call (ErrCrossSlot), and the -MOVED of another node's slot leaving
the served slot up. The fake cluster records the SSUBSCRIBE commands it gets.
…d call (Refs #1204)

A cluster refuses a SSUBSCRIBE or SUNSUBSCRIBE whose channels span slots
(-CROSSSLOT, checked on Redis 8.0.2), and pubsub mode drops that reply.

- The reconnect replay sends one SSUBSCRIBE per slot instead of one for all
  channels (the base) or only the served slot's channels (#1154): the served
  slot first, then the others by slot, the channels of each sorted. A channel of
  another slot is subscribed again when the new node owns its slot, and its
  -MOVED is ignored when it does not (unchanged). A PubSub that is not routed by
  slots (a plain Client) still sends one command.
- PubSub.SSubscribe on the PubSub of a ClusterClient, and
  ClusterClient.SSubscribe, return ErrCrossSlot for a call whose channels span
  slots, before anything is sent or tracked.
- PubSub.SUnsubscribe on such a PubSub sends one SUNSUBSCRIBE per slot.

Docs of ClusterClient.SSubscribe, PubSub.SSubscribe/SUnsubscribe and ErrCrossSlot
updated.
@FumingPower3925 FumingPower3925 added this to the v1.6.0 milestone Nov 1, 2026
@coderabbitai

coderabbitai Bot commented Nov 1, 2026 •

Copy link
Copy Markdown

Review in Change Stack →

Caution

Review failed

The pull request is closed.

  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@codecov

codecov Bot commented Nov 1, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 97.53086% with 2 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
driver/redis/pubsub.go 98.66% 1 Missing ⚠️
driver/redis/pubsub_route.go 66.66% 1 Missing ⚠️

📢 Thoughts on this report? Let us know!

…ce probes of #1204 (Refs #1204)

Review r5 of #1210:
- a held channel whose slot loses its owner (cluster-require-full-coverage no)
  is answered -CLUSTERDOWN Hash slot not served at every replay, which is a
  reroute trigger: the PubSub reroutes for ever (fails on the current head: 3
  more conns in 1.2 s; main is stable). The fake cluster can leave a slot
  unbound (unbind) and answers -CLUSTERDOWN for it.
- the replay sends another node's slot only to its owner: the test that
  expected the -MOVED round trip now expects the served slot's command alone.
- a refused cross-slot constructor call dials no conn (the up-front check is
  pinned).
- redis_cluster conformance: two slots of one node survive a replay (CLIENT
  KILL), the served slot moving to the node of the second slot, ErrCrossSlot for
  a cross-slot SSubscribe, SUnsubscribe across slots.
- stale "documented undefined behaviour" wording of the two-slot tests.
…it (Refs #1204)

Review r5 of #1210 (F1, regression): the per-slot replay brought back the slots
a PubSub does not serve at every reroute. One that has no owner (DELSLOTS with
cluster-require-full-coverage no) is answered -CLUSTERDOWN Hash slot not served,
which onControlError takes for a reroute trigger (the reply carries no slot
number), so the replay of every reroute started the next one, for ever (real
Redis 8.0.2: attempt 6 -> 9 over 10 s; main stable).

The replay now sends the served slot always and any other slot only if the
reroute's own topology (it has just refreshed it) says the node it landed on
owns it (pubsubRoute.owns). A slot of another node would only draw -MOVED (and
was ignored), a slot with no owner draws the -CLUSTERDOWN. The -CLUSTERDOWN
trigger itself is unchanged: the reply cannot say which slot it is for, and one
for the served slot is a real reason to look again. Also: the stale comment of
onShardUnsubscribed (a channel of another slot is replayed, not dropped) and the
SSubscribe doc.
@FumingPower3925
FumingPower3925 marked this pull request as ready for review November 1, 2026 12:40
@FumingPower3925
FumingPower3925 merged commit e2de1b3 into main Nov 1, 2026
56 checks passed
@FumingPower3925
FumingPower3925 deleted the fix/redis-pubsub-replay-by-slot-1204 branch November 1, 2026 12:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant