Skip to content

driver/redis: an older CLUSTER SLOTS read ending after a newer one no longer overwrites the slot table (v1.6.0, Refs #1219) - #1221

Merged
FumingPower3925 merged 3 commits into
mainfrom
fix/redis-topology-apply-order-1219
Nov 2, 2026
Merged

FumingPower3925 merged 3 commits into
mainfrom
fix/redis-topology-apply-order-1219

Conversation

@FumingPower3925

@FumingPower3925 FumingPower3925 commented Nov 1, 2026 •

Copy link
Copy Markdown
Contributor

driver/redis: an older CLUSTER SLOTS read that ends after a newer one no longer overwrites the slot table. Refs #1219 (found by the review of #1215; pre-existing).

The problem

applyTopology rebuilt the slot table from whatever CLUSTER SLOTS reply reached it, with no ordering. Two overlapping reads could apply out of order: a read that took its table before a slot moved, but ended after a newer read had applied the move, put the slot back to its pre-move owner (extra -MOVED rounds or a stale route until the next refresh). #1215's shared reads make overlapping reads more likely.

The change

  • Each read already has a registration sequence (topoRefresh.seq, assigned under refMu when the read is registered: driver/redis: sharded PubSub reroutes share one CLUSTER SLOTS read (v1.6.0, Refs #1203) #1211/driver/redis: commands that meet the same MOVED or lost primary share one topology read (v1.6.0, Refs #1214) #1215). runRefresh now passes it through readTopology to applyTopology, which, under c.mu, drops a result whose sequence is lower than the last applied one (ClusterClient.topoApplied) and otherwise applies it and records its sequence. A dropped read returns nil (a newer table is in place, so the caller's goal is met; PubSub/command joiners only share reads that started after their trigger, and the newer read is one).
  • No clock. The order is that of registration, not of the requests reaching the server (the best a client can tell without a clock; stated in the code).
  • A read that fails, or is cut short by its caller's context, applies nothing and does not advance topoApplied.
  • Sentinel path (checked, no change): SentinelClient.handleSwitchMaster swaps the master client, and it is called only from watchFailover, which subscribeLoop runs inline: +switch-master notifications are handled one at a time in delivery order, so a slow dial of an older failover cannot install its client after a newer one. There is no second writer. A comment on handleSwitchMaster says so. (queryMaster at construction is a single read.)

Commits: e2a6695 the tests (RED on main), b6958a9 the fix.

RED / GREEN

RED, e2a6695 alone (macOS arm64, go test ./driver/redis -run 1219 -v):

topology_order_1219_test.go:42: the table went back to the pre-move owner: the older read, which ended last, overwrote the newer one
--- FAIL: TestAnOlderTopologyReadEndingLastDoesNotOverwriteANewerOne1219
--- PASS: TestTopologyReadsThatEndInOrderAllApply1219    (guard, passes on main by design)

The test uses the fake cluster of #1211: hold read 1 after its table (pre-move) is taken, move the slot, run read 2 to the end (applied, the slot routes to the new owner), release read 1, and assert the table still names the new owner. The second test checks the guard is not too strong: reads that end in the order they started all apply (four moves back and forth), and a read cut short by its caller's context applies nothing and does not block the next one. No sleep and no clock decides a result.

GREEN, b6958a9: both pass.

Mutants (each alone, go test -run '1219|1214' -count=2)

mutant tests that fail
M1 no ordering (apply every result) TestAnOlderTopologyReadEndingLastDoesNotOverwriteANewerOne
M2 the applied number never advances TestAnOlderTopologyReadEndingLastDoesNotOverwriteANewerOne
M3 inverted (drop the newer one) TestTopologyReadsThatEndInOrderAllApply and 12 of the 1214 tests
M4 runRefresh passes 0 (unordered; that branch was removed in the r8 follow-up) TestAnOlderTopologyReadEndingLastDoesNotOverwriteANewerOne

Checks

  • -race -run '1219|1214' -count=20 Linux docker golang:1.27.3 --cpus 2: ok; whole driver/redis -race -count=1: ok (175 s). macOS -race -count=20: ok.
  • GOOS=windows go vet ./driver/redis: clean. golangci-lint v2.15.0 GOOS=linux: only the two pre-existing errcheck findings in untouched files.
  • Not run locally: the windows-latest and macos-latest Portable fallback jobs.

Release note

driver/redis (cluster): the slot table is no longer replaced by an older CLUSTER SLOTS read: a read that finishes after a newer one (registered later) is dropped instead of putting a slot back to its previous owner, which cost extra -MOVED rounds or a stale route until the next refresh. The order is that of registration, not of the cluster's truth: a read that hits a lagging node can still install an older table, which the next -MOVED (or the 60 s refresh) corrects. The sentinel client is not affected (failover notifications are handled in order).

Review r8 follow-up (commit 45a63cb)

  • F1, the guarantee (reworded): the code comment and the release note above now state only what holds: the table is never replaced by a read registered before the one in place. They name what it does not guarantee: registration order is not freshness order. A newer-registered read sent to a lagging node, or an older read that retries a second node (topologyProbes) after the newer one was sent, can install the older table and the fresher answer is dropped; the reviewer's probe P2 shows head leaving the slot on the old owner where main had it right by luck, corrected by one -MOVED. This is the mirror image of driver/redis: an older CLUSTER SLOTS read that finishes after a newer one overwrites the slot table #1219 and cannot be avoided without a server epoch.
  • F2, tests: TestAnOlderReadStillAppliesWhenTheNewerOneFailed1219 (the reviewer's probe P1): an older read carrying the move is in flight, the newer read is cut short, the older read must still apply. Mutants "a failed read advances the order" (in readTopology and in runRefresh) and "drop a result when a newer read is merely registered" fail it (they passed every other test).
  • F3: the "sequence 0 always applies" branch is gone (no production caller passed 0); the iouring: TestPort0HoldIsReleasedWhenListenFails895 failed once on main CI (1 socket still bound after a failed Listen); re-run passed #1104 test passes 1.
  • F4, known limitation (no issue): ordering by the cluster's truth would need a config epoch. CLUSTER SLOTS has none; CLUSTER NODES reports config-epoch per node. Not done here.
  • Checks on 45a63cb: -race -run '1219|1214' -count=20 Linux docker --cpus 2 ok, whole driver/redis -race -count=1 Linux ok (176 s); macOS -race ok (1214 in two batches of 10: 20 runs of the herd tests exhaust the macOS ephemeral ports, an environment limit); GOOS=windows go vet clean; golangci-lint only the two pre-existing errcheck findings.

@FumingPower3925 FumingPower3925 added this to the v1.6.0 milestone Nov 1, 2026
@coderabbitai

coderabbitai Bot commented Nov 1, 2026 •

Copy link
Copy Markdown

Review in Change Stack →

Caution

Review failed

The pull request is closed.

  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

…for an older read when the newer one fails; drop the unordered branch (Refs #1219)
@codecov

codecov Bot commented Nov 1, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@FumingPower3925
FumingPower3925 marked this pull request as ready for review November 2, 2026 18:42
@FumingPower3925
FumingPower3925 merged commit 8244030 into main Nov 2, 2026
55 of 56 checks passed
@FumingPower3925
FumingPower3925 deleted the fix/redis-topology-apply-order-1219 branch November 2, 2026 18:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant