Skip to content

Fix duplicate Pub/Sub subscription restoration after Sentinel failover - #3337

Open
vkyow wants to merge 2 commits into
seredis:mainfrom
vkyow:fix/sentinel-subscription-restoration-race
Open

vkyow wants to merge 2 commits into
seredis:mainfrom
vkyow:fix/sentinel-subscription-restoration-race

Conversation

@vkyow

@vkyow vkyow commented Nov 6, 2026

Copy link
Copy Markdown

Fixes #3336.

During Sentinel failover, connection establishment and reconfiguration can both try to restore the same single-node subscription before either SUBSCRIBE/PSUBSCRIBE acknowledgement arrives. Both physical subscriptions can survive, even though the client records only one owner. A single publication can then reach the callback twice through different Redis nodes.

This change serializes restoration through the acknowledgement, including internal fire-and-forget requests. Replies are associated with their restoration attempt, late obsolete registrations are cleaned up, and unsubscribe is coordinated with pending restoration. Delivery is accepted only from the physical connection that owns the subscription. It preserves the existing server-selection policy and multi-node subscription behavior, with no public API or dependency changes.

A second commit moves the existing Sentinel reconfiguration gate ahead of primary discovery. This prevents reconnect-event storms from starting parallel blocking discovery queries before checking that another reconfiguration is already running.

Validation

Tested against upstream main at 9d17974fcade6c89b29b233030fe281e75f9ab1c:

  • The initial deterministic acknowledgement-barrier regression fails all 8 cases on unmodified source. The completed restoration and Sentinel coalescing suites pass 26/26 cases under RESP2 and RESP3.
  • Existing unit tests: 2,034 passed, no failures or skips. In-process Pub/Sub tests: 36 passed, 12 existing skips.
  • All five shipped target frameworks build with the upstream analyzers. Runtime HA tests used .NET 10.
  • Isolated Docker tests used Redis and Sentinel 8.0.3, with both three-pair and five-pair topologies (one primary, remaining Redis replicas, and one Sentinel per pair). Each topology included three fresh clusters with ten primary switches, plus additional restart, outage, authentication and reconnection scenarios.
  • Across 104 accepted recovery events, the slowest recovery was 16.78 seconds after an independently confirmed primary switch. Traffic continued for at least two minutes after each recovery. All 251,734 expected post-recovery-ready deliveries arrived once, with no missing ready deliveries or lingering duplicate business subscriptions.
  • Unmodified 3.4.31 controls reproduced duplicate inbound frames from different Redis nodes for publication IDs sent exactly once. The reproduction and supporting evidence are attached to Duplicate Pub/Sub messages after Redis Sentinel failover #3336.

Pub/Sub messages can still be lost during the failover transition; this does not add replay or an exactly-once guarantee. Already queued handlers may finish after unsubscribe. The Docker results are local validation, not a production-load benchmark.

To run the focused regressions:

dotnet test --project tests/StackExchange.Redis.Tests/StackExchange.Redis.Tests.csproj -f net10.0 --filter-class StackExchange.Redis.Tests.SubscriptionRestorationTests
dotnet test --project tests/StackExchange.Redis.Tests/StackExchange.Redis.Tests.csproj -f net10.0 --filter-class StackExchange.Redis.Tests.SentinelSwitchCoalescingTests

Checklist

  • I fully and freely contribute this code in accordance with the project license (and am legally able to do so)
  • I take responsibility for this contribution's quality and correctness, including any portions produced with AI assistance (see CONTRIBUTING.md).

@mgravell

mgravell commented Nov 9, 2026

Copy link
Copy Markdown
Collaborator

looking shortly

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Duplicate Pub/Sub messages after Redis Sentinel failover

2 participants