fix(queue): retry first-time provisioning of a replicated stream - #219
Merged
Conversation
Concurrent *first* provisioning of a replicated stream does not degrade gracefully. Measured against a 3-node JetStream cluster at replicas=3: creating a fresh stream and its two durable consumers takes ~290ms from a single process and succeeds every time; two processes doing it at once both exceed the client's 5s request timeout and neither completes. It is a cliff, not a slope -- eight concurrent cold starts landed 0-3 successes. The server logs give the mechanism: the competing CREATEs drive the stream and consumer RAFT groups into repeated leader elections, which continue for minutes after every caller has given up. The retry works because the state that makes an attempt expensive does not survive it. Once one process wins, the stream and consumers exist, and a repeat CREATE for an identical config causes no further election -- eight concurrent processes against an existing stream provision in 19-34ms with no election at all. Measured after the fix: 8/8 at eight concurrent cold starts, 2/2 at two (both previously 0). This is the first pod that ever touches a queue, not a scale-up: a KEDA 0->N expansion runs against a stream that already exists and is the cheap case above. That distinction matters -- the failure was originally read as a scale-up hazard because the benchmark's --cleanup default made every run a cold one. Reading the current config first, to skip a no-op write, was tried and made it worse: the extra round trips spend the same 5s budget, taking eight concurrent cold starts from 8/8 down to 7/8. No automated coverage, and none that would mean anything: the failure needs a replicated cluster, and this package's E2E suite runs single-node, where the retry can be deleted and everything stays green. appwrite/cloud carries the reproduction as benchmarks/nats-provisioning.php.
levivannoort
requested review from
ChiragAgg5k,
abnegate and
lohanidamodar
as code owners
September 7, 2026 06:44
loks0n
approved these changes
Sep 8, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Broker\Nats::ensure()now retries first-time provisioning with full jitter over an exponentially growing window.Why
Concurrent first provisioning of a replicated stream falls off a cliff. Measured against a 3-node JetStream cluster at
replicas=3:A single cold provision is fast and reliable; two at once both blow the client's 5 s request timeout and neither completes. The server logs give the mechanism — the competing CREATEs drive the stream and consumer RAFT groups into repeated leader elections, which continue for minutes after every caller has given up:
The retry works because the expensive state does not survive the attempt: once one process wins, the stream exists and every later caller takes the warm path, which is clean at eight concurrent processes even with the retry removed. So a retry is not a rerun of the same race.
This is a queue's very first deploy, not a scale-up. A KEDA 0→N expansion runs against an existing stream — the warm row. The failure was originally read as a scale-up hazard because the benchmark's
--cleanup=1default dropped the streams after every run, making every run a cold one.Rejected alternative
Reading the current config first to skip a no-op write made it worse — the extra round trips spend the same 5 s budget, taking 8 concurrent cold starts from 8/8 down to 7/8. Measured, not assumed.
Coverage
None, and none that would mean anything here: reproducing this needs a replicated cluster, and this package's E2E suite runs single-node, where the retry can be deleted and everything stays green.
appwrite/cloudcarries the reproduction asbenchmarks/nats-provisioning.php(--reset=1first, or you measure the warm path).Tests
phpunit --testsuite unit— 113 tests, 289 assertions. Pint clean.