Repository navigation
[go] Fix InsertManyFast and QueueUpdate panics on drivers without listener support - #1558
Merged
Merged
Conversation
`InsertManyFast` inserts with `JobInsertFastManyNoReturning`, so its insert results are all nil, but it passed them to `notifyProducerWithoutListenerJobFetch`, which reads `Job.Queue` from each one. On a client with producers (one configured with `Queues`) and a driver that doesn't support a listener, `InsertManyFast` panicked after its transaction had already committed. Notify using the queues from insert params instead. A new `notifyProducerWithoutListenerJobFetchForQueues` takes queue names, and `notifyProducerWithoutListenerJobFetch` keeps its signature because it's handed to the pilot as `NotifyNonTxJobInsert`. To get at insert params, `insertManyFast` now takes `[]*rivertype.JobInsertParams` like `insertMany` does, and its callers build them first. Signed-off-by: Yaomeng1749 <176285760+Yaomeng1749@users.noreply.github.com>
`queueUpdate` returns a nil control event when metadata isn't being changed, but `QueueUpdate` passed it on to the producer anyway when the driver doesn't support a listener. The producer's fetch loop then dereferenced it, and with nothing recovering in that goroutine, the program exited. Only notify the producer when there's a control event. Signed-off-by: Yaomeng1749 <176285760+Yaomeng1749@users.noreply.github.com>
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
Contributor
|
Thanks @Yaomeng1749! Could you sign our CLA please? https://github.com/riverqueue/rivercla |
Contributor
Author
|
Thanks! I've signed the CLA in riverqueue/rivercla#39. |
A small optimization on top of riverqueue#1558 so that in case of an insertion-only client, skip unnecessary queue slice allocation.
brandur
approved these changes
Oct 29, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
These are two nil dereferences on the path that wakes a client's own producers when its driver doesn't support a listener, as with a
riverdatabasesqldriver created withNewrather thanNewWithPgxListener, orriverpgxv5on YugabyteDB withoutyb_enable_listen_notify. Both only affect clients that work jobs, since those are the ones with producers to wake.InsertManyFastpanics after its transaction commits. The fast path inserts withJobInsertFastManyNoReturning, so its insert results are all nil, butInsertManyFastpasses them tonotifyProducerWithoutListenerJobFetch, which readsJob.Queuefrom each one. The jobs are already committed by the time it panics, so a job that fans out work withInsertManyFastfails and is retried, inserting the batch again on every attempt. This has been the case since #1001 changed the notify function from taking queue names to taking insert results.This adds
notifyProducerWithoutListenerJobFetchForQueues, which takes queue names, andInsertManyFastcalls it with the queues from its insert params so that fast inserts still wake producers right away, likeInsertManydoes. Skipping nil results would also avoid the panic, but producers would then wait for the next fetch poll.notifyProducerWithoutListenerJobFetchkeeps its signature since it's handed to the pilot asNotifyNonTxJobInsert. To get at insert params,insertManyFastnow takes[]*rivertype.JobInsertParamslikeinsertMany, and its two callers build them first, soInsertManyFastalso no longer opens a transaction for params that fail validation.QueueUpdatewithout metadata crashes the program.queueUpdatereturns a nil control event when metadata isn't being changed (and sends no notification in that case), butQueueUpdatepasses it tonotifyProducerWithoutListenerQueueControlEventregardless. If the client works that queue, the producer'sfetchAndRunLoopthen dereferences it atswitch msg.Action, and since nothing recovers in that goroutine, the program exits.QueueUpdateParams.Metadatadocuments nil as leaving metadata unchanged, so it's a valid call. This has been the case since #928 added out-of-transaction notifications for drivers without a listener.QueueUpdatenow only notifies the producer when there's a control event, which is what the Rust client already does (it only signals producers whenmetadata.is_some()).#1361 adds a nil check to
TriggerQueueControlEventas part of reworking it, which would also stop this crash once it lands. It's a broader change that's still in review, so this fixes the caller in the meantime. In code, the two only overlap on the oneQueueUpdateline, and I'm happy to drop this commit if you'd rather rely on #1361.Each fix is its own commit with tests:
Test_Client_InsertManyFast/ProducerFetchLimiterCalled, like the one inTest_Client_Insert, usesNewDriverWithoutListenNotify, inserts into two queues, and checks that each producer is triggered once. Without the fix, it panics innotifyProducerWithoutListenerJobFetch. ItsProducerFetchLimiterNotCalledcounterpart checks that a driver with a listener still isn't notified this way.Test_Client_QueueUpdate/ProducerControlEventNotSentWithoutMetadata, also withNewDriverWithoutListenNotify, checks that nil metadata sends no control event and that the producer still handles a metadata change afterward. Without the fix, the test binary crashes infetchAndRunLoop.The other ports aren't affected: none of them offers fast insertion like
InsertManyFast, and none of their queue updates signal producers when there's no metadata to change.Locally against Postgres 16, the new subtests pass with
-race -count=20, andmake test,make test/race TEST_DATABASE=postgres, andmake lintpass.