Skip to content

[go] Fix InsertManyFast and QueueUpdate panics on drivers without listener support - #1558

Merged
brandur merged 3 commits into
riverqueue:masterfrom
Yaomeng1749:fix-poll-only-panics
Oct 29, 2026
Merged

brandur merged 3 commits into
riverqueue:masterfrom
Yaomeng1749:fix-poll-only-panics

Conversation

@Yaomeng1749

Copy link
Copy Markdown
Contributor

These are two nil dereferences on the path that wakes a client's own producers when its driver doesn't support a listener, as with a riverdatabasesql driver created with New rather than NewWithPgxListener, or riverpgxv5 on YugabyteDB without yb_enable_listen_notify. Both only affect clients that work jobs, since those are the ones with producers to wake.

InsertManyFast panics after its transaction commits. The fast path inserts with JobInsertFastManyNoReturning, so its insert results are all nil, but InsertManyFast passes them to notifyProducerWithoutListenerJobFetch, which reads Job.Queue from each one. The jobs are already committed by the time it panics, so a job that fans out work with InsertManyFast fails and is retried, inserting the batch again on every attempt. This has been the case since #1001 changed the notify function from taking queue names to taking insert results.

This adds notifyProducerWithoutListenerJobFetchForQueues, which takes queue names, and InsertManyFast calls it with the queues from its insert params so that fast inserts still wake producers right away, like InsertMany does. Skipping nil results would also avoid the panic, but producers would then wait for the next fetch poll. notifyProducerWithoutListenerJobFetch keeps its signature since it's handed to the pilot as NotifyNonTxJobInsert. To get at insert params, insertManyFast now takes []*rivertype.JobInsertParams like insertMany, and its two callers build them first, so InsertManyFast also no longer opens a transaction for params that fail validation.

QueueUpdate without metadata crashes the program. queueUpdate returns a nil control event when metadata isn't being changed (and sends no notification in that case), but QueueUpdate passes it to notifyProducerWithoutListenerQueueControlEvent regardless. If the client works that queue, the producer's fetchAndRunLoop then dereferences it at switch msg.Action, and since nothing recovers in that goroutine, the program exits. QueueUpdateParams.Metadata documents nil as leaving metadata unchanged, so it's a valid call. This has been the case since #928 added out-of-transaction notifications for drivers without a listener. QueueUpdate now only notifies the producer when there's a control event, which is what the Rust client already does (it only signals producers when metadata.is_some()).

#1361 adds a nil check to TriggerQueueControlEvent as part of reworking it, which would also stop this crash once it lands. It's a broader change that's still in review, so this fixes the caller in the meantime. In code, the two only overlap on the one QueueUpdate line, and I'm happy to drop this commit if you'd rather rely on #1361.

Each fix is its own commit with tests:

  • Test_Client_InsertManyFast/ProducerFetchLimiterCalled, like the one in Test_Client_Insert, uses NewDriverWithoutListenNotify, inserts into two queues, and checks that each producer is triggered once. Without the fix, it panics in notifyProducerWithoutListenerJobFetch. Its ProducerFetchLimiterNotCalled counterpart checks that a driver with a listener still isn't notified this way.
  • Test_Client_QueueUpdate/ProducerControlEventNotSentWithoutMetadata, also with NewDriverWithoutListenNotify, checks that nil metadata sends no control event and that the producer still handles a metadata change afterward. Without the fix, the test binary crashes in fetchAndRunLoop.

The other ports aren't affected: none of them offers fast insertion like InsertManyFast, and none of their queue updates signal producers when there's no metadata to change.

Locally against Postgres 16, the new subtests pass with -race -count=20, and make test, make test/race TEST_DATABASE=postgres, and make lint pass.

`InsertManyFast` inserts with `JobInsertFastManyNoReturning`, so its
insert results are all nil, but it passed them to
`notifyProducerWithoutListenerJobFetch`, which reads `Job.Queue` from each
one. On a client with producers (one configured with `Queues`) and a
driver that doesn't support a listener, `InsertManyFast` panicked after
its transaction had already committed.

Notify using the queues from insert params instead. A new
`notifyProducerWithoutListenerJobFetchForQueues` takes queue names, and
`notifyProducerWithoutListenerJobFetch` keeps its signature because it's
handed to the pilot as `NotifyNonTxJobInsert`. To get at insert params,
`insertManyFast` now takes `[]*rivertype.JobInsertParams` like
`insertMany` does, and its callers build them first.

Signed-off-by: Yaomeng1749 <176285760+Yaomeng1749@users.noreply.github.com>
`queueUpdate` returns a nil control event when metadata isn't being
changed, but `QueueUpdate` passed it on to the producer anyway when the
driver doesn't support a listener. The producer's fetch loop then
dereferenced it, and with nothing recovering in that goroutine, the
program exited. Only notify the producer when there's a control event.

Signed-off-by: Yaomeng1749 <176285760+Yaomeng1749@users.noreply.github.com>
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 28, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ⚠️ Failed 2026-10-28T23:12:03.105699Z
We couldn’t complete this review. Comment @codex review to try again.
6123852 PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@brandur

brandur commented Oct 29, 2026

Copy link
Copy Markdown
Contributor

Thanks @Yaomeng1749! Could you sign our CLA please? https://github.com/riverqueue/rivercla

@Yaomeng1749

Copy link
Copy Markdown
Contributor Author

Thanks! I've signed the CLA in riverqueue/rivercla#39.

A small optimization on top of riverqueue#1558 so that in case of an
insertion-only client, skip unnecessary queue slice allocation.
@brandur
brandur merged commit 53ce383 into riverqueue:master Oct 29, 2026
15 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants