Skip to content

Pgmq: poll again right after a full batch instead of waiting for the next signal - #6

Open
iGrog wants to merge 2 commits into
thesis-php:0.5.xfrom
iGrog:fix/pgmq-consumer-drains-backlog
Open

iGrog wants to merge 2 commits into
thesis-php:0.5.xfrom
iGrog:fix/pgmq-consumer-drains-backlog

Conversation

@iGrog

@iGrog iGrog commented Oct 7, 2026 •

Copy link
Copy Markdown
Contributor

Fixes #2.

After reading one batch the consumer waited for the next poll signal. With a backlog no new inserts notify, so it slept pollInterval (5 s by default) between batches: a queue of N messages took ~N / batchSize × pollInterval.

Change

PgmqTransport::startConsumer(): after a batch that came back full, push the next poll request right away. Partial or empty batches do not re-poll, so the idle query count is unchanged. A stopped consumer is respected ($polls->isComplete()).

The PR also carries the one-line phpstan fix from #5 (an @phpstan-ignore that no longer matches). Without it the CI is red on 0.5.x regardless of the change. The change is identical in both PRs, so they merge in either order.

Effect (same workload and data before and after)

before after
1 unit of work (~450 messages per queue, several queues in a chain) 60 s 8 s
10 such units at once 421 s 75 s
idle CPU (consumers + Postgres) ~0 % ~1 %

Edge cases checked with a stress harness (real Postgres, pgmq 1.11.1 and 1.13.0)

scenario result
backlog of 0 / 5 / 10 / 30 / 35, batchSize 10, pollInterval 10 s drained in ≤ 0.21 s (without the fix, backlogs larger than one batch wait for the timer — see the test in #8)
backlog of 500, 4 consumers 1.0 s, no duplicates
consumer stopped in the middle of a backlog stops in 0.02 s, nothing handled after the stop
handler throws in the middle of a batch error surfaces, no busy loop; the rest of the batch returns after the visibility timeout
producers keep sending while a backlog of 500 drains all 800 delivered, no duplicates ¹
20 delayed (3 s) + 25 visible messages, pollInterval 2 s visible in 0.2 s; delayed at 4.2 s
idle, pollInterval 1 s no busy loop

¹ Separate transactions inserting within the 30 ms notify throttle window can leave the last messages waiting for the timer; this happens with or without this change, see #4.

Tests

The repository has no Postgres in its dev environment, so this PR is code only. The integration tests are in #8: consumerDrainsBacklogWithoutWaitingForPollInterval is red on 0.5.x and green with this change.

Checks (run locally, PHP 8.4 / 8.5 × lowest / highest)

Tests ✅, phpstan ✅ (with the fix from #5), fixer-check ✅, rector-check ✅, composer checks ✅.

…next signal

A backlog that is already in the queue produces no further insert notifications,
so after reading one batch the consumer slept until the poll interval elapsed:
a queue of N messages took ~N/batch * pollInterval (5 s by default) to drain.
Recent PHPStan 2.2.x infers the nested property assignment correctly, so the ignore
is unmatched and phpstan fails on every cell of the CI matrix (8.4/8.5 x lowest/highest).

(cherry picked from commit ec45573)
@iGrog

iGrog commented Oct 7, 2026

Copy link
Copy Markdown
Contributor Author

Added the one-line phpstan fix from #5 (identical change, merges in either order), so this PR is green on its own. The description now also lists the edge cases checked with a stress harness: stop in the middle of a backlog, a failing handler, concurrent producers, delayed messages, idle polling.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

PgmqTransport: consumer reads one batch per poll signal, a backlog waits pollInterval between batches

1 participant