Repository navigation
Conversation
Consumer::consume() called enable_notify_insert() on every start, which drops and recreates the trigger on the queue table. That takes a table lock: a (re)starting worker waits for every open transaction that inserted into the queue, blocks new inserts meanwhile, and two workers of one queue starting together can deadlock. Add ensureNotifyInsert()/isNotifyInsertEnabled(): the consumer now only checks the catalog and enables notifications when they are missing or use another throttle.
…alidity and the throttle row - check and change run in one transaction under pgmq.acquire_queue_lock(), so consumers starting together no longer race into enable_notify_insert() (concurrent cold starts failed with 'trigger ... already exists'); - the trigger counts as valid only when enabled and calling notify_queue_listeners(); - a missing throttle row (UNLOGGED: emptied by crash recovery) re-enables notifications; - a different throttle interval uses update_notify_insert(), without DDL.
…etups - 4 consumers starting at once on a queue without notifications create the trigger exactly once (counted with an event trigger); without the queue lock they raced and failed with 'trigger ... already exists'; - a lost throttle row (crash recovery), a disabled trigger and a trigger calling another function are restored; - a different throttle interval is applied without recreating the trigger.
|
Reworked after testing with a stress harness (details in #20):
New tests: concurrent cold start (event trigger counts |
Lock, check and change now run in a single DO block, so ensureNotifyInsert() needs no transaction of its own: it works on any PostgresLink (a pool, a connection, or an open transaction, where the queue lock is held until that transaction ends) and costs one round trip instead of five. The trigger-validity condition is defined once and shared with isNotifyInsertEnabled().
|
One more change: |
Fixes #20.
Problem
Consumer::consume()calledenableNotifyInsert()on every start. That drops and recreates the trigger onpgmq.q_<queue>, taking a table lock: a (re)starting consumer waits for every open transaction that inserted into the queue, blocks new inserts meanwhile, and consumers starting together race into the same DDL.pgmq.notify_insert_throttleisUNLOGGED: crash recovery (or a promoted replica) empties it while the trigger survives, and the trigger then never notifies. Recreating everything on every start used to heal this by accident, so the fix has to keep that case working.Change
ensureNotifyInsert(queue, ?throttleInterval)(+Queue::ensureNotifyInsert()), as one statement (aDOblock):pgmq.acquire_queue_lock(queue): the lock pgmq itself takes to create and drop the queue, so concurrent starts pass one by one;tgenabledO/A) and callspgmq.notify_queue_listeners(); the throttle row is present;pgmq.update_notify_insert()(no DDL); otherwise →pgmq.enable_notify_insert().Being one statement, it needs no transaction of its own. It works on any
PostgresLink: a pool, a connection, or an open transaction (then the queue lock is held until that transaction ends). It costs one round trip.isNotifyInsertEnabled(queue, ?throttleInterval)(+Queue::isNotifyInsertEnabled()): the same check, read-only. The trigger-validity condition is defined once and shared by both.Consumer::consume()usesensureNotifyInsert().enableNotifyInsert()keeps its "(re)create" semantics.The default throttle (30 ms) is shared through an internal
notifyThrottleMs()instead of being repeated.thesis/message-busneeds the same behaviour in itsPgmqTransport; thesis-php/message-bus#7 calls this function.Tests
testConsumerStartDoesNotWaitForProducerTransactions: notifications set up, a producer keeps an insert open,consume()must return within 1 sCancelledExceptiontestConcurrentColdStartsCreateTheTriggerOnce: 4 consumers (separate pools, like worker processes) start at once on a queue without notifications while a producer forces their starts to overlap;CREATE TRIGGERcounted with an event trigger must be exactly 1trigger … already existstestEnsureNotifyInsertRestoresLostThrottleRowtestEnsureNotifyInsertRecreatesDisabledTriggertestEnsureNotifyInsertRecreatesTriggerCallingAnotherFunctiontestEnsureNotifyInsertChangesThrottleWithoutRecreatingTrigger(0CREATE TRIGGER)testEnsureNotifyInsertKeepsExistingSetupAndEnablesMissingOnetestEnsureNotifyInsertWorksInsideAnOpenTransaction(and is rolled back with it)The event-trigger tests need a superuser connection, which the compose setup provides.
Stress harness (real Postgres, pgmq 1.11.1 and 1.13.0, real crashes via
kill -9)already existsChecks (run locally in the project's Docker images)
PHP 8.3 / 8.4 / 8.5 × lowest / highest: tests ✅ (43), phpstan ✅.
fixer-check,rector-check, composer checks ✅. (On PHP 8.5 + lowest deps, the same twoamphp/ampdeprecations as on0.1.x; exit code 0.)Independent of #21. Both add
use Amp\TimeoutCancellation;to the test; git merges them cleanly in either order (verified).