Repository navigation
fix(worker): read the endpoint before each attempt - #47
Merged
Merged
Conversation
The worker read an endpoint once for each group of its deliveries in a batch and used that copy for the whole group. A change made while the group was being sent did not apply to its remaining deliveries: after a disable or a delete they were still sent, after a URL change they went to the old URL, and after rotateSecret() with overlap: 0 they were signed with the old secret only. The worker now reads the endpoint before each attempt, so a change applies from the next attempt; an attempt already in flight finishes as it started. The local patches of enabled and failingSince after each attempt are no longer needed and are removed. This also corrects the error recorded for the rest of a group after a 410 Gone disables the endpoint: it now reads "is disabled (gone)" instead of "is disabled (manual)", because the patch set enabled to false without the disabled reason. Closes #46
thekhegay
reviewed
Oct 24, 2026
thekhegay
left a comment
Contributor
There was a problem hiding this comment.
Tried it with store whose getEndpoint() takes 2s (controllable clock like lease spec in worker.spec.ts), delivery.timeout: '10s', worker.lease: '14s', two deliveries to one endpoint:
lease left when each send starts: master 12000 and 11000 ms, PR 12000 and 9000 ms
second worker polling at lease end: master sends message 2 once, PR twice with same webhook-id
stop() during read: master sends 0 and releases 2, PR sends 1 and releases 1
50ms read, first response 3.93s: lease left at second send master 10020 ms, PR 9970 ms
With suggestion every send starts with 10s+ of lease left, message 2 goes out once, stop() sends nothing and full suite stays green, relased group just pays one extra read
Comment on lines
232
to
+239
| // Never start an attempt that could outlive the lease: another worker could claim the delivery and send it too. | ||
| if (this.stopping || Date.now() + this.config.timeoutMs > leaseUntil) { | ||
| return this.release(group.slice(i), owner, result); | ||
| } | ||
|
|
||
| // Read before each attempt, not once per group: the endpoint may have been disabled, deleted, moved or had | ||
| // its secret rotated (or the previous attempt changed its health) since. | ||
| const endpoint = await this.endpoints.getEndpoint(endpointId); |
Contributor
There was a problem hiding this comment.
read now sits between stop/lease check and send, so its latency eats into lease margin and attempt can start with less than timeoutMs left or after stop()
Suggested change
| // Never start an attempt that could outlive the lease: another worker could claim the delivery and send it too. | |
| if (this.stopping || Date.now() + this.config.timeoutMs > leaseUntil) { | |
| return this.release(group.slice(i), owner, result); | |
| } | |
| // Read before each attempt, not once per group: the endpoint may have been disabled, deleted, moved or had | |
| // its secret rotated (or the previous attempt changed its health) since. | |
| const endpoint = await this.endpoints.getEndpoint(endpointId); | |
| // Read before each attempt, not once per group: the endpoint may have been disabled, deleted, moved or had | |
| // its secret rotated (or the previous attempt changed its health) since. | |
| const endpoint = await this.endpoints.getEndpoint(endpointId); | |
| // Never start an attempt that could outlive the lease: another worker could claim the delivery and send it too. | |
| if (this.stopping || Date.now() + this.config.timeoutMs > leaseUntil) { | |
| return this.release(group.slice(i), owner, result); | |
| } | |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
PR Checklist
Please check if your PR fulfills the following requirements:
PR Type
What kind of change does this PR introduce?
What is the current behavior?
The worker reads an endpoint once for each group of its deliveries in a batch (up to
worker.batchSize, 50 by default) and uses that copy for every delivery in the group. A change made while the group is being sent doesn't apply to the rest of it: afterupdate(id, { enabled: false })ordelete(id)the remaining deliveries are still sent, afterupdate(id, { url })they go to the old URL, and afterrotateSecret(id, { overlap: 0 })they're signed with the old secret only. The docs say pending deliveries of a disabled or deleted endpoint fail "when their turn comes", andWebhookEndpointStore's JSDoc says the worker reads it before each attempt.A related detail: after a 410 Gone disables the endpoint mid-group, the worker patched its copy to
enabled: falsewithout the reason, so the rest of the group failed with "is disabled (manual)" instead of "(gone)".Issue Number: #46
What is the new behavior?
The worker reads the endpoint before each attempt, after the lease check, so deliveries it releases cost no read. A disable, delete, URL change or rotation applies from the next attempt; an attempt already in flight finishes as it started. Since the store already records
enabled,disabledReasonandfailingSince, the worker's local patches of its copy (added in #43) and the twoOutcomefields that fed them are removed. The order of deliveries within a group is unchanged.Thanks to @johnnyhuirilef for the report, the reproduction and the idea of reading the endpoint again for each delivery.
Two choices compared to the proposal in the issue:
i > 0. The first attempt then uses the same code path as the others, and the localenabled/failingSincepatches can go entirely instead of staying alongside a fresh read. The first read also moves after the stop/lease check. The cost is the same: one primary-key lookup per attempt, next to the store write each attempt already makes and the HTTP request itself.updatedAt) instead of a full read.addEndpointSecret()doesn't bumpupdatedAt(in the in-memory and PostgreSQL stores), so a compare would miss a secret rotation. It would also need a read anyway, and a new store method that every custom store would have to implement.Tests (
tests/outgoing.e2e.spec.ts, "an endpoint changed while the worker sends a batch to it"): a disable, a delete, a URL change and a rotation withoverlap: 0during the first of three requests, plus the "(gone)" reason after a 410. All five fail on master.Does this PR introduce a breaking change?
Other information
The worker now makes one endpoint read per attempt instead of one per group of deliveries.