Skip to content

fix: grant no locks once the application starts shutting down - #41

Merged
kamilmysliwiec merged 1 commit into
masterfrom
fix/no-locks-during-shutdown
Oct 23, 2026
Merged

kamilmysliwiec merged 1 commit into
masterfrom
fix/no-locks-during-shutdown

Conversation

@kamilmysliwiec

Copy link
Copy Markdown
Member

PR Checklist

Please check if your PR fulfills the following requirements:

PR Type

What kind of change does this PR introduce?

  • Bugfix
  • Feature
  • Code style update (formatting, local variables)
  • Refactoring (no functional changes, no api changes)
  • Build related changes
  • CI related changes
  • Other... Please describe:

What is the current behavior?

beforeApplicationShutdown() releases a snapshot of the held locks, and each release wakes the acquire() calls waiting for that key. A woken acquire() gets the lock straight away, and that lock is added after the snapshot, so nothing releases it. It keeps renewing after onApplicationShutdown() has closed the store's connection: each renewal fails, and the lock is later reported lost. An acquire() whose store call was in flight when the hook began ends up the same way.

Issue Number: #40

What is the new behavior?

From the start of beforeApplicationShutdown(), Locks grants no new locks:

  • acquire() (and the internal tryAcquire()) resolves null without a store call.
  • A grant that lands after the hook began goes back to the store, and the call resolves null.
  • Callers waiting for a key stop waiting at once, including a wait on a key held by another instance, which would otherwise sleep until its next backoff (up to 1s).
  • A grant whose lock the hook had already released, because the shutdown began between the store's answer and acquire() returning, resolves null instead of a released Lock.
  • withLock() rejects with LockNotAcquiredError with reason: 'shutdown' and the message Could not acquire the lock "k": the application is shutting down. reason is a new optional constructor argument that defaults to 'taken', so the existing messages are unchanged.

Thanks to @johnnyhuirilef for the report, the reproduction and the closing flag, which this PR builds on. On top of it, this PR wakes the sleeping waiters, returns null for a grant whose lock the hook already released, and adds the reason on LockNotAcquiredError.

Does this PR introduce a breaking change?

  • Yes
  • No

Other information

A grant that only lands after onApplicationShutdown() has closed the store can't be given back, so that key stays taken for one ttl. Elections and ticks already behave this way at shutdown, and the hook doesn't wait for such grants, so a stalled store doesn't hold up the deploy.

The docs (content/reliability/locks.md, the acquire()/withLock() paragraph, the shutdown paragraph and the errors table) need a follow-up in docs.nestjs.com.

beforeApplicationShutdown() released a snapshot of the held locks, and
each release woke the acquire() calls waiting for its key. A woken
acquire() took the lock at once and added it to the held set after the
snapshot, so nothing released it: it kept renewing after
onApplicationShutdown() had closed the store's connection, logged
"Could not renew the lock", and was then reported lost. An acquire()
whose store call was still in flight when the hook began had the same
problem.

From the start of the hook, acquire() resolves null without a store
call, a grant that lands after that goes back to the store, and the
callers waiting for a key stop waiting at once. withLock() then rejects
with a LockNotAcquiredError whose new `reason` is 'shutdown', with a
message that says so.

Closes #40
@kamilmysliwiec
kamilmysliwiec merged commit 9ebfc03 into master Oct 23, 2026
10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant