IdempotencyStore has no processing lease: a crashed claim submission is stuck returning 409 #429

Problem

The idempotency store treats a processing record as permanent until a matching request calls complete(), but complete() is fire-and-forget and there is no lease or TTL on in-flight records. In middleware.ts:

// app/backend/src/idempotency/middleware.ts
res.send = (body: any) => {
  // Fire and forget cache save (log on failure)
  store.complete(key, recordStatus, status, bodyString).catch(err => console.error(...));
  return originalSend(body);
};

And in store.ts, tryAcquire inserts status='processing' and never times it out; the only expiry is cleanup(maxAgeHours) which deletes by created_at with no regard for status.

Consequences, distinctly:

• A crash orphans the key. If the process dies after tryAcquire inserts processing but before complete runs, the row stays processing forever. Every retry with the same Idempotency-Key returns 409 AlreadyProcessingError indefinitely (until a manual cleanup with a large window deletes it), permanently stranding a claim submission — the exact class of operation idempotency exists to protect.
• Replay-after-success races to 409. Because complete is not awaited, a fast retry that arrives immediately after the first response can still read processing and return 409 instead of the cached response, so the "same key ⇒ same result" guarantee is violated under concurrency.
• The record has no state-machine recovery. There is no failed-terminal distinction for a crashed in-flight request; a caller cannot distinguish "still running" from "abandoned", so safe retry policy is impossible.

Root cause

The store models three statuses (processing/succeeded/failed) but no notion of a lease/ownership window or an abandoned terminal state, and the middleware treats complete as best-effort rather than as part of the request lifecycle.

Why this is architecturally hard

1. This is a concurrency-control problem, not a timeout. A correct fix needs a lease/heartbeat on the processing record (e.g. lease_expires_at refreshed while the handler runs) plus a transition rule for abandoned leases; naively deleting processing rows older than N seconds can delete a still-running long request and allow double-execution — the opposite of idempotency.
2. The response-caching path is coupled to res.send. Caching must be awaited or moved to an onFinished/interceptor that runs before the connection is released, so a replay can never observe processing after the first response is committed.
3. Two storage backends. store.ts targets Postgres via Pool, but the module must keep working with whatever IdempotencyStore the app boots (and its tests); the lease protocol must be expressed in the store interface, not in one SQL dialect.
4. Claim submission is the trust boundary. The key protects exactly-once claim submissions to the on-chain adapter; getting the lease wrong risks either stranding claims (current bug) or double-submitting them (a naive fix).

Proposed design

Add a lease_expires_at (or processing_until) column set on acquire and refreshed via a heartbeat while the handler runs; treat expired-lease processing rows as abandoned (transition to a terminal failed/abandoned state and allow re-acquire). Await complete (or use a response-finish hook) so a succeeded result is visible before the client's connection closes. A table of the target semantics:

Acceptance criteria

Service

• A simulated crash between acquire and complete does not permanently strand the key: a subsequent request with the same key re-acquires within a bounded lease window.
• A retry arriving after the first request has responded replays the cached result (200) rather than returning 409.
• An abandoned lease cannot cause the handler to run twice concurrently.

Tests

• Unit tests cover: lease expiry re-acquire, replay-after-success, and concurrent acquire contention.

Out of scope

Redis ZSET rate limiting and webhook HMAC are separate, already-tracked concerns.

Getting started

Files: app/backend/src/idempotency/store.ts, app/backend/src/idempotency/middleware.ts, app/backend/src/idempotency/error.ts, and the idempotency specs under app/backend/src/idempotency/.

cd app/backend
npm test

Good first files to read: store.ts (the tryAcquire/complete/cleanup contract) and middleware.ts (the fire-and-forget complete and 409 paths).