[clonder]

continues #7075 after #7133 took the batching half.

Backstory and Issue

Every worker now tries to claim sequences by inserting rows. With 64 workers, about half of ~1M claim attempts lost the race, and the contention made requests so slow that workers hit their 30s timeout and crash-looped (101 restarts in the baseline run). #7133 already improved performance by batching insertions, this PR adds leases to prevent workers "fighting" each other.

A result was stored for whoever held the row, nobody checked which worker. A batch slower than the stale timeout overwrites the result of the worker that re-claimed it, and that worker gets a 422 for its whole batch.

Now every sequence is already in the queue as an UNPROCESSED row. A worker claims a batch atomically (FOR UPDATE SKIP LOCKED) and gets a lease: a claim ID + expiry time. If the worker dies, the lease expires and the batch goes back to the queue. Submitting results requires the matching claim ID, so a worker that lost its lease can't overwrite anyone else's work.

On the 64 pod runs from #7075 claim attempts went from 1,008,177 to 25,468, worker restarts from 101 to 0.

BREAKING CHANGES

Pipelines must send back the processingAttemptId they got from extract-unprocessed-data.

PR Checklist

• All necessary documentation has been adapted.
• The implemented feature is covered by appropriate, automated tests.
• Any manual testing that has been done is documented (i.e. what exactly was tested?)
  • benchmark runs in feat!(backend, preprocessing): lease-based preprocessing work queue #7075

🚀 Preview: Add preview label to enable

[claude]

Claude finished @clonder's task in 2m 32s —— View job

---

Review: lease-based preprocessing work queue

• Gather context
• Review backend changes
• Review migrations
• Review preprocessing changes
• Post review

The design looks solid. Claiming is now a single UPDATE ... FOR UPDATE SKIP LOCKED on a partial index. Submitting a result requires the matching processing_attempt_id. Edits and expired leases put the row back as UNPROCESSED instead of deleting it. I checked the edge cases I could think of, and they hold:

• Status view: an UNPROCESSED row still shows as RECEIVED / NO_ISSUES, the same as having no row before.
• Revocations: still get no queue rows.
• Deletes: the ON DELETE CASCADE foreign key still covers them.
• Old pipeline versions: the controller already returns 422 for an outdated version before the table lock is taken, so old workers can't keep re-taking it.
• Switching to a new pipeline version: findNewPreprocessingPipelineVersion doesn't switch early, because UNPROCESSED rows of the new version never count as successfully processed.
• Migrations: splitting them into parts, filling organism before the ACCESS EXCLUSIVE lock, and NOT VALID followed by VALIDATE all look right.

Main concern (inline): while a new version is being backfilled, every polling worker blocks on the FOR UPDATE of the registration row and holds a DB connection the whole time. With many pods this can use up the connection pool for the entire backend. SKIP LOCKED would avoid it.

Smaller points:

• The API docs say late results for an expired claim are rejected. In practice they are only rejected once the clean-up task has put the row back in the queue (inline).
• getPipelineVersionStatistics and findNewPreprocessingPipelineVersion still join sequence_entries to get the organism. They could now read sepd.organism, which matches the reasoning behind V1.39. This is optional.
• website/src/types/backend.ts unprocessedData doesn't include processingAttemptId or leaseUntil. The website doesn't use that endpoint, so this is only for consistency.

Tests: coverage is good: concurrent claims, re-claim after expiry rejecting the old attempt, a rolled-back mixed batch, migration tests, and carrying the attempt id through the Python pipeline. What's missing is a test for concurrent first-time extracts of a new pipeline version, which is the backfill path above.

I did not run the backend or Python tests. This review comes from reading the code only.

[anna-parker]

It would be great to also add to the PR description why there are 3 separate db migration files, why split up the work? also can they all be released to prod together in one rollout?