Background import job, Phase 1: imports run in a worker, round shows import status

[lgelauff]

Phase 1 of #621 (backend) and #622 (frontend). Builds on #623 (batched inserts); the base branch is fix/large-import-timeout-618, so this diff shows only the background-import work.

What changes

With MONTAGE_IMPORT_MODE=worker, importing entries into a round no longer happens inside the HTTP request:

• Enqueue. POST /admin/round/<id>/import validates the request, records an import_jobs row, and returns at once with the job.
• Worker. montage/import_worker.py runs as a Toolforge continuous job, with exactly one replica. It claims the oldest queued job with a compare-and-set UPDATE and fetches the files with no transaction open. It then runs the whole import (entries, round entries, round source, autodisqualify, audit log) in one transaction. The UPDATE that marks the job succeeded is fenced and comes last, so "finished" only appears once the data is committed. A failure is recorded separately, with a readable reason.
• Stuck jobs. At startup the worker fails jobs still running (a restart or deploy interrupted them), and it fails jobs running longer than 60 minutes. Fetches now have timeouts: HTTP is (15 s, 600 s); the Commons database is 30 s to connect and 45 min per query. A hung fetch therefore fails its job and the worker moves on.
• Activation gate. A paused round cannot be activated while any of its imports is queued or running, or while its latest import failed. There is a non-locking pre-check, then a round lock and a locking re-check. The API returns 400 with a specific message.
• UI.
  • The round-creation dialog closes at once with "Import started… refresh to see when it has finished".
  • The round view, the round header and the campaign cards show the import state whenever they load: queued, running, finished (with counts and warnings), or failed (with the reason).
  • Activate is disabled with an explanation. Until Phase 2, a failed import's hint is "cancel this round and create it again".
  • There is no polling. All new strings are in en.json and qqq.json.
• Rollback switch. MONTAGE_IMPORT_MODE defaults to sync outside dev and devtest. In sync mode the import runs in the request as before (no locks are held across the fetch) and a succeeded job row is recorded. Deploying this code changes nothing until worker mode is switched on.
• Deploy.
  • New import_jobs table: tools/migrate_import_jobs.sql, with tools/revert_import_jobs.sql to undo it.
  • tools/deploy.sh runs a schema pre-flight before restarting the webservice, then restarts the worker job.
  • deployment.md covers the worker job, the migration, rollback, and a hung worker.

Tests

• pytest montage/tests/ gives 73 passed, 1 xfailed, also with network access blocked.
• New test_import_jobs.py covers:
  • enqueue, the claim compare-and-set, the fenced success and fenced failure;
  • the startup and 60-minute sweeps;
  • the activation gate, the payloads (admin yes, juror no) and 400s on bad requests;
  • timeouts;
  • sync mode, including a rollback check.
• The existing import tests drain the queue and assert the job outcome.
• A mutation pass removed the safety checks one at a time (the fence, the gate, the claim compare-and-set, both sweeps, sync-mode locking, the timeouts); the suite catches each of them.
• Not covered on SQLite: the locked re-checks in activation and enqueue (SQLite ignores FOR UPDATE). Those need MariaDB.
• Frontend: npx prettier --check src/, npx eslint src/ and npm run build all pass.

Not verified yet

• Nothing has run on MariaDB or Toolforge yet. Still to check on beta:
  • that the migration runs;
  • that the env vars and ~/replica.my.cnf reach the job pod;
  • the exit code of toolforge jobs run --wait;
  • the worker's memory on the largest category.
• The UI journeys have not been clicked through. Local login on the dev box fails, so this needs montage-dev.

Known limitations (Phase 2)

• No heartbeat, so a deploy fails a running import ("interrupted by a worker restart").
• A queued job never times out if no worker is running.
• No Retry/Dismiss: a failed import blocks activation until the round is recreated.
• Lock and deadlock errors in activation and enqueue give a 500, not a friendly message.

Refs #621, #622, #618

Written with Claude (Claude Code) on behalf of @lgelauff.

🤖 Generated with Claude Code

[lgelauff]

Superseded by #630 (backend) and #631 (frontend), which split this PR and add Phase 2. Closing.

Written with Claude (Claude Code) on behalf of @lgelauff.