Background import job: imports run in a worker (backend) by lgelauff · Pull Request #630 · hatnote/montage

lgelauff · 2026-10-02T19:40:11Z

Backend for #621 (Phase 1 and Phase 2) and #622; the frontend is a separate PR on top of this one, which closes both issues. Based on #623 (batched inserts). Replaces #624, which mixed backend and frontend.

Deploy order: run tools/migrate_import_jobs.sql BEFORE deploying this, also in sync mode. The app checks the schema at startup and exits if the import_jobs table is missing, and sync mode writes a job row for every import. tools/deploy.sh checks this in the new image before it restarts anything, and aborts if the table is missing.

The default MONTAGE_IMPORT_MODE=sync keeps the current behaviour: the import runs in the request, and the old UI works unchanged. Don't switch to worker mode before the frontend PR is deployed. Worker mode on production is also gated by the tracking issue #629 ("Before enabling worker mode on production").

How to read this PR

19 commits. They are meant to be read in four groups (SHAs as of this branch; they change on rebase):

Phase 1, core: the job table and queue, the worker, the activation gate, the API, and the docs.
- cde012c feat: background import jobs and worker (Phase 1)
- c15172b test: background import job tests; drain the queue in existing tests
- 8be3074 docs(deploy): import worker job, schema pre-flight, rollback
Phase 1, fixes found while testing it:
- 8449a9d keep the worker's polls out of the log with db_echo on
- 561027b no locks across the fetch in sync import mode
- c4017b8 time out hung fetches so the worker moves on
- 047496e tests that fail when the worker's safety checks go
- 2ccd091 find wikireplica credentials in Toolforge job pods
Phase 2: recovery, retry and dismiss, lock errors. 2d5235a rewrites part of the Phase 1 worker, so read the worker at the end of this group rather than commit by commit.
- 2d5235a heartbeat, stale-job recovery and SIGTERM release in the worker
- 262768b retry and dismiss failed imports
- 1a19b5d MySQL lock errors in enqueue/activate are a retryable 400
- df31e45 round_sources.params MEDIUMTEXT, compose worker, deploy docs
- 4cf8a71 tests: the stale re-check and activate's final flush
Review fixes (from the PR check and the review panel):
- 01da1be keep retry_of, fast shutdown, no import after a lost claim
- 2c68dc8 compose: both services point at the lowercase dockerfile
- 53055c1 10 s wikireplica connect timeout, as before
- d33a254 the deploy pre-flight checks MONTAGE_IMPORT_MODE
- 57c15eb docs: never suggest toolforge envvars list
- b33f938 docs: fix drift in deployment.md, deploy.sh and a docstring

Please merge with a merge commit, so the groups stay readable.

What changes

Enqueue. With MONTAGE_IMPORT_MODE=worker, POST /admin/round/<id>/import validates the request, records an import_jobs row and returns at once with the job. A missing body or missing keys now give 400 instead of 500.
Worker (montage/import_worker.py, Procfile import-worker, a Toolforge continuous job):
- claims jobs with a compare-and-set UPDATE and fetches the files with no transaction open;
- runs the whole import in one transaction. The fenced UPDATE that marks the job succeeded comes last, so "finished" means committed;
- records failures separately, with a readable reason;
- has timeouts on every fetch: HTTP 15 s to connect and 600 s per read; Commons database 10 s to connect (pymysql's default, as before) and 45 min per query;
- finds the wikireplica credentials in job pods (TOOL_REPLICA_*, then $TOOL_DATA_DIR/replica.my.cnf). In a job pod HOME=/.
Recovery:
- A heartbeat every 30 s. A job with no heartbeat for 5 min is requeued once, then failed.
- SIGTERM, i.e. a deploy, hands the running job back to the queue.
- Jobs are failed after 60 min. This limit is soft: a fetch already under way is not cut short.
- Job rows stay correct with more than one replica, but keep one: overlapping imports contend for locks.
Activation gate. A paused round cannot be activated while any of its imports is queued or running, or while its latest import failed and was not dismissed. There is a non-locking pre-check, then a round lock and a locking re-check. MySQL lock errors (1205/1213) become a 400 "busy, please retry".
API for the frontend:
- import_state per round in the admin payloads;
- GET /admin/round/<id>/imports and GET …/import/<job_id>;
- POST …/import/<job_id>/retry and …/dismiss.
- Jurors never see import_state.
Sync mode (default) records a succeeded job row and holds no locks across the fetch.
Schema and deploy:
- tools/migrate_import_jobs.sql / revert_import_jobs.sql;
- round_sources.params → MEDIUMTEXT. The class, column and SQL are identical to the Commons-lookups PR, so the two merge cleanly;
- tools/deploy.sh runs a pre-flight in the new image before restarting the webservice: the schema check, and since this round of fixes also MONTAGE_IMPORT_MODE (an unknown value stops the app from starting, so a typo now aborts the deploy instead of taking the site down). Then it restarts the worker;
- deployment.md covers the worker job, migrations, rollback and a hung worker;
- docker-compose.yml gains a worker service.

Fixes from the PR check and the review

retry_of kept: a retried job that succeeded in worker mode lost its "retry of #N" (the success UPDATE replaced flags). The worker now merges the existing flags.
Shutdown: the heartbeat thread is joined once, for at most 2 s, so a stuck heartbeat can no longer push the SIGTERM release past the pod's grace period.
Lost claim: if the claim expired (60 min) or was lost during the fetch, the worker no longer opens the write transaction, which would hold locks against the run that now owns the job.
Compose: both services name the tracked dockerfile (lowercase); the build failed on case-sensitive filesystems.
Timeouts: the wikireplica connect timeout is back to 10 s (this branch had raised it to 30 s). In sync mode the fetch runs inside the web request, and 30 s ran into gunicorn's 30 s kill (a 502) instead of failing cleanly. MONTAGE_LABS_CONNECT_TIMEOUT still overrides it.
Import mode in the pre-flight, and no toolforge envvars list in deployment.md (it prints secrets; the docs now use toolforge envvars show <NAME>).
Doc drift: the import_entries docstring (sync is the default), the credential source, the deploy.sh header path, the replica and max-runtime wording.

Each code fix has a test.

Merging after the Commons-lookups PR (#628)

If #628 merges first:

run_import conflicts. Both sides make the same fix; take this PR's side.
Read Commons category members from the links replica, and batch file lookups #628's two API tests (selected-import and CSV warnings) already handle worker mode: when the response carries a job, they run the queue with run_import_jobs and read the warnings from GET /admin/round/<id>/import/<job_id>. No follow-up commit is needed.
The MEDIUMTEXT change then comes from Read Commons category members from the links replica, and batch file lookups #628, and the duplicate hunks collapse on rebase.

Tests

pytest montage/tests/ gives 117 passed, 1 xfailed on this branch (re-run after rebasing on master).
A mutation pass removed each safety check (the fences, the gate, the claim compare-and-set, the sweeps and heartbeat, SIGTERM release, the timeouts, sync-mode locking, the lock-error mapping, and after the review: retry_of, the shutdown join, the lost-claim check); each removal was caught.
Not covered on SQLite: the locked re-checks (SQLite ignores FOR UPDATE), and every other lock behaviour.

Verified on montage-dev (worker mode)

Deployed on 2026-10-02 as part of an integration branch (master + #623 + #628 + #626 + this PR + the frontend PR), after running both migrations:

Images from Wiki Loves Monuments 2026 in Russia: 15,676 files imported by the worker in 13 s; queued to running in 1 s.
A file list of 1,503 names (79 KB): 1,499 files in about 1 s, with a warning for each of the 3 names that do not exist.
Three imports of a 427-file category: all succeeded, and the round could be activated afterwards.
Activation, with this PR's lock: a double click in the UI sent one request; two activation requests sent at the same moment gave 200 and 400, and created one task per entry.
Earlier, on the first combined branch: a 21,521-file category, and the fix for replica credentials in job pods (1045 Access denied).
Not yet run: synchronous mode on montage-dev (production will run in sync mode first), heartbeat recovery, SIGTERM release, retry and dismiss, and MariaDB lock behaviour under load.
Found there, not caused by this PR: listing the entries of a 15.7k-entry round (/admin/round/<id>/entries and its CSV download) times out at 30 s.

Known limits and open decisions

Decisions still open:

Retry and dismiss work on any failed job of the round, not only the latest. Retrying an old failure can unblock a round whose newest import also failed; dismissing an older failure has no effect. The UI only offers the latest job. Restricting the endpoints to the latest job is a one-line policy change, not made yet.
Lock errors and releases. A lock error inside the worker fails the job instead of requeueing it once, which can happen when a released job is rerun while the old transaction is still rolling back. And a job whose pod keeps getting SIGTERM is released without using an attempt, so it never fails visibly. Requeue-on-lock-error and a cap on releases are not implemented.
Strict mode in the revert SQL. tools/revert_round_sources_params.sql only fails on long rows under strict sql_mode; otherwise MariaDB truncates them. Run the documented length check first; adding SET SESSION sql_mode='STRICT_ALL_TABLES' is not done yet.

Other limits:

A queued job has no timeout if no worker runs. The UI shows "queued" for as long as that lasts.
The 60-minute limit is soft (see above).
Worker memory: about 100k files is the ceiling under --mem 1Gi (1,043 MiB measured at 120k files, off Toolforge).
Activation still creates its tasks inside the request, one row at a time; very large rounds may hit 30 s (refs Can not activate a round, because of 500 error #619).

These, and the other items to do before worker mode is switched on in production, are tracked in #629 ("Before enabling worker mode on production").

Written with Claude (Claude Code) on behalf of @lgelauff.

🤖 Generated with Claude Code

POST /admin/round/<id>/import now validates the request, records a queued import_jobs row and returns at once; a Toolforge continuous job (python -m montage.import_worker, Procfile "import-worker", one replica) claims it with a compare-and-set UPDATE, fetches the files with no transaction open, and runs the existing import code in one transaction whose last statement marks the job succeeded (fenced on a claim token). Failures roll back and are recorded as failed jobs with a sanitised reason and an import_failed audit entry. - ImportJob model + LongJSONEncodedDict (MEDIUMTEXT on MySQL); tools/migrate_import_jobs.sql / revert_import_jobs.sql - import_state_for(): admin payloads (/admin, /admin/campaigns/all, /admin/campaign/<id>, /admin/round/<id>) carry import_state - activate_round refuses while any job is queued/running or the latest failed (non-locking pre-check, round lock, locking re-check) - GET /admin/round/<id>/imports and /admin/round/<id>/import/<job_id> - stuck-job rule: running jobs failed at worker startup and after 60 min - MONTAGE_IMPORT_MODE=sync keeps the synchronous import as a rollback switch (default sync outside devtest until the worker runs) - 400 instead of 500 for missing import keys; the selected-import warning is a dict again (was a set literal) - make_engine(config) extracted from create_app for the worker Refs #621, #618 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…g tests - conftest.run_import_jobs(app): run queued jobs in-process (import_worker.run_once) while external APIs are mocked - test_web_basic: drain after each import (imports are queued now); test_category_import_batches_inserts reads counts from the job - test_import_entries: runs in import_mode 'sync' (asserts on the synchronous response; same run_import code path as the worker) - new test_import_jobs.py: enqueue without external calls, worker success/failure/rollback, CAS claim, fenced commit, startup and 60-min stuck-job sweeps, loop survives DB errors, activation gate, import_state in admin (not juror) payloads, 400 on missing keys, sync rollback mode, schema check, MEDIUMTEXT columns Refs #621 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

- deploy.sh: check the schema inside the new image with a one-off schema-check job (tools/check_schema.py) and abort before restarting if a migration is missing; restart the import-worker continuous job after the webservice (or print the command to create it) - deployment.md: MONTAGE_IMPORT_MODE; tools/migrate_import_jobs.sql; new step 7b (create the import-worker job, exactly one replica); worker logs; "Imports stay queued" troubleshooting with the rollback SQL; missing import_jobs table at startup The Toolforge behaviour the script relies on (envvars in job pods, jobs run --wait exit status and working directory, jobs restart pulling :latest, ~/replica.my.cnf in a job pod) is still to be verified on montage-beta. Refs #621 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

With db_echo the worker logged the SQL of every 5-second poll; it now only logs its own lines (seen on the hague dev setup). Refs #621 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

check_no_active_import took the round row lock and a FOR UPDATE read on import_jobs, then the same request fetched and imported everything. On MariaDB that read's gap lock is held for the whole import, so imports into other new rounds wait or deadlock (1205/1213) and get a 500. Sync mode (the rollback switch, and the default) now only does the plain active-job pre-check, which restores the pre-#621 lock footprint. The residual race (an enqueue from a worker-mode process during a mode flip) is documented in the docstring. Refs #621 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

A dead wikireplica connection or a stalling CSV/API host blocked the single import worker for ever, so every later import stayed queued. - requests_get/requests_post: default timeout (15 s connect, 600 s per read), overridable with timeout= - labs.fetchall_from_commonswiki: connect_timeout 30 s, read_timeout 45 min (one query, under the worker's 60-min cap), write_timeout 60 s; MONTAGE_LABS_CONNECT_TIMEOUT / MONTAGE_LABS_READ_TIMEOUT override - worker: readable reasons for fetch timeouts and wikireplica errors; a failing rollback no longer skips marking the job failed - check_schema closes its session and disposes its engine (the worker kept an idle connection open for its lifetime) - deployment.md: "hung worker" recovery note Refs #621 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

From the PR check's mutation run (M3, M7, M8, M10 survived): - main() runs the startup sweep before the first loop iteration - run_loop_iteration runs the 60-min sweep - a failure after a sweep is fenced (keeps the sweep's error, one audit) - claim CAS goes through claim_next_job with a racing writer between its SELECT and UPDATE (replaces the hand-written UPDATE test) Also: sync mode takes no lock before the fetch; default HTTP and wikireplica timeouts; a timed-out fetch fails its job readably and the next job runs; a failing rollback still marks the job failed; the sync failure test checks entries/round_entries/round_sources roll back; every run_import_jobs call in test_web_basic asserts its outcome. Refs #621 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

The import worker runs as a Toolforge job, where HOME is / and not the tool's home, so ~/replica.my.cnf did not exist and pymysql connected without a password: on montage-dev every worker category import failed with 1045 "Access denied ... (using password: NO)". replica_credentials() now uses TOOL_REPLICA_USER/TOOL_REPLICA_PASSWORD when Toolforge sets them, else replica.my.cnf in TOOL_DATA_DIR, else ~ (the old behaviour), and raises a clear error if none exists. Checked in a montage-dev job pod: HOME=/, TOOL_DATA_DIR set, replica.my.cnf present there, TOOL_REPLICA_* set. Refs #621 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…e worker Phase 2 of the background import job (#621, origin #618). - A heartbeat thread refreshes heartbeat_date every 30 s (fenced on id, status='running' and claim_token) and fails the job after the 60-minute maximum runtime. It is stopped and joined before the final fenced success UPDATE. - recover_stale_jobs replaces the Phase 1 startup and max-runtime sweeps: a running job without a heartbeat for 5 minutes is requeued while attempts < 2, then failed ("the import worker stopped responding"). Conditional, fenced UPDATEs only, so several worker replicas are safe. - SIGTERM/SIGINT: while a job is fetched or imported the handler raises WorkerShutdown; the worker drops the import connection (server-side rollback) and releases the job back to 'queued' without using up an attempt, then exits 0. Between jobs it only sets a flag. - Jobs of rounds that are no longer paused (e.g. cancelled while queued) are failed before any fetch. - MySQL lock errors (1205/1213): the loop logs them and continues; a job failing on one gets a "database was busy, retry" reason. Tests: stale requeue then fail, rerun after requeue, no startup sweep (second replica), loop recovery, fenced failure after recovery, heartbeat tick/max runtime, heartbeat stopped before the final UPDATE, release, real SIGTERM during the fetch, shutdown during the import, cancelled round skipped without fetching, lock errors in the loop. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Phase 2 of the background import job (#621). - POST /admin/round/<id>/import/<job_id>/retry: imports again with the failed job's method and params (a new job; queued in worker mode, run in the request in sync mode). The new job records retry_of. - POST /admin/round/<id>/import/<job_id>/dismiss: sets dismiss_date and dismiss_user on a failed job and logs dismiss_import; the activation gate (import_state_for) already ignores dismissed failures. An empty round still cannot be activated. - The gate's failed-import message now says "Retry the import, or dismiss it" instead of "cancel this round and create it again". - Job details no longer expose claimed_by (worker hostname:pid) to coordinators. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Phase 2 of the background import job (#621; risk 4 of the Phase 1 review). enqueue_import and activate_round run inside lock_errors_as_busy: a MySQL/MariaDB lock wait timeout (1205) or deadlock (1213) becomes InvalidAction "the server is busy, please retry (...)" (HTTP 400; the request is rolled back as before) instead of a 500. activate_round now flushes inside the guard, so lock errors of the task inserts are mapped too. Other database errors are unchanged. The worker side (log and continue) is in the recovery commit. Not changed: the locking reads themselves (gap locks on import_jobs); that needs Lodewijk's decision. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…y docs Phase 2 of the background import job (#621). - round_sources.params uses LongJSONEncodedDict (MEDIUMTEXT on MySQL, unchanged on SQLite); tools/migrate_round_sources_params.sql and revert_round_sources_params.sql. The schema check ignores column types, so the migration can run before or after the deploy. - docker-compose.yml: an import-worker service for contributors (same image, repo mount and SQLite DB; MONTAGE_IMPORT_MODE=worker on both). - tools/deploy.sh: find the worker with `toolforge jobs list`, so a missing job and an API error get different messages; a failed restart no longer aborts the deploy before the smoke test. - deployment.md: heartbeat/stale recovery, SIGTERM release, more than one replica is safe, Retry/Dismiss, the new migration, and what to verify on montage-beta (launcher forwards SIGTERM, grace period, jobs list). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Phase 2 of the background import job (#621). Two mutations survived the suite: dropping the staleness re-check from the recovery UPDATE, and dropping the flush inside activate_round's lock-error guard. These tests fail for each. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Fixes from the PR check of the import worker (#621). - M1: the worker's success UPDATE replaced the whole flags column and erased retry_of. The claim now carries the job's flags and the UPDATE merges the import's flags into them, as sync mode already did. - M2: on shutdown the heartbeat thread is joined once, for at most SHUTDOWN_JOIN_TIMEOUT (2 s), instead of up to 60 s twice, so the job is released within the pod's grace period. Heartbeat.stop() joins only on its first call. Releasing is fenced, so a late tick is harmless. - M3: after the fetch, if the heartbeat reports the claim expired or lost, the worker returns 'lost' without opening the write transaction, so a zombie run holds no locks against the run that owns the job. - L3: the recovery fence uses an explicit IS NULL for a running row without a claim token. Tests (each fails without its fix): retry_of after a worker success; a stuck heartbeat tick does not delay shutdown (single short join); no import after the claim expired or was lost during the fetch; the heartbeat thread stops after lost/expired; recovery of running rows with NULL heartbeat_date and claim_token. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

The tracked file is `dockerfile`; `dockerfile: Dockerfile` only worked on case-insensitive filesystems. Fixes the existing montage service and the import-worker service added for #621 (PR check L7). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

The worker branch raised the connect timeout to 30 s. In sync import mode (the default) the replica fetch runs inside the web request, so an unreachable replica ran into gunicorn's 30 s kill (a 502) instead of failing cleanly. Go back to pymysql's own default of 10 s, which master used; MONTAGE_LABS_CONNECT_TIMEOUT still overrides it. Refs #621 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

The web app refuses to start on an unknown import mode, so a typo in the envvar took the site down at the webservice restart. tools/check_schema.py (the schema-check job deploy.sh runs in the new image before restarting anything) now validates the configured mode with utils.get_import_mode, so a bad value aborts the deploy instead. deployment.md says so, and that a manual webservice restart is not covered. Refs #621 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

It prints every envvar value, secrets included. The hung-queue checklist now uses `toolforge envvars show MONTAGE_IMPORT_MODE`, and the envvars section says to look at variables one at a time. Refs #621 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

- import_entries: sync is the default import mode, worker the opt-in. - deployment.md: where the worker finds the wikireplica credentials (TOOL_REPLICA_*, then $TOOL_DATA_DIR, then ~); keep one replica; the 60-minute limit is soft; Dismiss does not make an empty round activatable; move what montage-dev already showed out of the beta list and note the ~100k-file memory ceiling under --mem 1Gi. - deploy.sh header: the deploy clone is ~/www/python/src, as in deployment.md. Refs #621 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

lgelauff · 2026-10-02T20:44:22Z

Test update, montage-dev, 2026-10-02: an integration branch with #623, #626, #628, #632 and #631, in synchronous mode (MONTAGE_IMPORT_MODE=sync, the default this PR keeps for production):

Images from Wiki Loves Monuments 2026 in Russia: 15,676 files in 8.1 s; … 2025 in Russia: 21,521 files in 15.0 s; a 1,503-name file list: 1,499 files in 1.3 s with 3 warnings.
Each import came back with its counts and was recorded as a succeeded job.
The schema check at startup passed with both migrations applied.

Still not tested: Activate being blocked while an import is queued or running, Retry, Dismiss, the 60 s poll dialog, heartbeat recovery, SIGTERM release, and MariaDB locking under load.

Written with Claude (Claude Code) on behalf of @lgelauff.

lgelauff and others added 19 commits October 1, 2026 14:01

fix(import): keep the worker's polls out of the log with db_echo on

8449a9d

With db_echo the worker logged the SQL of every 5-second poll; it now only logs its own lines (seen on the hague dev setup). Refs #621 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

lgelauff marked this pull request as ready for review October 2, 2026 20:41

lgelauff added the final review label Oct 2, 2026

This was linked to issues Oct 3, 2026

Run round imports as a background job and record their status in the database #621

Open

Large category import killed by gunicorn's 30 s worker timeout (buildservice deploy) #618

Open

lgelauff mentioned this pull request Oct 4, 2026

docs(deployment): clone master for the deploy script, not the deleted tools/buildservice branch #646

Merged

This was referenced Oct 4, 2026

Narrow the file-name lookup in imports by upload date, so it does not scan the whole entries table #651

Closed

Check first-round imports before importing; save a first round with its files (#510, #509, #447) #656

Merged

mahmoud mentioned this pull request Oct 10, 2026

October 2026 integration: import check (#656), quorum reduction (#617), final-review PRs, sentry/timeout/export/import-lock fixes #659

Merged

Navigation Menu

Repository navigation

Background import job: imports run in a worker (backend) - #630

Background import job: imports run in a worker (backend)#630
lgelauff wants to merge 19 commits into
fix/large-import-timeout-618from
feat/import-worker

lgelauff commented Oct 2, 2026

Uh oh!

lgelauff commented Oct 2, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

lgelauff commented Oct 2, 2026

How to read this PR

What changes

Fixes from the PR check and the review

Merging after the Commons-lookups PR (#628)

Tests

Verified on montage-dev (worker mode)

Known limits and open decisions

Uh oh!

lgelauff commented Oct 2, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant