Batch entry inserts so large category imports finish within the request timeout

[lgelauff]

Fixes #620, the immediate cause of #618. Not a final fix for large imports: that is the background import job (#621 backend, #622 frontend).

Why

On the buildservice deploy, gunicorn's 30 s worker timeout kills large category imports. On montage-beta, a ~21.5k-file category was aborted at 30.5 s and left an empty round (details in #618).

The time goes into the database writes. Every imported file costs two single-row INSERTs, one into entries and one into round_entries, so 21.5k files mean ~43k round trips inside one request. Lookups have been chunked since 2016, but inserts have been one row at a time since then. The old webservice had no 30 s limit, so this only started failing after the move to the buildservice.

What changes

• CoordinatorDAO.add_entries: new entries in each 200-entry chunk are inserted with one multi-row INSERT, then read back by name to get their ids.
• CoordinatorDAO.add_round_entries: all round entries are inserted with one executemany. The round's round_entries collection is then expired so total_entries is not stale.
• Everything still runs in the request's transaction, so a failure rolls back exactly as before.

For a 4,000-file import (local SQLite), the number of statements drops from 8,048 to 89. On MariaDB, pymysql turns both executemany calls into multi-row INSERT statements: a test checks that the compiled statements match the pattern pymysql batches, but the suite does not run against MariaDB. The real-database evidence is the montage-dev run below.

Tests

• New montage/tests/test_import_entries.py, with synthetic data only:
  • Parity: the same scenario runs through the previous per-row code and the new code, and both must leave identical rows and identical API responses. The scenario mixes seeded and new entries over three chunks, with non-ASCII, quoted, %, emoji and space-containing names, reuploads, missing file_id, and every autodisqualify reason.
  • Edge cases: chunk boundaries (1/199/200/201/401), an empty category, exact and case-variant duplicates, re-importing into the same round, and entries shared between campaigns.
  • Failure: an error after the inserts leaves nothing behind.
  • Activation: creates quorum × eligible tasks.
  • Stale count: total_entries is right when the round's entries were already loaded.
  • MySQL batching: pymysql batches both INSERTs.
• test_web_basic.py: a statement-count regression test (450 files must need at most 3 INSERTs per table).
• The new tests fail when the change is deliberately broken (a dropped column, a missing expire, or the old per-row code put back).
• pytest montage/tests/ → 29 passed, 1 xfailed, also with network access blocked.
• conftest.py: Wikimedia user-id lookups are now mocked for the whole suite. App creation used to call the live Commons API, and the extra tests made CI hit Commons often enough to get throttled.

Verified on montage-dev

Deployed 3bd7eed to montage-dev (default gunicorn 30 s timeout) and imported Images from Wiki Loves Monuments 2025 in Russia into a new round:

• POST /v1/admin/round/130/import succeeded in 26.7 s, with no WORKER TIMEOUT or traceback in the webservice log.
• The round shows 21,521 files from 386 uploaders. The Commons category page lists 21,527; the 6-file difference has not been investigated (the import skips deleted files and case-variant duplicate names, the same as before this change).
• On montage-beta, the same import was killed at 30.5 s before this change.

Known temporary limitation

The gunicorn timeout stays at the 30 s default; we are deliberately not raising it. The margin is small (26.7 s for ~21.5k files), and most of the remaining time is no longer the inserts. Categories somewhat larger than ~21.5k files, or a slow wikireplica, can still exceed 30 s and fail as in #618. This is accepted as temporary until imports run in the background (#621, #622).

Not in this PR

• The background import job (imports independent of the request; round blocked until the import finishes): Run round imports as a background job and record their status in the database #621 and Show "import in progress" instead of keeping the browser waiting, and show import status on rounds #622.
• Activating a round still creates its tasks one row at a time (one Vote INSERT per entry × quorum, see create_initial_rating_tasks), so activating a very large round can hit the same 30 s limit. Possibly related to Can not activate a round, because of 500 error #619; not yet verified.
• Autodisqualify still updates disqualified round entries one at a time.
• sentry-sdk 2.8.0 crashes while capturing exceptions on Python 3.13 (FrameLocalsProxy). Separate issue.

Refs #618, #621, #622

Written with Claude (Claude Code) on behalf of @lgelauff.

🤖 Generated with Claude Code

[lgelauff]

Tested on montage-dev in synchronous import mode (the way production runs), on 2026-10-02, on an integration branch with #623, #626, #628, #630 and #631:

• Images from Wiki Loves Monuments 2026 in Russia: 15,676 files in 8.1 s.
• Images from Wiki Loves Monuments 2025 in Russia: 21,521 files in 15.0 s (26.7 s with Batch entry inserts so large category imports finish within the request timeout #623 alone).
• A file list of 1,503 names (79 KB): 1,499 files in 1.3 s, with a warning for each of the 3 names that do not exist (the 1,500 real names hold 1,499 that are unique ignoring case).

Activating the 21.5k-file round afterwards got the webservice OOMKilled. That is activation, not changed by this PR; it is fixed in #632, with which the same round activated in 4.5 s.

Written with Claude (Claude Code) on behalf of @lgelauff.