Read Commons category members from the links replica, and batch file lookups

[lgelauff]

Fixes #625. Fixes #627 (file-list and filename-only CSV imports time out).

Based on master and code-independent of #623. Deploy together with or after #623 (batched inserts): without it a ~15k-file category import still writes one row per file and can pass gunicorn's 30 s.

Why

Two problems with how Montage reads files from Commons:

• Category imports miss files. Since 2026-09-08 the Commons links tables (categorylinks, linktarget, …) live on their own database cluster (x4), and the copies on the main Commons replica are no longer written (Wikitech, T398709). get_files() joined categorylinks with file/filerevision in one query on the main replica, so it silently missed every file added to a category after that date. Images from Wiki Loves Monuments 2026 in Russia imported 4,989 of 15,678 files. The stale copy has links up to 2026-09-08 and none after; the links replica has all 15,678.
• File-list imports time out (tracked in File-list and filename-only CSV imports time out (one query per file) #627). File-list ("selected") imports, filename-only CSVs and /utils/file looked up each file with its own query and its own connection. On montage-beta a 1000+ name list was killed by gunicorn's 30 s worker timeout. This is not caused by the x4 move; it is a separate bug on the same code path.

What changes

• Category imports: labs.get_files() reads the category's file names from the Commons links replica (categorylinks ⋈ linktarget ⋈ page; page exists on both clusters). It then looks up file/filerevision/actor/filetypes on the main replica, 500 names per query, and joins in Python. Output keys and order are unchanged. The host can be overridden with MONTAGE_COMMONS_LINKS_DB_HOST.
• File-list imports, filename-only CSVs and /utils/file share the batched labs.get_files_info_by_names(). Results keep the input order, and every missing name still gets its warning.
• One replica connection per lookup. A whole get_files() call now uses one connection per replica host (links + main), and get_files_info_by_names() one, instead of one connection per 500-name chunk. Connections are closed when the lookup ends, also on errors; before, they were never closed and were dropped by garbage collection. fetchall_from_commonswiki() keeps its signature and also closes its connection now.
• Three bugs on these paths:
  • A file-list import with any unknown name returned 500: the warning was a set, {'import issues', msg}.
  • A filename-only CSV with the local source could never succeed: it unpacked the file-info dict into (edict, warnings).
  • A CSV import's warning was a plain string, which the import dialog showed as its last character only. It is now a dict ({'import issues': ...}) like the other warnings. This was pre-existing, but became reachable here because the filename-only CSV import used to crash before it got that far.
• round_sources.params → MEDIUMTEXT on MySQL (LongJSONEncodedDict). A 1000+ name list is over TEXT's 64 KB. Migration: tools/migrate_round_sources_params.sql; revert: tools/revert_round_sources_params.sql.
• get_files_legacy() and its xfail parity test are removed. They read the dropped image/oldimage tables and the stale categorylinks; the docstring asked for their removal after 28 May 2026.
• Docs: deployment.md lists MONTAGE_COMMONS_LINKS_DB_HOST and the migration.
• Test suite offline: the conftest change from Batch entry inserts so large category imports finish within the request timeout #623 (fdfa606) is included, so this branch's tests don't call the live Commons API. The patch is identical and merges cleanly with Batch entry inserts so large category imports finish within the request timeout #623.

Tests

• pytest montage/tests/ gives 29 passed, 1 xfailed (the new live test, Toolforge only). External HTTP and the replica connections are mocked.
• New tests:
  • category members come from the links replica and file data from the main replica;
  • chunking, binary (case-sensitive) order, missing names, duplicates and space/underscore pairs, empty categories;
  • one connection per replica host for a multi-chunk lookup (2 for get_files, 1 for get_files_info_by_names), all closed, also when a query fails;
  • filename-only CSV;
  • the missing-name warning and the CSV warning shape through the API (written to work in both the synchronous and the background-import mode of Run round imports as a background job and record their status in the database #621);
  • a 2,000-name file list survives the round trip, and the column compiles to MEDIUMTEXT.
• The category tests, the connection tests and the warning tests fail on the old code.

Verified

• From the Toolforge bastion: the links replica answers with the tool's replica credentials (database commonswiki_p) and counts 15,678 members for the category above.
• On montage-dev on 2026-10-02, deployed as part of an integration branch with Batch entry inserts so large category imports finish within the request timeout #623, Lock the round when activating it, so a double click can't create every task twice #626 and the background-import PRs, in background-import mode (the import runs in a worker job pod):
  • Images from Wiki Loves Monuments 2026 in Russia: 15,676 files in 13 s (the links replica counts 15,678 members; 20 disqualified).
  • A file list of 1,503 names (79 KB, over the old 64 KB limit): 1,499 files in about 1 s, and a warning for each of the 3 names that do not exist.
• Still to check: the same imports in synchronous mode from the webservice, as on production, and on montage-beta.

Deploy note

• Deploy together with or after Batch entry inserts so large category imports finish within the request timeout #623 (see the top).
• Run tools/migrate_round_sources_params.sql. The deploy works either way round, but file lists over 64 KB keep failing until it has run. No other schema change.

Written with Claude (Claude Code) on behalf of @lgelauff.

🤖 Generated with Claude Code

[lgelauff]

Tested on montage-dev in synchronous import mode (the way production runs), on 2026-10-02, on an integration branch with #623, #626, #628, #630 and #631:

• Images from Wiki Loves Monuments 2026 in Russia: 15,676 files in 8.1 s.
• Images from Wiki Loves Monuments 2025 in Russia: 21,521 files in 15.0 s (26.7 s with Batch entry inserts so large category imports finish within the request timeout #623 alone).
• A file list of 1,503 names (79 KB): 1,499 files in 1.3 s, with a warning for each of the 3 names that do not exist (the 1,500 real names hold 1,499 that are unique ignoring case).

Activating the 21.5k-file round afterwards got the webservice OOMKilled. That is activation, not changed by this PR; it is fixed in #632, with which the same round activated in 4.5 s.

Written with Claude (Claude Code) on behalf of @lgelauff.

[lgelauff]

Added a commit (8508612): both wikireplica hosts are now written the same way, and both can be overridden. COMMONS_DB_HOST was the legacy alias commonswiki.labsdb, which resolves to s4.analytics.db.svc.wikimedia.cloud. The default is now commonswiki.analytics.db.svc.wikimedia.cloud (the same replica), and MONTAGE_COMMONS_DB_HOST overrides it, like MONTAGE_COMMONS_LINKS_DB_HOST. deployment.md lists both.

Tested on montage-dev in synchronous mode after this commit: Images from Wiki Loves Monuments 2026 in Russia imported 15,676 files in 13.6 s through both replicas, and the round (15,656 tasks) activated in 5.2 s with #632. Tests: 31 passed, 1 xfailed; two new tests check both defaults and both overrides.

Written with Claude (Claude Code) on behalf of @lgelauff.

[lgelauff]

Check before merging: old vs new category query on last year's production imports

I took every category import on production since 2025-10-04 (284 imports, 175 distinct categories) and ran, for each category, both the old query (master: one join on the main Commons replica) and the new one (this PR: members from the links replica, files from the main replica). Since 2026-09-08 the main replica's copy of categorylinks is no longer updated, so the old query sees category membership as of that date; files added to a category after it were left out of the comparison.

Results:

Duplicates: 0 in every category, in both queries.
Files only the new query returns that were added before 2026-09-08: 0 in every category. No subcategory members or other extra files come in.
110 or so categories gave identical results, including the largest (46,264 and 25,878 files). Where the new count is higher, the extra files were all added after 2026-09-08.
26 files only the old query returns, in 6 categories. All 26 still exist on Commons and are on the links replica, but were moved out of the category after 2026-09-08 (for example to "Images from WLM 2026 India with poor quality", or from a Haiti category to "Abidjan"). The old query still lists them only because its copy is frozen.
So no difference comes from the code. The file-detail part of the query is unchanged in this PR and was not compared; file-list imports were not part of this check.