Check first-round imports before importing; save a first round with its files (#510, #509, #447)

[lgelauff]

Closes #510, closes #509, closes #447, closes #517, closes #146. Fixes the link part of #208. Prevents #645 within one import, and against files already in Montage where entries.name has a unique index. Related: #513, #635, #645, #650, #654.

What changes

First-round imports become two steps: check, then save the round together with exactly what was checked.

Check (POST /v1/admin/campaign/<id>/import/check, campaign coordinators)

• Montage fetches the source: a CSV, gist or Google Sheet link, a File List, or a category.
• The File List box takes file names, file IDs or both, one per line. A line of digits only is a file ID (a Commons file name always has an extension). If the first line is a header with filename, img_name or file_id, the box is read as a CSV, so the check's own download can be pasted back in.
• It looks every row up on Commons, by file_id if the row has one, otherwise by name. The lookup by file_id is a new wikireplica query, file.file_id IN (...), 500 per query, with the same joins as the name lookup.
• Only the name column (filename, or else img_name; header names are case-insensitive and a BOM is ignored) and an optional file_id column are read. A CSV may have only a file_id column. Commons supplies all metadata (Entry metadata should be identical regardless of import method #509).
• Names are made canonical the way MediaWiki does for File: titles:
  • a File:/Image: prefix is removed, in any case;
  • runs of spaces and underscores become one underscore;
  • the first letter is upper-cased.

Row results:

Row	Status	Result
file_id known, same name	ok	imported
file_id known, other name	renamed	imported under Commons' current name if the file is new to Montage; shown as information
a bare number that is a file_id	by_file_id	imported; shown as "Found by file ID" with the file's name, so a wrong number is seen before saving
same Commons file twice	duplicate	imported once (#146)
name known (no file_id)	ok	imported
name unknown (no file_id)	unknown_name	left out, warning
file_id unknown, or not a plain whole number	unknown_file_id, malformed_file_id	blocks
a bare number that is no file_id but the page ID or a revision ID of a file page	page_id, revision_id	blocks; the message names the file and says to use its file ID or name
two different files in this import whose names the database treats as equal	same_name	blocks CSV, gist, Sheet and File List imports (the names shown side by side); in a category, the first file is kept and the others are left out with a warning
a new file whose name the database treats as equal to a different file already in Montage	same_name_existing	the same as same_name. Only where entries.name has a unique index (beta, installs built from the model); production has none (#650), so the lookup is skipped there

A file ID is the number of the file itself (the file_id column), not the page ID shown under Page information. Almost every file_id is also the page ID of some other file page, so a bare number is always taken as a file_id. Page and revision IDs are looked up only for numbers that are no file_id, to explain the error.

The same-name rule is the database's own:

• On MariaDB, the check asks the server for WEIGHT_STRING(CONVERT(name USING utf8mb4) COLLATE utf8mb4_unicode_ci), batched and without reading any table.
• In tests and local development (SQLite), a clearly marked approximation is used. Anywhere else the check refuses rather than approximate.
• One switch: import_check.SAME_NAME_CHECK.

Translated reasons: every checked row and every source error carries reason_code and reason_params. The form shows them through i18n keys (montage-round-check-reason-*, montage-round-check-error-*) and falls back to the English server text (marked lang="en") only when a key is missing. The dialog after saving is built from the check's rows, translated.

Check file (MONTAGE_IMPORT_CHECK_PATH)

• The check result is saved as a JSON file under a random token, mode 0600.
• It holds the campaign id, the source, each row as written, and Commons' data per file, including Commons uploader names. It holds no Montage users.
• A check can be used to save a round for 1 hour (files are renamed and deleted on Commons all the time). The file itself is deleted after at most 7 days: every check, import, download and app start deletes older files, and at most 5,000 are kept. The downloads work until then.
• The folder must be Montage's own. Montage refuses to start if the variable is missing outside local development, or if the path is relative, not writable, or holds other files.

Downloads

• The upload file has filename (Commons' current name) and file_id, one row per file the import would take. Uploading it again, or pasting it into File List, passes the check unchanged. When the check blocks, the link says the file holds only the rows without problems, and the steps follow as text.
• A separate issues report lists every row that is not ok.
• Cells that a spreadsheet would run as a formula get a leading ', which Montage removes again when reading.

Saving a first round (POST /v1/admin/campaign/<id>/add_round with import: {import_method, check_token}, #447)

• The round is created and its checked files imported in one transaction. The import reads the check file only: no fetch, no Commons lookup.
• If the check blocks, is older than 1 hour, belongs to another campaign or source, or the import brings no files (import_empty), nothing is saved (no round). The coordinator fixes the source (or checks it again) and saves again.
• POST /v1/admin/round/<id>/import also needs check_token (for imports into an existing round). Without one it returns import_check_required.
• Advancing a round (import_method: round) is unchanged.

Matching existing entries: by file_id in every campaign. #654's 2026-06-01 cutoff is removed, because every checked file has a file_id. Entry.file_id is declared indexed (production already has ix_entries_file_id).

Round form

• A Check button and a result panel:
  • counts, and a verdict: blocked, or "You can save the round" (also when only warnings were found);
  • rows with problems, marked in text, not by colour only, with their row numbers in the text box;
  • same-name groups;
  • the upload-file and issues-report downloads.
• Save is enabled only after a check that does not block and finds at least one file. It is disabled while a check runs. Changing the source clears the check.
• While saving, the source fields and Cancel are disabled.
• Errors are shown next to the check or next to Save, by type. An expired check says that nothing was saved and why. A save that got no answer reloads the campaign and says the round may exist.
• Screen readers: the result panel is a named region; check results and errors are announced through a live region outside the busy container; focus moves to the result or the save error; Save is described by the reason it is disabled.
• Phones: buttons wrap instead of cutting labels off; the problems table scrolls sideways; file names are in <bdi> for right-to-left names.
• Texts follow one glossary (problems, file ID, "check the source again", Commons, group). Labels match the help: "File names or file IDs (one per line)", "Link to the sheet or file list".

Source links

• https only, and every address of the host must be public; each redirect (at most 3) is checked the same way.
• At most 20 MB and 100,000 rows.
• The gist bug is fixed: a gist is looked up on the instance's own replica (it used to go through production's public utils endpoint).
• Only gist.github.com links are rewritten to /raw; other links are fetched as given (Montage File List URL image load interface is broken #208).
• An empty import no longer warns "all entries disqualified" (Montage File List URL image load interface is broken #208).

Round sources store a fingerprint of the check (check_id), not the token; a new check of the same source reuses the round source.

Deploy: deploy.sh stops before building if a required envvar is missing. deployment.md has the folder rules, troubleshooting and rollback notes, and docs/api.md the new endpoints.

Behaviour changes

• Coordinators press Check before Save on a first round.
• A first round is saved together with its files. A failed import leaves no empty round.
• Every CSV, gist and Google Sheet import needs the wikireplica; if it is unavailable, the check fails (no round created).
• Metadata columns in a CSV have no effect. Files no longer on Commons are left out, or block if their row has a file_id.
• A bare number in a File List is a file ID. A page ID or revision ID blocks, with a message naming the file.
• Sheets that show file_id in a number format (1.49673E+08, 149,673,020) block until the column holds plain whole numbers.
• Source links must be https.
• Two files whose names differ only in an emoji count as the same name (the database cannot tell them apart either): blocked in a list, first kept in a category.

Choices made in this PR (easy to change)

• First round creation without images succeeds with zero entries, blocking subsequent round creation #447 is in this PR, because its atomic save replaces Support file_id column in CSV imports as alternative to filename lookup #510's interim "round created, retry the import" path.
• There is no host allowlist for source links, because it would break plain links (Montage File List URL image load interface is broken #208). Instead, only public addresses are fetched.
• The caps: 20 MB and 100,000 rows (categories not capped), 5,000 check files.
• The clash with names already in Montage is checked only where entries.name has a unique index. Without the index both rows can be stored, and the lookup would be a full table scan.

Open questions (not changed here)

1. Refuse checked imports into rounds other than the first?
2. add_round without import still creates an empty round, for API and script use only (the form always sends import). Require import when the campaign has no rounds yet?
3. A first round whose files are all disqualified by its settings is saved: refuse it?
4. Read the collation of entries.name from the database instead of assuming utf8mb4_unicode_ci?
5. A kill switch (envvar) for the same-name check?

Known limits

• Production has no index on entries.name (Category imports time out on production: every file-name lookup reads the whole entries table #650). A file stored before 2026-05-31 (no file_id) gets a second entries row when imported again. This is accepted until the backfill (Backfill file_id for entries imported before #505 migration #513). /v1/entry/<name> fails for names stored twice.
• A clash with an existing name is caught by the check. A file stored by another import between the check and the save can still clash at save time (a 500 on an instance with the unique name index).
• A renamed file that Montage already has keeps its stored row and old name.
• Commons can probably give a new upload the file_id of a deleted file with the same name (the file table keeps one row per name; a deleted file keeps its row, marked file_deleted) [not verified]. A stored file_id would then point to a different image after a delete and re-upload under the same name.
• Two imports into one round at the same moment: see Two simultaneous imports into one round add every file twice #635, which merges with this PR after resolving three simple conflicts (keep both sides). The merged result passed 144 tests, 2 xfailed (before the 1-hour change).
• In local dev (MONTAGE_ENV=dev, lookups through production's public utils endpoint) there is no lookup by file_id. Rows with a file_id are compared by name, and renames are not detected.
• The background import worker (Run round imports as a background job and record their status in the database #621/Background import job: imports run in a worker (backend) #630) has to adapt (import by token) when it lands.

Tests

• Backend: pytest montage/tests/ gives 158 passed, 3 xfailed (master: 56 passed, 1 xfailed). New or extended: test_import_check.py, test_loaders.py, test_import_entries.py. A pr-check panel review (2026-10-06) led to fixes, each with a failing test first: 24 of the 28 tests added for them fail without the fix; the other 4 guard behaviour that was already correct. Later tests cover reason codes per row and per source error, the clash with existing names (app level, the database comparison, and the skip without a unique index), bare file IDs, and a live-replica test of the page/revision ID query (xfail off Toolforge).
• Frontend: prettier --check src/, eslint src/ and npm run build pass. The Cypress spec asserts the save request, a double click saving once, and a refused import keeping the form open. It has not been run (not in CI).
• Same-name rule on MariaDB, confirmed on ToolsDB (10.6.22-MariaDB, 2026-10-06, read-only query): under utf8mb4_unicode_ci, е/ё, upper/lower case, accents and ß/ss each compare equal, and so do names that differ only in an emoji (this collation gives every emoji the same weight). Different names stay different.
• On montage-dev (2026-10-07, scripts in a buildservice shell, rolled back): a public https CSV (5 rows, 0.2 s); http and 127.0.0.1 refused; the WLM 2026 Russia category (15,678 files) checked in 6.5–7.4 s; renamed, unknown and malformed file IDs, unknown names and duplicates classified as above; a blocked check refused at save before anything was written; a check older than an hour, or from another campaign, refused. A 307-line File List imported without delay.
• On montage-beta (2026-10-07..10): a save of the WLM 2026 Russia category hit IntegrityError 1062 on a name equal to an existing row's (…71А vs …71а); that led to the same_name_existing check. A File List of file IDs was blocked as "ambiguous" before dcc90b1; after it, a quick test worked.
• Not verified: the form in a browser by script (the Cypress spec needs Xvfb, which is not installed on the test machine); it is tested by hand on staging (checks below).

After deploy: checks on montage-beta

1. A Google Sheet with filename,file_id, one id changed: Check shows the row as blocking and Save is disabled. Fix it, check again, Save: the round exists with its files, and the entries carry the ids.
2. A renamed file (old name + its file_id) shows "Renamed on Commons"; a file new to Montage is stored under its current name.
3. Two different files whose names differ only in accents or case break category imports (unicode_ci unique index on entries.name) #645: a File List with Церковь_Богоявления,_Вышний_Волочек.jpg and Церковь_Богоявления,_Вышний_Волочёк.jpg → both rows block, side by side. Then a category containing both: the first is imported, the others left out with a warning, no 500.
4. Existing name: a File List with a file whose name differs only in case from a different file already in the campaign's database → blocks with "Same name as a file already in Montage"; in a category it is left out with a warning.
5. File IDs: a File List of bare file IDs → each row "Found by file ID" with its name. A number that is no file ID but the page ID of a file page → blocks and names the file.
6. Pasted CSV: download the upload file, paste it into File List, check → all rows OK. The issues report lists the rest.
7. Montage File List URL image load interface is broken #208: a plain https text or CSV link that is not a gist loads; an http link is refused.
8. Inconsistency: load_partial_csv fallback permanently fails during Google Sheet imports #517: a Google Sheet with a filename column and other columns imports by name.
9. First round creation without images succeeds with zero entries, blocking subsequent round creation #447: a source whose files are all missing from Commons → Save disabled with a message. A save refused by the server leaves no round in the campaign.
10. A category of a few thousand files: time the check and the save (each must stay under 30 s).
11. With a screen reader: press Check and hear the summary; with Save disabled, hear why.
12. Double-click Save on a clean check: one round is created, with its files, and the form closes.
13. Check a source, wait more than an hour, then Save: the form says the check has expired and nothing was saved, no round is created, and Check plus Save then works.
14. Close the import-warning dialog with the X: the form closes and the new round appears in the campaign.
15. In a non-English interface language, the reasons in the result panel are translated (or marked English where TranslateWiki has no translation yet).

Deploying to production

MONTAGE_IMPORT_CHECK_PATH must exist on the montage tool before this runs there: Montage refuses to start without it (production went down on 2026-10-09 when this branch was deployed by mistake without it). Steps are in deployment.md.

Written with Claude (Claude Code) on behalf of @lgelauff.