Repository navigation
Faster import lookups: lookup by file_id in batches of 199 (#513) - #654
Merged
Merged
Conversation
The import looks up existing entries by name in batches of 200, twice per batch (lookup, and read-back after the multi-row insert). Two things make that slow on production (#513, #650): - Production has no index on entries.name (lost when the tables were converted to utf8mb4_unicode_ci / Compact on 2025-07-01), so every batch reads the whole 2M-row table. On a copy of production (#652) a ~6,000-file category import was killed after 31 s, nothing saved. - MariaDB's eq_range_index_dive_limit is 200: from 200 values in an IN (...) list on, the optimizer estimates from index statistics instead of counting in the index. Measured on production (read-only, 2026-10-05) for newest, oldest, random and new file_ids: 50, 150 and 199 ids always use ix_entries_file_id (1-7 ms, also with 0 hits), 200, 201 and 400 ids always read the whole table (0.56 s warm, 9.4 s cold). Change: for campaigns opened on or after 2026-06-01, when every file in the import has a file_id (wikireplica imports; CSV imports have none), add_entries finds existing rows, and reads new rows back, by file_id in batches of 199 (FILE_ID_LOOKUP_CHUNK_SIZE). No index hint is needed; FORCE INDEX is documented next to the constant as a fallback. On production the last category-imported entry without a file_id dates from 2026-05-30 13:21 and the first with one from 2026-05-31 20:52, so every file new to Montage after the cutoff has its file_id. Files not found by file_id are looked up by name only where entries.name is indexed (beta, fresh installs, production once #650's index is restored); a row found that way is reused if it has no file_id, so older files keep their row and the import does not fail on beta's unique name index. Without a name index (production today) that lookup is skipped; such files get a second row. The indexes are read from the database and cached for ten minutes. A renamed file (same file_id) keeps its existing row. Other imports are unchanged; no schema change. Tests: no name lookup after the cutoff with production's indexes, and re-imports keep row ids; name lookup before the cutoff and without a file_id; a renamed file keeps its row; a file without file_id keeps its row where name is indexed; every file_id lookup stays below 200 values. Each fails without its part of the change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Found on montage-dev: a 17,000-file import failed after 4 s with "ResourceClosedError: This Connection is closed [SQL: 'SHOW CREATE TABLE entries']" from _entries_indexes. inspect(engine) reflects through connectionless execution, on a connection that closes itself once its first result is exhausted. The app's engine_connect listener (set_mysql_session_charset_and_collation) runs SET NAMES on every new connection; that result has no rows, so it is exhausted at once and the connection closes before the reflection query runs. Reflecting on rdb_session.connection() uses the request's own connection instead (no new connection, same transaction). Reproduced and verified on MariaDB with the app's engine and listeners (old: fails with the same error; new: indexes read, session still usable). Test: an engine_connect listener that runs a statement without rows (like SET NAMES) no longer breaks _entries_indexes; fails with the old code with the same ResourceClosedError. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This was referenced Oct 5, 2026
Open
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Speeds up the import's lookup of existing entries. Related: #513, #650, #652.
entries.name(Category imports time out on production: every file-name lookup reads the whole entries table #650), so the import's lookup by name reads the whole 2M-row table, twice per batch. A ~6,000-file category import times out after 30 s (reproduced on a copy of production, Give montage-dev a production-sized database: a copy of production with the jury votes shuffled #652).eq_range_index_dive_limitis 200: with 200 or more values inIN (...), the optimizer ignores the index and reads the whole table. Measured on production, lookup byfile_id: 199 values always use the index (1–7 ms); 200 values always scan (0.56–9.4 s).Change
file_id: look existing entries up byfile_id, in batches of 199 (ix_entries_file_id). Every file new to Montage since 2026-05-31 has one.file_idare looked up by name only whereentries.nameis indexed (beta; production after Category imports time out on production: every file-name lookup reads the whole entries table #650), so older files keep their row. Without that index they get a second row.SET NAMESlistener closes connectionless connections; found on montage-dev).FORCE INDEXdocumented as fallback.Tests
6 new tests in
test_import_entries.py, each checked to fail without its part of the change. Full suite: 56 passed, 1 xfailed.On montage-dev (copy of production, production's indexes, no name index): a category import of 17,898 new files took ≤ 13 s; on
mastera 6,000-file import times out.Written with Claude (Claude Code) on behalf of @lgelauff.