Repository navigation
Create round tasks with batched inserts, so activating a large round fits in memory - #632
Merged
Merged
Conversation
Activating a round built one Vote object per task, plus one RoundEntry object per entry, and inserted the tasks one row at a time. On montage-dev, activating a round of 21,516 tasks ran the 512 MiB webservice container out of memory (OOMKilled after 6.6 s, the round rolled back), which is very likely the production activation failure in #619. create_initial_rating_tasks and create_ranking_tasks now read only the shuffled round-entry ids and insert the tasks with executemany in chunks of 5,000 (pymysql turns each chunk into multi-row INSERTs). The assignment is unchanged. Measured locally on SQLite for 21,500 tasks: peak Python memory 219 MiB -> 12 MiB, time 15.6 s -> 1.0 s. The new test checks one INSERT for a 40-task round (master: 40), quorum distinct jurors per entry, and the same row contents (status active, empty flags, a create_date). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This was referenced Oct 2, 2026
Open
lgelauff
added a commit
that referenced
this pull request
Oct 4, 2026
Resolves the conflict with #632 (#619) in montage/tests/test_web_basic.py: both PRs added a test at the same spot; both are kept unchanged (the concurrent-activation test for #334, then the batched task insert test). rdb.py merged cleanly: activate_round takes the round lock before create_initial_tasks, which now inserts in batches. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #619 (very likely: see below).
What goes wrong
Activating a round builds one
Voteobject per task, and loads oneRoundEntryobject per entry, all held inthe request until it commits, and inserts the tasks one row at a time. For large rounds that needs more memory
than the webservice has.
On montage-dev (512 MiB container, 4 gunicorn workers), activating a round of 21,516 tasks got the container
OOMKilled after 6.6 s: the browser got a 503, the round stayed paused with no tasks (the transaction rolled
back), and nothing was logged, because the process was killed rather than raising an error. That matches #619,
an activation failure on production without a clear cause.
The fix
create_initial_rating_tasksandcreate_ranking_tasksnow read only the shuffled round-entry ids and insertthe tasks with
executemanyin chunks of 5,000; pymysql turns each chunk into multi-row INSERTs. Which jurorgets which entry is unchanged. The session is flushed before the inserts and expired after them, so nothing
loaded earlier in the request is stale.
Measured locally (SQLite, 21,500 tasks): peak Python memory 219 MiB → 12 MiB, time 15.6 s → 1.0 s.
Tests
test_activation_inserts_tasks_in_batches: a 40-task round is inserted with 1 INSERT (master: 40), eachentry gets quorum different jurors, and the rows look as before (status
active, empty flags, acreate_date). It fails on master. It works with and without the background-import PR (Background import job: imports run in a worker (backend) #630).Not in this PR
time, so it can hit the same limit on very large rounds.
Written with Claude (Claude Code) on behalf of @lgelauff.
🤖 Generated with Claude Code