mutants: split each night between new code and a durable backlog - #919
Merged
Merged
Conversation
The nightly job spent its whole budget on new mutants, and the backlog cursor lived only in a 14-day artifact. The run now starts at 00:47 UTC and keeps going for 8 hours as two jobs, because a hosted runner dies at 6. New batches stop once half of the first job's budget has elapsed when both queues still have work. The rest of the night walks the backlog, and the second job does not open another new window. The cursor and MISSED lines are pushed to the mutants-state branch.
A saved new_skip past the current new-mutant list dropped those names and then never moved new_base, so later diffs stayed hidden behind the old skip. Split now closes that phase the way advance() did when new_skip >= n_new, and a backlog step does the same if it is still set.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
47 0 * * *) and runs for 8 hours as two 4-hour jobs. A GitHub-hosted job dies at 6 hours, so one job cannot hold the whole budget. Each job's timeout is 270 minutes (30 minutes of slack, under the 6-hour cap). The per-mutant timeout stays 20 minutes.MISSEDstill does not fail the run, and mutants stay off the required PR checks.MISSEDlines are pushed to themutants-statebranch after each job. The next run resumes from that branch even when the 14-day artifact is gone. A name that is still in the regenerated list is the resume point; a missing name uses the saved index modulo the current backlog length. The branch wins when it disagrees with the artifact. A failed push fails the workflow. Two nights cannot overwrite each other:cancel-in-progressis false.Test plan
./scripts/mutants-queue.test.sh(fake elapsed clock, name and index resume, cron, 8-hour budgets, state-branch publish to a local remote, artifact fallback)MUTANTS_STATE_DRY_RUN=1 ./scripts/publish-mutants-state.shbash -non the nightly script, the publisher, and the restore helperqc(runs the queue self-test),test,windows,macos,coverage