Skip to content

fix(cron): skip a shutdown-killed recurring slot instead of pausing (RIG-8801) - #47

Merged
mattwilkinsonn merged 5 commits into
rigel-releasefrom
hermes-sync/rig-8801-skip-a-shutdown-killed-recurring-slot
Nov 10, 2026
Merged

mattwilkinsonn merged 5 commits into
rigel-releasefrom
hermes-sync/rig-8801-skip-a-shutdown-killed-recurring-slot

Conversation

@rigel-mintaka

@rigel-mintaka rigel-mintaka commented Nov 9, 2026 •

Copy link
Copy Markdown
Collaborator

What does this PR do?

A recurring cron job whose run is killed by a gateway shutdown no longer pauses forever. On mattmini, nemo-dev-oversight ended paused with "Canonical cron unknown" and missed about 100 runs silently.

Related Issue

RIG-8801 (Linear).

Type of Change

  • 🐛 Bug fix (non-breaking change that fixes an issue)

Changes Made

  • cron/jobs.py: a shutdown kill stamps shutdown_interrupted_execution with the run's execution id. New skip_unknown_occurrence books that slot as skipped, advances next_run_at, keeps the job scheduled, does not spend a repeat count, and lifts the recovery pause.
  • cron/scheduler_authority.py: an unknown receipt follows one ordered, retry-safe path. If already booked, or its slot already completed (for example by the feat(cron): keep a run cut off by a gateway restart as pending replay (RIG-8084) #45 replay), the receipt is retired. If it matches the shutdown stamp, the slot is skipped. Anything else pauses once and sends one notice via _deliver_crash_failure.
  • cron/executions.py, cron/scheduler.py: the execution id goes through the shutdown sweep and the skip is recorded in run history.

How to Test

  1. tests/cron/test_unknown_outcome_pause.py: 11 passed. The first 3 cases failed before the fix ('paused' == 'scheduled', 0 == 1 notices). Also covered: both shutdown phases, replay ordering, crashes between writes, a missing ledger row.
  2. scripts/run_tests.sh tests/cron -j 2: 1547 passed. The 2 failures are test_cron_script.py cases that already fail on rigel-release.

Checklist

Code

  • Conventional Commits
  • Only this fix
  • Tests fail before and pass after

Documentation & Housekeeping

  • cron-internals.md updated

For New Skills

N/A.

Screenshots / Logs

N/A.

Refs RIG-8801

Co-authored-by: Matt Wilkinson matt@rigel.build

rigel-mintaka and others added 5 commits November 9, 2026 04:00
…RIG-8801)

A recurring job whose run was killed by a gateway shutdown and whose outcome is unknown now skips only that slot and stays scheduled. Any pause that remains notifies through the cron failure delivery path once.

Co-authored-by: Matt Wilkinson <matt@rigel.build>
…RIG-8801)

Co-authored-by: Matt Wilkinson <matt@rigel.build>
…(RIG-8801)

Co-authored-by: Matt Wilkinson <matt@rigel.build>
…IG-8801)

Co-authored-by: Matt Wilkinson <matt@rigel.build>
…very (RIG-8801)

Co-authored-by: Matt Wilkinson <matt@rigel.build>
@linear-code

linear-code Bot commented Nov 9, 2026

Copy link
Copy Markdown

RIG-8801

@rigel-mintaka

Copy link
Copy Markdown
Collaborator Author

Review rounds (single review lens, pre-submit), final head 7a5ccaf:

Round Finding Action
1 medium: completed-execution early return moved below the owner fence Fixed: restored above the fence; test added
1 open: replay finishing before recovery could be skipped again Fixed: a completed slot retires the receipt; real-path test
1 open: crash between the jobs, ledger and journal writes Fixed and tested
1 low: tests hand-set the unknown row Fixed: uses mark_running_jobs_interrupted and give_up_execution
2 open: replay cut off a second time paused instead of retiring Fixed: already-booked execution retires; test
3 low: completed-slot lift untested; half-recorded skip Fixed and tested
4 high: staged skip_detail could leave the job paused after a failed booking Fixed: replaced by one ordered, retry-safe recovery path (no staged journal state)
5 low (unverified reachability): shutdown-stamped unknown whose ledger row is pruned AND whose slot already completed would book a skip over a completed slot Accepted: needs ledger pruning before recovery and a lost replay booking; the stamp is cleared by any later booked run

All-clear on 7a5ccaf. Tests: test_unknown_outcome_pause.py 11 passed; tests/cron 1547 passed, 2 failures shared with rigel-release.

@rigelbuild-bot
rigelbuild-bot marked this pull request as ready for review November 9, 2026 11:35
@mattwilkinsonn
mattwilkinsonn merged commit e46f94e into rigel-release Nov 10, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants