[rigel-mintaka]

What does this PR do?

A gateway restart that cuts off a cron run no longer loses that occurrence. Shutdown stores it as pending work. The normal admitted tick replays it at most once, under ESTOP, drain and the profile gate.

This is T1 of the orion-docs design nemo-cron-replay-pending.md, with Matt's Q2 ruling. A run is counted at its replay, so a cut-off final repeat still replays. A cut-off replay never replays again. T2 and the Q1 Discord read follow later.

Related Issue

RIG-8084 (Linear). Replaces the startup re-fire in #32.

Type of Change

• ✨ New feature (non-breaking change that adds functionality)

Changes Made

• cron/jobs.py, cron/occurrences.py, cron/scheduler.py and related files: the gateway shutdown sweep stamps a replay marker, in the scheduler's zone, and it moves through pending_slot, fire_claim and unreachable_retry. A malformed marker counts the run and never replays.
• Tests cover the real shutdown sweep and a non-UTC zone. cron-internals.md documents the replay.

How to Test

1. pytest tests/cron -q -k 'not TestRunJobScript': 1518 passed, 1 skipped. The sandbox has no /bin/bash.
2. ruff check on the changed files: clean. ruff format drift is already on rigel-release.

Submitted with rjc --on rigel-release, because this fork's PR base is rigel-release, not main.

Checklist

Code

• My commit messages follow Conventional Commits
• My PR contains only changes related to this feature
• I've run pytest tests/cron -q and all tests pass
• python scripts/check passes: its two failures are already on rigel-release
• I've added tests for my changes
• I've tested on my platform: Linux (NixOS)

Documentation & Housekeeping

• I've updated relevant documentation (cron-internals.md)
• I've updated cli-config.yaml.example if I added/changed config keys: N/A
• I've updated CONTRIBUTING.md or AGENTS.md if I changed architecture or workflows: N/A

For New Skills

N/A: no skill changes.

Screenshots / Logs

N/A.

Refs RIG-8084

Co-authored-by: Matt Wilkinson matt@rigel.build

[linear-code]

RIG-8084

[rigel-mintaka]

Review round 1 at 443c3bcb: not all-clear

Lenses run: correctness, design, security and style. The fixes are in progress.

High

1. Real shutdown never creates a replay. mark_running_jobs_interrupted calls mark_job_run without shutdown_interrupted=True. The run is counted and pending_slot is dropped. Probe: pending_slot=None, completed=1.
2. The stamp stores a UTC string. pending_slot_stamp(owned, ...) stores the claim instant in UTC form. For a cron job west of UTC, the timezone-shift repair moves the slot to the next day and the replay is lost. Retry rungs outside UTC lose it the same way. restore_unstarted_occurrence already says to store the scheduler's zone.

Medium

• The caller's ladder_rung is ignored, because mark_job_run works it out again from the claim. A schedule edit during a run counts a rung twice. Base code honored the argument.
• A malformed replay marker (not a dict) is read as "no replay". The run is not counted and gets a fresh replay. That breaks at-most-once. Fail closed instead.
• Test gaps:
  • Nothing calls _record_fire_ownership_lost or mark_running_jobs_interrupted, and nothing uses a non-UTC zone. That is why both highs passed.
  • The multiplex test patches tick, so the ticker never dispatches the replay.
  • The crash test stays in one process.
• The _advance_after_run ladder_rung docstring is now wrong.
• Replay lookup is written three times, with two ways of comparing instants.
• Code comments cite design case numbers and RIG/ruling references.

Low

• A manual fire while a replay is pending drops the marker silently.
• restored_replay stays set after the slot is re-anchored.
• The second-shutdown assertion is an either/or, and two tests duplicate others.
• No test checks that an ordinary run is counted exactly once.
• The comment saying a heartbeat never moves started_at is wrong for legacy claims.
• The _claim_replay wrapper adds nothing.
• _carry_replay uses a backslash line continuation.
• The restored_replay shape is not documented.
• The lock setup is repeated in two tests.
• Two sentences in the cron-internals.md replay section are unclear.

Dismissed: the clear-order and claim-read-order questions. Both were traced and are correct.

[rigel-mintaka]

Review round 1: fixes at 414232bc

Finding	Fix	Test
High: real shutdown never stamps	The shutdown sweep and the ownership-loss path record the interruption as a replay	test_shutdown_sweep_preserves_interrupted_occurrence, test_ownership_loss_records_interruption
High: UTC stamp lost west of UTC	Stamp stored in the scheduler's zone	test_non_utc_replay_survives_due_scan
Caller's ladder_rung ignored	Honored	test_midrun_schedule_edit_keeps_retry_count
Malformed marker fails open	Counts the run, never replays	test_malformed_replay_counts_without_another_replay
Manual fire drops marker	Kept	test_manual_run_preserves_pending_replay
Stale restored_replay	Cleared on re-anchor and edit	test_schedule_edit_clears_restored_replay
Ordinary run counted once	New test	test_normal_completion_counts_once_without_replay
One-shot restore untested	New test	test_restore_unstarted_one_shot_releases_claim
Second-shutdown assertion was either/or	Asserts the replay key is absent	test_second_shutdown_does_not_stamp_again
Test names claimed crash and dispatch coverage	Renamed to their real scope	test_scan_before_claim_keeps_replay, test_multiplex_gate_ticks_only_admitted_profile_with_replay
Replay lookup written three times; _claim_replay wrapper; backslash continuation	One lookup helper using scheduled_instant; wrapper removed	n/a
Flag docstring, case numbers, RIG references, started_at comment, restored_replay shape, duplicated lock setup, doc sentences	Fixed	n/a
Shutdown mocks	test_shutdown_interrupt.py mocks require and assert shutdown_interrupted=True	that file

Each high's test failed before its fix.

pytest tests/cron -q -k 'not TestRunJobScript' gives 1517 passed, 1 skipped (re-run at the final commit). ruff check on the changed files is clean.

[rigel-mintaka]

Review round 2 fix at 9b77dc2: ladder_replay now returns {} for a present but malformed rung marker, matching slot_replay and claim_replay, so a cut-off replay is counted and never replayed again. Regression test test_malformed_rung_replay_does_not_replay_again failed before the fix. Cron suite: 1518 passed, 1 skipped (TestRunJobScript deselected, no /bin/bash in the sandbox); ruff clean. I re-ran the shutdown-sweep, non-UTC (direct and retry rung), mid-run edit and malformed-marker tests by name: 8 passed.