feat(roadmap): a suspended intake row retries after a set time #4065

Problem

The FailureLedger (run.py) suspends a row after 3 failed rounds on one input digest. The suspension ends only when the digest changes (FailureLedger.suspended, _round_input).

Which lanes are affected. _lane_reads_changes is true only for a lane with a row that has a due_rule. Only the triage lane has one (row intake). The status lane and the model lane put the hour of the round into the digest (#2757). A suspended row there runs again in the next hour. Only the row intake stays suspended until an item changes. An outage that ends on its own does not end the suspension of intake.

The ledger stores the exception class name as the cause of each round (failed(), _count_failure). No alert exists on the gauge devops_cli_service_job_suspended. The gauge has no class label.

Expected

Add an optional retry_after time to the ledger record. Set it from RETRY_AFTER of the class (the issue #4060).

• A row with retry_after in the past runs one probe round (half open).
• A success clears the record.
• A failure sets a new time.
• Keep the digest rule: a new input restarts the row at once.

Store the class value as the cause of each round. Add the class to the suspension log line (MESSAGES.serve.job_suspended). Add the label class to the gauge devops_cli_service_job_suspended. Add one gauge with the seconds left.

Add the alert RoadmapServiceRowSuspended (suspended for 3 hours, warning, with the label class) to k8s/monitoring/prometheus-values.yaml. Add a case for it in the fixture tests/fixtures/prometheus_rules/roadmap_service_tests.yaml. Add a row in the alert tables of k8s/README.md.

Rollout rule for the old file. The new field is optional. _FailedRounds already accepts entries of 4 and 5 fields. Add the 6-field form. An old record loads with no retry_after and keeps the old rule. An unreadable file still raises, as today (read_state_record).

Acceptance criteria

• uv run pytest tests/roadmap/test_roadmap_ledger_retry_after.py -k retry_after - a suspended row runs after its time with no new input.
• uv run pytest tests/roadmap/test_roadmap_ledger_retry_after.py -k half_open - a failed probe round sets a new time. A success clears the record.
• uv run pytest tests/roadmap/test_roadmap_ledger_retry_after.py -k new_input - a new digest runs the row at once.
• uv run pytest tests/roadmap/test_roadmap_ledger_retry_after.py -k old_record - a record of 5 fields loads and the row keeps the old rule. It does not raise.
• uv run pytest tests/roadmap/test_roadmap_ledger_retry_after.py -k class_cause - the ledger file holds the class value, and the gauge has the label class.
• grep -q 'RoadmapServiceRowSuspended' k8s/monitoring/prometheus-values.yaml && grep -q 'RoadmapServiceRowSuspended' tests/fixtures/prometheus_rules/roadmap_service_tests.yaml && uv run pytest tests/prometheus/test_k8s_monitoring_alerting.py - the alert and its promtool case exist and pass. The fixture is tests/fixtures/prometheus_rules/roadmap_service_tests.yaml.

Every command above fails on the unchanged code. The pull request cites the fail-to-pass run (devops ci fail-to-pass --base <ref>).

Feasibility

I read these files at origin/main: run.py (FailureLedger, _FailedRounds, _round_input, _lane_reads_changes, STATUS_LANE_ROWS, TRIAGE_LANE_ROWS, MODEL_LANE_ROWS), the alert group in k8s/monitoring/prometheus-values.yaml, tests/prometheus/test_k8s_monitoring_alerting.py and the fixture. I read pybreaker 1.4.1 in the virtualenv. I did not run the Service or promtool.

A cheaper option exists. Put the hour into the digest of the triage lane too, as #2757 did for the other lanes. It retries every hour with no class and no probe. It costs about 10k tokens. I chose the class time because the stage and refine waits use the same table.

Existing tool considered

FailureLedger with a time field. pybreaker 1.4.1 is already a dependency (ai/rag/investigator.py, valkey/cache.py). I compared it and did not use it, for three reasons:

1. The state must survive a restart in the ledger file. A custom storage copies that persistence (about 10 methods).
2. The digest rule (a new input ends the suspension) is not a pybreaker idea. The breaker and the ledger would keep two states that must agree.
3. pybreaker wraps the call and raises its own CircuitBreakerError. This changes the control flow of _execute_due_rows.

Cost

Build with the ledger: about 50k tokens and 2.5 h (about 40 lines, 6 tests, one alert). Build with pybreaker and a custom file storage: about 80k tokens and 4 h. Keep: about 3k tokens per quarter. Fix: about 15k tokens and 1 h.

Related

Part of #3027. #3666 (a round that fails before its first row), #3655 (dashboard and alerts). Order: after the issues #4060 and #4061. All change run.py.