feat(roadmap): implement keeps phase marks and the spend of a killed Job #4070

Problem

Facts that I checked in implement_run.py.

1. Every Job runs _pull_request, which calls _comment(pull, _NOTHING_YET). _comment builds its table from self.outcomes[since.outcomes:]. A fresh Job that continues a run has an empty outcomes list. So it posts a comment with the body - and no table. The earlier table is not lost on GitHub, but the new comment is empty.
2. The scoped checks (_scoped_checks, about 3.5 minutes) and the watch of the PR checks run again in every Job. No mark says that they passed for the same head.
3. implement.spend is written only at a stop or at ready (stop, _ready).
   • The group mark (_record(group=..., sha=...)) does not carry the spend.
   • _item_spend() sums self.outcomes of this Job only.
   • A killed Job leaves no spend. A continuation starts with a full item budget. With 2 continuations an item can spend three budgets.
   • At a stop, implement.spend is overwritten with the spend of the last Job only.
4. The wait for the PR checks runs up to 3600 s (monitor_timeout_seconds, watch_pull). The runner writes no log line during it. The stall limit is 4200 s because of this wait (defaults.py).

Expected

This issue is blocked by #3677 (checkpoint of each passed group). #3677 is open and has no pull request. Both change _groups and _start. Start this issue after #3677 merges, and read the final order of the group steps from main. Do not copy the push of #3677.

The steps below name phases, not positions in _groups. They hold for any order that #3677 sets.

• Add the keys implement.checked_sha and implement.published_sha to ImplementRecordKey (stores/store.py).
  • Write checked_sha after the scoped checks pass.
  • Write published_sha after the push.
  • Use the existing marks writes. A Job with the same head skips the phase.
• Put a marker line for the run into the PR comment. Read the comments of the PR once per Job. Update the comment that holds the marker. Do not post a comment whose body is -.
• Write the spend of each passed group in the group mark that exists (_record(group, sha, spend)). Start the Job budgets (_item_spend) from the recorded sum of the earlier Jobs. At a stop or at ready, write the sum, not the spend of the last Job only.
• Write one log line at a set interval during the watch of the PR checks (watch_pull). Use the existing DEFAULT_ROADMAP_IMPLEMENT_POLL_SECONDS loop.
• Request cost per Job: 1 read of the PR comments for the marker. Add 1 read and 1 PATCH for each of the two phase marks. This gives 5 requests. The spend rides on the group mark and adds none. This is inside the request budget (fix(service): the metrics row sends too many counted requests #3941, fix(service): the request budget counts only what GitHub counts #3946).

Update docs/ROADMAP_WORKFLOW.md and the spec roadmap-implement-job for the two new keys.

Not in this issue:

• the typed result and the hand-off of the check log (the issue feat(roadmap): implement writes a typed result and hands off its log #4071);
• the comment on a stop (fix(roadmap): comment on pull request when implement stops #3997).

Acceptance criteria

• uv run pytest tests/roadmap/implement/test_roadmap_implement_phase_marks.py -k checked_sha - a second Job at the same head runs zero scoped checks.
• uv run pytest tests/roadmap/implement/test_roadmap_implement_phase_marks.py -k pr_comment_marker - a rerun with no new outcome posts no - comment. One comment holds the table.
• uv run pytest tests/roadmap/implement/test_roadmap_implement_phase_marks.py -k spend_survives_kill - a continuation starts with the spend of the killed Job and stops at the item budget as a sum.
• uv run pytest tests/roadmap/implement/test_roadmap_implement_phase_marks.py -k watch_progress_line - the watch of the checks logs a line at the set interval.
• uv run pytest tests/roadmap/implement/test_roadmap_implement_phase_marks.py -k request_count - a Job sends at most 5 extra requests for the marker and the two phase marks.

Every command above fails on the unchanged code. The pull request cites the fail-to-pass run (devops ci fail-to-pass --base <ref>).

Feasibility

I read these files at origin/main: implement_run.py (_groups, _record, _item_spend, _spend, _push, _scoped_checks, _pull_request, _comment, _checks, _ready, stop), implement_pull.py (watch_pull), stores/store.py (ImplementRecordKey) and defaults.py. I did not run the code.

I checked that #3677 is open and that no open pull request holds a branch for it. An earlier claim said that a rerun posts a second copy of the comment table. That was wrong. The real fault is an empty comment.

Existing tool considered

Item record marks (set_marks), the branch and the ticks of tasks.md as resume state, and the checkpoint of #3677. No library is needed.

Cost

Build: about 90k tokens and 4 h (two keys, the comment marker, the spend sum, the log line, 6 tests). Keep: about 4k tokens per quarter. Fix: about 20k tokens and 1.5 h.

Related

Part of #3027. Blocked by #3677 (must merge first). #3066, #3997 (comment on a stop), #3941, #3946, #3105.