Skip to content

Checkpoint runner recovery, save task work and bound worker lifetimes - #692

Merged
n0rpan merged 4 commits into
mainfrom
agent/333-work-state-main
Nov 16, 2026
Merged

n0rpan merged 4 commits into
mainfrom
agent/333-work-state-main

Conversation

@n0rpan

@n0rpan n0rpan commented Nov 16, 2026 •

Copy link
Copy Markdown
Collaborator

Tasks currently retain a branch but cannot reliably recover which worker/check/save action finished. This main-based replacement for #387 adds durable checkpoints, normal sandboxed saves, read-only status, and the decided task deadline (#333; #629 prerequisite). Tier T3: subprocess lock ownership and sandboxed Git callbacks.

Each transition has a repository-wide revision. A durable pending observation is replayed before any later allocation, with one bounded counter and a public CheckpointObserver hook; append failures stop the batch. Session IDs are saved immediately, repairs continue that session, and recovery uses the saved base, checks, offline policy and deadline. Unsupported formats are preserved and refused, following #331's current-format-only rule.

The runner saves successful, red, blocked and interrupted edits through ordinary Git hooks inside the existing sandbox. Read-only status never acquires a task lock, including when an orphaned worker/supervisor still owns the inherited descriptor. Task max_time defaults to 45 minutes; the child receives its remaining allowance through --max-time, with TERM at +60 seconds and KILL 10 seconds later. Interrupted work is kept.

The CLI observer is explicitly a no-op until the separately owned L1 store (#684) is integrated. This PR does not claim events/situation/standing commands are connected. Saved head/tree IDs describe saved work; commit-bound acceptance and object-integrity verification remain blocking prerequisites for publication in #397.

Validation:

  • Exact-head ready run 43864226792 on 11dc511 is GREEN: all 3,521 inventoried workspace tests passed, zero runtime skips; both mandatory-bwrap shards, report guard, strict Clippy, build, formatting, fuzz controls, scans and links pass.
  • The recovery-planning P2 is fixed and answered. New CLI regressions cover a deleted base, invalid new done settings, cached success and explicit new attempts. All 17 runner core tests also pass locally under real bwrap.
  • The real CLI suite covers parent death during a blocked commit hook, inherited save/worker locks, nonlocking status, same-session recovery, sticky offline policy, shared deadlines and saved head/tree IDs.
  • Failing-first proofs cover advancement after append failure, missing checkpoint/counter fields, a status flock syscall, a slow TERM sender postponing kill, and deleted-base recovery.
  • Preliminary same-path x86 ci footprint on earlier head 2d24e08: stripped binary +206,336 bytes; active runner median 5,976 KiB / 3 threads; no persistent idle coordinator. A current-head footprint rerun is blocked by the local workspace refusing fork/thread creation; the measurement is labeled with its actual commit.

See oxa/docs/work.md and oxa/docs/work-state.md for the recovery and observer contracts.

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-16T12:13:29.653153Z 11dc511 Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 43321ee9f5

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread oxa/oxa-cli/src/work_cmd.rs Resolved
@n0rpan
n0rpan marked this pull request as draft November 16, 2026 12:03
@n0rpan
n0rpan marked this pull request as ready for review November 16, 2026 12:10
@n0rpan

n0rpan commented Nov 16, 2026

Copy link
Copy Markdown
Collaborator Author

gatekeeper: Independent read of 11dc511, including the complete earlier 43321ee diff and this heads recovery-planning correction.

Findings: no P1–P3 findings established in this checkpoint/save/deadline scope.

Checked all changed code/tests/docs and surrounding task planning, trusted executable selection, inherited lock, sandbox supervisor and worker contracts. Repository revision reservation precedes task publication; observation/acknowledgement failures poison the writer and stop advancement, and pending observations recover before another allocation. Saved execution inputs precede mutable base/settings lookup, with a second revision check under the task lock. Worker-session capture is durable before continuation; worker completion and acceptance entry have separate revisions. Saves use ordinary sandboxed Git/hooks, preserve unresolved work and inspect cleanliness afterward. Status does not acquire flock, and inherited-descriptor ownership is considered after parent death. One saved deadline covers workers/repair/recovery; TERM sending cannot delay the final KILL. Tests exercise real CLI/bwrap, crash during commit, lock contention, frozen inputs, failed checkpoints and deadline work retention. This is a sizable addition, but remains sequential with bounded individual checkpoints and one bounded repository cursor rather than a daemon or per-task replay table.

The observer is deliberately still a no-op; commit-bound publication evidence and object integrity are separately documented prerequisites, not claims of this PR. Current-head gate is green; author records the local process-limit limitation and preliminary footprint. I ran no builds and read no credential files.

Touches oxa-work and sandboxed Git control: the orchestrator must still perform its own independent read after this comment.

@n0rpan

n0rpan commented Nov 16, 2026

Copy link
Copy Markdown
Collaborator Author

Independent security read at 11dc511: MERGE-OK, approved for the queue. No blocking finding. Verified: one child start for first run, resume and repair, always with the runner deciding trust and OXA_TRUST_PROJECT and GIT_* stripped; every host git call through the hardened path; recovery takes base and commands from saved state, never from current refs; state files 0600 in the 0700 storage outside the binds with temp, fsync, rename; a worker cannot reach or forge them; saving runs entirely in the sandbox; the deadline is wall-clock, persisted and never yields done; an append failure aborts before checks or saving.

Carried forward (write these into the next slices):

  1. Before publication (blocking there): 'done' is still decided on the dirty tree before the commit it is attached to (execute.rs:172-181, then lib.rs:606, work_checks.rs:81-98): a gitignored helper can make the frozen command pass and is not committed; save first and run acceptance on a clean export of head_sha, or re-run checks after the save and refuse ignored files. A cached done must re-check that the branch still equals head_sha. The shared objects and refs directories are still worker-writable and nothing verifies the base before checkout.
  2. Before unattended use: the lifetime bound signals one pid (worker_wait.rs:56-67) and then joins the pipe readers until EOF (worker.rs:146-147): a descendant that inherited stderr (for example a user-level MCP server) keeps the runner hanging past the limit while it holds the task lock. Start the worker in its own process group, signal the group, give the reader joins a deadline.
  3. Low: an existing task branch is adopted (tree.rs:95-113, already on main): record the branch sha at plan time or refuse a branch the runner did not create. One unreadable checkpoint fails the whole status listing (status.rs:37).

🤖 Generated with Claude Code

@n0rpan
n0rpan added this pull request to the merge queue Nov 16, 2026
Merged via the queue into main with commit 0628a52 Nov 16, 2026
47 checks passed
@n0rpan
n0rpan deleted the agent/333-work-state-main branch November 16, 2026 12:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant