Skip to content

A recovered agent step reuses the sandbox without fencing the abandoned call #636

Description

@czpython

Problem

When the web process restarts mid-agent-call — a deploy, a crash — DBOS recovery re-executes the workflow, the in-flight step has no checkpoint, and it re-runs with a fresh call id; the old call is marked abandoned. But the abandoned call's process is still alive: the sandbox VM does not care that the supervisor restarted, and the harness keeps working inside it.

With steps_reuse_sandbox, the recovered step then starts in that same sandbox. The fresh agent walks into a working tree with the abandoned sibling's uncommitted edits and a live harness process. Observed end to end after a deploy landed two minutes into an implement:

The refusal was the good outcome. The bad outcomes are live: the abandoned harness runs unsupervised with push credentials until its lease expires (an untracked push to the PR branch is possible), and a less careful agent would have raced it.

Proposal

Fence the abandoned call before the recovered step touches a workspace, any of:

  1. Kill the abandoned call's process — its pid is already recorded in the run directory — and reset the checkout to the last pushed state before starting the fresh call.
  2. Lease a fresh sandbox for a recovered step and release the old one, paying one provision for a clean room; the janitor reaps the orphan.

Option 1 keeps sandbox reuse; option 2 is simpler and cannot miss a stray child process. Either way, an abandoned call must not keep credentials and a working tree it shares with its replacement.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions