Skip to content

feat(server): keep running Claude turns alive through a server restart - #29

Closed
greaber wants to merge 1 commit into
grant/mainfrom
grant/agents-survive-restart
Closed

greaber wants to merge 1 commit into
grant/mainfrom
grant/agents-survive-restart

Conversation

@greaber

@greaber greaber commented Nov 20, 2026

Copy link
Copy Markdown
Owner

Updating T3 restarts the server, and the server killed every running Claude CLI on the way down (the process guard stops the group when the server's pipe closes, and the agent-scope sweep stops whatever is left on startup). Every mid-turn thread was cut short and then sent "Continue where you left off", which redoes work and loses the turn's real ending.

This keeps a mid-turn Claude CLI running through a graceful restart and lets the next server pick up its output.

Design

Mechanism. On Linux, each native Claude CLI now runs under a slightly different guard script, inside its systemd scope:

  • the CLI's stdout goes through a FIFO into tee -a <userdata>/agent-turns/<id>/stdout.jsonl and on to the server's pipe, so the file keeps growing after the pipe closes (SIGPIPE is ignored; tee drops the dead pipe and keeps the file);
  • the guard still stops the whole group when the server's pipe closes, unless the server first wrote detach; then it waits for the CLI to exit (polling every 2 s, at most 12 h) and stops what it left behind;
  • the file is truncated each time a prompt is sent, so it only holds the current turn.

On graceful shutdown, prepareForShutdown now returns the runs it scheduled restart continuations for. For each CLI whose active run is one of those, the server writes record.json (thread, run, native session, pid, /proc start time, scope unit, uuid of the last message the adapter handled), tells the guard detach, stalls that session's message stream, and makes the SDK's kill() a no-op. The SDK then closes stdin as usual. I checked the real CLI (2.1.329): with stdin closed and stdout gone, it finishes the turn and exits 0.

On startup, the registry (process/detachedAgentTurns.ts) loads the records. It keeps a turn whose process is still alive (same pid and start time), or whose file contains the turn's final result. It deletes everything else. The agent-scope sweep stops only the scopes it does not keep.

The existing restart continuation then runs. When the turn can be adopted, its message is a note ("T3 Code restarted while this turn was running…"). The Claude runner gets adoptRunId and, instead of spawning claude --resume, returns an adopted session. That session follows the file from the resume point through the normal adapter mapping, and sends no prompt. It first re-sends any tool calls still waiting for a result, so their results land on the right item. It ends when the process has exited and the file is fully read; then the run finalizes as usual and the scope is reaped.

The next message after that spawns claude --resume <session> exactly as today. Any other turn in the session (a user message that arrives first) stops the leftover CLI before resuming. A turn nobody adopts within 10 minutes is stopped. Stop, interrupt, archive or a disabled provider close the adopted session, which kills the process group.

Deviation from the agreed plan. The record lives in a JSON file next to the output, not in SQLite. The agent-scope startup sweep has to read it before any orchestration service exists, and the file shares the output's lifetime, so a crash cannot leave a row pointing at deleted output.

What survives:

  • a mid-turn Claude CLI in a thread whose project has "continue threads after server update" on (the default), on Linux, across a graceful stop or update of the server or service;
  • a turn that ends while the server is down: its output is replayed at startup and the run completes;
  • a second restart while an adopted turn is still running: the adopted session is detached again with an updated resume point.

What doesn't (limitations):

  • The turn shows as two runs. The original run still ends "cancelled because the server shut down" (shutdown reconciliation is unchanged). The rest of the turn (tool card, reply, completion) appears in a continuation run after the note.
  • No stdin channel after detach. Permission prompts, AskUserQuestion, MCP elicitations and SDK hook callbacks fail at once with "Stream closed". I checked this with the real CLI: the model sees the tool error and wraps up. In full-access mode this only affects questions and plan approvals.
  • T3's own MCP tools fail for the rest of that turn. MCP credentials are held in memory, so the new server rejects the old token. Tool calls made while the server is down fail as connection errors.
  • Scope of the platforms and paths covered. Not supported on macOS or Windows (liveness uses /proc), or for a script (.js) Claude binary or a custom spawner. A server crash still kills mid-turn CLIs and uses the old "continue" path.
  • If the CLI exits without finishing during the gap between the "can adopt?" check and the turn start, the continuation sends the note text as a prompt. This window is a few milliseconds.

Upgrade path. An old server never writes records, so a new server adopts nothing and behaves as before. A new server shutting down for a downgrade leaves detached CLIs running; the old server's sweep stops the whole agent slice at startup, and the old "continue" prompt runs. This is no worse than today. The record has version: 1; an unreadable record is deleted, and its scope is stopped.

Failure modes:

  • Agent finishes while the server is down: adopted from the file; the run completes (tested end to end).
  • Agent waits on a question: the question fails fast (see above) and the turn continues.
  • Output file growth: truncated at every prompt; deleted when the CLI exits while owned and when an adopted turn finishes; leftovers are deleted on the next start.
  • Crash loops: a crash never detaches anything. The adoption claim deletes record.json first, so a crash while adopting cannot adopt the same output twice; the orphaned CLI is then stopped by the scope sweep. The existing three-continuations-per-chain limit still applies.

Codex (codex app-server), assessed, not prototyped. One app-server is shared by many threads (shared scope), and Codex rejects a second initialize on it. So surviving a restart means keeping the whole shared app-server alive behind a small relay that owns its stdio, buffers output to a file, and accepts a reconnect over a Unix socket. The new server would then need to re-attach without initialize and re-subscribe loaded threads. That needs Codex protocol support I have not verified (whether it accepts a reconnect without initialize, and what thread/resume does on a loaded thread). Per-turn finalization could instead come from Codex's own rollout JSONL. I estimate this as larger than the Claude work, so it is left as a plan.

Checks

The test, typecheck and check:changed rows ran on the working tree that became ac0de82e55. The only later edit was renaming one function's span (Effect.fnUntraced → Effect.fn); after it I re-ran typecheck, lint and the four directly affected tests (all pass).

Check Result
vp test run on the 11 touched test files (ClaudeAdapterV2, ClaudeAgentSdkQueryRunner, ClaudeDetachedOutput, detachedAgentTurns, agentScope, RestartContinuation, serverRuntimeStartup ×2, ProviderRuntimeRecoveryService ×2, FoundationPersistence) 535 passed
vp run typecheck (apps/server) pass
vp run check:changed greaber/grant/main format, lint, knip, graphql, typecheck, tests: all pass
Real CLI, stdin and stdout closed mid-tool-call (claude -p --input-format stream-json …, haiku) finished the turn (20 s sleep) and exited 0; the transcript shows the final reply
Real CLI, AskUserQuestion after stdin closed tool error "Stream closed"; the agent wrapped up; exit 0
E2E, isolated server (t3 serve --base-dir /tmp/…/base --port 38917): t3 thread start --provider claudeAgent --model claude-haiku-5-5 with a 45 s Python wait as the tool call; SIGTERM to the server 3 s into the tool call; t3 serve again record written; CLI, guard and tee alive in their scope after the old server exited; the new server adopted the turn; the thread completed with FINISHED survived-restart-ok in run 2, plus a completed command_execution card and the restart note; the output directory and scope were cleaned up
E2E, same setup, server kept down for 90 s (the CLI finished while no server ran) the CLI exited during the gap; on start the turn was adopted from the file and completed within 4 s with the same reply
E2E, next message after an adopted turn (t3 thread send --wait … "What exact output did the command print…") resumed the same Claude session (--resume), answered survived-restart-ok

🤖 Generated with Claude Code

A server restart (for example an update) used to kill every running Claude
CLI, so each mid-turn thread was cut short and later told to "continue".

Now, on graceful shutdown, a Claude CLI that is mid-turn on a run the server
will continue is detached instead of killed: its guard waits for the turn to
end, tee keeps writing its stdout to a per-turn file under userdata, and its
systemd scope survives the startup sweep. The next server's restart
continuation adopts it: it replays the file from the last message the old
server handled through the normal adapter mapping, without sending a prompt,
and the run finishes as usual. Any other turn in that session stops the
leftover CLI first; an unclaimed one is stopped after ten minutes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions github-actions Bot added vouch:trusted PR author is trusted by repo permissions or the VOUCHED list. size:XL labels Nov 20, 2026
@greaber

greaber commented Nov 21, 2026

Copy link
Copy Markdown
Owner Author

Closing as a completed prototype, not a change to ship. The end-to-end test proved a running command and its output can survive a graceful Linux server restart. But this approach closes the input channel: the surviving turn cannot deliver questions or approval requests, and it loses access to T3 tools. A replacement should demonstrate a post-restart question, answer, and T3 tool call in the same turn before replacing the existing continuation behavior.

@greaber greaber closed this Nov 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size:XL vouch:trusted PR author is trusted by repo permissions or the VOUCHED list.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant