Repository navigation
Conversation
A server restart (for example an update) used to kill every running Claude CLI, so each mid-turn thread was cut short and later told to "continue". Now, on graceful shutdown, a Claude CLI that is mid-turn on a run the server will continue is detached instead of killed: its guard waits for the turn to end, tee keeps writing its stdout to a per-turn file under userdata, and its systemd scope survives the startup sweep. The next server's restart continuation adopts it: it replays the file from the last message the old server handled through the normal adapter mapping, without sending a prompt, and the run finishes as usual. Any other turn in that session stops the leftover CLI first; an unclaimed one is stopped after ten minutes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Owner
Author
|
Closing as a completed prototype, not a change to ship. The end-to-end test proved a running command and its output can survive a graceful Linux server restart. But this approach closes the input channel: the surviving turn cannot deliver questions or approval requests, and it loses access to T3 tools. A replacement should demonstrate a post-restart question, answer, and T3 tool call in the same turn before replacing the existing continuation behavior. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Updating T3 restarts the server, and the server killed every running Claude CLI on the way down (the process guard stops the group when the server's pipe closes, and the agent-scope sweep stops whatever is left on startup). Every mid-turn thread was cut short and then sent "Continue where you left off", which redoes work and loses the turn's real ending.
This keeps a mid-turn Claude CLI running through a graceful restart and lets the next server pick up its output.
Design
Mechanism. On Linux, each native Claude CLI now runs under a slightly different guard script, inside its systemd scope:
tee -a <userdata>/agent-turns/<id>/stdout.jsonland on to the server's pipe, so the file keeps growing after the pipe closes (SIGPIPEis ignored; tee drops the dead pipe and keeps the file);detach; then it waits for the CLI to exit (polling every 2 s, at most 12 h) and stops what it left behind;On graceful shutdown,
prepareForShutdownnow returns the runs it scheduled restart continuations for. For each CLI whose active run is one of those, the server writesrecord.json(thread, run, native session, pid,/procstart time, scope unit, uuid of the last message the adapter handled), tells the guarddetach, stalls that session's message stream, and makes the SDK'skill()a no-op. The SDK then closes stdin as usual. I checked the real CLI (2.1.329): with stdin closed and stdout gone, it finishes the turn and exits 0.On startup, the registry (
process/detachedAgentTurns.ts) loads the records. It keeps a turn whose process is still alive (same pid and start time), or whose file contains the turn's finalresult. It deletes everything else. The agent-scope sweep stops only the scopes it does not keep.The existing restart continuation then runs. When the turn can be adopted, its message is a note ("T3 Code restarted while this turn was running…"). The Claude runner gets
adoptRunIdand, instead of spawningclaude --resume, returns an adopted session. That session follows the file from the resume point through the normal adapter mapping, and sends no prompt. It first re-sends any tool calls still waiting for a result, so their results land on the right item. It ends when the process has exited and the file is fully read; then the run finalizes as usual and the scope is reaped.The next message after that spawns
claude --resume <session>exactly as today. Any other turn in the session (a user message that arrives first) stops the leftover CLI before resuming. A turn nobody adopts within 10 minutes is stopped. Stop, interrupt, archive or a disabled provider close the adopted session, which kills the process group.Deviation from the agreed plan. The record lives in a JSON file next to the output, not in SQLite. The agent-scope startup sweep has to read it before any orchestration service exists, and the file shares the output's lifetime, so a crash cannot leave a row pointing at deleted output.
What survives:
What doesn't (limitations):
AskUserQuestion, MCP elicitations and SDK hook callbacks fail at once with "Stream closed". I checked this with the real CLI: the model sees the tool error and wraps up. In full-access mode this only affects questions and plan approvals./proc), or for a script (.js) Claude binary or a custom spawner. A server crash still kills mid-turn CLIs and uses the old "continue" path.Upgrade path. An old server never writes records, so a new server adopts nothing and behaves as before. A new server shutting down for a downgrade leaves detached CLIs running; the old server's sweep stops the whole agent slice at startup, and the old "continue" prompt runs. This is no worse than today. The record has
version: 1; an unreadable record is deleted, and its scope is stopped.Failure modes:
record.jsonfirst, so a crash while adopting cannot adopt the same output twice; the orphaned CLI is then stopped by the scope sweep. The existing three-continuations-per-chain limit still applies.Codex (
codex app-server), assessed, not prototyped. One app-server is shared by many threads (shared scope), and Codex rejects a secondinitializeon it. So surviving a restart means keeping the whole shared app-server alive behind a small relay that owns its stdio, buffers output to a file, and accepts a reconnect over a Unix socket. The new server would then need to re-attach withoutinitializeand re-subscribe loaded threads. That needs Codex protocol support I have not verified (whether it accepts a reconnect withoutinitialize, and whatthread/resumedoes on a loaded thread). Per-turn finalization could instead come from Codex's own rollout JSONL. I estimate this as larger than the Claude work, so it is left as a plan.Checks
The test, typecheck and
check:changedrows ran on the working tree that becameac0de82e55. The only later edit was renaming one function's span (Effect.fnUntraced→Effect.fn); after it I re-ran typecheck, lint and the four directly affected tests (all pass).vp test runon the 11 touched test files (ClaudeAdapterV2, ClaudeAgentSdkQueryRunner, ClaudeDetachedOutput, detachedAgentTurns, agentScope, RestartContinuation, serverRuntimeStartup ×2, ProviderRuntimeRecoveryService ×2, FoundationPersistence)vp run typecheck(apps/server)vp run check:changed greaber/grant/mainclaude -p --input-format stream-json …, haiku)AskUserQuestionafter stdin closedt3 serve --base-dir /tmp/…/base --port 38917):t3 thread start --provider claudeAgent --model claude-haiku-5-5with a 45 s Python wait as the tool call; SIGTERM to the server 3 s into the tool call;t3 serveagainFINISHED survived-restart-okin run 2, plus a completedcommand_executioncard and the restart note; the output directory and scope were cleaned upt3 thread send --wait … "What exact output did the command print…")--resume), answeredsurvived-restart-ok🤖 Generated with Claude Code