Repository navigation
Description
Before submitting
- I searched existing issues and did not find a duplicate.
- I included enough detail to reproduce or investigate the problem.
Area
apps/server
Steps to reproduce
Self-hosted server on Linux, web client, Codex provider. This needs #26188 (b88da91), so that the server recovers from a dead app-server at all.
- In a Codex thread, ask the agent to spawn a native subagent that runs
sleep 120, and to end its own turn without waiting for it. - Wait until the root replies and the subagent row still shows
Running. The composer shows "Waiting on subagent …". - Send
SIGKILLto the sharedcodex app-serverprocess only (thecodex app-serverPID in thet3code-codex-*.scopecgroup). Leave the T3 server running. - Wait 15 s.
A second case with the same cause: in step 1, ask for a foreground sleep 90 in a shell instead, and kill the app-server while it runs. The turn ends as failed, but the command item still shows Running: /bin/bash -lc 'sleep 90'.
Expected behavior
When the provider process dies, the subagent and its child work end as failed (or stopped), and the composer stops waiting on them. The command item of the killed turn ends too.
Actual behavior
The subagent row stays Running, and the composer keeps waiting on it. Nothing changes until the user presses Stop or the server restarts. The command item of the killed turn also stays Running.
From reading the source on main (14f295e):
- The adapter fails its event stream (fix(server): Codex works again after its app-server is killed #26188), and
RunExecutionService's ingestion-failure handler runs. That handler returns early when the root run is already finalized (if (yield* Ref.get(rootRunFinalized)) return;, apps/server/src/orchestration-v2/RunExecutionService.ts:1829). So nothing ends the run-owned subagents that outlived the root. The same happens after a failed root withchildTurnsContinue: true. - On the root-active path, failure finalization cascades through subagent records and child items, but not through ordinary root
command_executionitems. That is why thesleep 90item stays running. - The user's Stop clears both:
thread.stop→dispatchRunInterrupt→settleBackgroundWork(Orchestrator.ts:12226), which ends pending items and native subagents whose provider session is no longer live (hasLiveSession).
We tried a direct fix that wrote the cascaded child terminal events from RunExecutionService when the stream fails after the root settles. We dropped it after review, for three reasons:
- It writes from a cached projection without a gate, so it can overwrite a Stop result that is already committed.
- It empties its tracking before the write and only logs a failed write, so a storage error leaves the rows running for good.
- That handler also runs for ingestion and storage failures while the provider is alive, so it would mark live children failed.
A safer shape is probably to run the existing background-work settlement through the orchestrator on confirmed runtime loss only (the session released with runtime_error), so the same serialized, liveness-checked path that Stop uses does the cleanup.
Impact
Major degradation or frequent failure
Version or commit
main 14f295e (found on a fork at 2026-11 with #26188 applied)
Environment
Linux (Arch), self-hosted server as a systemd user service, web client in Chrome, Codex provider with the shared app-server.
Logs or stack traces
WARN orchestration V2 provider event ingestion failed
cause: ProviderAdapterEventStreamError (provider-session:provider-instance:codex:shared)
In our production incident the app-server was OOM-killed during a long subagent run. The parent thread kept showing the subagent as Running for 12 hours.
Workaround
Press Stop in the thread (the composer's Stop, or the Stop on the background-work banner), or restart the server.
Activity
fix(server): a Codex subagent that outlived its run ends when the app…
test(server): work left running when the Codex app-server dies stays …
fix(server): ending a dead provider's work no longer overwrites a Stop