Skip to content

[Bug]: Codex notification handlers block the app-server stdout reader, so a busy permit makes the shared app-server drop its T3 connection #28697

Closed as not planned

Description

@esukram

What happened

Note

AI-generated issue: drafted by Claude Opus 5.5 (Claude Code via t3 triage) on behalf of @esukram.

The shared Codex app-server went silent three times in ~80 minutes on Oct 31 (17:29, 18:30 and 18:51 UTC), and again on Nov 1 (13:25 UTC) on a newer nightly. Each time:

  • it was in the middle of streaming item/agentMessage/delta;
  • Codex logged Failed to write to stdout: Resource temporarily unavailable (os error 11);
  • it marked its only client connection as disconnected, and dropped every later response and notification while the process stayed alive.

T3 kept sending requests and waited forever. The consequences (stuck workers, a blocked worktree lease) are tracked in #16636. This issue is about the trigger on T3's side.

Diagnosis

Likely cause: read-loop backpressure.

  • packages/effect-codex-app-server/src/protocol.ts (~L406–430) handles each stdout line before reading the next chunk, and it runs onNotification handlers inline.
  • In CodexAdapterV2.ts, the item/started, item/completed, thread/closed and thread/goal/* handlers run under turnTerminalizationPermit.withPermits(1) (~L4528, L4694, L4835, L5157). That is one Semaphore(1) for all threads on the shared app-server.
  • So any permit holder that waits stops T3 from reading stdout. Codex fills its non-blocking stdout socket and gets EAGAIN.

Not yet identified: which holder waits. The goal-turn turn/interrupt (~L4511) is ruled out, since it is forked. The terminalization paths (~L5964) are unverified.

Ruled out: event-loop blocking (t3_server_event_loop_delay_max ≤15 ms at the drops), host load (CPU ≤14 %) and cgroup throttling (flat counters, PSI 0).

Fix idea: dispatch handlers off the reader, through a queue or a fork, and/or never hold the permit across an await.

Steps to reproduce

Not yet reduced to a minimal repro.

  1. A Claude orchestrator repeatedly delegate_tasks to Codex (reviews), with long streamed replies, on one shared app-server.
  2. Within tens of minutes Codex logs os error 11 on stdout, then dropping message for disconnected connection for every message.
  3. Every later request goes unanswered, and running turns get no more events.

Deterministic idea: make a permit holder sleep while another thread streams deltas.

Version

0.1.6-nightly.20261030.3940 (three times on Oct 31) and 0.1.7-nightly.20261031.4004 (Nov 1), both with codex-cli 0.166.0, so it is not specific to one nightly. Triage CLI: 0.1.6.

Environment

  • Arch Linux x64, kernel 7.2.11; t3 serve as a systemd user service; web UI client on the LAN
  • codex-cli 0.166.0 as the shared app-server; Claude Agent (claude-opus-5-5) as orchestrator

Evidence

Codex `logs_2.sqlite`, first drop:

2026-10-31 17:29:04 ERROR codex_app_server_transport::transport::stdio  Failed to write to stdout: Resource temporarily unavailable (os error 11)
2026-10-31 17:29:04 WARN  codex_app_server::transport  dropping message for disconnected connection: ConnectionId(0)

On Nov 1 (4004) it was the same: EAGAIN at 13:25:29, last message from Codex at 13:25:28.931, and `thread/start` ids 27 and 32 never answered. Timelines: https://github.com/pingdotgg/t3code/issues/16636#issuecomment-6547877689

Related issues

#16636 (consequences), #23854 (bounds control calls, not this trigger), #21321 (worker slots only), #19066, openai/codex#55242

Fix applied or workaround

Stop the wedged app-server: systemctl --user stop t3code-codex-<id>.scope. T3 then fails or retries the pending turns, and starts a fresh app-server on the next turn. No backend restart was needed. Delegating to Claude instead of Codex avoids it.

Filed by

Claude Opus 5.5 · Claude Code · orchestrated by T3 Code (t3 triage). AI-generated; reviewed by @esukram before submission

Activity

  1. changed the title [-]Codex notification handlers block the app-server stdout reader, so a busy permit makes the shared app-server drop its T3 connection[/-] [+][Bug]: Codex notification handlers block the app-server stdout reader, so a busy permit makes the shared app-server drop its T3 connection[/+] on Nov 1, 2026
  2. juliusmarminge commented on Nov 13, 2026

    @juliusmarminge
    Member

    A fix for this was proposed in #28711, but when the reported steps were run in the real app the problem did not appear.

    What was run on f313d0a6cb, the commit before this change:

    On each commit, asked an orchestrating agent to delegate fifteen sequential Codex reviews with long streamed reports, then queued two follow-ups during the first review. Both completed all fifteen and processed both follow-ups. Follow-up wording started additional sequences, so the same clarification was steered to both agents after their original fifteen finished. Base completed four additional reviews; head completed one additional review before cancellation took effect. Persisted incremental updates confirm streaming. Neither side exhibited EAGAIN, dropped-message symptoms, or a permanent connection failure.

    How: Real T3 development servers in separate checkouts with fresh isolated databases, driven through T3's project/thread CLI. Both used Codex CLI 0.166.0, gpt-6-luna, separate authenticated Codex homes, and one shared app-server session per server. Synthetic cache.js fixture; no replay provider or tests used as evidence.

    Could you share the exact steps, what you see, and the version and platform you are on? The pull request is closed for now; the fix is picked up again once the problem can be shown.

  3. juliusmarminge commented on Nov 13, 2026

    @juliusmarminge
    Member

    @esukram Can you capture T3's server log at the moment of a drop, with its timestamps next to the Codex EAGAIN line? We most need to know whether the drop came right after a Codex error, a capacity retry (serverOverloaded), a goal update, or a Stop/interrupt. Also: can you still reproduce it on a current nightly, and how many Codex threads were streaming at once when it happened?

  4. NOirBRight commented on Nov 20, 2026

    @NOirBRight

    Confirming we hit this exact failure mode repeatedly today under high parallel concurrency on a much newer nightly, and sharing why the automated repro in #28711 (which ran reviews sequentially) did not trigger it.

    Environment

    • T3 Code: 0.1.11-nightly.20261119.4954 (Linux x64 desktop AppImage)
    • Codex CLI: 0.170.0 (codex app-server -c thread_unload_delay_secs=0, shared session provider-session:provider-instance:codex:shared)
    • OS / Resources: Linux x64, 58 GiB RAM (~39 GiB available at all failure timestamps; zero OOM kills, zero core dumps, weekly rate limit at 5% used)

    Observed behavior (parallel vs low concurrency)

    • Lower concurrency: the shared codex app-server stayed up for 40+ minutes continuously.
    • High parallel concurrency (16 active Codex worker threads + parallel read-only delegated review subagents = 22+ concurrent active threads on the single shared codex app-server, ~940 MB RSS / 98 threads / 613 open FDs):
      • CodexAppServerProtocol.request.enqueue latency spiked to 6.1–6.6 seconds (e.g. thread/start / thread/resume enqueue durations of 6599 ms and 6144 ms in server.trace.ndjson).
      • The shared codex app-server then repeatedly exited/disconnected every 2–4 minutes (new initialize handshakes observed at 06:44:15Z, 06:46:38Z, 06:46:45Z, 06:50:12Z).
      • Every in-flight Codex worker and delegated subagent on provider-session:provider-instance:codex:shared failed simultaneously with:
        • UI / thread error: The provider event stream closed unexpectedly. Retry the turn; if it keeps failing, check the provider and server logs.
        • Server trace: CodexAppServerTransportError: Codex App Server transport operation 'read-process-exit-status' failed (ProviderAdapterEnsureThreadError / ProviderAdapterInterruptError).
      • Once those workers auto-retried together, the retry burst saturated the freshly restarted shared codex app-server again within 2–3 minutes.

    Why #28711's automated capture missed it

    The automated test in #28711 ran "three sequential reviews... twelve more sequential reviews". Sequential turns never contend on turnTerminalizationPermit (Semaphore(1)) while 15–20+ other threads are simultaneously streaming item/agentMessage/delta / item/started / item/completed over the single stdout pipe.

    Suggested minimal repro for reopening #28711: launch 16–20 parallel Codex turns/subagents simultaneously on the same shared codex instance, each running tool calls + streaming multi-thousand-character replies concurrently, and monitor CodexAppServerProtocol.request.enqueue latency and shared process uptime over 5–10 minutes.

    (Workaround that immediately stabilized our 16-way workload: moving parallel workers/reviews off the single shared codex session onto a per-thread-process provider or sharding across multiple codex provider instances.)

  5. juliusmarminge commented on Nov 20, 2026

    @juliusmarminge
    Member

    @esukram When the shared Codex app-server is running (ideally just before a drop), what does grep flags /proc/<codex app-server pid>/fdinfo/1 print? If its octal value includes 04000 (O_NONBLOCK), something in your setup is putting Codex's stdout into non-blocking mode: an MCP server, a wrapper, or a tool Codex starts. Can you tell which?

  6. NOirBRight commented on Nov 20, 2026

    @NOirBRight

    Replying to @juliusmarminge's question in #issuecomment-7033662449:

    On our live Linux x64 instance (T3 Code 0.1.11-nightly.20261120.4979, Codex CLI 0.170.0 codex app-server -c thread_unload_delay_secs=0, x86_64-unknown-linux-musl binary) running under t3code-codex-*.scope:

    • readlink /proc/<pid>/fd/1 -> socket:[<inode>] (Node.js child_process Unix socketpair)
    • cat /proc/<pid>/fdinfo/1:
      pos:	0
      flags:	02
      mnt_id:	12
      ino:	<inode>
      scm_fds: 0
      
    • Same for fd/0 and fd/2: both are socket:[<inode>] with flags: 02 (O_RDWR).

    So 04000 (O_NONBLOCK) is not set on fd/1 of codex app-server here — it remains blocking O_RDWR (02). Note that unlike the codex exec panic in openai/codex#55242 (where codex set FIONBIO on stdio), our high-concurrency drops did not report an EAGAIN line; instead, when 16–22+ parallel threads stream simultaneously and handleLine stalls the parent reader on turnTerminalizationPermit, CodexAppServerProtocol.request.enqueue latency spikes to 6+ seconds and the shared app-server socketpair backs up / drops (CodexAppServerTransportError: ... 'read-process-exit-status' failed). Decoupling handleLine from the stdout reader loop (as in #28711) directly prevents the parent reader from stalling and filling the socketpair buffer.

  7. juliusmarminge commented on Nov 20, 2026

    @juliusmarminge
    Member

    I tried to reproduce this on the current build and could not.

    Commit 2c33cbd (origin/main). I started an isolated server with T3CODE_DEV_SEED=0 node apps/server/src/bin.ts serve --base-dir <scratch> --port 47321 on a fresh database, added a scratch git project, and confirmed one Codex thread replied (t3 thread start ... --provider codex --model gpt-6-luna --wait). Codex CLI 0.171.0 ran as a single shared codex app-server -c thread_unload_delay_secs=0, a child of that server; its fd 1 was a socket with flags: 02 (blocking). Round 1: 18 parallel thread start --wait turns, each running ls -la once and then streaming an essay of at least 2,500 words. All 18 exited 0 in 116-143 s with 21-35 KB replies each. Round 2: 22 parallel turns of the same kind; all 22 exited 0. The app-server PID stayed alive through both rounds. Codex logs_2.sqlite, read-only, from the run start: 0 rows matching 'os error 11', 'Failed to write to stdout' or 'disconnected connection'. The T3 server log had 0 error or transport lines. So neither the reporter's EAGAIN followed by a dropped connection nor the second commenter's repeated app-server exits appeared. Separately, a scratch protocol.test.ts case, not pushed, showed the mechanism the issue names: when an onNotification handler waits on a Deferred, a response line after it is never read (Option.none after a 1 s TestClock timeout on main), and the case passes when the handler is forked off the reader. On main I found no turnTerminalizationPermit holder that actually waits: the holders I read do in-memory work, and the capacity-retry path deliberately leaves startNativeTurn outside the permit.

    Closing as not planned. If it still happens for you, reply with the exact steps and the version you are on and it will be looked at again.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    via-triageFiled through npx t3 triage

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions