Skip to content

[Bug]: Codex runs that share a provider session fall behind together while the writer is busy #39135

Open

Description

@navidnadali

Note

Written by Claude Opus 5.5 on behalf of Navid

Before submitting

  • I searched existing issues and did not find a duplicate.
  • I included enough detail to reproduce or investigate the problem.

Area

apps/server

Steps to reproduce

We saw this with a root Codex thread and its delegated children falling behind together on a busy server, and
reproduced it in a server test with the test clock (below).

  1. Run several Codex threads of one provider instance at once, e.g. a root thread that delegates to a few children.
    Codex declares supportsMultipleProviderThreadsPerSession (apps/server/src/orchestration-v2/Adapters/CodexAdapterV2.ts:263),
    so providerSessionIdFor gives all of them one provider session (Orchestrator.ts:1643-1659), and
    publishToSubscribers offers every event of that session to every live run's subscription
    (ProviderSessionManager.ts:646-658).
  2. Keep the SQLite writer busy with other work (other threads, projections), so a commit waits about a second.
  3. Let each thread stream as usual: usage reports (provider_turn.updated, running) and turn item snapshots.

Expected behavior

Each run's cost follows its own thread's events: a run drops another thread's events without waiting on the writer, and
the threads keep up while the writer is busy.

Actual behavior

A run drops another thread's event only when it processes it (routeEvent → routeProviderEvent,
RunExecutionService.ts:1667-1674), after consumeProviderEvents has handled it (RunExecutionService.ts:1975-1991):

  • before the root terminal another thread's snapshot is encoded and queued as a job in the shared provider commit
    (ProviderCommitBatch.ts:314-321);
  • another thread's usage report is a control, which runs only after the run's earlier replies settle
    (ProviderCommitBatch.ts:303-325), i.e. after the commit that carries the other thread's snapshots.

So with K Codex runs live each run waits for about K commits per round instead of its own, and the queued commit jobs
grow with K². When the writer is busy every run falls behind, all together, and the lag grows with time. Measured after
#38544, which stopped controls from queueing a no-op; it still grows.

Test-clock model through the real RunExecutionService and ProviderCommitBatch on main: the writer is a FIFO
one-permit lock (as the SQLite client's), ten other writers keep it busy (~1 s per commit), and every 10 s each thread
reports usage, writes two turn items and reports usage again. Rounds each run is behind:

runs on the session after 1 min after 5 min after 10 min
1, 5 0 0 0
9 0-1 4-5 9-10
12 1-2 8-9 18

Impact

Major degradation or frequent failure

While it lasts, every Codex thread of the instance shows its output late, and the delay grows the longer the threads
run.

Version or commit

main @ 6fd14f4

Environment

Server-side; any OS. Codex (any provider with supportsMultipleProviderThreadsPerSession).

Logs or stack traces

None; the model above is a server test.

Possible fix: route each run's events as they arrive and drop, before the commit queue, those routing rejects without a
routing change. A PR with that fix and a test that fails on main follows.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    vouch:trustedAuthor is trusted by repo permissions or the VOUCHED list.

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions