Repository navigation
Description
Note
Written by Claude Opus 5.5 on behalf of Navid
Before submitting
- I searched existing issues and did not find a duplicate.
- I included enough detail to reproduce or investigate the problem.
Area
apps/server
Steps to reproduce
We saw this with a root Codex thread and its delegated children falling behind together on a busy server, and
reproduced it in a server test with the test clock (below).
- Run several Codex threads of one provider instance at once, e.g. a root thread that delegates to a few children.
Codex declaressupportsMultipleProviderThreadsPerSession(apps/server/src/orchestration-v2/Adapters/CodexAdapterV2.ts:263),
soproviderSessionIdForgives all of them one provider session (Orchestrator.ts:1643-1659), and
publishToSubscribersoffers every event of that session to every live run's subscription
(ProviderSessionManager.ts:646-658). - Keep the SQLite writer busy with other work (other threads, projections), so a commit waits about a second.
- Let each thread stream as usual: usage reports (
provider_turn.updated, running) and turn item snapshots.
Expected behavior
Each run's cost follows its own thread's events: a run drops another thread's events without waiting on the writer, and
the threads keep up while the writer is busy.
Actual behavior
A run drops another thread's event only when it processes it (routeEvent → routeProviderEvent,
RunExecutionService.ts:1667-1674), after consumeProviderEvents has handled it (RunExecutionService.ts:1975-1991):
- before the root terminal another thread's snapshot is encoded and queued as a job in the shared provider commit
(ProviderCommitBatch.ts:314-321); - another thread's usage report is a control, which runs only after the run's earlier replies settle
(ProviderCommitBatch.ts:303-325), i.e. after the commit that carries the other thread's snapshots.
So with K Codex runs live each run waits for about K commits per round instead of its own, and the queued commit jobs
grow with K². When the writer is busy every run falls behind, all together, and the lag grows with time. Measured after
#38544, which stopped controls from queueing a no-op; it still grows.
Test-clock model through the real RunExecutionService and ProviderCommitBatch on main: the writer is a FIFO
one-permit lock (as the SQLite client's), ten other writers keep it busy (~1 s per commit), and every 10 s each thread
reports usage, writes two turn items and reports usage again. Rounds each run is behind:
| runs on the session | after 1 min | after 5 min | after 10 min |
|---|---|---|---|
| 1, 5 | 0 | 0 | 0 |
| 9 | 0-1 | 4-5 | 9-10 |
| 12 | 1-2 | 8-9 | 18 |
Impact
Major degradation or frequent failure
While it lasts, every Codex thread of the instance shows its output late, and the delay grows the longer the threads
run.
Version or commit
main @ 6fd14f4
Environment
Server-side; any OS. Codex (any provider with supportsMultipleProviderThreadsPerSession).
Logs or stack traces
None; the model above is a server test.
Possible fix: route each run's events as they arrive and drop, before the commit queue, those routing rejects without a
routing change. A PR with that fix and a test that fails on main follows.
Activity