Repository navigation
Description
Note
Written by Claude Opus 5.5 on behalf of Navid
Before submitting
- I searched existing issues and did not find a duplicate.
- I included enough detail to reproduce or investigate the problem.
Area
apps/server
Steps to reproduce
- Keep one delegated child as a long-lived worker: the parent delegates a task, then sends each new task to the same
finished child witht3_thread_send, which makes the follow-up a task of the child. The child collects thousands of
runs. - Give it one more task.
finalizeAppOwnedSubagent(child)runs under the parent's lock for that run'sstarting,waitingand terminal
updates, andqueuedif it queues (apps/server/src/orchestration-v2/Orchestrator.ts:16170-16175, filter
:16200-16215); for each provider-thread or subagent update that settles child work (:16303-16317); and after the
run's checkpoint capture while the child has a pending delegated completion (EffectWorker.ts:589-602→
Orchestrator.ts:16479-16531). Boot recovery (:16332-16345) and cancel (:16610) call it too. A task that starts
at once costs at least three calls.- Each call starts with
getThreadRecords(child, ["runs", "messages", "subagents", "providerThreads", "providerTurns", "attempts"], { messageRoles: ["user"] })
(Orchestrator.ts:14961-14965): every run, user message, task, provider turn and attempt the child has ever had.
Scripted, on a synthetic child: in a checkout of main with dependencies installed, save the script below as
apps/server/finalize-read-scaling.mjs and run node apps/server/finalize-read-scaling.mjs (Node 26). It runs main's
migrations on in-memory SQLite, gives a child thread N finished runs (one attempt, one completed provider turn, a user and
an assistant message each), and times that read through the real ProjectionStore.
finalize-read-scaling.mjs
// Save as apps/server/finalize-read-scaling.mjs in a checkout of main, then: node apps/server/finalize-read-scaling.mjs
// A delegated child thread with N finished runs (one attempt, one completed provider turn, a user and an assistant
// message each), read through the real ProjectionStore on in-memory SQLite with the call that
// finalizeAppOwnedSubagent starts with. Prints the rows it decodes, its SQLite VM steps and its median wall time.
import * as NodeSqlite from "node:sqlite";
import * as DateTime from "effect/DateTime";
import * as Effect from "effect/Effect";
import * as Layer from "effect/Layer";
import * as SqlClient from "effect/sql/SqlClient";
import * as Statement from "effect/sql/Statement";
import * as ProjectionStore from "./src/orchestration-v2/ProjectionStore.ts";
import * as SqlitePersistence from "./src/persistence/Sqlite.ts";
const modelSelection = { instanceId: "codex", model: "gpt-5.4" };
const measure = (runCount) =>
Effect.gen(function* () {
const store = yield* ProjectionStore.ProjectionStoreV2;
const sql = yield* SqlClient.SqlClient;
const now = yield* DateTime.now;
const at = DateTime.formatIso(now);
const threadId = "thread:child";
const providerThreadId = "provider-thread:child";
yield* store.apply({
id: "event:thread-created", type: "thread.created", threadId, occurredAt: now,
payload: {
createdBy: "agent", creationSource: "server", id: threadId, projectId: "project:1", title: "Worker",
providerInstanceId: "codex", modelSelection, runtimeMode: "full-access", interactionMode: "default",
branch: null, worktreePath: null, activeProviderThreadId: providerThreadId,
lineage: { parentThreadId: null, relationshipToParent: null, rootThreadId: threadId },
forkedFrom: null, createdAt: now, updatedAt: now, archivedAt: null, settledOverride: null, settledAt: null,
lastVisitedAt: null, deletedAt: null,
},
});
yield* sql`
INSERT INTO orchestration_v2_projection_provider_threads (
provider_thread_id, thread_id, owner_node_id, provider, provider_session_id, status, first_run_ordinal,
last_run_ordinal, updated_at, payload_json, driver, provider_instance_id
) VALUES (${providerThreadId}, ${threadId}, NULL, 'codex', NULL, 'idle', 1, ${runCount}, ${at},
json_object('id', ${providerThreadId}, 'driver', 'codex', 'providerInstanceId', 'codex',
'providerSessionId', NULL, 'appThreadId', ${threadId}, 'ownerNodeId', NULL, 'nativeThreadRef', NULL,
'nativeConversationHeadRef', NULL, 'status', 'idle', 'firstRunOrdinal', 1, 'lastRunOrdinal', ${runCount},
'handoffIds', json('[]'), 'forkedFrom', NULL, 'createdAt', ${at}, 'updatedAt', ${at}), 'codex', 'codex')`;
const series = (body) => sql.unsafe(
`WITH RECURSIVE n(i) AS (SELECT 1 UNION ALL SELECT i + 1 FROM n WHERE i < ${runCount}) ${body}`,
);
const ts = `'${at}'`;
yield* series(`INSERT INTO orchestration_v2_projection_runs (run_id, thread_id, ordinal, provider, provider_thread_id,
status, requested_at, completed_at, payload_json, provider_instance_id)
SELECT 'run:' || i, '${threadId}', i, 'codex', '${providerThreadId}', 'completed', ${ts}, ${ts},
json_object('id', 'run:' || i, 'threadId', '${threadId}', 'ordinal', i, 'providerInstanceId', 'codex',
'modelSelection', json('${JSON.stringify(modelSelection)}'), 'providerThreadId', '${providerThreadId}',
'userMessageId', 'message:user:' || i, 'rootNodeId', 'node:' || i, 'activeAttemptId', 'attempt:' || i,
'status', 'completed', 'requestedAt', ${ts}, 'startedAt', ${ts}, 'completedAt', ${ts},
'checkpointId', NULL, 'contextHandoffId', NULL, 'delegatedTaskId', 'node:task:' || i), 'codex'
FROM n`);
yield* series(`INSERT INTO orchestration_v2_projection_run_attempts (attempt_id, thread_id, run_id, attempt_ordinal,
root_node_id, provider, provider_thread_id, provider_turn_id, status, payload_json, provider_instance_id)
SELECT 'attempt:' || i, '${threadId}', 'run:' || i, 1, 'node:' || i, 'codex', '${providerThreadId}',
'turn:' || i, 'completed',
json_object('id', 'attempt:' || i, 'runId', 'run:' || i, 'attemptOrdinal', 1, 'rootNodeId', 'node:' || i,
'providerInstanceId', 'codex', 'providerThreadId', '${providerThreadId}', 'providerTurnId', 'turn:' || i,
'reason', 'initial', 'status', 'completed', 'startedAt', ${ts}, 'completedAt', ${ts}), 'codex'
FROM n`);
yield* series(`INSERT INTO orchestration_v2_projection_provider_turns (provider_turn_id, thread_id, provider_thread_id,
node_id, run_attempt_id, ordinal, status, started_at, completed_at, payload_json)
SELECT 'turn:' || i, '${threadId}', '${providerThreadId}', 'node:' || i, 'attempt:' || i, i, 'completed', ${ts}, ${ts},
json_object('id', 'turn:' || i, 'providerThreadId', '${providerThreadId}', 'nodeId', 'node:' || i,
'runAttemptId', 'attempt:' || i, 'nativeTurnRef', NULL, 'ordinal', i, 'status', 'completed',
'startedAt', ${ts}, 'completedAt', ${ts})
FROM n`);
for (const role of ["user", "assistant"])
yield* series(`INSERT INTO orchestration_v2_projection_messages (message_id, thread_id, run_id, node_id, role,
streaming, created_at, updated_at, payload_json)
SELECT 'message:${role}:' || i, '${threadId}', 'run:' || i, 'node:' || i, '${role}', 0, ${ts}, ${ts},
json_object('id', 'message:${role}:' || i, 'threadId', '${threadId}', 'runId', 'run:' || i,
'nodeId', 'node:' || i, 'role', '${role}', 'text', '${role === "user" ? "Next task" : "Done"} ' || i,
'attachments', json('[]'), 'streaming', json('false'), 'createdAt', ${ts}, 'updatedAt', ${ts},
'createdBy', '${role === "user" ? "agent" : "provider"}', 'creationSource', 'server')
FROM n`);
// The first read of finalizeAppOwnedSubagent (Orchestrator.ts), unchanged.
const read = store.getThreadRecords(
threadId,
["runs", "messages", "subagents", "providerThreads", "providerTurns", "attempts"],
{ messageRoles: ["user"] },
);
const statements = [];
const records = yield* read.pipe(
Effect.provideService(Statement.CurrentTransformer, (statement) => {
statements.push(statement.compile());
return Effect.succeed(statement);
}),
);
const times = [];
for (let i = 0; i < 5; i++) {
const start = performance.now();
yield* read;
times.push(performance.now() - start);
}
const copyPath = `file:/finalize-read-${runCount}?vfs=memdb`;
const copy = new NodeSqlite.DatabaseSync(copyPath);
yield* sql`VACUUM INTO ${copyPath}`;
const steps = statements.reduce((sum, [query, params]) => {
const statement = copy.prepare(query);
statement.all(...params);
return sum + statement.stat("vmStep");
}, 0);
copy.close();
return {
runs: records.runs.length, attempts: records.attempts.length, providerTurns: records.providerTurns.length,
messages: records.messages.length, steps, ms: times.sort((a, b) => a - b)[2],
};
}).pipe(Effect.provide(ProjectionStore.layer.pipe(Layer.provideMerge(SqlitePersistence.layerMemory))));
console.log("child runs | runs, attempts, provider turns, user messages decoded | VM steps | median ms (5 reads)");
for (const runCount of [100, 1_000, 3_000, 9_000]) {
const r = await Effect.runPromise(measure(runCount));
console.log(`${runCount} | ${r.runs}, ${r.attempts}, ${r.providerTurns}, ${r.messages} | ${r.steps} | ${r.ms.toFixed(1)}`);
}Expected behavior
Finalizing reads what its decisions use: the child's unfinished runs, the run the task reports from, and the runs of the
tasks it may settle. #34551 already did this for the parent's side (Orchestrator.ts:14974-14977).
Actual behavior
The script's output on main @ 22f006c. Rows and VM steps are exact and the same on macOS arm64, Linux arm64 and
Linux x86_64; read times are the range of three runs in a Linux arm64 container (Node 26.13.1) on a busy machine.
| Child runs | Rows decoded (runs, attempts, provider turns, user messages) | SQLite VM steps | Read |
|---|---|---|---|
| 100 | 400 | 6,278 | 7-10 ms |
| 1,000 | 4,000 | 61,178 | 41-52 ms |
| 3,000 | 12,000 | 183,178 | 127-137 ms |
| 9,000 | 36,000 | 549,178 | 391-429 ms |
That is one read; a task pays it at least three times, each while the parent's lock is held, so the parent's sends,
Stop and run-end reactions wait behind it.
One whole finalize call in an in-process bench (real ProjectionStore on SQLite, Node 26.13.1, a child with 9,002 runs,
8,001 provider turns and 9,003 messages), measured on a build of main from 2026-11-15 (the finalize code is unchanged on
22f006c). "Bounded" is a read limited to what the decisions need, checked to decide the same as the full read.
| Full read (main) | Bounded | |
|---|---|---|
| Finalize wall time | 586 ms | 18 ms |
| Decode | 482 ms | 4 ms |
| Run rows read | 18,004 | 57 |
| Provider turn rows read | 8,001 | 1 |
| Message rows read | 18,005 | 22 |
| Heap peak growth | 127 MB | 5 MB |
Impact
Major degradation or frequent failure
Every task a long-lived worker finishes makes the next finalize slower, and each finalize holds the parent's lock.
Version or commit
main @ 22f006c
Environment
Server-side; any OS and provider.
Workaround
Start a fresh delegated thread instead of reusing a long-lived one.
Related
- perf(server): delegated tasks start and finish without reading the whole parent thread #34551: the parent side of the same read.
- perf(server): task follow-ups no longer read the whole parent thread #38867: a follow-up task's read of its parent.
- fix(server): a waiting child run marks its parent task waiting #38112: added the call on a delegated run's
waitingupdate. - perf(server): boot no longer re-finalizes live subagent children #32211 (open): fewer finalize calls at boot; each call still reads the whole child.
- fix(server): a delegated task still reaches its parent when its first update fails #37225 (open): retries this call.
- A send or queued start to a long thread still reads 6–15 s of its history on main (follow-up to #29468) #36127: other whole-history reads under a thread's lock.
We have a bounded read with a test that its decisions match the full read; happy to port it and open a PR, or a
maintainer can take it.
Fixed in the performance fork since v0.1.9-perf.9 (2026-11-16).
Activity