Skip to content

[Bug]: Finalizing a delegated child reads the child's whole history under the parent's lock, several times per task #39262

Closed

Description

@navidnadali

Note

Written by Claude Opus 5.5 on behalf of Navid

Before submitting

  • I searched existing issues and did not find a duplicate.
  • I included enough detail to reproduce or investigate the problem.

Area

apps/server

Steps to reproduce

  1. Keep one delegated child as a long-lived worker: the parent delegates a task, then sends each new task to the same
    finished child with t3_thread_send, which makes the follow-up a task of the child. The child collects thousands of
    runs.
  2. Give it one more task.
  3. finalizeAppOwnedSubagent(child) runs under the parent's lock for that run's starting, waiting and terminal
    updates, and queued if it queues (apps/server/src/orchestration-v2/Orchestrator.ts:16170-16175, filter
    :16200-16215); for each provider-thread or subagent update that settles child work (:16303-16317); and after the
    run's checkpoint capture while the child has a pending delegated completion (EffectWorker.ts:589-602 →
    Orchestrator.ts:16479-16531). Boot recovery (:16332-16345) and cancel (:16610) call it too. A task that starts
    at once costs at least three calls.
  4. Each call starts with
    getThreadRecords(child, ["runs", "messages", "subagents", "providerThreads", "providerTurns", "attempts"], { messageRoles: ["user"] })
    (Orchestrator.ts:14961-14965): every run, user message, task, provider turn and attempt the child has ever had.

Scripted, on a synthetic child: in a checkout of main with dependencies installed, save the script below as
apps/server/finalize-read-scaling.mjs and run node apps/server/finalize-read-scaling.mjs (Node 26). It runs main's
migrations on in-memory SQLite, gives a child thread N finished runs (one attempt, one completed provider turn, a user and
an assistant message each), and times that read through the real ProjectionStore.

finalize-read-scaling.mjs
// Save as apps/server/finalize-read-scaling.mjs in a checkout of main, then: node apps/server/finalize-read-scaling.mjs
// A delegated child thread with N finished runs (one attempt, one completed provider turn, a user and an assistant
// message each), read through the real ProjectionStore on in-memory SQLite with the call that
// finalizeAppOwnedSubagent starts with. Prints the rows it decodes, its SQLite VM steps and its median wall time.
import * as NodeSqlite from "node:sqlite";
import * as DateTime from "effect/DateTime";
import * as Effect from "effect/Effect";
import * as Layer from "effect/Layer";
import * as SqlClient from "effect/sql/SqlClient";
import * as Statement from "effect/sql/Statement";
import * as ProjectionStore from "./src/orchestration-v2/ProjectionStore.ts";
import * as SqlitePersistence from "./src/persistence/Sqlite.ts";

const modelSelection = { instanceId: "codex", model: "gpt-5.4" };

const measure = (runCount) =>
  Effect.gen(function* () {
    const store = yield* ProjectionStore.ProjectionStoreV2;
    const sql = yield* SqlClient.SqlClient;
    const now = yield* DateTime.now;
    const at = DateTime.formatIso(now);
    const threadId = "thread:child";
    const providerThreadId = "provider-thread:child";
    yield* store.apply({
      id: "event:thread-created", type: "thread.created", threadId, occurredAt: now,
      payload: {
        createdBy: "agent", creationSource: "server", id: threadId, projectId: "project:1", title: "Worker",
        providerInstanceId: "codex", modelSelection, runtimeMode: "full-access", interactionMode: "default",
        branch: null, worktreePath: null, activeProviderThreadId: providerThreadId,
        lineage: { parentThreadId: null, relationshipToParent: null, rootThreadId: threadId },
        forkedFrom: null, createdAt: now, updatedAt: now, archivedAt: null, settledOverride: null, settledAt: null,
        lastVisitedAt: null, deletedAt: null,
      },
    });
    yield* sql`
      INSERT INTO orchestration_v2_projection_provider_threads (
        provider_thread_id, thread_id, owner_node_id, provider, provider_session_id, status, first_run_ordinal,
        last_run_ordinal, updated_at, payload_json, driver, provider_instance_id
      ) VALUES (${providerThreadId}, ${threadId}, NULL, 'codex', NULL, 'idle', 1, ${runCount}, ${at},
        json_object('id', ${providerThreadId}, 'driver', 'codex', 'providerInstanceId', 'codex',
          'providerSessionId', NULL, 'appThreadId', ${threadId}, 'ownerNodeId', NULL, 'nativeThreadRef', NULL,
          'nativeConversationHeadRef', NULL, 'status', 'idle', 'firstRunOrdinal', 1, 'lastRunOrdinal', ${runCount},
          'handoffIds', json('[]'), 'forkedFrom', NULL, 'createdAt', ${at}, 'updatedAt', ${at}), 'codex', 'codex')`;
    const series = (body) => sql.unsafe(
      `WITH RECURSIVE n(i) AS (SELECT 1 UNION ALL SELECT i + 1 FROM n WHERE i < ${runCount}) ${body}`,
    );
    const ts = `'${at}'`;
    yield* series(`INSERT INTO orchestration_v2_projection_runs (run_id, thread_id, ordinal, provider, provider_thread_id,
        status, requested_at, completed_at, payload_json, provider_instance_id)
      SELECT 'run:' || i, '${threadId}', i, 'codex', '${providerThreadId}', 'completed', ${ts}, ${ts},
        json_object('id', 'run:' || i, 'threadId', '${threadId}', 'ordinal', i, 'providerInstanceId', 'codex',
          'modelSelection', json('${JSON.stringify(modelSelection)}'), 'providerThreadId', '${providerThreadId}',
          'userMessageId', 'message:user:' || i, 'rootNodeId', 'node:' || i, 'activeAttemptId', 'attempt:' || i,
          'status', 'completed', 'requestedAt', ${ts}, 'startedAt', ${ts}, 'completedAt', ${ts},
          'checkpointId', NULL, 'contextHandoffId', NULL, 'delegatedTaskId', 'node:task:' || i), 'codex'
      FROM n`);
    yield* series(`INSERT INTO orchestration_v2_projection_run_attempts (attempt_id, thread_id, run_id, attempt_ordinal,
        root_node_id, provider, provider_thread_id, provider_turn_id, status, payload_json, provider_instance_id)
      SELECT 'attempt:' || i, '${threadId}', 'run:' || i, 1, 'node:' || i, 'codex', '${providerThreadId}',
        'turn:' || i, 'completed',
        json_object('id', 'attempt:' || i, 'runId', 'run:' || i, 'attemptOrdinal', 1, 'rootNodeId', 'node:' || i,
          'providerInstanceId', 'codex', 'providerThreadId', '${providerThreadId}', 'providerTurnId', 'turn:' || i,
          'reason', 'initial', 'status', 'completed', 'startedAt', ${ts}, 'completedAt', ${ts}), 'codex'
      FROM n`);
    yield* series(`INSERT INTO orchestration_v2_projection_provider_turns (provider_turn_id, thread_id, provider_thread_id,
        node_id, run_attempt_id, ordinal, status, started_at, completed_at, payload_json)
      SELECT 'turn:' || i, '${threadId}', '${providerThreadId}', 'node:' || i, 'attempt:' || i, i, 'completed', ${ts}, ${ts},
        json_object('id', 'turn:' || i, 'providerThreadId', '${providerThreadId}', 'nodeId', 'node:' || i,
          'runAttemptId', 'attempt:' || i, 'nativeTurnRef', NULL, 'ordinal', i, 'status', 'completed',
          'startedAt', ${ts}, 'completedAt', ${ts})
      FROM n`);
    for (const role of ["user", "assistant"])
      yield* series(`INSERT INTO orchestration_v2_projection_messages (message_id, thread_id, run_id, node_id, role,
          streaming, created_at, updated_at, payload_json)
        SELECT 'message:${role}:' || i, '${threadId}', 'run:' || i, 'node:' || i, '${role}', 0, ${ts}, ${ts},
          json_object('id', 'message:${role}:' || i, 'threadId', '${threadId}', 'runId', 'run:' || i,
            'nodeId', 'node:' || i, 'role', '${role}', 'text', '${role === "user" ? "Next task" : "Done"} ' || i,
            'attachments', json('[]'), 'streaming', json('false'), 'createdAt', ${ts}, 'updatedAt', ${ts},
            'createdBy', '${role === "user" ? "agent" : "provider"}', 'creationSource', 'server')
        FROM n`);
    // The first read of finalizeAppOwnedSubagent (Orchestrator.ts), unchanged.
    const read = store.getThreadRecords(
      threadId,
      ["runs", "messages", "subagents", "providerThreads", "providerTurns", "attempts"],
      { messageRoles: ["user"] },
    );
    const statements = [];
    const records = yield* read.pipe(
      Effect.provideService(Statement.CurrentTransformer, (statement) => {
        statements.push(statement.compile());
        return Effect.succeed(statement);
      }),
    );
    const times = [];
    for (let i = 0; i < 5; i++) {
      const start = performance.now();
      yield* read;
      times.push(performance.now() - start);
    }
    const copyPath = `file:/finalize-read-${runCount}?vfs=memdb`;
    const copy = new NodeSqlite.DatabaseSync(copyPath);
    yield* sql`VACUUM INTO ${copyPath}`;
    const steps = statements.reduce((sum, [query, params]) => {
      const statement = copy.prepare(query);
      statement.all(...params);
      return sum + statement.stat("vmStep");
    }, 0);
    copy.close();
    return {
      runs: records.runs.length, attempts: records.attempts.length, providerTurns: records.providerTurns.length,
      messages: records.messages.length, steps, ms: times.sort((a, b) => a - b)[2],
    };
  }).pipe(Effect.provide(ProjectionStore.layer.pipe(Layer.provideMerge(SqlitePersistence.layerMemory))));

console.log("child runs | runs, attempts, provider turns, user messages decoded | VM steps | median ms (5 reads)");
for (const runCount of [100, 1_000, 3_000, 9_000]) {
  const r = await Effect.runPromise(measure(runCount));
  console.log(`${runCount} | ${r.runs}, ${r.attempts}, ${r.providerTurns}, ${r.messages} | ${r.steps} | ${r.ms.toFixed(1)}`);
}

Expected behavior

Finalizing reads what its decisions use: the child's unfinished runs, the run the task reports from, and the runs of the
tasks it may settle. #34551 already did this for the parent's side (Orchestrator.ts:14974-14977).

Actual behavior

The script's output on main @ 22f006c. Rows and VM steps are exact and the same on macOS arm64, Linux arm64 and
Linux x86_64; read times are the range of three runs in a Linux arm64 container (Node 26.13.1) on a busy machine.

Child runs Rows decoded (runs, attempts, provider turns, user messages) SQLite VM steps Read
100 400 6,278 7-10 ms
1,000 4,000 61,178 41-52 ms
3,000 12,000 183,178 127-137 ms
9,000 36,000 549,178 391-429 ms

That is one read; a task pays it at least three times, each while the parent's lock is held, so the parent's sends,
Stop and run-end reactions wait behind it.

One whole finalize call in an in-process bench (real ProjectionStore on SQLite, Node 26.13.1, a child with 9,002 runs,
8,001 provider turns and 9,003 messages), measured on a build of main from 2026-11-15 (the finalize code is unchanged on
22f006c). "Bounded" is a read limited to what the decisions need, checked to decide the same as the full read.

Full read (main) Bounded
Finalize wall time 586 ms 18 ms
Decode 482 ms 4 ms
Run rows read 18,004 57
Provider turn rows read 8,001 1
Message rows read 18,005 22
Heap peak growth 127 MB 5 MB

Impact

Major degradation or frequent failure

Every task a long-lived worker finishes makes the next finalize slower, and each finalize holds the parent's lock.

Version or commit

main @ 22f006c

Environment

Server-side; any OS and provider.

Workaround

Start a fresh delegated thread instead of reusing a long-lived one.

Related

We have a bounded read with a test that its decisions match the full read; happy to port it and open a PR, or a
maintainer can take it.

Fixed in the performance fork since v0.1.9-perf.9 (2026-11-16).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    vouch:trustedAuthor is trusted by repo permissions or the VOUCHED list.

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions