Skip to content

[Bug]: Grok: session/load replay leaks into the V2 ACP projection, pins "background work" forever and blocks handoff recovery (ContextHandoffBackgroundWorkError) #40319

Closed

Description

@silinin

Before submitting

  • I searched existing issues and did not find a duplicate.
  • I included enough detail to reproduce or investigate the problem.

Area

apps/server

Steps to reproduce

  1. Create a thread on Grok and grow the native session large (several MB of history, many tool calls).
  2. Switch the thread to another provider (e.g. Codex), run one turn, switch back to Grok so a delta_since_target_last_seen handoff is pending.
  3. Send a message. Turn start fails with ContextHandoffBackgroundWorkError on every retry (4 attempts), and on every later message.
  4. Wait 30 min: idle-release-deferred is logged for the provider session although nothing runs.

Expected behavior

Replayed history (_meta.isReplay: true) during session/load must not create wake/background state; recovery into a fresh native session should proceed. Also, for providers that do not report usage, a zero budget should not be derived from raw history bytes alone.

Actual behavior

A long-lived thread was switched Grok -> Codex -> Grok. Every new turn on Grok now fails before any prompt is sent with:

ProviderAdapterTurnStartError ... [cause]: ContextHandoffBackgroundWorkError: The target provider still has background work running.

Only initialize + session/load are sent to Grok; no session/prompt. The thread is unusable on Grok; provider-session.detach or a restart does not help because the next session/load replays the same history again.

Diagnosis (from reading the 0.1.9 bundle):

  1. Handoff budget is 0. Grok does not report context usage, so contextUsageForHandoff falls back to estimating the native context from history size (~6.36 MB) which exceeds the 256k window. The pending delta handoff (~116 KB) is rejected with ContextHandoffBudgetError.
  2. loadFromProvider then takes the recovery branch (fresh native session + recovery handoff), but first checks session.hasPendingBackgroundWork, which is true -> ContextHandoffBackgroundWorkError.
  3. Why it is true on a freshly started runtime: AcpSessionRuntime filters replay (sessionUpdateIsReplay(...) -> return) only inside its own acp.handleSessionUpdate callback. But the runtime also exposes handleSessionUpdate: acp.handleSessionUpdate, and the V2 adapter subscribes projectAcpRuntimeSessionUpdate through it (wireAcpRuntimeHandlers -> targetRuntime.handleSessionUpdate(...)). That path has no isReplay check; the only guard is flavor.ignoreSessionUpdate, which the Grok flavor does not define. With enablePostSettleContinuation (Grok), replayed tool_call/agent_message_chunk updates reach bufferPostSettleWake/trackRunningBackgroundTools when activeTurn === null. For a large session the tail of the replay is processed after activeSessionId is set, so wakeBuffer / runningBackgroundTaskIds become non-empty and stay that way.
  4. Evidence that the state is stuck (not in-flight): 30 min after the failed turn, with no turns running, the idle reaper logs orchestration-v2.driver-session.idle-release-deferred (pinnedForMs: 0) for that provider session, i.e. hasPendingBackgroundWork is still true. Overnight the same session was deferred every 30 min until idle-release-pin-expired after 4h. Grok itself reports nothing running: all replayed _x.ai/session/update events carry _meta.isReplay: true, every task_backgrounded has a matching task_completed, and the final live background_tasks snapshot is [].

Small sessions do not reproduce because the replay finishes before session/load returns.

Impact

Blocks work completely (the thread cannot start any turn on Grok)

Version or commit

T3 Code 0.1.9 (server, orchestrator V2), self-hosted on Linux. Observed on 0.1.9 (runtime logs + bundle). Not re-run on 0.1.10/0.1.11. Source of tag v0.1.11 was read (not executed): the same path is unchanged. projectAcpRuntimeSessionUpdateEffect in packages/provider-acp/src/server/adapter.ts still has only the flavor.ignoreSessionUpdate guard (no isReplay check) and is attached via targetRuntime.handleSessionUpdate(...); AcpSessionRuntime filters replay only in its own acp.handleSessionUpdate callback; ProviderTurnStartService still throws ContextHandoffBackgroundWorkError when hasPendingBackgroundWork is true after the fresh-session budget check. hasPendingBackgroundWork additionally checks midTurnUnreportedCompletedTaskIds. Release notes 0.1.10/0.1.11 do not mention this.

Environment

Linux x86_64; Grok CLI 1.0.92 (grok agent --always-approve stdio), model grok-4.7-build-fast, effort xhigh. Native Grok session ~6.4 MB of history (~7k replayed notifications on session/load).

Logs or stack traces

ProviderAdapterTurnStartError ... [cause]: ContextHandoffBackgroundWorkError: The target provider still has background work running.

Only initialize + session/load are sent to Grok; no session/prompt. After 30 min idle with no turns running: orchestration-v2.driver-session.idle-release-deferred (pinnedForMs: 0). Overnight the same session was deferred every 30 min until idle-release-pin-expired after 4h.

Workaround

Detach the provider session (provider-session.detach) and clear the provider thread's nativeThreadRef so T3 creates a new native Grok session (loses native history; context must be re-supplied manually).

Suggested fix: skip sessionUpdateIsReplay(...) notifications in projectAcpRuntimeSessionUpdateEffect (or default ignoreSessionUpdate to isReplay for all ACP flavors), and/or ignore replay-derived wake state when evaluating hasPendingBackgroundWork for the handoff-recovery gate.

Related but different: #3560 (Grok resume replay floods thread, closed), #29211 (Copilot CLI session/load replay leak, closed), #25610, #20180, #29359, #32296 (send refused while background work runs), #38257 (Grok wait_all leaves subagents running).

(Report drafted with an AI agent; diagnosis from the 0.1.9 bundle and server logs.)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions