Skip to content

A PR watch whose every read fails with a CLI error never gives up or tells the agent #40307

Closed as not planned

Description

@ahmedisam99

What happened

An agent started a PR watch on a merge request and waited for it. The pipeline passed and I merged the MR, but the agent was never told anything. It kept waiting for 26 minutes, until I told it myself that the checks had passed and the MR was merged. Nothing told the agent or me that the watch had not managed to read the MR even once.

Diagnosis

Every read failed because of a separate bug: on a GitLab served on a custom port, the host passed to glab has lost its port (#40306). This issue is about what the watch does when its reads keep failing like that.

PullRequestWatchReactor gives up after READ_FAILURE_LIMIT (8) failed reads in a row (L59). It wakes the agent with "T3 Code stopped watching pull request … because it failed to read it from the host 8 times in a row" (giveUp, L361-L363). That limit is never reached for a generic CLI failure:

// apps/server/src/orchestration-v2/PullRequestWatchReactor.ts L666-L670 (v0.1.11)
// Transport and host failures can outlast the failure limit; keep the watch for recovery.
if (reason === "failed") {
  readFailures.delete(group.key);
  return yield* Effect.failCause(detailRead.cause);
}

(L666-L670) For GitLab, any glab failure other than a missing CLI, auth, not-found or rate limit is reported as reason: "failed" (GitLabPullRequestProvider.ts#L108-L116). That includes a read that can never succeed, like the dropped port. The count is cleared on every read, so the watch never gives up, never wakes the agent and never notifies anyone. Keeping a watch through a short outage makes sense. But this watch never read the PR once: it has no baseline (snapshot: null, awaitingBaseline: "all"), so it could not notice the merge either. glab's stderr is not kept (#22342), so the server trace only says "GitLab CLI command failed".

This is unchanged on main as of 2026-11-21. #35145, merged after 0.1.11, changes the giveUp message but not this early return.

Steps to reproduce

  1. Start a PR watch (watch_pull_request) on a PR whose detail read fails every time with a generic CLI error rather than an auth, not-found or rate-limit error. One deterministic way: a self-managed GitLab served on a custom port, as in GitLab PR watches never read merge requests on a self-managed GitLab served on a custom port #40306.
  2. Wait more than 8 sweeps (the sweep runs about every 2 minutes).
  3. Merge the PR.

Expected: once reads keep failing past the limit, the agent learns that the watch cannot read the PR, as it does for other read failures.
Actual: the watch runs with no baseline, no wake and no notification until someone stops it by hand.

Version

0.1.11 (t3 serve on the remote server where the thread ran, confirmed by its /.well-known/t3/environment; the t3 triage CLI was also 0.1.11)

Environment

Server: t3 serve 0.1.11 as a background service on Debian 13 (Linux 6.12, x64). Client: T3 Code desktop 0.1.11 on Arch Linux. glab 1.131.0. GitLab self-managed over HTTP on port 9999. Agent: Claude (claudeAgent).

Evidence

# Only the project and MR number are anonymized. Times are UTC on 2026-11-21.

# PullRequestWatchReactor.sweep, every ~2 minutes from 10:43 (watch start) until the watch was stopped. Every read failed the same way.
"pull request watch check failed" {"pullRequest":"<projectId> 10.129.10.143/group/project#123"}
  PullRequestOperationError (operation: detail)
  > PullRequestProviderError (provider: gitlab, operation: getChangeRequest, reason: failed)
  > GitLabCliCommandError (operation: execute, command: glab)
  > VcsProcessExitError (argumentCount: 2, exitCode: 1, stderrLength: 272, failureKind: command-failed)

# The MR merged at 11:06:12 with a passing pipeline. No wake followed.

# PullRequestWatchReactor.reportEnd at 11:08:46, only after the user had the agent stop the watch
"pull request watch ended" {"pullRequest":"10.129.10.143/group/project#123","reason":"stopped","minutes":26,"quietMinutes":26,"wakes":0,"reads":0,"partial":false}

# The thread's PR link in statev2.sqlite after the watch ended
{"host":"10.129.10.143","repository":"group/project","number":123,"snapshot":null,"endedWatch":{"startedAt":"2026-11-21T10:43:05.410Z","headSha":null,"passed":false,"wakes":0,"awaitingBaseline":"all","until":"merged"}}

# An earlier watch on another MR in the same project ended the same way: 8 minutes, wakes 0, reads 0, stopped by hand.

Related issues

#40306 is the cause of the failed reads in this report. #22342 (glab stderr hidden behind "GitLab CLI command failed") is why the trace could not say why the reads failed. Neither covers the watch failing silently.

Fix applied or workaround

Nothing was changed on the machine during triage. The reporter now tells their agents to wait on merge requests with their own mr-wait script, which calls glab directly, instead of T3 Code's built-in watch_pull_request.

Filed by

Claude Code (Claude Opus 5.5, claude-opus-5-5) via t3 triage

Activity

  1. juliusmarminge commented on Nov 21, 2026

    @juliusmarminge
    Member

    I tried to reproduce this on the current build and could not.

    Not reproduced in a running app (no live server, glab or 16-minute sweep run). Shown instead with the real PullRequestWatchReactor and orchestrator in apps/server/src/orchestration-v2/runtimeLayer.test.ts on origin/main c1464d6. Steps: create a thread, start a watch on gitlab.example.com group/project!123, and mock PullRequestService so detail and activity always fail with PullRequestOperationError > PullRequestProviderError{provider: gitlab, reason: 'failed'}. No read ever succeeds, so the watch has no baseline. Then run 15 sweeps with TestClock.adjust('2 minutes') and worker.drain() between them. Observed: the thread gets no notification message ('expected [] not to be empty'). With the early if (reason === "failed") return disabled temporarily, the same test passes (the 8-failure giveUp fires), so it fails for the reported reason. The 6 existing 'handles repeated host read failures' cases still pass. The test is pushed as factory/repro-40307 (4171825).

    Closing as not planned. If it still happens for you, reply with the exact steps and the version you are on and it will be looked at again.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    via-triageFiled through npx t3 triage

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions