Skip to content

fix(reports): kill stalled side-branch git with createGitWithBlockTimeout - #1268

Merged
jeff-r2026 merged 9 commits into
Tencent:mainfrom
SaulMoro:fix/reports-git-timeout
Oct 22, 2026
Merged

jeff-r2026 merged 9 commits into
Tencent:mainfrom
SaulMoro:fix/reports-git-timeout

Conversation

@SaulMoro

@SaulMoro SaulMoro commented Oct 21, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Git on the teamai-reports and teamai-learnings side branches had no spawn timeout. Only teamai init pushes used the block-timeout factory. When the remote accepts the connection and never answers, updateReports never returns and keeps the per-machine reports lock. Every later hook, stats or run-record write on that machine then gets busy until the process dies.

The side-branch code now runs all its git through createGitWithBlockTimeout, with the same 30s no-output timeout and TEAMAI_INIT_PUSH_TIMEOUT_MS override that init pushes use. A killed git surfaces as the existing { status: 'failed' }, and the lock is released. A push killed by the timeout ends the retry loop, because fetch and rebase cannot fix a remote that stopped answering. The side-branch fetch passes --progress: on a pipe git is silent until it finishes, so without it a slow fetch of a large branch would be killed while it still moves data.

The learnings publish (learnings-publish.ts) fetched origin's learnings branch through plain createGit while it held the sync lock, so a stalled origin hung teamai contribute and teamai pull and kept the sync lock. It now uses createGitWithBlockTimeout too, and no longer imports plain createGit.

The block timeout kills git alone. Its git-remote-http helper kept the stalled connection open, so the CLI could not exit until the server dropped it. Block-timed git now also sets http.lowSpeedLimit=1 and http.lowSpeedTime to the block timeout plus one second. curl then aborts the stalled transfer and the helper exits on its own. The extra second keeps the block kill first, so a timed-out push still fails with the timeout error that stops the retry loop.

Type of Change

  • Bug fix (non-breaking change that fixes an issue)
  • New feature (non-breaking change that adds functionality)
  • Breaking change (fix or feature causing existing behavior to change)
  • Documentation only
  • Refactor / internal cleanup

Test Plan

  • npx tsc --noEmit passes
  • npm run lint passes
  • npx vitest run: 11485 passed, 21 skipped
  • New test in git-kind-reports.test.ts: points the clone at an HTTP server that never answers, with a 1s timeout. On main it times out at 30s. Without the retry stop it takes 10.7s (five stalled push and fetch rounds). With this branch it returns failed with a block-timeout reason within the test's 6s bound (the whole test takes about 4s), and the next write is not busy.
  • New test in learnings-import-remnants.test.ts: a publish with an older import remnant against the same kind of server. Before the learnings change it times out at 30s. With it the publish returns in about 3s, the remnant stays for a later run, and the sync lock is released.
  • New test in git-kind-reports.test.ts: after the failed write, every connection to the stalled server closes. Without the low-speed limit 2 connections are still open when the wait ends. With it they close within about 8s.

Real CLI, built from this branch and from main, sandbox HOME, local bare origin:

1. teamai session save --push, normal origin
main    ✔ Pushed: sessions/alice/2026-10.md, on origin teamai-reports
branch  ✔ Pushed: sessions/alice/2026-10.md, on origin teamai-reports

2. same, origin replaced by an HTTP server that never answers, TEAMAI_INIT_PUSH_TIMEOUT_MS=1000
main    reports lock held at 5s, still held when the 90s cap killed the process
branch  reports lock released by 5s

Run 1 is the regular reports write.

Head with --progress on both side-branch pushes: teamai init on a sandbox HOME with a local bare origin published teamai-reports through both pushes (Initialize reports branch by the initial -u push, Register member: alice by commitAndPushReports). On a pipe, a 20 MB git push printed 0 progress lines without --progress and 4 with it, so without the flag a healthy upload is as silent as a stalled one. Agents and providers other than this git origin are left to CI.

Process exit and the learnings publish, at head fe2b5e1 and at the same branch without each change, with TEAMAI_INIT_PUSH_TIMEOUT_MS=1000 and a 60s cap. "Status" is a server that sends only a status line, and "chunk" is one that sends headers and one body chunk, each then stalling:

3. teamai session save --push, stalled origin
without   write fails at 2.2s, process alive until the 60s cap, 4 git helpers still on the socket
status    write fails at 2.3s, process exits 0 at 3.3s, no git process left
chunk     write fails at 2.3s, process exits 0 at 8.3s, no git process left

4. teamai contribute with an older import remnant in the learnings checkout, stalled origin
without   sync lock held at 5s, process killed by the 60s cap, sync lock left behind
head      "Saved locally (block timeout reached)" at 4.5s, exits 0 at 5.4s, sync lock released, remnant kept, no git process left

Before both stalled runs, a normal session save --push and a normal contribute pushed to the local origin.

Related Issues

Found while designing #1084, which writes run snapshots through the same reports branch.

Notes for Reviewers

  • The first commit only renames createGitForInitPush and initPushBlockTimeoutMs to createGitWithBlockTimeout and gitBlockTimeoutMs, since they no longer serve init alone. The env var keeps its name, so an existing override still works.
  • branch-worktree.ts and learnings-publish.ts no longer import plain createGit, so a git call added there gets the timeout by default. createGitWithBlockTimeout now shares createGit's output handler, so silent commands skip simple-git's extra 50ms wait.
  • curl averages the transfer speed over about six seconds. When the server sent some bytes before it stalled, the helper exits up to about six seconds after the kill (the "chunk" run). At the default 30s timeout, the helper ends 31s to about 37s after the stall begins. A transfer that still moves at least 1 byte per second is not cut.
  • The low-speed limit applies to HTTP(S) remotes only. SSH remotes are out of scope here and were not tested.
  • Not fixed here. A stalled origin still holds the lock for one stalled fetch plus one stalled push, about 60s at the default timeout. teamai session save --push prints "Nothing new to push." for any unpublished result, a failed one included. That message is older than this change.

@github-actions

Copy link
Copy Markdown
  • [P2 non-blocking] Force progress output for pushes too — src/utils/branch-worktree.ts:610: The new timeout also applies to pushes, but neither the regular nor initial push passes --progress. Git suppresses transfer progress when stderr is piped, so a healthy upload taking over 30 seconds without other output can now be killed and reported as failed. Add --progress to both pushes, matching the fetch change.

The PR description documents sufficient testing: built real-CLI runs cover both normal publication and lock release against a stalled origin. Additional provider/agent coverage is explicitly deferred to CI.

No earlier findings were supplied. Nothing from the PR was run, built, or installed.

On a pipe git push prints nothing until it finishes, so a healthy upload longer than the timeout was killed as stalled. Both pushes now pass --progress, as the fetch does.
The learnings publish fetched origin's branch through plain git while holding the sync lock, so a stalled origin held the lock and the CLI forever. Every git call in the publish now uses the block-timeout factory.
@github-actions

Copy link
Copy Markdown

No new findings in the scoped PR diff.

  • Resolved: The earlier [P2 non-blocking] finding is fixed: both regular and initial side-branch pushes now pass --progress.
  • Testing: The PR description documents sufficient built real-CLI verification, covering normal publication and lock release with a stalled origin. Additional provider/agent coverage is explicitly deferred to CI.

Nothing from the PR was run, built, or installed.

The block timeout kills git alone, and its HTTP helper kept the stalled connection open, so the CLI could not exit until the server dropped it. Block-timed git now sets curl's low-speed limit a second past the block timeout, which ends the helper on its own.
@github-actions

Copy link
Copy Markdown

No new findings in the scoped PR diff.

  • Resolved: The earlier [P2 non-blocking] finding is fixed: both regular and initial side-branch pushes now pass --progress.
  • Testing: The PR description documents sufficient built real-CLI verification, including normal publication, stalled-origin lock release, process exit, and learnings publication. Additional provider/agent coverage is explicitly deferred to CI.

Nothing from the PR was run, built, or installed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants