Repository navigation
fix(reports): kill stalled side-branch git with createGitWithBlockTimeout - #1268
Merged
Merged
Conversation
The PR description documents sufficient testing: built real-CLI runs cover both normal publication and lock release against a stalled origin. Additional provider/agent coverage is explicitly deferred to CI. No earlier findings were supplied. Nothing from the PR was run, built, or installed. |
On a pipe git push prints nothing until it finishes, so a healthy upload longer than the timeout was killed as stalled. Both pushes now pass --progress, as the fetch does.
The learnings publish fetched origin's branch through plain git while holding the sync lock, so a stalled origin held the lock and the CLI forever. Every git call in the publish now uses the block-timeout factory.
|
No new findings in the scoped PR diff.
Nothing from the PR was run, built, or installed. |
The block timeout kills git alone, and its HTTP helper kept the stalled connection open, so the CLI could not exit until the server dropped it. Block-timed git now sets curl's low-speed limit a second past the block timeout, which ends the helper on its own.
|
No new findings in the scoped PR diff.
Nothing from the PR was run, built, or installed. |
8 of 9 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Git on the
teamai-reportsandteamai-learningsside branches had no spawn timeout. Onlyteamai initpushes used the block-timeout factory. When the remote accepts the connection and never answers,updateReportsnever returns and keeps the per-machine reports lock. Every later hook, stats or run-record write on that machine then getsbusyuntil the process dies.The side-branch code now runs all its git through
createGitWithBlockTimeout, with the same 30s no-output timeout andTEAMAI_INIT_PUSH_TIMEOUT_MSoverride that init pushes use. A killed git surfaces as the existing{ status: 'failed' }, and the lock is released. A push killed by the timeout ends the retry loop, because fetch and rebase cannot fix a remote that stopped answering. The side-branch fetch passes--progress: on a pipe git is silent until it finishes, so without it a slow fetch of a large branch would be killed while it still moves data.The learnings publish (
learnings-publish.ts) fetched origin's learnings branch through plaincreateGitwhile it held the sync lock, so a stalled origin hungteamai contributeandteamai pulland kept the sync lock. It now usescreateGitWithBlockTimeouttoo, and no longer imports plaincreateGit.The block timeout kills
gitalone. Itsgit-remote-httphelper kept the stalled connection open, so the CLI could not exit until the server dropped it. Block-timed git now also setshttp.lowSpeedLimit=1andhttp.lowSpeedTimeto the block timeout plus one second. curl then aborts the stalled transfer and the helper exits on its own. The extra second keeps the block kill first, so a timed-out push still fails with the timeout error that stops the retry loop.Type of Change
Test Plan
npx tsc --noEmitpassesnpm run lintpassesnpx vitest run: 11485 passed, 21 skippedgit-kind-reports.test.ts: points the clone at an HTTP server that never answers, with a 1s timeout. On main it times out at 30s. Without the retry stop it takes 10.7s (five stalled push and fetch rounds). With this branch it returnsfailedwith a block-timeout reason within the test's 6s bound (the whole test takes about 4s), and the next write is notbusy.learnings-import-remnants.test.ts: a publish with an older import remnant against the same kind of server. Before the learnings change it times out at 30s. With it the publish returns in about 3s, the remnant stays for a later run, and the sync lock is released.git-kind-reports.test.ts: after the failed write, every connection to the stalled server closes. Without the low-speed limit 2 connections are still open when the wait ends. With it they close within about 8s.Real CLI, built from this branch and from main, sandbox
HOME, local bare origin:Run 1 is the regular reports write.
Head with
--progresson both side-branch pushes:teamai initon a sandboxHOMEwith a local bare origin publishedteamai-reportsthrough both pushes (Initialize reports branchby the initial-upush,Register member: alicebycommitAndPushReports). On a pipe, a 20 MBgit pushprinted 0 progress lines without--progressand 4 with it, so without the flag a healthy upload is as silent as a stalled one. Agents and providers other than this git origin are left to CI.Process exit and the learnings publish, at head fe2b5e1 and at the same branch without each change, with
TEAMAI_INIT_PUSH_TIMEOUT_MS=1000and a 60s cap. "Status" is a server that sends only a status line, and "chunk" is one that sends headers and one body chunk, each then stalling:Before both stalled runs, a normal
session save --pushand a normalcontributepushed to the local origin.Related Issues
Found while designing #1084, which writes run snapshots through the same reports branch.
Notes for Reviewers
createGitForInitPushandinitPushBlockTimeoutMstocreateGitWithBlockTimeoutandgitBlockTimeoutMs, since they no longer serve init alone. The env var keeps its name, so an existing override still works.branch-worktree.tsandlearnings-publish.tsno longer import plaincreateGit, so a git call added there gets the timeout by default.createGitWithBlockTimeoutnow sharescreateGit's output handler, so silent commands skip simple-git's extra 50ms wait.teamai session save --pushprints "Nothing new to push." for any unpublished result, a failed one included. That message is older than this change.