Skip to content

fix(server): checkpoints keep big files staged under batch fsync - #39927

Open
juliusmarminge wants to merge 2 commits into
mainfrom
factory/issue-39913
Open

juliusmarminge wants to merge 2 commits into
mainfrom
factory/issue-39913

Conversation

@juliusmarminge

Copy link
Copy Markdown
Member

On macOS and Windows, a checkpoint could be lost when a turn added a small file plus a file larger than Git's big-file threshold (512 MiB by default, such as a video). Under the batch fsync mode checkpoints use there, Git 2.43–2.56.1 drops the big file's pack, and git add exits 128. Checkpoint staging now retries a failed batch-fsync add once with per-object fsync. The existing embedded-repository recovery also stages with per-object fsync, so it can no longer hit the same Git bug. Linux is unchanged.

Fixes #39913

Reproduction on main

Commit origin/main a1b9e1c, on Linux. (1) Raw Git, with the system Git 2.43.0 and with Git 2.56.1 built from source. The reporter's exact steps passed in every variant, because only the big file is new. Adding a small new file that sorts first reproduces it: git init; git commit --allow-empty -m init; echo small > a.txt; head -c 2097152 /dev/urandom > big.bin; mkdir .git/objects/tmp_objdir-priv; GIT_OBJECT_DIRECTORY=$PWD/.git/objects/tmp_objdir-priv GIT_INDEX_FILE=../idx git -c core.bigFileThreshold=1m -c core.fsync=objects -c core.fsyncMethod=batch add -A exits 128 with fatal: unable to rename temporary file to '.../tmp_objdir-priv/pack/pack-<sha>.pack': No such file or directory. The same happens without the private object folder. With core.fsyncMethod=fsync it succeeds in every case. (2) Through T3: in GitVcsDriver.test.ts, captureCheckpoint was run with HostProcess.Platform set to darwin or win32 on a repo with a small new file, a 2 MiB file and bigFileThreshold=1m. The logged Git commands show the first add -A (with core.fsyncMethod=batch) exiting 128. The recovery retry add --ignore-errors -A then succeeded, which hides the bug in small cases. That retry gives up after 5 s or when more than 64 untracked embedded repositories are listed. With 65 committed nested repositories the checkpoint fails with VcsProcessExitError exit 128 on darwin and win32, and passes on linux as a control. (3) Realistic size: with a 600 MiB file plus a small new file, the first add exited 128 after about 13 s and the second took about 14 s. That is well over the 5 s recovery timeout, so a real checkpoint with a file over the default 512 MiB threshold is lost. Not run on real macOS or Windows; batch mode is a Git setting that behaves the same on Linux.

Evidence

Being captured: before and after in the real app. This section is filled in before the pull request can merge.

Tests

  • vp test run src/vcs/GitVcsDriver.test.ts -t "captures a big file" before the fix: 2 failed (darwin, win32), exit 128, which reproduces the bug
  • vp test run src/vcs/GitVcsDriver.test.ts after the fix: 114 passed, 1 failed. The failure is "lists the fetch and push URLs of a partial clone remote", caused by this VM's global url.insteadOf rewrite and unrelated to this change
  • GIT_CONFIG_GLOBAL=/dev/null vp test run src/vcs/GitVcsDriver.test.ts: 115 passed
  • vp run check:changed: format, knip and typecheck passed; lint failed on prefer-catch-tags (now fixed and clean under vp lint); tests failed only on the insteadOf test above
  • commit hook: vp fmt, vp lint, knip:check passed

Not in this change

Who did the work

  • Written by Claude Opus 5.5, in Claude Code
  • Reviewed by GPT-6.1 Sol, in Codex, and by Claude Opus 5.5, each on its own
  • Proof captured by GPT-6.1 Sol and checked by Claude Opus 5.5

Run by the t3code factory in T3 Code.

juliusmarminge and others added 2 commits November 20, 2026 21:59
…er small new files

Reproduces #39913: with core.fsyncMethod=batch (darwin, win32), git add
fails with exit 128 when a loose object is written before a file over
core.bigFileThreshold is streamed into a pack.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
On macOS and Windows checkpoint staging uses core.fsyncMethod=batch, and
Git 2.43 through 2.56.1 loses a file over core.bigFileThreshold when a
smaller loose object was written earlier in the same add (exit 128).
Retry a failed batch add once with per-object fsync, and stage the
embedded-repository recovery with per-object fsync too.

Fixes #39913

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions github-actions Bot added vouch:trusted Author is trusted by repo permissions or the VOUCHED list. size:S 10-29 changed lines (additions + deletions). labels Nov 20, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Thread transfer impact

✅ Thread transfer remains within every enforced ceiling.

Provider Metric Main baseline This PR Impact PR ceiling
Codex Total thread wire 5.0 KiB 5.0 KiB 0 B (0.0%) 6.8 KiB ✅
Codex Thread snapshot wire 3.8 KiB 3.8 KiB 0 B (0.0%) 4.9 KiB ✅
Codex Live turn WebSocket wire 1.2 KiB 1.2 KiB 0 B (0.0%) 2.0 KiB ✅
Codex Live turn WebSocket decoded 21.5 KiB 21.5 KiB 0 B (0.0%) 24.4 KiB ✅
Codex Live turn messages 1 1 0 (0.0%) 8 ✅
Claude Total thread wire 5.0 KiB 5.0 KiB 0 B (0.0%) 6.8 KiB ✅
Claude Thread snapshot wire 3.8 KiB 3.8 KiB 0 B (0.0%) 4.9 KiB ✅
Claude Live turn WebSocket wire 1.2 KiB 1.2 KiB 0 B (0.0%) 2.0 KiB ✅
Claude Live turn WebSocket decoded 21.9 KiB 21.9 KiB 0 B (0.0%) 24.4 KiB ✅
Claude Live turn messages 1 1 0 (0.0%) 8 ✅

Other scenarios

All 64 scenario measurements match main.

All 64 scenario measurements
Provider Scenario Metric Main baseline This PR Impact PR ceiling
Codex short turn resume WebSocket decoded 10.5 KiB 10.5 KiB 0 B (0.0%) 12.7 KiB ✅
Codex short turn resume WebSocket messages 2 2 0 (0.0%) 2 ✅
Codex long turn running, cold open HTTP decoded 281.7 KiB 281.7 KiB 0 B (0.0%) 302.7 KiB ✅
Codex long turn running, cold open rows 75 75 0 (0.0%) 75 ✅
Codex long turn running, history page HTTP decoded 92.4 KiB 92.4 KiB 0 B (0.0%) 107.4 KiB ✅
Codex long turn running, history page rows 75 75 0 (0.0%) 75 ✅
Codex long turn finished, cold open HTTP decoded 381.5 KiB 381.5 KiB 0 B (0.0%) 410.2 KiB ✅
Codex long turn finished, cold open rows 151 151 0 (0.0%) 151 ✅
Codex long turn finished, socket cold open WebSocket decoded 381.6 KiB 381.6 KiB 0 B (0.0%) 410.2 KiB ✅
Codex long turn finished, socket cold open WebSocket messages 2 2 0 (0.0%) 2 ✅
Codex long turn finished, socket cold open rows 151 151 0 (0.0%) 151 ✅
Codex long turn finished, work page HTTP decoded 186.2 KiB 186.2 KiB 0 B (0.0%) 200.2 KiB ✅
Codex long turn finished, work page rows 150 150 0 (0.0%) 150 ✅
Codex idle thread resume WebSocket decoded 65 B 65 B 0 B (0.0%) 200 B ✅
Codex idle thread resume WebSocket messages 1 1 0 (0.0%) 1 ✅
Codex fleet shell snapshot HTTP decoded 42.4 KiB 42.4 KiB 0 B (0.0%) 53.7 KiB ✅
Codex fleet shell snapshot rows 27 27 0 (0.0%) 27 ✅
Codex fleet shell subscribe WebSocket decoded 65 B 65 B 0 B (0.0%) 200 B ✅
Codex fleet shell subscribe WebSocket messages 1 1 0 (0.0%) 1 ✅
Codex fleet shell resume check HTTP decoded 18 B 18 B 0 B (0.0%) 100 B ✅
Codex fleet shell resume check rows 0 0 0 0 ✅
Codex stale shell reload HTTP decoded 42.4 KiB 42.4 KiB 0 B (0.0%) 53.7 KiB ✅
Codex stale shell reload rows 27 27 0 (0.0%) 27 ✅
Codex stale shell resume WebSocket decoded 140 B 140 B 0 B (0.0%) 400 B ✅
Codex stale shell resume WebSocket messages 2 2 0 (0.0%) 2 ✅
Codex stale todo list cold open HTTP decoded 72.4 KiB 72.4 KiB 0 B (0.0%) 83.0 KiB ✅
Codex stale todo list cold open rows 75 75 0 (0.0%) 75 ✅
Codex stale todo list cold open nodes 2 2 0 (0.0%) 4 ✅
Codex stale todo list cold open subagents 2 2 0 (0.0%) 4 ✅
Codex stale todo list cold open plans 1 1 0 (0.0%) 2 ✅
Codex stale todo list cold open providerTurns 1 1 0 (0.0%) 2 ✅
Codex stale todo list cold open checkpoints 1 1 0 (0.0%) 2 ✅
Claude short turn resume WebSocket decoded 10.6 KiB 10.6 KiB 0 B (0.0%) 12.7 KiB ✅
Claude short turn resume WebSocket messages 2 2 0 (0.0%) 2 ✅
Claude long turn running, cold open HTTP decoded 283.3 KiB 283.3 KiB 0 B (0.0%) 302.7 KiB ✅
Claude long turn running, cold open rows 75 75 0 (0.0%) 75 ✅
Claude long turn running, history page HTTP decoded 93.4 KiB 93.4 KiB 0 B (0.0%) 107.4 KiB ✅
Claude long turn running, history page rows 75 75 0 (0.0%) 75 ✅
Claude long turn finished, cold open HTTP decoded 384.2 KiB 384.2 KiB 0 B (0.0%) 410.2 KiB ✅
Claude long turn finished, cold open rows 151 151 0 (0.0%) 151 ✅
Claude long turn finished, socket cold open WebSocket decoded 384.3 KiB 384.3 KiB 0 B (0.0%) 410.2 KiB ✅
Claude long turn finished, socket cold open WebSocket messages 2 2 0 (0.0%) 2 ✅
Claude long turn finished, socket cold open rows 151 151 0 (0.0%) 151 ✅
Claude long turn finished, work page HTTP decoded 188.1 KiB 188.1 KiB 0 B (0.0%) 200.2 KiB ✅
Claude long turn finished, work page rows 150 150 0 (0.0%) 150 ✅
Claude idle thread resume WebSocket decoded 66 B 66 B 0 B (0.0%) 200 B ✅
Claude idle thread resume WebSocket messages 1 1 0 (0.0%) 1 ✅
Claude fleet shell snapshot HTTP decoded 43.0 KiB 43.0 KiB 0 B (0.0%) 53.7 KiB ✅
Claude fleet shell snapshot rows 27 27 0 (0.0%) 27 ✅
Claude fleet shell subscribe WebSocket decoded 66 B 66 B 0 B (0.0%) 200 B ✅
Claude fleet shell subscribe WebSocket messages 1 1 0 (0.0%) 1 ✅
Claude fleet shell resume check HTTP decoded 18 B 18 B 0 B (0.0%) 100 B ✅
Claude fleet shell resume check rows 0 0 0 0 ✅
Claude stale shell reload HTTP decoded 43.0 KiB 43.0 KiB 0 B (0.0%) 53.7 KiB ✅
Claude stale shell reload rows 27 27 0 (0.0%) 27 ✅
Claude stale shell resume WebSocket decoded 140 B 140 B 0 B (0.0%) 400 B ✅
Claude stale shell resume WebSocket messages 2 2 0 (0.0%) 2 ✅
Claude stale todo list cold open HTTP decoded 73.4 KiB 73.4 KiB 0 B (0.0%) 83.0 KiB ✅
Claude stale todo list cold open rows 75 75 0 (0.0%) 75 ✅
Claude stale todo list cold open nodes 2 2 0 (0.0%) 4 ✅
Claude stale todo list cold open subagents 2 2 0 (0.0%) 4 ✅
Claude stale todo list cold open plans 1 1 0 (0.0%) 2 ✅
Claude stale todo list cold open providerTurns 1 1 0 (0.0%) 2 ✅
Claude stale todo list cold open checkpoints 1 1 0 (0.0%) 2 ✅

Baseline: a1b9e1c · PR result: 2b07bda · Source CI: success

Scenario and decoded snapshot size

10 historical turns, 5 command tools per turn, 878.9 KiB retained MCP result per historical turn, and a 1.05 MiB retained result in the measured turn.

  • Codex decoded thread snapshot: 108.5 KiB
  • Claude decoded thread snapshot: 108.8 KiB

Updated in place by a trusted workflow. PR artifacts are strictly validated and never executed.

@coderabbitai

coderabbitai Bot commented Nov 20, 2026

Copy link
Copy Markdown
Contributor

Explain Change →

No actionable comments were generated in the recent review.

Walkthrough

Checkpoint writes now use platform-specific fsync settings. On macOS and Windows, staging retries with per-object fsync after a VcsProcessExitError. Nested-repository recovery staging also uses per-object fsync.

Changes: Checkpoint staging

Layer / File(s) Summary
Configure checkpoint fsync settings
apps/server/src/vcs/GitVcsDriver.ts
Durable-write settings separate per-object fsync from batch fsync. stageFiles accepts an fsync argument list and defaults to the durable-write settings.
Add staging fallback and coverage
apps/server/src/vcs/GitVcsDriver.ts, apps/server/src/vcs/GitVcsDriver.test.ts
On macOS and Windows, staging retries with per-object fsync after a VcsProcessExitError. Nested-repository recovery staging uses per-object fsync. The test captures checkpoints on macOS and Windows with a 2 MiB file and 65 nested repository fixtures.

Estimated code review effort: 2 (Simple) | ~10 minutes

Change: Bug fix · Severity of issue fixed: Medium

Possibly related issues

Suggested reviewers: stienswout, amv146, muhammad-auon-marham

Merge Risk: Minimal · up to 2b07b

Checkpoint staging now falls back to a safer fsync mode when a large file triggers a batch-fsync failure on macOS or Windows. The change is narrow, the main path is unchanged, and no merge-blocking risk was identified.

Pre-merge checks: Passed 4

Check name Status Explanation
Title check Passed The title names the main change: checkpoint staging for big files under batch fsync is fixed. It follows a conventional commit format. It is specific to the server fix described in the objectives and …
Description check Passed The description covers the problem (checkpoint loss on macOS/Windows with a large file under batch fsync), the change (batch fsync retry with per-object fsync and recovery path), reproduction steps, a…
Linked Issues check Passed Issue #39913 asks that checkpoint staging stop failing on macOS when a file exceeds core.bigFileThreshold under batch fsync with a private GIT_OBJECT_DIRECTORY. The PR adds a single retry of a failed …
Out of Scope Changes check Passed The changes fall in two files: GitVcsDriver.ts (checkpoint staging retry and per-object fsync for embedded-repository recovery) and GitVcsDriver.test.ts (the new darwin/win32 big-file checkpoint test)…
Full details: Title check

Explanation

The title names the main change: checkpoint staging for big files under batch fsync is fixed. It follows a conventional commit format. It is specific to the server fix described in the objectives and summary, so it matches the changeset.

Full details: Description check

Explanation

The description covers the problem (checkpoint loss on macOS/Windows with a large file under batch fsync), the change (batch fsync retry with per-object fsync and recovery path), reproduction steps, and a verification section with test results. It also notes scope and linked issue #39913. The Scope section does not state a maintainer request or a full bug-scope rationale beyond the linked issue, and the Evidence section is a placeholder that says it will be filled in before merge. The template asks for before/after evidence for UI changes only, so this is not a blocker. The description is mostly complete and follows the template structure.

Full details: Linked Issues check

Explanation

Issue #39913 asks that checkpoint staging stop failing on macOS when a file exceeds core.bigFileThreshold under batch fsync with a private GIT_OBJECT_DIRECTORY. The PR adds a single retry of a failed batch-fsync add with per-object fsync, limited to darwin and win32 VcsProcessExitError cases (GitVcsDriver.ts, per the change summary). It also makes the embedded-repository recovery staging use per-object fsync. The new parameterized test in GitVcsDriver.test.ts covers darwin and win32 with a small file, a large file, and 65 nested repositories, and checks the staged object hash. The PR reports that this test failed before the fix (exit 128) and passed after it under GIT_CONFIG_GLOBAL=/dev/null (115 passed). Limits: the GitVcsDriver.ts diff could not be read from the checkout, so this verdict rests on the change summary, the PR description, and the reported test runs. Real macOS and Windows runs were not performed. The issue's requirement is coding-only, so no non-coding task applies.

Full details: Out of Scope Changes check

Explanation

The changes fall in two files: GitVcsDriver.ts (checkpoint staging retry and per-object fsync for embedded-repository recovery) and GitVcsDriver.test.ts (the new darwin/win32 big-file checkpoint test). Both connect directly to the batch-fsync staging failure in #39913. The PR lists bulk-fsync temp folder cleanup (#26227) and the default batch-fsync first attempt as excluded, and the summary shows no code for either. The PR also mentions a lint fix (prefer-catch-tags) and a test-environment note; the summary does not show these as separate unrelated changes. The full diff could not be read, so this verdict relies on the change summary.

Finishing Touches

Generate unit tests (beta)

  • Commit to this branch
  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
Recent review info

Run configuration

Commits

Reviewing files that changed from the base of the PR and between a1b9e1c and 2b07bda.

Files selected for processing: 2

  • apps/server/src/vcs/GitVcsDriver.test.ts
  • apps/server/src/vcs/GitVcsDriver.ts

Included review availability: This review used your included allowance. Your plan provides up to 10 included reviews per hour; 2 remain after this review.

@juliusmarminge

Copy link
Copy Markdown
Member Author

The automated pipeline stopped on this pull request and will not merge it:

the fix is written and reviewed, but proving it needs something no worker has:

  • Permanent blocker: native macOS is required to pursue the reported macOS failure after the Windows diagnostic did not reproduce it; this worker has no permitted native macOS environment.
  • Checkpoint behavior on base and head remains unverified in the real app. Neither false verdict means the fix failed; the comparison was not completed.
  • The default 512 MiB threshold and realistic 600 MiB reproduction were not exercised.

It stays open for a person to finish or close.

@navidnadali

Copy link
Copy Markdown
Contributor

Note

Written by Claude Opus 5.5 on behalf of Navid

The pipeline stopped this for lack of native macOS, so here is a run on one: macOS 27.0.1 (arm64), Apple Git 2.54.0 (Apple Git-157), Node 26.13.1, no core.* or fsync settings in the system or global Git config. Base: main e2035934a6 with gh pr diff 39927 (head 2b07bda19f) applied; it applies with line offsets only.

This PR's rows (vp test run src/vcs/GitVcsDriver.test.ts -t "captures a big file" in apps/server):

  • main plus only the new test rows: 2 failed, checkpoint on darwin captures a big file staged after small new files and checkpoint on win32 captures a big file staged after small new files, both VcsProcessExitError from captureCheckpoint, git exited with 128.
  • with the fix: both pass.

The win32 row only sets T3's platform; Git is still Apple Git.

Raw Git on the same Mac, outside T3 (repo with one empty commit, core.bigFileThreshold=1m, a 2 MiB random file, git -c core.fsync=objects -c core.fsyncMethod=batch add -A, 3 runs each):

  • a small new file sorting before the big one, private GIT_OBJECT_DIRECTORY (this PR's steps): 3/3 exit 128, fatal: unable to rename temporary file to '.../tmp_objdir-priv/pack/pack-<sha>.pack'. Without the private folder: 3/3 exit 128. With core.fsyncMethod=fsync: 0/3.
  • Checkpoint staging fails on macOS for files over the big-file threshold #39913's exact steps, where no smaller new file sorts before big.bin (only the index's own idx.lock, after it): 0/3. Here too, as in the PR's Linux run, the add failed only when a smaller new file came first.
  • Git's default threshold, a small file plus a 600 MiB file into a private object folder: batch exit 128, per-object fsync exit 0 (one run each).

Whole file (vp test run src/vcs/GitVcsDriver.test.ts): main 112 passed, 3 failed; with this PR 114 passed, 3 failed. The same three fail on both, checkpoint bulk staging cleans operation-owned packs after interrupt|timeout|success and can retry, with NotFound: FileSystem.readDirectory (.../.git/objects/tmp_objdir-t3-checkpoint-*/pack). These are the failures #39913 describes. They come from the test: it looks for the operation's unfinished pack only in <private objects>/pack, and on a Mac, where staging uses batch fsync, it is not there (Linux never adds batch, so CI does not reach this). #37404's test change, which searches the whole private object folder, fixes them: with it on top of this PR the file passes 117/117, and on main the three rows pass with it too.

To reproduce from a checkout of main e2035934a6:

gh pr diff 39927 | git apply --include=apps/server/src/vcs/GitVcsDriver.test.ts
(cd apps/server && vp test run src/vcs/GitVcsDriver.test.ts -t "captures a big file")  # 2 failed, exit 128
gh pr diff 39927 | git apply --include=apps/server/src/vcs/GitVcsDriver.ts
(cd apps/server && vp test run src/vcs/GitVcsDriver.test.ts -t "captures a big file")  # 2 passed
(cd apps/server && vp test run src/vcs/GitVcsDriver.test.ts)                           # 114 passed, 3 failed
gh pr diff 37404 | git apply --include=apps/server/src/vcs/GitVcsDriver.test.ts
(cd apps/server && vp test run src/vcs/GitVcsDriver.test.ts)                           # 117 passed

Made with Claude Opus 5.5 via Claude Code. Reviewed by GPT-6.1 Sol via Codex.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size:S 10-29 changed lines (additions + deletions). vouch:trusted Author is trusted by repo permissions or the VOUCHED list.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Checkpoint staging fails on macOS for files over the big-file threshold

2 participants