Repository navigation
test+fix: red tests and fixes for upstream merge (stacked on #3) - #4
kmannislands wants to merge 7 commits into
Conversation
The conflict resolution in sender.updateProgress kept both the fork's increment under s.mu and upstream's increment under progressCurrentMu, so every packet is added to progressCurrent twice whenever progressCb is set. The final total reported to progressCb is exactly double the STAT+DATA bytes the sender put on the wire. TestSendProgressTotal wraps the sender stream, sums the Size() of each non-terminal STAT packet and each non-empty DATA packet, and asserts the last=true total equals it. It fails on the merge (205440 vs 102720) and passes on both parents (644b083 and 91a3fc4). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The merge put the fork's receiver StatusStat verboseProgressCb call
before upstream's filepath.FromSlash normalisation, so on Windows the
receiver reports StatusStat with the unix wire path ("dir/b") but
StatusReceiving/StatusReceived with the native r.files key ("dir\b").
The sender has the mirror problem: StatusStat gets the native walk
path while StatusSent gets the ToSlash'd wire path from s.files.
TestVerboseProgressPathsConsistent asserts that every StatusSent path
was also reported with StatusStat on the sender, and that the receiver
reports the same native paths for StatusStat, StatusReceiving and
StatusReceived as it does to NotifyHashed.
FromSlash/ToSlash are no-ops on Unix, so this passes on macOS/Linux
and is expected to fail only on Windows (cross-compiled with
GOOS=windows; not executed).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configuration
📒 Files selected for processing (5)
Included review availability: This review used your included allowance. Your plan provides up to 1 included review per hour; 0 remain after this review. 📝 WalkthroughWalkthroughThe CI workflow now uses updated triggers and pinned action references. Sender and receiver progress callbacks report native-format paths. Sender progress totals increment only when a progress callback is configured, with new tests for paths and packet totals. ChangesCI workflow
Transfer progress reporting
Priority: ➖ Normal Estimated code review effort: 3 (Moderate) | ~20 minutes Change: Bug fix Merge Risk: ⚪ Minimal · up to No actionable regression is established in this PR. It is mergeable after normal checks. Pre-merge checks |
|
| Check name | Status | Explanation |
|---|---|---|
| Linked Issues check | Issue #3 requires the fsutil change set that supports moby/buildkit v0.14.1 and the stacked merge order. The PR is based on the reported #3 merge commit 890baae, and the PR description requires #3 to … |
|
| Out of Scope Changes check | The source fixes address regressions attributed to the #3 upstream merge: send progress accounting, native verbose-progress paths, and CI triggers for stacked branches. The added tests verify those fi… |
|
| Description Check | Check skipped - CodeRabbit’s high-level summary is enabled. | |
| Title check | The title accurately identifies the added regression tests and fixes related to the upstream merge. It also notes the stacked relationship to issue #3. The CI updates are secondary and do not need to … |
Full details: Docstring Coverage
Explanation
Docstring coverage is 66.67% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 3 functions across 4 files. (1 skipped: 1 unsupported.)
- Fix all pre-merge checks with AI
✨ Finishing Touches 💡 2
📝 Generate docstrings 💡
- Commit to this branch
- Create a new PR
🛠️ Fix failing CI checks 💡
- Commit to this branch
- Create a new PR
🧪 Generate unit tests (beta)
- Commit to this branch
- Create a new PR
- Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.
Comment @coderabbitai help to get the list of available commands.
The upstream merge took upstream's trigger block. That dropped the fork's push trigger (earthly-main, gh_test_ci) and limited pull_request to master/main. So CI no longer ran for PRs into earthly-main, including #3 itself. This brings back the push trigger from earthly-main (644b083). It also removes the branch filter from pull_request, so the workflow runs for every PR, whatever its base. A PR into earthly-main would have been enough, but stacked PRs (like this one, based on chore-merge-v0.14.1) need CI too. We keep upstream's workflow_dispatch and schedule triggers. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
When the merge resolved the updateProgress conflict, it kept both sides. The fork increments progressCurrent under s.mu, and upstream increments it again under progressCurrentMu. So whenever progressCb is set, every STAT and DATA packet is counted twice. The callback also ran while s.mu was held. Upstream 216a08f added progressCurrentMu to avoid exactly that, because s.mu also guards s.files and the send queue. This takes upstream's version: a single increment under progressCurrentMu, and only when progressCb is set. Nothing else reads progressCurrent, so it no longer needs s.mu. Fixes TestSendProgressTotal. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Upstream now sends unix-style wire paths: the sender applies ToSlash
and the receiver applies FromSlash. The merge put the fork's
VerboseProgressCb calls around that change in a way that mixes the two
forms on Windows.
- Receiver: StatusStat was reported with p.Stat.Path before
FromSlash, so it got the wire path ("dir/b"). StatusReceiving and
StatusReceived get the native r.files key ("dir\b"). The call also
ran before the "unrepresentable path" check, so a rejected path
was still reported as stat'ed.
- Sender: walk reports StatusStat with the native walk path, but
sendFile reported StatusSent with h.path, which is the ToSlash'd
value stored in s.files.
Each side now reports native paths:
- Receiver: StatusStat moves to after normalisation and the
unrepresentable-path check, and reports the normalised path.
- Sender: StatusSent reports filepath.FromSlash(h.path).
Why native paths:
- They match what the fork reported before the merge, when no
conversion happened.
- They match the receiver's r.files keys, pipeNames and NotifyHashed.
- They match the paths the sender's walk callback gets.
- EarthBuild's fsutilprogress keys per-file byte counts by this path
across StatusReceiving and StatusReceived, so the two must agree.
The receiver's StatusStat still fires before the order and hardlink
validators, as it did before the merge. A validator error aborts the
whole transfer anyway, so moving the call past them would add nothing
useful.
On Unix FromSlash and ToSlash do nothing, so behaviour there is
unchanged. Fixes TestVerboseProgressPathsConsistent on Windows.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The EarthBuild org requires every action to be pinned to a full commit SHA; with tag pins every job failed at "Set up job" before running. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
bake-action < v5 refuses to run with the buildx now shipped on ubuntu-latest runners. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Needs #6 first to see real CI impact. We should probably merge that and rebase the stack |
Windows stat handling does not populate hardlink linknames, so TestHardlinkFilter cannot assert the Unix hardlink normalization behavior there. Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> (cherry picked from commit a0b4937) Cherry-picked rather than fixed: the failure is pre-existing upstream behaviour, not something the merge or #4 introduced. TestHardlinkFilter arrived with upstream 91a3fc4 (tonistiigi#198), and setUnixOpt in stat_windows.go is a no-op, so the inode map that fills Stat.Linkname for hardlinks never runs on Windows. Upstream's own CI masked the failure the same way until tonistiigi#259, which added `shell: bash` and this skip together (its bash-only run 26987953480 failed all three Windows test-os jobs). With the bash fix alone, workflow_dispatch probes failed the same way on Windows (Go 1.20 and 1.21) on upstream 91a3fc4 (run 38000549628) and on #4's code e66a226 (run 38000552489), with `foo2: The system cannot find the file specified.` and an empty Linkname where "foo" was expected. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…nner) (#6) * ci: run CI for pushes to earthly-main and PRs into any branch The upstream merge took upstream's trigger block. That dropped the fork's push trigger (earthly-main, gh_test_ci) and limited pull_request to master/main. So CI no longer ran for PRs into earthly-main, including #3 itself. This brings back the push trigger from earthly-main (644b083). It also removes the branch filter from pull_request, so the workflow runs for every PR, whatever its base. A PR into earthly-main would have been enough, but stacked PRs (like this one, based on chore-merge-v0.14.1) need CI too. We keep upstream's workflow_dispatch and schedule triggers. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit f2e55f5) * ci: pin actions to full-length commit SHAs The EarthBuild org requires every action to be pinned to a full commit SHA; with tag pins every job failed at "Set up job" before running. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit e66a226) * ci: bump docker/bake-action to v5 for buildx >= 0.20 bake-action < v5 refuses to run with the buildx now shipped on ubuntu-latest runners. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit e2fbb01) * ci: log in to Docker Hub before bake to avoid pull rate limits Anonymous pulls of tonistiigi/xx and mvdan/shfmt hit Docker Hub's 429 rate limit on shared runners. Use the EarthBuild org's CI account (vars.DOCKERHUB_USERNAME / secrets.DOCKERHUB_TOKEN); skipped for fork PRs, which don't receive secrets. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * ci: run FreeBSD tests in a vmactions VM on ubuntu-latest test-freebsd-amd64 targeted the retired macos-13 runner, so it queued forever and the workflow run never completed. Port upstream tonistiigi/fsutil's approach (vmactions/freebsd-vm on Linux) for the FreeBSD leg, keeping the fork's Go 1.21 and actions pinned by SHA. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * ci: skip Go 1.21 on macOS where its test binaries cannot load Go 1.21 binaries abort with "dyld: missing LC_UUID load command" on the current macos-latest image before any test runs. Go 1.20 keeps macOS coverage; Windows and Linux still run 1.21. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * ci: run OS tests with bash Force the test-os workflow step to use bash so a failing go test command is not masked by a later coverage command on Windows PowerShell. Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> (cherry picked from commit 818582b) Without a shell, windows-latest runs this step in PowerShell, where a failing native command does not stop the script: `go tool cover` succeeding overwrote `go test`'s exit code. Run 37996598366 (jobs 114044153320, 114044153527) logged `--- FAIL: TestHardlinkFilter` yet both Windows jobs concluded success. The other multi-command run steps already stop on failure: the FreeBSD Prepare step sets `shell: bash`, and the FreeBSD Test step runs on ubuntu-latest (default `bash -e`) with `set -ex` inside the VM script. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test: skip hardlink filter on Windows Windows stat handling does not populate hardlink linknames, so TestHardlinkFilter cannot assert the Unix hardlink normalization behavior there. Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> (cherry picked from commit a0b4937) Cherry-picked rather than fixed: the failure is pre-existing upstream behaviour, not something the merge or #4 introduced. TestHardlinkFilter arrived with upstream 91a3fc4 (tonistiigi#198), and setUnixOpt in stat_windows.go is a no-op, so the inode map that fills Stat.Linkname for hardlinks never runs on Windows. Upstream's own CI masked the failure the same way until tonistiigi#259, which added `shell: bash` and this skip together (its bash-only run 26987953480 failed all three Windows test-os jobs). With the bash fix alone, workflow_dispatch probes failed the same way on Windows (Go 1.20 and 1.21) on upstream 91a3fc4 (run 38000549628) and on #4's code e66a226 (run 38000552489), with `foo2: The system cannot find the file specified.` and an empty Linkname where "foo" was expected. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Tonis Tiigi <tonistiigi@gmail.com>
Stacked on #3 (
chore-merge-v0.14.1). Merge #3 first, or merge this into #3's branch.Structure
This PR is red first, then green:
earthly-main644b083 and upstream 91a3fc4).Findings
send.gosender.updateProgress: the conflict resolution kept both the fork'ss.mulock andprogressCurrent += sizeand upstream'sprogressCurrentMuand increment. Progress is double-counted, ands.muis held during the callback, which upstream 216a08f was written to avoid.TestSendProgressTotalgo test -race -run TestSendProgressTotal .receive.go: the fork'sverboseProgressCb(p.Stat.Path, StatusStat, …)call runs beforefilepath.FromSlashand before the unrepresentable-path rejection. The send side reportsStatusStatwith the native path andStatusSentwith the wire (ToSlash) path. So on Windows the paths each side reports don't match.TestVerboseProgressPathsConsistentFromSlash/ToSlashdo nothing on Unix). Compile it withGOOS=windows go test -c .; it runs on thewindows-latestCI job..github/workflows/ci.yml: the merge dropped the fork'spushtrigger (earthly-main,gh_test_ci), andpull_requestonly targetsmaster/main. So CI doesn't run for PRs intoearthly-main, including #3 and this PR.Known issue, not fixed here
TestCopySimplefails on macOS, and it fails on upstream too. It passes on Linux.Commit map
TestSendProgressTotalfailsTestVerboseProgressPathsConsistentfails on Windows onlypush(earthly-main, gh_test_ci) and runpull_requestfor any base branch, so stacked PRs get CIupdateProgress: one increment underprogressCurrentMu, ands.muis no longer held during the callbackTestSendProgressTotalStatusStatmoves afterFromSlashand the unrepresentable-path check, and the sender'sStatusSentusesfilepath.FromSlash(h.path)TestVerboseProgressPathsConsistentVerification (at cc66d52)
go test -race -count=1 ./...: everything passes except the knownTestCopySimplefailure.docker run --rm -v "$PWD":/src -w /src golang:1.21 go test -race -count=1 ./...: everything passes.GOOS=windows go vet ./...andGOOS=windows go test -c .both succeed. The test has not been run on Windows yet.Windows results (update)
These come from throwaway
workflow_dispatchprobes that ran #6's workflow (shell: bash, so failures aren't hidden) withgo test -v, onwindows-latestwith Go 1.20 and 1.21. The probe branches have been deleted.TestSendProgressTotalTestVerboseProgressPathsConsistentTestHardlinkFilter[]string{"a", "dir", "dir\\b", …} does not contain "dir/b"--- PASS--- PASSTestHardlinkFilteris the only failing test.TestHardlinkFilterfails on upstream 91a3fc4 the same way (an empty linkname, thenfoo2: The system cannot find the file specified.), so this PR didn't cause it. ci: make the workflow run again (pinning, Docker Hub auth, FreeBSD runner) #6 cherry-picks upstream's Windows skip for it (0c1f49d, upstream a0b4937). No code change is needed here.ci.ymlstill runs the Windows Test step in PowerShell, which hides failures. Its "green" Windows jobs (e.g. run 37989776036) do log--- FAIL: TestHardlinkFilter. That is fixed in ci: make the workflow run again (pinning, Docker Hub auth, FreeBSD runner) #6 (ef0123f,shell: bash), and this PR gets that fix once ci: make the workflow run again (pinning, Docker Hub auth, FreeBSD runner) #6 lands in chore: update buildkit to v0.14.1 #3. The fix wasn't duplicated here, so test+fix: red tests and fixes for upstream merge (stacked on #3) #4's history and its pinned code head stay as they are.CI did not run on this PR
GitHub disabled the
ciworkflow for inactivity (state: disabled_inactivity), so f2e55f5's trigger change has no effect until a maintainer turns it back on (gh workflow enable ci --repo EarthBuild/fsutil). Two things to watch once it is on:sha_pinning_required: true, and this workflow references actions by tag (actions/checkout@v4, etc.). That may fail at "Set up job".test-freebsd-amd64usesmacos-13, and GitHub has retired that runner image.🤖 Generated with Claude Code
Summary by CodeRabbit