Skip to content

test+fix: red tests and fixes for upstream merge (stacked on #3) - #4

Draft
kmannislands wants to merge 7 commits into
chore-merge-v0.14.1from
review/pr3-red-tests
Draft

kmannislands wants to merge 7 commits into
chore-merge-v0.14.1from
review/pr3-red-tests

Conversation

@kmannislands

@kmannislands kmannislands commented Oct 9, 2026 •

Copy link
Copy Markdown

Stacked on #3 (chore-merge-v0.14.1). Merge #3 first, or merge this into #3's branch.

Structure

This PR is red first, then green:

  1. Red: two commits that only add tests. Each test fails on chore: update buildkit to v0.14.1 #3's merge commit (890baae) and passes on both parents (earthly-main 644b083 and upstream 91a3fc4).
  2. Green: one fix commit per finding. Each fix turns its test green.

Findings

# Finding Test How to run
1 send.go sender.updateProgress: the conflict resolution kept both the fork's s.mu lock and progressCurrent += size and upstream's progressCurrentMu and increment. Progress is double-counted, and s.mu is held during the callback, which upstream 216a08f was written to avoid. TestSendProgressTotal go test -race -run TestSendProgressTotal .
2 receive.go: the fork's verboseProgressCb(p.Stat.Path, StatusStat, …) call runs before filepath.FromSlash and before the unrepresentable-path rejection. The send side reports StatusStat with the native path and StatusSent with the wire (ToSlash) path. So on Windows the paths each side reports don't match. TestVerboseProgressPathsConsistent Red only on Windows (FromSlash/ToSlash do nothing on Unix). Compile it with GOOS=windows go test -c .; it runs on the windows-latest CI job.
3 .github/workflows/ci.yml: the merge dropped the fork's push trigger (earthly-main, gh_test_ci), and pull_request only targets master/main. So CI doesn't run for PRs into earthly-main, including #3 and this PR. (CI config) Check that CI runs on this PR

Known issue, not fixed here

TestCopySimple fails on macOS, and it fails on upstream too. It passes on Linux.

Commit map

Commit Kind Finding Test turns green
88b32bf red #1 double-counted send progress TestSendProgressTotal fails
2cf3609 red #2 verbose progress paths (Windows) TestVerboseProgressPathsConsistent fails on Windows only
f2e55f5 fix #3 CI: restore push (earthly-main, gh_test_ci) and run pull_request for any base branch, so stacked PRs get CI n/a
59f0fd7 fix #1 take upstream's updateProgress: one increment under progressCurrentMu, and s.mu is no longer held during the callback TestSendProgressTotal
cc66d52 fix #2 both sides report native paths: the receiver's StatusStat moves after FromSlash and the unrepresentable-path check, and the sender's StatusSent uses filepath.FromSlash(h.path) TestVerboseProgressPathsConsistent
e66a226 ci pin actions to full-length commit SHAs (also in #6) n/a
e2fbb01 ci bump docker/bake-action to v5 for buildx >= 0.20 (also in #6) n/a

Verification (at cc66d52)

  • macOS, go test -race -count=1 ./...: everything passes except the known TestCopySimple failure.
  • Linux, docker run --rm -v "$PWD":/src -w /src golang:1.21 go test -race -count=1 ./...: everything passes.
  • Windows: GOOS=windows go vet ./... and GOOS=windows go test -c . both succeed. The test has not been run on Windows yet.

Windows results (update)

These come from throwaway workflow_dispatch probes that ran #6's workflow (shell: bash, so failures aren't hidden) with go test -v, on windows-latest with Go 1.20 and 1.21. The probe branches have been deleted.

Code Run TestSendProgressTotal TestVerboseProgressPathsConsistent TestHardlinkFilter
red 2cf3609 38001592592 (1.20, 1.21) FAIL FAIL: []string{"a", "dir", "dir\\b", …} does not contain "dir/b" FAIL
fixes e66a226 38000552489 (1.20, 1.21) --- PASS --- PASS FAIL
upstream 91a3fc4 38000549628 n/a n/a FAIL

CI did not run on this PR

GitHub disabled the ci workflow for inactivity (state: disabled_inactivity), so f2e55f5's trigger change has no effect until a maintainer turns it back on (gh workflow enable ci --repo EarthBuild/fsutil). Two things to watch once it is on:

  • The repo has sha_pinning_required: true, and this workflow references actions by tag (actions/checkout@v4, etc.). That may fail at "Set up job".
  • test-freebsd-amd64 uses macos-13, and GitHub has retired that runner image.

🤖 Generated with Claude Code

Summary by CodeRabbit

  • Bug Fixes
    • File paths shown in verbose transfer progress now use the platform’s native path format, keeping sender and receiver status messages consistent.
    • Transfer progress totals now advance only when a progress callback is configured, and completion is reported once for transfers that include empty, small, or multi-part files.

kmannislands and others added 2 commits October 9, 2026 13:04
The conflict resolution in sender.updateProgress kept both the fork's
increment under s.mu and upstream's increment under progressCurrentMu,
so every packet is added to progressCurrent twice whenever progressCb
is set. The final total reported to progressCb is exactly double the
STAT+DATA bytes the sender put on the wire.

TestSendProgressTotal wraps the sender stream, sums the Size() of each
non-terminal STAT packet and each non-empty DATA packet, and asserts
the last=true total equals it. It fails on the merge (205440 vs 102720)
and passes on both parents (644b083 and 91a3fc4).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The merge put the fork's receiver StatusStat verboseProgressCb call
before upstream's filepath.FromSlash normalisation, so on Windows the
receiver reports StatusStat with the unix wire path ("dir/b") but
StatusReceiving/StatusReceived with the native r.files key ("dir\b").
The sender has the mirror problem: StatusStat gets the native walk
path while StatusSent gets the ToSlash'd wire path from s.files.

TestVerboseProgressPathsConsistent asserts that every StatusSent path
was also reported with StatusStat on the sender, and that the receiver
reports the same native paths for StatusStat, StatusReceiving and
StatusReceived as it does to NotifyHashed.

FromSlash/ToSlash are no-ops on Unix, so this passes on macOS/Linux
and is expected to fail only on Windows (cross-compiled with
GOOS=windows; not executed).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@coderabbitai

coderabbitai Bot commented Oct 9, 2026 •

Copy link
Copy Markdown

Review in Change Stack →

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration
  • Configuration used: defaults
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: b6ac248b-ea89-43e2-8263-fc0d4139e942

📥 Commits

Reviewing files that changed from the base of the PR and between 890baae and e2fbb01.


📒 Files selected for processing (5)
  • .github/workflows/ci.yml
  • receive.go
  • send.go
  • send_progress_test.go
  • verbose_progress_test.go

Included review availability: This review used your included allowance. Your plan provides up to 1 included review per hour; 0 remain after this review.



📝 Walkthrough

Walkthrough

The CI workflow now uses updated triggers and pinned action references. Sender and receiver progress callbacks report native-format paths. Sender progress totals increment only when a progress callback is configured, with new tests for paths and packet totals.

Changes

CI workflow

Layer / File(s) Summary
Workflow triggers and pinned actions
.github/workflows/ci.yml
Push triggers now target earthly-main and gh_test_ci; pull requests have no branch filter. Action references use commit SHAs.

Transfer progress reporting

Layer / File(s) Summary
Native paths in progress callbacks
receive.go, send.go, verbose_progress_test.go
The receiver converts STAT paths before reporting them, and the sender converts sent-file paths to native format. Tests compare callback paths across transfer statuses.
Progress totals with callbacks
send.go, send_progress_test.go
The sender increments progress totals only when a callback is configured. The test compares the final total with counted STAT and DATA packet sizes and checks that the final callback occurs once.

Priority: ➖ Normal

Estimated code review effort: 3 (Moderate) | ~20 minutes

Change: Bug fix


Merge Risk: ⚪ Minimal · up to e2fbb

No actionable regression is established in this PR. It is mergeable after normal checks.

Pre-merge checks | Passed 4 | Failed 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage Warning Docstring coverage is 66.67% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 3 functions across 4 files. (1 skipped: 1… Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Linked Issues check Passed Issue #3 requires the fsutil change set that supports moby/buildkit v0.14.1 and the stacked merge order. The PR is based on the reported #3 merge commit 890baae, and the PR description requires #3 to …
Out of Scope Changes check Passed The source fixes address regressions attributed to the #3 upstream merge: send progress accounting, native verbose-progress paths, and CI triggers for stacked branches. The added tests verify those fi…
Description Check Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check Passed The title accurately identifies the added regression tests and fixes related to the upstream merge. It also notes the stacked relationship to issue #3. The CI updates are secondary and do not need to …

Full details: Docstring Coverage

Explanation

Docstring coverage is 66.67% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 3 functions across 4 files. (1 skipped: 1 unsupported.)



  • Fix all pre-merge checks with AI
✨ Finishing Touches 💡 2
📝 Generate docstrings 💡
  • Commit to this branch
  • Create a new PR

🛠️ Fix failing CI checks 💡
  • Commit to this branch
  • Create a new PR

🧪 Generate unit tests (beta)
  • Commit to this branch
  • Create a new PR


  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

kmannislands and others added 4 commits October 9, 2026 13:37
The upstream merge took upstream's trigger block. That dropped the
fork's push trigger (earthly-main, gh_test_ci) and limited
pull_request to master/main. So CI no longer ran for PRs into
earthly-main, including #3 itself.

This brings back the push trigger from earthly-main (644b083). It also
removes the branch filter from pull_request, so the workflow runs for
every PR, whatever its base. A PR into earthly-main would have been
enough, but stacked PRs (like this one, based on
chore-merge-v0.14.1) need CI too. We keep upstream's
workflow_dispatch and schedule triggers.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
When the merge resolved the updateProgress conflict, it kept both
sides. The fork increments progressCurrent under s.mu, and upstream
increments it again under progressCurrentMu. So whenever progressCb is
set, every STAT and DATA packet is counted twice. The callback also
ran while s.mu was held. Upstream 216a08f added progressCurrentMu to
avoid exactly that, because s.mu also guards s.files and the send
queue.

This takes upstream's version: a single increment under
progressCurrentMu, and only when progressCb is set. Nothing else reads
progressCurrent, so it no longer needs s.mu.

Fixes TestSendProgressTotal.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Upstream now sends unix-style wire paths: the sender applies ToSlash
and the receiver applies FromSlash. The merge put the fork's
VerboseProgressCb calls around that change in a way that mixes the two
forms on Windows.

- Receiver: StatusStat was reported with p.Stat.Path before
  FromSlash, so it got the wire path ("dir/b"). StatusReceiving and
  StatusReceived get the native r.files key ("dir\b"). The call also
  ran before the "unrepresentable path" check, so a rejected path
  was still reported as stat'ed.
- Sender: walk reports StatusStat with the native walk path, but
  sendFile reported StatusSent with h.path, which is the ToSlash'd
  value stored in s.files.

Each side now reports native paths:

- Receiver: StatusStat moves to after normalisation and the
  unrepresentable-path check, and reports the normalised path.
- Sender: StatusSent reports filepath.FromSlash(h.path).

Why native paths:

- They match what the fork reported before the merge, when no
  conversion happened.
- They match the receiver's r.files keys, pipeNames and NotifyHashed.
- They match the paths the sender's walk callback gets.
- EarthBuild's fsutilprogress keys per-file byte counts by this path
  across StatusReceiving and StatusReceived, so the two must agree.

The receiver's StatusStat still fires before the order and hardlink
validators, as it did before the merge. A validator error aborts the
whole transfer anyway, so moving the call past them would add nothing
useful.

On Unix FromSlash and ToSlash do nothing, so behaviour there is
unchanged. Fixes TestVerboseProgressPathsConsistent on Windows.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The EarthBuild org requires every action to be pinned to a full commit
SHA; with tag pins every job failed at "Set up job" before running.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kmannislands
kmannislands added this pull request to stack #5 October 9, 2026 20:47
bake-action < v5 refuses to run with the buildx now shipped on
ubuntu-latest runners.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kmannislands

Copy link
Copy Markdown
Author

Needs #6 first to see real CI impact. We should probably merge that and rebase the stack

kmannislands pushed a commit that referenced this pull request Oct 9, 2026
Windows stat handling does not populate hardlink linknames, so
TestHardlinkFilter cannot assert the Unix hardlink normalization behavior
there.

Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
(cherry picked from commit a0b4937)

Cherry-picked rather than fixed: the failure is pre-existing upstream
behaviour, not something the merge or #4 introduced. TestHardlinkFilter
arrived with upstream 91a3fc4 (tonistiigi#198), and
setUnixOpt in stat_windows.go is a no-op, so the inode map that fills
Stat.Linkname for hardlinks never runs on Windows. Upstream's own CI
masked the failure the same way until tonistiigi#259, which
added `shell: bash` and this skip together (its bash-only run
26987953480 failed all three Windows test-os jobs).

With the bash fix alone, workflow_dispatch probes failed the same way
on Windows (Go 1.20 and 1.21) on upstream 91a3fc4 (run 38000549628)
and on #4's code e66a226 (run 38000552489), with
`foo2: The system cannot find the file specified.` and an empty
Linkname where "foo" was expected.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
janishorsts pushed a commit that referenced this pull request Oct 10, 2026
…nner) (#6)

* ci: run CI for pushes to earthly-main and PRs into any branch

The upstream merge took upstream's trigger block. That dropped the
fork's push trigger (earthly-main, gh_test_ci) and limited
pull_request to master/main. So CI no longer ran for PRs into
earthly-main, including #3 itself.

This brings back the push trigger from earthly-main (644b083). It also
removes the branch filter from pull_request, so the workflow runs for
every PR, whatever its base. A PR into earthly-main would have been
enough, but stacked PRs (like this one, based on
chore-merge-v0.14.1) need CI too. We keep upstream's
workflow_dispatch and schedule triggers.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit f2e55f5)

* ci: pin actions to full-length commit SHAs

The EarthBuild org requires every action to be pinned to a full commit
SHA; with tag pins every job failed at "Set up job" before running.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit e66a226)

* ci: bump docker/bake-action to v5 for buildx >= 0.20

bake-action < v5 refuses to run with the buildx now shipped on
ubuntu-latest runners.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit e2fbb01)

* ci: log in to Docker Hub before bake to avoid pull rate limits

Anonymous pulls of tonistiigi/xx and mvdan/shfmt hit Docker Hub's 429
rate limit on shared runners. Use the EarthBuild org's CI account
(vars.DOCKERHUB_USERNAME / secrets.DOCKERHUB_TOKEN); skipped for fork
PRs, which don't receive secrets.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* ci: run FreeBSD tests in a vmactions VM on ubuntu-latest

test-freebsd-amd64 targeted the retired macos-13 runner, so it queued
forever and the workflow run never completed. Port upstream
tonistiigi/fsutil's approach (vmactions/freebsd-vm on Linux) for the
FreeBSD leg, keeping the fork's Go 1.21 and actions pinned by SHA.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* ci: skip Go 1.21 on macOS where its test binaries cannot load

Go 1.21 binaries abort with "dyld: missing LC_UUID load command" on
the current macos-latest image before any test runs. Go 1.20 keeps
macOS coverage; Windows and Linux still run 1.21.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* ci: run OS tests with bash

Force the test-os workflow step to use bash so a failing go test command
is not masked by a later coverage command on Windows PowerShell.

Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
(cherry picked from commit 818582b)

Without a shell, windows-latest runs this step in PowerShell, where a
failing native command does not stop the script: `go tool cover`
succeeding overwrote `go test`'s exit code. Run 37996598366 (jobs
114044153320, 114044153527) logged `--- FAIL: TestHardlinkFilter` yet
both Windows jobs concluded success.

The other multi-command run steps already stop on failure: the FreeBSD
Prepare step sets `shell: bash`, and the FreeBSD Test step runs on
ubuntu-latest (default `bash -e`) with `set -ex` inside the VM script.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test: skip hardlink filter on Windows

Windows stat handling does not populate hardlink linknames, so
TestHardlinkFilter cannot assert the Unix hardlink normalization behavior
there.

Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
(cherry picked from commit a0b4937)

Cherry-picked rather than fixed: the failure is pre-existing upstream
behaviour, not something the merge or #4 introduced. TestHardlinkFilter
arrived with upstream 91a3fc4 (tonistiigi#198), and
setUnixOpt in stat_windows.go is a no-op, so the inode map that fills
Stat.Linkname for hardlinks never runs on Windows. Upstream's own CI
masked the failure the same way until tonistiigi#259, which
added `shell: bash` and this skip together (its bash-only run
26987953480 failed all three Windows test-os jobs).

With the bash fix alone, workflow_dispatch probes failed the same way
on Windows (Go 1.20 and 1.21) on upstream 91a3fc4 (run 38000549628)
and on #4's code e66a226 (run 38000552489), with
`foo2: The system cannot find the file specified.` and an empty
Linkname where "foo" was expected.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Tonis Tiigi <tonistiigi@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant