[kmannislands]

Problem

On Docker Desktop (macOS, Windows, WSL2), every build waits about 90 seconds after printing Earth Build ✅ SUCCESS before earth exits. A client that holds a connection to the registry proxy open can delay exit for as long as it likes.

This is a regression on main and has not shipped. Since #914, a proxied connection stays open for as long as its client keeps it, which is what a kept-alive docker pull needs. But stopping the proxy closed only its listener, then waited for every connection handler to return. It waited on the build's context, which is never cancelled at that point.

The proxy holds one of those connections itself. On Docker Desktop, startDarwinProxy checks readiness with a GET /v2/ through the socat support container, using http.DefaultClient, and closes the body unread. Since Go 1.27 (#883), the transport drains a small unread body on Close and pools the connection. That connection sits idle in DefaultTransport for its 90s IdleConnTimeout. It runs back through socat and host.docker.internal into the proxy, so earth was waiting on itself. docker also keeps the connection it pulled a SAVE IMAGE image over.

flowchart LR
    subgraph host["macOS host"]
        probe["earth: readiness check<br/>GET /v2/ with http.DefaultClient"]
        proxy["earth: registry proxy<br/>127.0.0.1:P"]
    end
    subgraph vm["Docker Desktop VM"]
        socat["earthly-darwin-proxy<br/>socat container"]
        dockerd["docker daemon"]
    end
    bk["buildkitd<br/>embedded registry"]

    probe -- "127.0.0.1:S<br/>connection pooled, idle 90s" --> socat
    dockerd -- "docker pull of a SAVE IMAGE<br/>connection kept alive" --> socat
    socat -- "host.docker.internal:P" --> proxy
    proxy -- "one gRPC stream per connection" --> bk

Loading

Evidence:

• Goroutine dump (SIGQUIT on a stalled run):
  • the deferred closer is in its select (controller.go:91)
  • serve is in wg.Wait
  • one handle is in registry.Copy, with one goroutine in stream.RecvMsg and another in conn.Read
  • an idle net/http persistConn is in the same process: the readiness check's connection
• lsof: shows the loop: earth → 127.0.0.1:<socat port> (held by com.docker.backend), and com.docker.backend → earth's proxy listener.
• Timing: on main, three runs took 88.7s, 89.4s and 89.5s from SUCCESS to exit. The verbose log shows "Starting Darwin proxy" and the container removal exactly 90.0s apart.
• Leaked containers: the support container is only removed after the proxy stops, so killing a stalled earth leaked its earthly-darwin-proxy-* container.

Change

sequenceDiagram
    participant B as convertAndBuild
    participant S as proxy stop
    participant H as connection handlers
    participant C as clients (readiness check, docker)

    Note over B: bkClient.Build has returned, so every pull is done
    B->>S: deferred stop
    S->>S: close the listener
    alt Before
        S->>H: wait for every handler, on the build context (never cancelled)
        Note over H,C: A handler returns only when its client closes.<br/>The readiness check's idle connection: 90s. docker: as long as it likes
    else After
        S->>H: wait up to 250ms for handlers to return
        S->>H: cancel the connections' context
        H->>C: close the connection
        H-->>S: handlers return
    end
    S-->>B: stopped, earth exits

Loading

• regproxy/controller.go
  • The readiness check (now waitForRegistry) sets req.Close, so its connection is closed after the response instead of pooled.
  • Connections are served under their own context. Stopping the proxy closes the listener, gives open connections a 250ms grace (defaultCloseGrace) to finish, then cancels that context.
  • The goroutine draining the proxy's errors closes its done channel rather than sending on an unbuffered one.
  • Errors from connections closed on purpose are no longer logged.
  • A failure to start the Darwin support container now also stops the proxy, and the listener is closed on the address error path.
• regproxy/proxy.go: handle closes its connection when the context is done (context.AfterFunc), so shutdown never depends on a client letting go.

The buildkit fork's registry.Copy behaves as designed, and this needs no fork change.

No changelog entry, because the stall never shipped.

Why closing after 250ms can't cut off a pull

The proxy is stopped by a defer in convertAndBuild, which runs after bkClient.Build returns. The build can't return while a pull is in flight:

• The earthlyoutputs exporter sends a pull-ping and waits for the client's reply ("wait for the client to finish pulling") before the export finishes.
• The client replies from onPull, only after DockerPullLocalImages has returned for every image.

So on a successful build, any connection still open at stop is idle, and the grace only lets one that is already ending finish cleanly. If the build fails or is cancelled, cutting off a pull still in flight is the intended outcome.

I checked this end to end. Each of 10 iterations builds two images in parallel, each with a new ~150MB random layer, and loads them through the proxy. After each earth exit, a container verifies the sha256 of the layer inside each loaded image:

	result
grace closed a leftover connection (docker's keep-alive)	10/10
loaded images with intact layer content	20/20
SUCCESS → exit	0.57–1.20s (was ~90s)
leaked earthly-darwin-proxy-* containers	0

Tests

• New tests:
  • TestProxyStopDoesNotWaitOnAConnectionTheClientKeepsOpen runs the real gRPC chain against buildkit's Server.Proxy. It has two cases: an idle keep-alive, and an unfinished request.
  • TestWaitForRegistryDoesNotKeepItsConnection
• Without the fix: with the behaviour reverted and the test hooks kept, all three cases fail (the stop cases at their 10s deadline, the readiness-check case at 5s).
• go test -race -count=50 ./regproxy/...: pass.
• End to end: a rebuilt CLI on Docker Desktop exits 0.2–0.6s after SUCCESS in about 70 runs, including:
  • SAVE IMAGE builds
  • runs with a connection deliberately held open through socat (both an unfinished request and an idle keep-alive)
  • a remote-cache round trip
• earth +lint: 0 issues.

Not in this PR

• A second proxy that is never stopped: buildFunc starts a proxy each time it runs. When the push phase runs a second build, the first proxy's stop function is overwritten, and its Darwin container is never removed. That only happens when RunPush.HasState is set, which the converter forbids with wait blocks. So it's limited to VERSION 0.6 Earthfiles, or builds with wait blocks disabled.
• Fork hardening: registry.Copy could end the copy whenever its context is cancelled (context.AfterFunc(ctx, ...)). Today that only works because both callers pass the stream's own context.

Prepared with AI assistance (Claude Code).

🤖 Generated with Claude Code

[github-actions]

➖ Are we earthbuild yet?

No change in "earthly" occurrences

📈 Overall Progress

Branch	Total Count
main	2640
This PR	2640
Difference	+0

---

Keep up the great work migrating from Earthly to Earthbuild! 🚀

💡 Tips for finding more occurrences

Run locally to see detailed breakdown:

./.github/scripts/count-earthly.sh

Note that the goal is not to reach 0.
There is anticipated to be at least some occurrences of earthly in the source code due to backwards compatibility with config files and language constructs.

[coderabbitai]

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

• Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

• Autopilot · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}