Skip to content

main: overlap Docker daemon startup with builder start - #145

Open
aayushshah15 wants to merge 2 commits into
mainfrom
aayushshah15/overlap-docker-startup
Open

aayushshah15 wants to merge 2 commits into
mainfrom
aayushshah15/overlap-docker-startup

Conversation

@aayushshah15

@aayushshah15 aayushshah15 commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

The setup step waits for the Docker daemon on its critical path. Its first Docker call is docker version in the "Docker info" group, and that call blocks until dockerd has finished starting. On our own step-duration-regression run on main (run 39735830544), docker version took 0.96 s to print anything. Buildx setup and the Blacksmith builder start then ran another 0.77 s, and neither needs the daemon.

This starts that wait in the background as soon as the step begins, and prints the Docker info group once buildx and the builder are set up:

let dockerWarmup: Promise<unknown> | null = null;
if (process.env.BLACKSMITH_DOCKER_WARMUP === "false") {
  await printDockerInfo();
} else {
  dockerWarmup = execAsync("docker version").catch(() => undefined);
}
try {
  // ... Setup buildx, Starting Blacksmith builder ...
} finally {
  if (dockerWarmup) {
    await dockerWarmup;
    await printDockerInfo();
  }
}

The Docker info group has the same contents and now comes right after "Starting Blacksmith builder" in the log. It's printed in a finally, so it still appears when the builder start throws (nofallback). The daemon starts at the same point in the step as before; the step just doesn't wait for it before moving on.

It also polls for the buildkitd PID every 50 ms instead of 300 ms. The PID usually shows up within a few tens of ms, so one missed pgrep costs a full 300 ms today.

Estimated impact. Across ~23k setup runs per hour in production (2026-10-21, 12:00-13:00 UTC), "Docker info" averaged 1.14 s, the builder start 1.4 s, and the buildkitd PID wait had p50 0.32 s. Hiding most of the daemon start behind the builder start and trimming the PID wait saves about 0.8-1.1 s per run, or ~5-7 runner-hours per hour: ~0.02-0.025% of all whole-job time (denominator: ~28,500 whole-job hours, same window). These fleet numbers are modeled.

Measured. On the step-duration-regression job, the setup step (from Run ./ to the next step) took 2.12 s, 2.04 s and 2.34 s on the last three scheduled runs on main (39735830544, 39557284551, 39382020583) and 1.13 s on this PR (39875460145). The Docker info group dropped from 1.09 s to 0.15 s, because the daemon was already up by the time it ran. That's one run, so treat it as a sanity check, not a distribution.

Edge cases:

  • Docker missing or the daemon fails to start: the warm-up swallows its error, and the Docker info group prints the same message it does today.
  • Daemon hangs: the step hangs at the same total point as before, since the old code also waited on docker version with no timeout.
  • Fallback to the local builder: Docker info is printed before the fallback runs, as before.
  • Ordering: nothing in buildx setup or the builder start talks to dockerd, writes its config or touches its socket. They only use the buildx CLI plugin, the sticky disk agent and buildkitd.
  • CPU: dockerd and buildkitd now start concurrently. On a 2 vCPU runner that can slow each a little, but they ran back to back before, so the overlapped total should stay under the old serial time.
  • PID polling: at most ~200 pgrep calls in the 10 s timeout instead of ~33, and in practice one or two.

BLACKSMITH_DOCKER_WARMUP=false restores the old order. dist/ is rebuilt in this PR.


View with [code]smith Test with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need.


Devin Review

Matches Daemon start overlaps builder start

Before, buildx and the builder start waited ~1.3 s for docker version; after, they ran while dockerd started, Docker info printed after the builder with identical contents, and the step averaged ~0.38 s faster over 5 runs each.

Merge-base

Cold daemon timeline

This PR fe4fcc13

Cold daemon timeline

Also checked · 4 match

Matches buildkitd PID poll at 50 ms
Matches Docker info on nofallback failure
Matches Daemon unavailable message unchanged
Matches BLACKSMITH_DOCKER_WARMUP=false

Proof on fe4fcc13


View with [code]smith View full Proof
Need help on this PR? Tag @codesmith-bot with what you need. Autofix is enabled. (Staging)

aayushshah15 and others added 2 commits October 22, 2026 23:59
The first Docker command in the step waits for the daemon to finish
starting. Run it in the background right away and print Docker info
after buildx setup and the builder start, so the wait overlaps them.
BLACKSMITH_DOCKER_WARMUP=false restores the old order.

Also poll for the buildkitd PID every 50ms instead of 300ms.

Co-authored-by: Codesmith Staging <codesmith-bot@users.noreply.github.com>
Co-authored-by: Codesmith Staging <codesmith-bot@users.noreply.github.com>

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Devin Review: No Issues Found

Devin Review analyzed this PR and found no bugs or issues to report.

Devin Review

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant