Skip to content

post: fstrim the buildkit sticky disk before unmount at commit - #143

Open
magma-beef wants to merge 2 commits into
mainfrom
devin/1792451281-buildkit-fstrim
Open

magma-beef wants to merge 2 commits into
mainfrom
devin/1792451281-buildkit-fstrim

Conversation

@magma-beef

@magma-beef magma-beef commented Oct 20, 2026 •

Copy link
Copy Markdown

Summary

Feature — the post step runs a best-effort fstrim on the buildkit sticky disk (/var/lib/buildkit) right before unmounting a disk it is about to commit, for at most the number of seconds the VM agent sends. Nothing changes until a host is configured: the agent sends 0 today, and the hypervisor does not negotiate discard yet.

Start here: src/setup_builder.ts (trim helper + where it sits in cleanup), then src/main.ts (reads the budget from the mount response, saves it in state). dist/ is generated.

Why

The buildkit sticky disk is the largest sticky-disk population by bytes. It is an RBD image; layers buildkit prunes stay allocated in Ceph until the guest discards them (fleet-wide ~47 % of allocated sticky-disk bytes are filesystem-free, measured 2026-10-19). Same change as useblacksmith/stickydisk#82 for the bare action.

What

  • In post, when the run is still on track to commit (integrity check passed, no commitEarlyDeny, no SIGKILL, no failed steps) and the host enabled the trim, run timeout <secs> sudo fstrim -v -- <mount> after buildkitd stops and before the df sample and umount. Any exit code or exception only logs (fstrim <mount>: <output> in Nms | failed in Nms: timed out after Ns | ...). The previous-step check is evaluated once and reused for the commit decision, so skip reasons are unchanged.
  • The budget comes from the VM agent: GetStickyDiskResponse.fstrim_timeout_secs (FastActions/fa#6600, live config guest_fstrim_timeout_secs), saved in action state at setup. 0/absent = no trim, so this release does nothing until the agent sends a budget — the rollout and its tuning are agent-side.
  • fstrimTimeoutSecsFrom() reads the typed field when the bundled proto package has it, else unknown field 8 from $unknown (protobuf-es keeps unknown binary fields; the transport is gRPC), so this version keeps working once the agent starts sending the field without a package bump.

Evidence

Check Result
lefthook pre-commit (prettier, eslint, tsc, build, dist diff) clean
pnpm test 34 pass; new tests for fstrimTimeoutSecsFrom (typed field, unknown field 8 on a real response decoded with the field-less schema, wrong wire type), decodeVarint, parseFstrimTimeoutSecs
CI Build, verify-build, 5 setup-docker-builder e2e legs, step-duration gate, CodeQL green

Risks

Verdict: Ship. With the agent sending 0 (current default) post only reads one state value; the trim path is reachable only after a host is configured.
Worry about: once a host is configured and discard is negotiated, the first trim of a large never-trimmed buildkit disk runs until the budget expires and post is longer by up to that budget; later trims are incremental.
Checked, not worried: every trim failure mode (unsupported discard, missing binary, timeout) logs and continues; the trim runs after buildkitd is stopped and before the df sample, so the committed state and the sample agree; non-commit runs skip it.
If it goes wrong: post logs fstrim /var/lib/buildkit: failed ...; set guest_fstrim_timeout_secs back to 0 on the host — no action release needed.

Related: useblacksmith/stickydisk#82, useblacksmith/checkout#63, FastActions/fa#6600, useblacksmith/web#18629.

Link to Devin session: https://app.devin.ai/sessions/2e17815c306b4c3cb7af4f6bce008a66
Open in Devin Desktop: https://app.devin.ai/desktop/session/2e17815c306b4c3cb7af4f6bce008a66?variant=devin
Requested by: @magma-beef


Devin Review


View with [code]smith View full Proof
Need help on this PR? Tag @codesmith-bot with what you need. (Staging)


View with [code]smith Generate Proof with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need.

magma-beef and others added 2 commits October 19, 2026 23:08
Pruned layers keep their RBD extents allocated on the backing store until
the guest discards them. Run a best-effort fstrim on /var/lib/buildkit
before unmount, only when the run is still on track to commit (integrity
check passed, no early deny, no SIGKILL, no failed steps) and only when
the host enabled it: the VM agent returns
GetStickyDiskResponse.fstrim_timeout_secs (fa live config
guest_fstrim_timeout_secs), saved in action state at mount time; 0 means
no trim. Any failure or timeout only logs.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
…ackage predates it

The bundled GetStickyDiskResponse has no fstrim_timeout_secs yet, and protobuf-es keeps fields it does not know in $unknown (the gRPC transport is binary), so decode field 8 from there when the typed field is absent. Releases built before the field existed therefore honour the host budget once the agent sends it.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
@devin-ai-integration

Copy link
Copy Markdown
Contributor

I'll fix CI failures and address comments from users with write access. I'll skip comments containing "(aside)".

  • Disable automatic comment, CI, and merge conflict monitoring

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Devin Review: No Issues Found

Devin Review analyzed this PR and found no bugs or issues to report.

Devin Review

@blacksmith-staging

blacksmith-staging Bot commented Oct 20, 2026 •

Copy link
Copy Markdown

Checked Post trims freed blocks before unmount, then commits

Before, the deleted 512 MiB stayed allocated on the backing disk after post. After, post trimmed before the df sample and unmount, the backing disk dropped back to its empty size, and the commit still went out.

Merge-base

Post without trim

This PR 56214f0b

Post with trim

Also checked · 4 more claims, all matched observations

Checked Setup reads budget from unknown field 8
Checked No budget means no trim
Checked Failed earlier step: no trim, same skip reason
Checked fstrim timeout only logs

Proof on 56214f0b

View full Proof report

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant