Repository navigation
Serve the workspace with gitfs behind BLACKSMITH_USE_GITFS - #62
Draft
max-blacksmith wants to merge 3 commits into
Draft
max-blacksmith wants to merge 3 commits into
max-blacksmith wants to merge 3 commits into
Conversation
When the job's installation has use_gitfs_checkout on, mount the requested revision from the git mirror sticky disk with gitfs instead of fetching and checking it out, create .git through the mount so git works from the first step, and take gitfs down in the post step before the mirror is unmounted. Anything outside the plain Linux cohort, or any gitfs failure, falls through to the regular checkout. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Contributor
|
I'll fix CI failures and address comments from users with write access. I'll skip comments containing "(aside)".
|
…n mount) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
…ing first An empty sticky disk no longer costs a blocking git clone --mirror when the job uses gitfs: the action creates an empty bare mirror (same config as --mirror) and runs gitfs mount --ref <rev> --url <repo>, which fetches the revision --depth 1 into the mirror, serves the pack as it arrives and completes the mirror in the background. The same path replaces the blocking targeted fetch for a ref the mirror has not synced (pull-request merge refs). gitfs prints the mounted commit, which is how the action learns the sha before git has written the ref. The e2e test (self-skips without the binary) mounts from an empty mirror over file:// and checks the mirror ends up complete. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Per-job opt-in to serve
$GITHUB_WORKSPACEfrom the git mirror sticky disk with gitfs instead of fetch + checkout. The agent setsBLACKSMITH_USE_GITFS=truefrom the installation'suse_gitfs_checkoutflag (FastActions/fa#6520, useblacksmith/web#18247); gitfs mounting a--mirrorbare repo in place is useblacksmith/gitfs#2, the cold (streaming) mount is useblacksmith/gitfs#3.Flow in
getSourceInner:The cold branch covers the empty sticky disk (first job on a repo: no
clone --mirrorbefore the first step — babel from GitHub: checkout 3.0 s vs 13.0 s, whole job 69.9 s vs 77.1 s) and a ref the mirror has not synced (pull-request merge refs), replacing the blocking targeted fetch. The mirror ends the job as a complete ordinary mirror either way; the post step commits it.pointWorkspaceAtproduces whatgit checkout -B <b> origin/<b>/ detached checkout would (branch config,refs/remotes/pull/N/mergefor PR refs) without touching the tree;read-treewithoutupdate-index --refreshso nothing stats every file through FUSE — the firstgit statusverifies contents instead.Post step (
main.tscleanup, beforeblacksmithCache.cleanup):gitfs.teardownrunsgitfs unmount(ends a fetch still in flight; a mirror left shallow is completed by the next job's mount), then waits for / SIGTERMs anygitfs mount <mirror>daemon found in/proc, because the daemon maps the mirror's packs and the sticky-diskumountwould otherwise be busy.Not in the cohort →
[gitfs] not used: <reason>and the regular path; any gitfs failure → warning and the regular path (with an empty mirror that is a network fetch into the workspace, and the post step's deferred sync fills the mirror).serving_modetelemetry is unchanged (closed enum on the agent side). Caveat documented in BLACKSMITH.md: the workspace has the full history via alternates regardless offetch-depth.Tests:
__test__/gitfs.test.ts— eligibility, candidate mapping,coldRevision, and two end-to-end tests that run whengitfsis on PATH (skipped otherwise): a warm mount of a real--mirrorclone (.gitthrough the mount, status clean and HEAD stable before and after the swap, mirror untouched) and a cold mount from an empty mirror overfile://(sha reported, file readable, mirror ends complete and unshallowed).Testing this on the fleet
v0.1.0there (the release workflow publishes staticgitfs-<tag>.linux-{amd64,arm64}+.sha256)./usr/local/bin/gitfs, both patchroot and stage_bootstrap paths) with the hashes from step 1; merge with FastActions/fa#6520 and roll the agent.use_gitfs_checkouton one internal installation. In the checkout step look for[gitfs] … serves <sha> from the mirror (mounted in N ms)(warm) or… streamed from the remote(cold), and in the post step[gitfs] Unmounting ….[gitfs] not used: <reason>means the regular path ran. Kill switch: turn the flag off (orbypass_blacksmith_checkout); nothing persists on the disk but an ordinary mirror.Link to Devin session: https://app.devin.ai/sessions/5af1665023e24d5f97ffbb031bcb4920
Open in Devin Desktop: https://app.devin.ai/desktop/session/5af1665023e24d5f97ffbb031bcb4920?variant=devin
Requested by: @max-blacksmith
Need help on this PR? Tag
@codesmith-botwith what you need.