perf(git): optimize fetch for repos with many tags and branches by zendesk-slowery23 · Pull Request #453 · carvel-dev/vendir

zendesk-slowery23 · 2026-06-24T21:25:17Z

Problem

vendir sync is very slow for git repositories with large numbers of branches and tags. The root cause is that the git fetcher unconditionally:

Set remote.origin.tagOpt = --tags, causing all tag objects to be downloaded on every fetch
Did not pass a refspec to git fetch, causing all remote refs to be negotiated

On a repo with thousands of branches and tags, this turns every sync into a multi-megabyte download regardless of what ref you actually need.

Solution

When a fixed named ref is specified (tag or branch, not a SHA), use a targeted single-refspec fetch:

git fetch origin <ref> --no-tags [--depth N]

Then check out FETCH_HEAD (which git populates from a single-refspec fetch) instead of the ref name.

--tags is kept only where it's actually needed:

refSelection (semver matching must enumerate all tags)
Plain SHA refs with no OriginalRef (SHAs can't be fetched by name)

`--locked` path

--locked substitutes the locked SHA for the original ref before reaching the fetcher, which previously forced a full --tags fetch every time. This PR preserves the original ref in a new unexported OriginalRef field on DirectoryContentsGit (set by Lock(), not serialized to YAML). The fetcher uses it for the same targeted fetch path, then verifies the resulting commit SHA matches the lock — catching force-pushed tags.

Performance

Measured against an internal repo with thousands of branches and hundreds of tags:

Command	Before	After	Improvement
`vendir sync`	43s	11s	74% faster
`vendir sync --locked`	1m 26s	9s	89% faster

Changes

pkg/vendir/config/directory.go — add OriginalRef string (unexported from YAML) to DirectoryContentsGit; set it in Lock() before overwriting Ref with the SHA
pkg/vendir/fetch/git/git.go — targeted refspec fetch for named refs and locked-SHA-with-OriginalRef; conditional --tags/--no-tags; post-checkout SHA verification in locked mode; isHexSHA() helper
pkg/vendir/fetch/git/git_test.go — unit tests for each fetch path (named tag, origin/ branch, locked SHA+OriginalRef, plain SHA, depth ordering)
test/e2e/git_test.go — TestGitFetchOptimizations: verifies targeted fetch line in --json output, locked sync uses OriginalRef, tampered lock SHA is detected

Testing

./hack/build.sh
go test ./pkg/vendir/fetch/git/... -v
VENDIR_BINARY_PATH=./vendir go test ./test/e2e/... -v -run TestGitFetchOptimizations

🤖 Generated with Claude Code

When syncing a git source with a fixed ref (tag or branch name), vendir previously fetched all remote refs and all tags, which is very slow for repos with thousands of branches or hundreds of tags. Changes: - Named refs (tags, branches) now use a targeted refspec fetch (`git fetch origin <ref> --no-tags`) and check out FETCH_HEAD, avoiding negotiation of every remote ref. - `--no-tags` tagOpt is set for named refs; `--tags` is kept only for RefSelection (semver matching) and bare SHAs. - In `--locked` mode, `DirectoryContentsGit.Lock()` now preserves the original ref in a new `OriginalRef` field (not serialized) before overwriting `Ref` with the locked SHA. The fetcher uses `OriginalRef` for the targeted fetch, then verifies the resulting commit matches the locked SHA — catching force-pushed tags. - Plain SHA refs without an OriginalRef (edge case) fall back to the previous full fetch with `--tags`. Observed improvement on a repo with thousands of branches: vendir sync: 43s → 11s (74% faster) vendir sync --locked: 86s → 9s (89% faster) Adds unit tests covering each fetch path and e2e tests verifying the targeted fetch behaviour and tampered-lock detection. Signed-off-by: Steve Lowery <steve.lowery@zendesk.com>

zendesk-slowery23 · 2026-06-24T21:31:55Z

 	if hasUser {
-		password := string(secret.Data[
-			ctlconf.SecretK8sCorev1BasicAuthPasswordKey])
+		password := string(secret.Data[ctlconf.SecretK8sCorev1BasicAuthPasswordKey])


result of go fmt from hack/build.sh

Copilot

Pull request overview

This PR speeds up vendir sync for git sources by avoiding full ref negotiation and unconditional tag downloads when only a single named ref is needed, and adds test coverage around the new fetch behaviors (including --locked).

Changes:

Add an internal OriginalRef field to preserve the pre-lock ref name for optimized fetching in --locked mode.
Update the git fetcher to use targeted single-refspec fetches and conditionally disable tag fetching.
Add unit + e2e tests validating fetch command shapes and locked-SHA mismatch detection.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
`pkg/vendir/fetch/git/git.go`	Implements targeted fetch logic, conditional `--tags/--no-tags`, `FETCH_HEAD` checkout handling, and locked SHA verification.
`pkg/vendir/config/directory.go`	Adds `OriginalRef` and sets it during `Lock()` before overwriting `Ref` with the locked SHA.
`pkg/vendir/fetch/git/git_test.go`	Adds unit tests covering the new fetch paths and helper utilities for command assertions.
`test/e2e/git_test.go`	Adds an end-to-end test asserting optimized fetch behavior and tampered lock detection.
`pkg/vendir/fetch/http/sync.go`	Minor formatting change in basic auth password extraction.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

- Strip origin/ prefix from OriginalRef before using it as a fetch refspec in locked mode (origin/main is not a valid remote refspec) - In locked mode, verify FETCH_HEAD^{commit} matches the locked SHA before checkout, then check out the SHA directly rather than FETCH_HEAD; this ensures mutable refs like origin/main stay pinned to the locked commit rather than tracking the branch tip - Accept uppercase hex in isHexSHA (git SHAs are case-insensitive) - Fix e2e test bare-fetch assertion (needle had a spurious \n that prevented it from ever triggering) - Add unit test covering origin/ prefix stripping in locked mode - Update locked-mode unit test to match revised checkout behavior Signed-off-by: Steve Lowery <steve.lowery@zendesk.com>

zendesk-slowery23 · 2026-06-25T14:58:22Z

Are any of the maintainers available to approve the workflows for my PR and/or to discuss it? We use vendir pretty heavily and this performance improvement will have a big boost in our engineer's productivity.

zendesk-slowery23 · 2026-07-29T14:40:06Z

Nudge on this. Anybody still maintaining vendir?

joaopapereira

Sorry about the delay, Lgtm

joaopapereira · 2026-08-13T23:09:34Z

Do you mind reading this or so that we have the tests back? Thanks

The origin/-prefixed ref case in the targeted fetch switch didn't set useFetchHead, so checkout tried a local ref name (e.g. a tag) that was never created since fetch only populated FETCH_HEAD — breaking ref: origin/<tag> syncs. Also cleans up the lint issues flagged in CI: missing switch default, cognitive complexity, magic numbers, line length, and unhandled errors in the e2e tests. Signed-off-by: Steve Lowery <steve.lowery@zendesk.com>

zendesk-slowery23 · 2026-08-18T20:02:27Z

Pushed a fix for the fetch bug that was causing Test GH and lint to fail — a mismatched FETCH_HEAD/checkout case for origin/<tag> refs, plus the lint issues Copilot/CI flagged.

I ran both go test ./pkg/... and golangci-lint run ./... (pinned to v2.4.0, matching CI's config) locally and both pass on the changed files. I also manually reproduced the exact e2e scenarios that were failing (tag ref, origin/-prefixed tag ref, branch ref, --locked sync, and moved-tag detection) against a real repo and confirmed they now work correctly.

That said, I can't confirm the actual GitHub Actions checks will pass since they're still pending workflow approval on this PR — happy to fix anything else that surfaces once they run. Thanks for flagging this!

joaopapereira · 2026-08-21T02:23:46Z

I just let them run and they are failing do you mind taking a look?

Signed-off-by: Steve Lowery <steve.lowery@zendesk.com>

zendesk-slowery23 · 2026-08-25T15:37:52Z

Thanks for letting me know. I've tested the changes locally, including the previously failing Git/cache/locked E2E cases and lint, and I think they should address the checks that are awaiting validation. I pushed the fixes in d29bc66b; the updated workflows are currently awaiting approval to run.

zendesk-slowery23 · 2026-08-27T20:53:53Z

Looks like everything is green now 🎉 . Any reason to hold off on merging/releasing?

joaopapereira

Sorry for the late review, I think the PR is going in the right direction but there are some points that we will have to address in order to be able to get this merged

joaopapereira · 2026-08-28T01:39:43Z

+	// before checkout so a force-pushed tag is caught early.
+	if useFetchHead && isHexSHA(t.opts.Ref) {
+		rp := []string{"rev-parse", fetchHeadCommit}
+		out, _, runErr := t.cmdRunner.Run(rp, nil, dstPath)


Not passing env so we lose all the env variables that we might have set

joaopapereira · 2026-08-28T01:40:10Z

+		return remoteRefTag, nil
+	}
+
+	out, _, err := t.cmdRunner.Run(


Not passing env so we lose all the env variables that we might have set

joaopapereira · 2026-08-28T01:41:45Z

+	// In locked mode we fetch by the original named ref for speed, but must
+	// check out the exact locked SHA. Verify FETCH_HEAD resolves to that SHA
+	// before checkout so a force-pushed tag is caught early.
+	if useFetchHead && isHexSHA(t.opts.Ref) {


the "fetched SHA must equal locked SHA" hard-check runs even when OriginalRef is a branch (not a tag). Since branches move, running vendir sync --locked after the branch has advanced normally will now hard-fail with a "tag may have been moved" error, when the locked commit is still perfectly valid and reachable.

joaopapereira · 2026-08-28T01:43:21Z

+		// The original ref may have been deleted or renamed since the config
+		// was locked. Fetching the locked object directly retains locked-sync
+		// behavior without requiring the old name to remain available.
+		fallbackArgs := []string{"fetch", "origin", t.opts.Ref, noTagsFlag}


when the named-ref targeted fetch fails, it falls back to git fetch origin <sha> --no-tags. Many git servers disallow fetching an arbitrary non-tip SHA (uploadpack.allowReachableSHA1InWant not enabled). Pre-PR, a full mirror fetch would have pulled the SHA in via any other ref containing it; this fallback has no further recovery path.

joaopapereira · 2026-08-28T01:44:51Z

+		if len(fields) < minRemoteRefFields {
+			continue
+		}
+		switch fields[1] {


remoteRefType() returns on the first ls-remote match; since output is alphabetically sorted, refs/heads/<name> sorts before refs/tags/<name>, so a name that exists as both is always classified as a branch. This is the opposite of git's own precedence (tags win per gitrevisions(7)), so a repo with both v2 branch and v2 tag will now checkout a different commit than before.

joaopapereira · 2026-08-28T01:53:56Z

+// isHexSHA reports whether s looks like a full or abbreviated git commit SHA
+// (7–40 hex characters, case-insensitive). SHAs cannot be fetched by refspec
+// name and require a full fetch to be resolved.
+func isHexSHA(s string) bool {


This function can produce false positives because it is possible to have words or mix of words with numbers that look HEX but are just normal branch names ex cafebabe or something.

joaopapereira · 2026-08-28T01:55:04Z

 	return info, nil
 }

 func (t *Git) fetch(dstPath string, tempArea ctlfetch.TempArea, bundle string) error {


Despite pkg/vendir/fetch/git/sync.go:55 was not modified by this PR, the URL-only bundle cache key predates it. But git.go's new targeted-fetch behavior means a sync now typically populates the local bundle with only one ref's objects, instead of the full mirror the old unconditional full-fetch produced. Since the cache key doesn't include the ref, a vendir.yml with two directories: entries on the same URL but different refs can now overwrite each other's cached bundle each run, a caching regression this PR introduces by changing what goes into the bundle, even though it doesn't touch the cache-key code itself.

joaopapereira · 2026-08-28T02:04:14Z

+	// named ref to target. A targeted tag fetch explicitly creates just that
+	// tag; a targeted branch fetch uses Git's default auto-follow behavior,
+	// which fetches only tags reachable from the branch tip.
+	if !t.canTargetFetch() || refType == remoteRefUnknown ||


targeted branch fetches leave tagOpt at git's default (auto-follow tags on newly transferred objects only) instead of the old unconditional --tags, so git describe --tags HEAD in Retrieve() can silently produce worse/empty tags: data in vendir.lock.yml, especially with depth: 1 or a warm bundle cache.

Ex: vendir.yml has ref: main. Someone tagged an older commit on main (e.g., v1.2.0 on a commit from last week) as a release marker. On a sync where the bundle cache already has most of main's history, git only transfers the handful of new commits, v1.2.0's commit isn't among them, so auto-follow skips that tag. Later, Retrieve() calls git describe --tags HEAD to populate tags: in vendir.lock.yml, and it comes back empty or points to some other/older tag it does know about. Silently less accurate lock-file metadata than before the PR, with no error raised.

carvel-bot added this to Carvel Jun 24, 2026

zendesk-slowery23 changed the title ~~Optimize git fetch for repos with many tags and branches~~ perf(git): optimize fetch for repos with many tags and branches Jun 24, 2026

zendesk-slowery23 commented Jun 24, 2026

View reviewed changes

zendesk-slowery23 marked this pull request as ready for review June 24, 2026 21:32

Copilot AI review requested due to automatic review settings June 24, 2026 21:32

Copilot started reviewing on behalf of zendesk-slowery23 June 24, 2026 21:33 View session

Copilot AI reviewed Jun 24, 2026

View reviewed changes

joaopapereira approved these changes Aug 13, 2026

View reviewed changes

joaopapereira closed this Aug 15, 2026

github-project-automation Bot moved this to Closed in Carvel Aug 15, 2026

joaopapereira reopened this Aug 15, 2026

github-project-automation Bot moved this from Closed to In Progress in Carvel Aug 15, 2026

zendesk-slowery23 force-pushed the git-fetch-performance-optimization branch from 52b1819 to 3c251aa Compare August 18, 2026 19:57

Fix targeted git fetch regressions

d29bc66

Signed-off-by: Steve Lowery <steve.lowery@zendesk.com>

zendesk-slowery23 force-pushed the git-fetch-performance-optimization branch from e457e8f to d29bc66 Compare August 25, 2026 14:20

joaopapereira requested changes Aug 28, 2026

View reviewed changes

Conversation

zendesk-slowery23 commented Jun 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Problem

Solution

--locked path

Performance

Changes

Testing

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

zendesk-slowery23 commented Jun 25, 2026

Uh oh!

zendesk-slowery23 commented Jul 29, 2026

Uh oh!

joaopapereira left a comment

Choose a reason for hiding this comment

Uh oh!

joaopapereira commented Aug 13, 2026

Uh oh!

zendesk-slowery23 commented Aug 18, 2026

Uh oh!

joaopapereira commented Aug 21, 2026

Uh oh!

zendesk-slowery23 commented Aug 25, 2026

Uh oh!

zendesk-slowery23 commented Aug 27, 2026

Uh oh!

joaopapereira left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

zendesk-slowery23 commented Jun 24, 2026 •

edited

Loading

`--locked` path