Skip to content

perf(server): index a new worktree in ~40s, not 6–9 min, by starting from the main checkout's index - #178

Closed
raimonade wants to merge 1 commit into
ScriptedAlchemy:mainfrom
raimonade:raimonade/seed-worktree-code-index
Closed

raimonade wants to merge 1 commit into
ScriptedAlchemy:mainfrom
raimonade:raimonade/seed-worktree-code-index

Conversation

@raimonade

Copy link
Copy Markdown

Every agent that starts in a new worktree triggers a full code index of that worktree from nothing. On a large repo (16.5k files, ~10k indexed, ~210k symbols) that took 443–542 s per worktree in my server traces, and ~870 MB of index each. The agent usually finished its work before the index did, while two of those builds competed with everything else for CPU and memory, and the index was deleted again once the thread settled.

The worktree's sources are almost identical to its main checkout's, which usually already has a published index. Index paths are checkout-relative and every file record carries a content hash, so that index is valid for the worktree except for the files that differ.

What changed

buildIndexInWorker (apps/server/src/project/knowledge/codeIntelligenceWorker.ts): when a checkout has no published generation and is a linked git worktree, the build copies the main checkout's published database into its first generation, the same way a successor already copies its predecessor (FICLONE, so a reflink on APFS/btrfs), then runs the usual incremental sync. Only files whose hash differs are re-parsed.

  • The main checkout is found from git rev-parse --git-dir --git-common-dir; a worktree of a bare repository has none and builds from scratch as before.
  • The main checkout's generation is pinned with the existing reader lock while it is copied, so its own refresh cannot retire it mid-copy.
  • The seed is only used under the same rules that let a successor inherit its predecessor: matching configuration digest (.gitignore/codegraph.json), and a ranked listing when it is over the cap. Otherwise the build starts from scratch, as today.
  • Nothing about cleanup changes; the worktree's index is still released when its thread settles.

Evidence

Regression test: starts a linked worktree's first index from its main checkout's in CodeIntelligence.test.ts uses a real git worktree add. The worktree lists only its new file, so a symbol from an unlisted-but-checked-out file can only be in its index if it came from the main checkout's.

without the change:  AssertionError: expected +0 to equal 1    Tests  1 failed | 54 skipped
with the change:     Tests  1 passed | 54 skipped

vp test run CodeIntelligence.test.ts CodeIndexReactor.test.ts codeIndexLocks.test.ts: Test Files 3 passed (3), Tests 103 passed | 5 skipped (108).

Real repository: buildIndexInWorker called directly from a script on an APFS clone of a live main-checkout index in a temp directory (M-series Mac, load average ~20 from other work during the run, one run each):

Build Time Files already in the index
Main checkout, configuration changed so it starts over 873 s 0
Worktree first build, seeded from main (41 files differ) 42.8 s 9,999
Main checkout refresh, nothing changed 1.9 s –
Worktree first build, from scratch (lighter load) 368 s 0

SDK phase timings (CODEGRAPH_SYNTH_TIMINGS=1) for the seeded build: sync-scoped: 581ms (10011 paths, 9999 tracked), sync-failed-ref-retry: 4969ms (11666 refs) against 97964ms (71362 refs) for the from-scratch build.

vp run check:changed: format, lint, knip, graphql and typecheck pass. Its test step was skipped locally because the project-memory addon needs cargo, which this machine lacks; the test files above were run directly instead.

Details

  • A main-checkout index whose configuration digest is stale (for example after a .gitignore edit, before the main checkout's next build) is not used as a seed; that worktree builds from scratch, exactly as before.
  • Main checkouts registered through a symlinked path will not match git's physical path, so they fall back to the from-scratch build rather than seeding from the wrong index.
  • Windows: the worktree resolution only uses git rev-parse output and existing path helpers; it is untested there.

ELI5

When an agent opens a copy of a project to work in, T3 used to re-read the whole project from scratch to build its code map, which took longer than the agent's work. Now it starts from the map it already has for the original project and only re-reads the few files that are different.

@changeset-bot

changeset-bot Bot commented Nov 22, 2026

Copy link
Copy Markdown

⚠️ No Changeset found

Latest commit: 9c82541

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

@raimonade raimonade closed this Nov 22, 2026
@raimonade

Copy link
Copy Markdown
Author

apologies, meant to send clanker to do the pr on my fork

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant