Repository navigation
fix(import): recognise an origin-less --dir checkout by a hash of its git root - #1427
Conversation
No other confident findings in the specified diff. I did not run, build, or install PR code. |
|
Added the record to the body, under Real-CLI verification: Run 1's baseline reads back as The |
SaulMoro
left a comment
There was a problem hiding this comment.
This was generated by AI during triage.
@Smilewithoutfalling Thanks, the fix works. With the built CLI, import --dir <svc> --incremental gives these results on main and on this head:
| Scenario | main | this head |
|---|---|---|
| Origin-less checkout, rerun after editing one file | full run, 3 files | incremental, 1 file |
| A different checkout with the same basename | full run | full run |
| The checkout moved | full run | full run |
| Manifest written by main | — | one full run, then incremental |
| Checkout with an origin | incremental, as before | incremental, as before |
The new tests fail on main and pass here, also under a symlinked TMPDIR.
One change before approval:
[P1 blocking] localRoot is written for checkouts that have a remote, and nothing reads it there. sourceRepositoryMetadata sets localRoot for every checkout (src/codebase-extract.ts:656, :673, :684), so every import puts the member's absolute path (/Users/<name>/…, or D:\… as in the body's own record) into source-manifest.json, which the team repo commits. For a checkout with a remote, manifestMatchesSource compares repoIdentity and never looks at localRoot.
The stated reason, "dropping an origin afterwards does not throw away a usable baseline", does not hold. A manifest that names a remote is refused as a baseline for an origin-less checkout (:633-634). With the built CLI, removing origin after a remote import gives a full run on this head too.
Write localRoot only when there is no remote identity. Then adjust test case 2 and drop that line from the comment and the body.
[P2 non-blocking] For origin-less checkouts the raw path is still published to the team. Storing a hash of the real git root (for example sha256(realGitRoot)) would match exactly the same way without exposing the username.
Heads-up for whichever of this and #1417 lands second: #1417's sameRepository decides whether curated docs carry over by comparing repoUrl, separately from manifestMatchesSource. An origin-less checkout will still not carry its docs across re-imports there, even though this PR now gives it an incremental baseline. We found this by reading the source, not by building both together.
CI review: it reported a missing real-CLI record. We agree, and the body's built-CLI run now resolves it. It did not flag the point above.
Verified at 27b41bd:
27b41bd to
c1600a3
Compare
Resolved: The earlier missing real-CLI verification finding is resolved: the description now documents a post-build CLI run demonstrating origin-less incremental baseline reuse. I only read the specified diff; I did not run, build, or install PR code. |
…t root A `--dir` checkout with no `origin` remote has no `repoIdentity`, so `manifestMatchesSource` rejected the published baseline on every run and `--incremental` silently re-extracted everything. Record `sha256(realpath(git root))` as `localRootHash` — only when the checkout has no remote, because `repoIdentity` identifies a remote checkout and nothing reads a hash there. `sameRepository` compares the same two values, so an origin-less checkout keeps its curated docs across a re-import instead of having them replaced. Both the hash and `sourceSubdir` have to match: two subdirectories of one origin-less repository share the git root and are still separate codebases. Refs Tencent#1422 (gap 3).
c1600a3 to
ee188f8
Compare
|
Pushed What was wrong. Evidence, real CLI. One repository containing The hash is identical in all three runs, so it identifies the repository and not the codebase; Tests. Nine cases now. The new one asserts both halves — that re-importing
Gate. One thing for whoever merges. #1440, opened after this PR, replaces |
|
No new confident findings in the specified diff. Resolved
The documented testing satisfies the review criteria. I only read the specified diff; I did not run, build, or install PR code. |
|
@SaulMoro @jeff-r2026 — the review items and the One ordering question, because If |
SaulMoro
left a comment
There was a problem hiding this comment.
This was generated by AI during triage.
@Smilewithoutfalling Thanks, all three items are resolved. With the built CLI at ee188f8:
- A checkout with a remote no longer writes a local root: its manifest has only
repoUrlandrepoIdentity. - An origin-less checkout publishes
localRootHash, which matches the sha256 of the real path, and no path appears in the manifest. - The same checkout with
--incrementalre-extracts 1 file and keeps the curated docs. - A different checkout with the same basename, a sibling subdirectory, and a checkout whose origin was removed each get a full run that replaces the docs, so the
sameRepositoryproblem from #1417 is gone too.
[P3 nit] The hash is unsalted, so someone with read access to the team repo could confirm a guessed path such as /Users/<name>/src/<repo>. Not blocking.
On ordering: this goes first. It is approved and smaller, and #1440 can carry the port on its rebase. That port needs CodebaseSourceSchema to accept a manifest without repoUrl and to declare localRootHash and sourceSubdir. Otherwise an origin-less manifest parses to null and the curated-doc carry-over from this PR is silently lost.
Verified at ee188f8: build, tsc and lint pass; the 10 related unit files (158 tests) and 4 e2e files pass, the new file also under a symlinked TMPDIR; CI is green. We agree with the CI review that both earlier items are resolved.
…n guarding its evidence Tencent#1427 compares localRootHash in sameRepository, which this branch replaced with publishedSource. Parse the published and staged sources as a remote or a local root, and treat two origin-less checkouts as the same source when the hash and the directory match. A differing hash stays unknown, so a checkout moved to another path re-imports instead of being refused. An origin-less checkout over evidence a remote repo published is now refused rather than replacing it, as the other refusals are.
…ce slug (Tencent#1422) A --dir slug is the checkout's basename, so /a/svc and /b/svc both wrote teamwiki/evidence/code/svc/ and the second import replaced the first's evidence. importLocalCheckout decides the slug once, before anything is seeded or extracted: a git repository answers by its remote identity — every checkout of one repo keeps sharing its evidence from any path (Tencent#823) — and a checkout without one answers by its hashed local root. That root now covers directories without a repository too: sourceRepositoryMetadata hashes the source root itself, extending Tencent#1427's localRootHash, so no raw path is published in the teamwiki. Differing identities publish under <slug>-<hash8(realpath)> and leave the first codebase's evidence untouched; a legacy sidecar with neither marker keeps today's behaviour, so nothing already imported moves.
…ce slug (Tencent#1422) A --dir slug is the checkout's basename, so /a/svc and /b/svc both wrote teamwiki/evidence/code/svc/ and the second import replaced the first's evidence. importLocalCheckout decides the slug once, before anything is seeded or extracted: a git repository answers by its remote identity — every checkout of one repo keeps sharing its evidence from any path (Tencent#823) — and a checkout without one answers by its hashed local root. That root now covers directories without a repository too: sourceRepositoryMetadata hashes the source root itself, extending Tencent#1427's localRootHash, so no raw path is published in the teamwiki. Differing identities publish under <slug>-<hash8(realpath)> and leave the first codebase's evidence untouched; a legacy sidecar with neither marker keeps today's behaviour, so nothing already imported moves.
…ce slug (Tencent#1422) A --dir slug is the checkout's basename, so /a/svc and /b/svc both wrote teamwiki/evidence/code/svc/ and the second import replaced the first's evidence. importLocalCheckout decides the slug once, before anything is seeded or extracted: a git repository answers by its remote identity — every checkout of one repo keeps sharing its evidence from any path (Tencent#823) — and a checkout without one answers by its hashed local root. That root now covers directories without a repository too: sourceRepositoryMetadata hashes the source root itself, extending Tencent#1427's localRootHash, so no raw path is published in the teamwiki. Differing identities publish under <slug>-<hash8(realpath)> and leave the first codebase's evidence untouched; a legacy sidecar with neither marker keeps today's behaviour, so nothing already imported moves.
…n guarding its evidence Tencent#1427 compares localRootHash in sameRepository, which this branch replaced with publishedSource. Parse the published and staged sources as a remote or a local root, and treat two origin-less checkouts as the same source when the hash and the directory match. A differing hash stays unknown, so a checkout moved to another path re-imports instead of being refused. An origin-less checkout over evidence a remote repo published is now refused rather than replacing it, as the other refusals are.
…and keep AI components on --skip-enrich (#1440) * fix(import): name a --dir checkout with an origin after its remote repo Two --dir checkouts with the same basename shared one evidence directory, so importing /b/svc replaced the evidence of /a/svc. A checkout root whose origin names a repo now takes the slug --from-repo, --from-mr and ci extract-mr give that repo. * fix(import): refuse to replace evidence published from a different repo A checkout named after its directory (no origin, or a subdirectory) can still share a slug with another codebase. When the published source-manifest.json names a repository and the checkout being imported is a different one, or has no remote, stop with an actionable error instead of replacing that codebase's evidence. * fix(import): move a checkout's old directory-named evidence to its remote slug Before this change, --dir published a checkout with an origin under its directory name. Now that it takes the remote's slug, its next import moves that evidence, curated docs included, to the new slug instead of leaving a duplicate codebase in the teamwiki. When the remote slug already has evidence, the old copy is kept and a warning names it. * fix(import): keep AI-written components on a --skip-enrich import A daily static sync (import --skip-enrich --incremental) replaced the published AI _manifest.json with the static fallback, losing each component's category and responsibilities and the edges between them. For the same repository, keep those entries for every component still in the code, refresh their entrypoints, and give new components the static entry. * docs(knowledge): say which slug import --dir uses * fix(import): treat SSH and HTTPS clones of one repo as the same source They now share one slug, so comparing scheme-sensitive repo identities refused an import that CI over HTTPS and a member's SSH clone both make. * fix(import): refuse to replace a same-named subdirectory of the same repo --dir repo/apps/api and --dir repo/libs/api both take the slug api. Compare the extracted directory as well as the repository before treating the published evidence as this checkout's. * refactor(codebase): resolve a checkout's anchors once per slug lookup * fix(import): name roles and projects still scoping the old --dir slug A role or project that lists a checkout's directory-named slug under resources.wiki no longer matches once the evidence takes the remote's slug, and recall then shares that codebase with everyone. Warn with the entries to change. * docs: describe the --dir slug and refusal precisely, in the wiki skill too * fix(import): point kept AI components at the evidence they now live in A skipped-enrichment import that moves directory-named evidence to the remote slug kept each component's old docPath, so the manifest pointed at pages under the removed directory. Take the doc path from the fresh extraction, as the entrypoints already are. * fix(import): recognise an origin-less checkout by its hashed root when guarding its evidence #1427 compares localRootHash in sameRepository, which this branch replaced with publishedSource. Parse the published and staged sources as a remote or a local root, and treat two origin-less checkouts as the same source when the hash and the directory match. A differing hash stays unknown, so a checkout moved to another path re-imports instead of being refused. An origin-less checkout over evidence a remote repo published is now refused rather than replacing it, as the other refusals are. * fix(import): warn about directory-named evidence that names no repository Evidence published before the per-codebase source manifest, or before the checkout had an origin, cannot be shown to be this repo's, so it is not moved to the remote slug. It used to stay beside the new slug with no message, and recall kept serving its stale facts. Name it, so the team can move the curated docs and remove it. * style(import): keep the --max-files comment on parseMaxFiles * docs: scope a wiki entry by the slug teamai import prints * fix(import): suggest adding the origin back before renaming a refused checkout A checkout whose origin was removed is refused over its own evidence. Adding the origin back keeps the curated docs, since the import then moves the directory-named evidence to the remote slug. * docs(changelog): note the --dir slug change for #1422 * fix(import): suggest a new wiki scope only when the old name holds no evidence A checkout of other/svc next to evidence/code/svc/ from acme/svc leaves that evidence in place, and a role scoping svc still means it. Telling the role to switch to github__other__svc pointed it at the wrong repo.
What
--incrementalpromises to re-extract only the files changed since the last import (docs/guide/knowledge.md:151). For a--dircheckout with nooriginremote it never did: every run was a full extraction. Once the baseline does match, a second gap shows up — the re-import replaces the curated docs, becausesameRepository(src/import-codebase.ts:74, added by #1417) decides that byrepoUrl, which an origin-less checkout does not have.This is gap 3 of #1422. Gap 4 is #1425, gap 1 is #1417. Gap 2 (two checkouts with the same basename sharing one evidence directory) is not fixed here: they still share
evidence/code/<slug>/. What changes is that the second checkout no longer inherits the first one's curated docs.Ordering with #1440. That PR, opened after this one, also rewrites
sameRepository— aspublishedSource— and it refuses an origin-less import over evidence published by a remote, where this PR replaces the docs. Whichever lands first, the other needs a rebase and one test changes:replaces the docs when the published manifest names a remoteasserts today's replace, and refusing is the stricter reading, so I would drop my version of that case rather than theirs. The overlap is described on #1440.Reproduction
main@97a762ce, the realextractCodebase(no mocks), two temp git repos that differ only bygit remote add origin:originversion,lastScan,project,files,sourceSubdir,headShano matching source baseline, running full extraction, 2 files scannedoriginrepoUrl,repoIdentityincremental: 1 files changed, 0 deleted, 1 file scannedThe two rows share every input except that one config line, so the difference is the remote and nothing else.
Why
manifestMatchesSourcerequiredsource.repoIdentity(src/codebase-extract.ts:609).sourceRepositoryMetadatareturned only{sourceSubdir}when the checkout had no remote URL (:650).incrementalBaselineMatches(:700) was false on every run, and the run printed the fallback notice.Fix
For a checkout with no remote identity, record
sha256(realpath(git root))aslocalRootHashand compare it inmanifestMatchesSource; compare it insameRepositoryas well, so the docs carry over too.It is a separate field rather than a value written into
repoIdentitybecausemanifestRepoIdentityinsrc/codebase-freshness.ts:64feedsrepoIdentityinto remote-versus-remote matching — a filesystem path stored there would make freshness compare a path against URLs.What changed since the reviews
Both review points were right, including the part of my rationale that was wrong.
[P1]
localRootis no longer written for a checkout that has a remote.sourceRepositoryMetadatanow returns it only on the path whereremoteUrlis empty, sorepoIdentityidentifies a remote checkout and nothing else is added to a manifest the team commits. The reason I gave for writing it everywhere — "dropping anoriginafterwards does not throw away a usable baseline" — does not hold, and the measurement confirms your reading:manifestMatchesSourcerefuses a manifest that names a remote for an origin-less checkout (:638-639), so removingoriginafter a remote import is a full run either way.[P2] The value is a hash, not the raw path.
sha256(realpath(git root))matches exactly the same pairs of runs, andsource-manifest.jsonno longer carries/Users/<name>/….sameRepositoryhandles the origin-less case. Now that #1417 is in, comparingrepoUrlalone leaves an origin-less checkout with no identity, so its curated docs were replaced on every re-import. Both sides blank now fall through to the samelocalRootHashandsourceSubdircomparison, and a manifest without the hash (written by an older CLI) is treated as "not the same source" rather than as a match.[P1, bot review] Two subdirectories of one origin-less repo no longer share curated docs.
service-a/svcandservice-b/svcof one repository hash to the same git root, so the hash on its own called them the same source and handed the first one's docs to the second — the exact case in the review.sameRepositorynow requires the recordedsourceSubdirto be equal too, which is the conditionmanifestMatchesSourcealready applied to the baseline. Both values are in the manifest the review pointed at; the real-CLI run below is that scenario end to end.Compatibility
localRootHash, so each origin-less codebase pays one more full extraction and increments from then on.--incrementalfalls back to one full extraction, and a re-import throughimport --dirreplaces the curateddocs/instead of carrying it over. That is the safe direction. The alternative — falling back to the slug — is the same-basename collision this PR refuses to reintroduce, and a moved checkout cannot be told apart from "another checkout with the same name"; only one of those two readings is ever safe, and the basename is the unsafe one. So a move costs one full run and the AI-written docs, and the trade-off is deliberate, not an oversight.Tests
src/__tests__/codebase-incremental-no-origin.test.ts, nine cases against real git repos:localRootHashexisted;service-a/svcthenservice-b/svc, and the case first asserts that re-importingservice-a/svcstill keeps its docs, so it cannot pass by never carrying anything;Three reverse arms, each reverting one part of the fix and leaving the tests intact:
sameRepositoryback tomain's versionlocalRootHashwrite onlysourceSubdircomparison onlyCase 8 fails in all three because it needs the whole chain: a hash that matches, the write that records it, and the directory check that tells two codebases apart. The others fail only where their own part is removed, and 3–5, 7, 9 are a guard rather than a restatement of the fix.
Real-CLI verification
npm run build(tsup), then the built CLI against a git repo with no remotes at all (git remote -vempty), inside a single-mode team repo:The baseline published by run 1, read back from
teamwiki/evidence/code/svc3/source-manifest.json:No
localRoot, norepoUrl, norepoIdentity, and no path anywhere in the file.sha256(realpath(.../verify/a/svc3))computed independently from the shell gives the same digest, so the field is a hash of the checkout and not of something else.Two subdirectories of one origin-less repo, which is the case the bot review raised — one repository with
service-a/svcandservice-b/svc, imported under the basename slug both of them derive:Same hash in all three runs, so the hash genuinely identifies the repository and not the codebase; the subdirectory is what separates them.
Three more runs with the same binary:
docs/, then one source file editedincremental: 1 files changed;docs/curated-by-ai.mdis still there afterwardsoriginrepoUrl+repoIdentityand nolocalRootHashorigin, thengit remote remove origin, then--incrementalno matching source baseline, running full extraction, 2 files — dropping an origin does not reuse the old baseline--incrementalno matching source baseline, running full extraction, 3 files;localRootHashbecomes24cb91f7…, which issha256(realpath(new path))computed separately, and the curateddocs/is goneThe
autoPush failedline in these runs comes from the stand-ingit@github.com:acme/team.gitorigin on the sandbox team repo, not from the source checkout.Verification
tsc --noEmitoxlint --type-aware --deny-warnings --report-unused-disable-directivescodebase-incremental-no-origin+import-codebase+codebase+codebase-freshnesscodebase-reconcile.test.tsstill fails one case on a Windows checkout; it fails identically withsrc/codebase-extract.tsreverted tomain, so it predates this change. CI covers the rest.