diff --git a/docs/kg-entity-key-fix-design.md b/docs/kg-entity-key-fix-design.md new file mode 100644 index 0000000..cc75deb --- /dev/null +++ b/docs/kg-entity-key-fix-design.md @@ -0,0 +1,219 @@ +# ADR: Fix Entity primary-key / project-scope mismatch in the kg indexer + +Status: **Accepted — Option A (A1 variant)**. Multi-project-per-DB is a supported +scenario (product decision, 2026-10-09), so the global-PK bug is fixed by a +namespaced primary key, not by foreclosing multi-project DBs. +Date: 2026-10-09 + +## Context + +`kg index` emits large volumes of warnings of the form: + +``` +Note: COPY FROM JSON unavailable (... Found duplicated primary key value ...); falling back to row-by-row insert +Warning: insert entity :: ... Found duplicated primary key value , which violates the uniqueness constraint ... +``` + +These are **not benign**. Investigation (source-verified) shows: + +1. **The `Entity` primary key is global `id` alone** — `src/kglib/schema.go:45` + (`id STRING PRIMARY KEY`). Same for `Observation` (`schema.go:57`). +2. **Every clear/delete and nearly every read is scoped by `project_id`**: + - `clearCodeDerivedData` — `src/kg/internal/knowledge/indexer.go:280-288` + - `clearProjectData` — `indexer.go:373-399` + - `deleteProjectRelations` — `indexer.go:209-226` + - reads in `graph.go:136-178`, `ListEntities` (`src/kglib/entity.go:326`) +3. **In-run entity dedup is airtight**: `writeEntityVis` (`indexer.go:584`) is the + *only* append site and guards every append with `if seen[id] { return false }`. + So a batch never contains duplicate ids — collisions are strictly against rows + **already in the DB**. +4. **Relations match endpoints by bare `id` with no project filter in the MATCH** — + `CreateRelation`, `src/kglib/relation.go:24-28` + (`WHERE from.id = $from_id AND to.id = $to_id`). + +### Failure mechanism + +Because the PK is global but the clear is project-scoped, any pre-existing +code-derived row under a **different `project_id`** survives the clear. When the +current run writes the identically-id'd entity it collides and is **silently +dropped** (first/old row wins; `CREATE` at `indexer.go:659-677` and the COPY path +at `indexer.go:644-653`; the log indexer repeats the pattern at +`log_indexer.go:278-335`). Downstream: + +- The current project's entity, plus its freshly-computed observations/relations + (which attach **by id**), never land. +- Relations can bind to the *other* project's surviving node (relation.go MATCH is + project-blind), or be dropped as dangling (`batchCreateRelations`, indexer.go:763). +- `kg stats` / graph reads scoped to the live `project_id` under-return. + +A mismatched-`project_id` partition arises from any of: legacy `project_id = ""` +rows, a changed `projectIDFromCwd` (`src/kg/index.go:140`) across versions, a +moved/renamed repo, or two scope configs whose `cfg.Database` point at the same DB +file (`indexScopeDB`, `src/kg/index.go:103-118`). The *presence* of the warnings +proves such a partition exists — a same-project prior run would have been emptied +by the clear, producing no collision. + +### Hard constraint + +Kuzu **v0.11.3** (`src/kg*/go.mod`) node tables support exactly **one** PRIMARY KEY +column. A literal composite `(project_id, id)` node PK is **not expressible**. The +realistic options are a namespaced single key, or a structural one-project-per-DB +guarantee. + +## Gating open question (needs a human decision) + +**Is "multiple `project_id`s in one DB file" a supported scenario, or an accident?** + +- The deployment path opens **one DB per scope** and derives `project_id` from cwd + (`index.go:103-140`), i.e. one project per file in practice. +- But `KGMeta` keys by `project_id` (`schema.go:69`), and reads filter by + `project_id` — the schema *looks* built for many projects per file. +- Federated linking (`graph_federated.go`) links across **separate** graph files, + not multiple projects in one file. + +The answer selects the recommended option: + +- **One project per DB is the real model → Option B (+ C).** Minimal, correct. +- **Multi-project-per-DB must be supported → Option A.** Larger, fully correct. + +Recommendation absent other input: **one project per DB is the real model → Option B +primary, C as hardening.** Multi-project-per-file appears to be the accident that +creates stale partitions. + +## Accepted design — Option A1 (namespaced `pk`, `id` stays local) + +Decision: multi-project-per-DB **must** be supported, so Option B is rejected and +the key is made project-safe via a namespaced primary key. Within Option A, the +**A1** variant is chosen: keep the existing `id` column human-facing and unchanged +(`file:…`, `function:…`, UUIDs), and add a separate namespaced primary-key column. +This preserves the id contract that search, CLI display, and the journal +`stableIDHint` all depend on; A2 (rewriting `id` itself) is rejected as needlessly +breaking those. + +Concrete decisions: + +1. **Schema** (`src/kglib/schema.go`): Entity and Observation gain + `pk STRING PRIMARY KEY`; the current `id` becomes a plain (non-unique) STRING + attribute. `entity_id` on Observation stays, but observation→entity joins key on + the pair `(project_id, id)` (see 4). +2. **Key derivation** — one central helper: + `pk(projectID, id) = projectID + "\x00" + id`. NUL is the separator because + neither a `project_id` nor a code-derived/UUID `id` ever contains one; document + that invariant next to the helper. All writes derive `pk` here, never ad hoc. +3. **Writes populate `pk`**: COPY column lists (`indexer.go:646`, the log-indexer + COPY in `log_indexer.go`, the observation COPY) and every `CREATE` + (`indexer.go:659-677`, `entity.go:148`, `log_indexer.go` fallback) add `pk`. +4. **Lookups/endpoint matches become project-qualified** (the real correctness + fix): every `{id: …}` / `WHERE e.id = …` that is not already project-scoped must + match on `(project_id, id)` (equivalently on `pk`): + - `CreateRelation` MATCH — `relation.go:24-28` (currently project-blind — this is + the cross-project-bleed site). Add `from.project_id = $pid AND to.project_id = + $pid`, or match on `from.pk`/`to.pk`. + - `GetEntity` — `entity.go` (confirm it filters project_id; if it keys on id + alone, qualify it). + - Observation joins / `HAS_OBSERVATION` creation keyed on `entity_id`. + - `clearCodeDerivedData`, `clearProjectData`, `deleteProjectRelations` already + filter `project_id`; with a project-unique `pk` the DETACH DELETE now both + targets only this project's rows **and** stops colliding — no logic change + beyond confirming they read the new column where needed. +5. **Upsert**: with `pk` project-safe, switch entity writes `CREATE` → `MERGE` on + `pk` with last-write-wins refresh of `name`/`type`/`updated_at`. This makes + re-index idempotent and eliminates the warnings outright. (`MERGE` was unsafe + under the old global PK; it is correct now.) +6. **Federated linking** — `graph_federated.go`: cross-graph joins must key on the + **local `id`** (and/or name), never on `pk`, since `pk` embeds a per-graph + `project_id` that differs across files. Audit every cross-graph comparison. +7. **Migration**: Kuzu has no `ALTER TABLE … ADD PRIMARY KEY`, so the new `pk` + column forces a table recreate → rebuild-from-source. Bump `KGMeta.kg_version` + (`StampMeta`, `index.go:216`); on version mismatch, ride `archiveIfMismatch` + (`index.go:124`) to archive the old DB and rebuild fresh, then replay hand-writes + via `restoreHandWrites` (`index.go:210`). The journal is the only copy of + hand-writes — archive, never delete. + +Open sub-question for the engineer to resolve during implementation: whether Kuzu +rel tables in v0.11.3 let `MERGE`/`MATCH` resolve endpoints efficiently by a +non-PK `(project_id, id)` pair, or whether endpoint matching should go through the +`pk` column directly (likely faster, since `pk` is the indexed PK). Prefer `pk`. + +## Options (for the record) + +### Option A — Namespaced single primary key (supports multi-project-per-DB) + +Store PK as `pk = project_id + "\x00" + localID`; keep `id` (the current +`file:…`/`function:…`/UUID value) as a **non-PK indexed** attribute for display, +search, and the journal hint. + +- Rewrite all MATCH/CREATE/COPY on Entity/Observation to key on `pk`, including + relation endpoint matching (`relation.go:24-28` must match on `pk`, not `id`). +- Blast radius (each must change): + - id construction sites: `indexer_treesitter.go:198` (`qualifiedSymbolID`), + and `fmt.Sprintf("file:%s"|"type:%s:%s"|"function:%s:%s"|"import:%s")` across + `indexer_yaml.go`, `indexer_makefile.go`, `indexer_asm.go`, + `indexer_jsonschema.go`, `indexer_graphql.go`, `indexer_html.go`, + `indexer_markdown.go`, `indexer_pdf.go`, `indexer_cmake.go` — these now feed + `localID`, with `pk` derived centrally in `writeEntityVis`. + - `codeDerivedIDClause` / `codeDerivedIDPrefixes` — `kg_types.go:34-52` + (prefix test must run against `localID`, not `pk`). + - COPY column lists — `indexer.go:646`, `log_indexer.go` COPY, observation COPY. + - reads — `graph.go`, `entity.go` (`ListEntities`, `GetEntity`), + `src/kglib/search.go`. + - journal `stableIDHint` and replay — `relation.go` note, `index.go:210` + (`restoreHandWrites`); the hint must remain the *localID* so replay still works. + - federated cross-graph linking — `graph_federated.go` (joins by id across files). +- Pro: preserves multi-project-per-DB; fully correct. Con: wide, touches the + human-facing/stable id contract and the journal. + +### Option B — Enforce one `project_id` per DB file (recommended primary) + +At index time, read the DB's existing `KGMeta.project_id`. If it differs from the +current run's `project_id` (or legacy `""` rows exist), treat it exactly like a +storage-format mismatch: **archive the old DB and rebuild fresh** — reusing the +existing `archiveIfMismatch` lever (`index.go:124`). With one project per file, the +global `id` PK is project-scoped by construction and collisions cannot occur. + +- Change surface is small and localized to `index.go` (a project_id check beside + `archiveIfMismatch`) plus a KGMeta read helper. +- Journal safety: archived DB keeps its journal (`index.go:128-132`); hand-writes + replay into the fresh DB via `restoreHandWrites` (`index.go:210`), which is + idempotent. +- Pro: minimal, correct, matches real deployment. Con: forecloses + multi-project-per-DB (acceptable iff the gating question resolves that way). + +### Option C — Global code-derived clear + MERGE upsert (hardening, pairs with B) + +Under the one-project-per-DB premise, code-derived ids are globally unique within a +DB, so: + +- Drop the `{project_id: $project_id}` filter from the **code-derived** DETACH DELETE + in `clearCodeDerivedData` (indexer.go:280-288) so a rebuild clears *all* + code-derived rows regardless of a stale project_id — making re-index idempotent and + warning-free even on a legacy DB. +- Switch entity writes `CREATE` → Kuzu **`MERGE`** (idempotent upsert) with + last-write-wins refresh of `name`/`type`/`updated_at`. + - **MERGE is unsafe under the current global PK** (one project would overwrite + another's row). It is correct only after B (or A) makes the key project-safe. + +## Migration & schema versioning + +- Bump the schema/kg version stamp (`KGMeta.kg_version`, written by `StampMeta`, + `index.go:216`) so older DBs are recognized. +- Option B needs no table rewrite (fresh rebuild on project_id mismatch). +- Option A requires a one-time rebuild (new PK column); ride `archiveIfMismatch`: + on version mismatch, archive + rebuild-from-source (code-derived data is + regenerable), then journal-replay hand-writes. +- In both, the journal is the **only** copy of hand-written entities — migration must + archive (not delete) the prior DB and run `restoreHandWrites` against the fresh one. + +## Acceptance criteria + +- [ ] `cd src/kg && go build ./... && go test ./...` exits 0 +- [ ] `cd src/kglib && go build ./... && go test ./...` exits 0 +- [ ] New regression test: index project A, then project B that shares entity ids + (e.g. both `import:logging`, `file:common.py`); assert **both** projects retain + their own entities and `kg stats` per project is correct. +- [ ] Re-indexing an already-indexed project emits **zero** `duplicated primary key` + warnings and **no** `COPY FROM JSON unavailable` note. +- [ ] Hand-written entities/relations (`kg add`) survive a re-index and a migration + rebuild (journal replay verified). +- [ ] `MATCH (e:Entity) RETURN DISTINCT e.project_id` on a migrated DB shows exactly + one project_id (Option B) — a field diagnostic to confirm the fix landed. diff --git a/src/kg/graph.go b/src/kg/graph.go index acf011a..8e70a31 100644 --- a/src/kg/graph.go +++ b/src/kg/graph.go @@ -302,7 +302,7 @@ func runFederatedGraph(cmd *cobra.Command, s graphSettings, out io.Writer) (know graph, report, err := knowledge.LoadFederatedGraph(knowledge.FederatedGraphOptions{ AIDir: aiDir, Scope: scope, - ProjectID: filepath.Base(root), + ProjectID: projectIDFromCwd(root), OnlyLayers: graphLayers, MaxJoinLayers: graphMaxJoin, JoinTypes: parseJoinTypes(cmd), diff --git a/src/kg/handle_server.go b/src/kg/handle_server.go index 5a96e26..d0a78d0 100644 --- a/src/kg/handle_server.go +++ b/src/kg/handle_server.go @@ -23,7 +23,7 @@ func handleServer(cmd *cobra.Command) { } projectRoot := findProjectRoot(cwd) aiDir := filepath.Join(projectRoot, ".ai") - projectID := filepath.Base(projectRoot) + projectID := projectIDFromCwd(projectRoot) // Determine which scope to use for the MCP server // Priority: default scope > legacy mode (knowledge.db) diff --git a/src/kg/helpers.go b/src/kg/helpers.go index 9d05861..a1ef03c 100644 --- a/src/kg/helpers.go +++ b/src/kg/helpers.go @@ -62,10 +62,32 @@ func findProjectRoot(dir string) string { return dir } +// pkSeparator mirrors the Unit Separator (0x1F) that kglib.PK joins project_id +// and id with to form a collision-free primary key. project_id must never +// contain it, or the pk for a whole project would be corrupted. +const pkSeparator = '\x1f' + // projectIDFromCwd returns the project ID for the given working directory. // The project ID is the base name of the project root. +// +// It also guards the one invariant kglib.PK rests on — that project_id never +// contains the 0x1F separator. This is the derivation helper every path-derived +// project_id flows through (index, search, stats, show, meta, push, the MCP +// server, and the federated-graph filter all route through it), so guarding the +// byte here guards it everywhere the value is matched against the stored +// project_id column. A repo directory basename holding 0x1F is effectively +// impossible, so this defends the invariant rather than handling a real case: +// if one ever appeared we replace the byte (non-fatally, with a warning) +// instead of letting a raw separator reach PK and silently collapse every pk +// for the project. func projectIDFromCwd(cwd string) string { - return filepath.Base(findProjectRoot(cwd)) + id := filepath.Base(findProjectRoot(cwd)) + if strings.ContainsRune(id, pkSeparator) { + fmt.Fprintf(os.Stderr, + "Warning: project_id %q contains the reserved 0x1F separator; replacing with '_' to preserve primary-key uniqueness\n", id) + id = strings.ReplaceAll(id, string(pkSeparator), "_") + } + return id } // openStore locates the project root, opens the knowledge graph store in @@ -120,5 +142,5 @@ func openStoreModeWithScope(readOnly bool, scopeName string) (*knowledge.Store, if err != nil { return nil, "", err } - return store, filepath.Base(root), nil + return store, projectIDFromCwd(root), nil } diff --git a/src/kg/helpers_test.go b/src/kg/helpers_test.go new file mode 100644 index 0000000..eb246f3 --- /dev/null +++ b/src/kg/helpers_test.go @@ -0,0 +1,29 @@ +package main + +import ( + "os" + "path/filepath" + "strings" + "testing" +) + +// A project_id derived from a repo directory basename must never contain the +// 0x1F Unit Separator that kglib.PK uses to join project_id and id, or the pk +// for the whole project would collapse. projectIDFromCwd guards that invariant +// at its single derivation site. +func TestProjectIDFromCwdReplacesSeparator(t *testing.T) { + base := t.TempDir() + dir := filepath.Join(base, "weird\x1fname") + if err := os.Mkdir(dir, 0o755); err != nil { + t.Fatalf("mkdir: %v", err) + } + + got := projectIDFromCwd(dir) + + if strings.ContainsRune(got, pkSeparator) { + t.Fatalf("project_id still contains reserved 0x1F separator: %q", got) + } + if got != "weird_name" { + t.Fatalf("project_id = %q, want %q", got, "weird_name") + } +} diff --git a/src/kg/index.go b/src/kg/index.go index 8c77660..281a290 100644 --- a/src/kg/index.go +++ b/src/kg/index.go @@ -45,7 +45,7 @@ Examples: // Index each scope totalStart := time.Now() for _, scopeName := range scopes { - if err := indexScopeDB(root, aiDir, scopeName); err != nil { + if err := indexScopeDB(root, aiDir, scopeName, nil); err != nil { return fmt.Errorf("index scope %s: %w", scopeName, err) } } @@ -100,7 +100,10 @@ func determineScopesToIndex(aiDir, requestedScope string, all bool) ([]string, e return names, nil } -func indexScopeDB(root, aiDir, scopeName string) error { +// missingPKHint is forwarded to archiveIfMismatch: the `kg migrate` caller has +// already computed databaseMissingPK for this database and passes it in to avoid +// a second read-only open; the standalone `kg index` caller passes nil. +func indexScopeDB(root, aiDir, scopeName string, missingPKHint *bool) error { var dbPath string var scopeConfig *knowledge.ScopeConfig @@ -121,7 +124,7 @@ func indexScopeDB(root, aiDir, scopeName string) error { // all, so the check has to happen before the open, not in response to its // error. Indexing is exactly the operation that repairs one: the derived // content is about to be regenerated from the tree anyway. - aside, err := archiveIfMismatch(dbPath, os.Stdout) + aside, err := archiveIfMismatch(dbPath, os.Stdout, missingPKHint) if err != nil { return fmt.Errorf("rebuild %s: %w", dbPath, err) } diff --git a/src/kg/internal/knowledge/graph_federated_test.go b/src/kg/internal/knowledge/graph_federated_test.go index 6e6dff4..6404522 100644 --- a/src/kg/internal/knowledge/graph_federated_test.go +++ b/src/kg/internal/knowledge/graph_federated_test.go @@ -269,8 +269,8 @@ func seedEntityWithID(t *testing.T, aiDir, layer, id, name, entityType string) { defer store.Close() result, err := store.QueryParams(` - CREATE (e:Entity {id: $id, name: $name, type: $type, project_id: $project_id}) - `, map[string]any{"id": id, "name": name, "type": entityType, "project_id": fedProject}) + CREATE (e:Entity {pk: $pk, id: $id, name: $name, type: $type, project_id: $project_id}) + `, map[string]any{"pk": PK(fedProject, id), "id": id, "name": name, "type": entityType, "project_id": fedProject}) if err != nil { t.Fatalf("seed entity %s in %s: %v", id, layer, err) } diff --git a/src/kg/internal/knowledge/indexer.go b/src/kg/internal/knowledge/indexer.go index bc5e996..1483775 100644 --- a/src/kg/internal/knowledge/indexer.go +++ b/src/kg/internal/knowledge/indexer.go @@ -626,6 +626,7 @@ func (idx *Indexer) batchCreateEntities(entities []entityRecord) error { enc := json.NewEncoder(f) for _, ent := range entities { row := map[string]string{ + "pk": PK(ent.ProjectID, ent.ID), "id": ent.ID, "name": ent.Name, "type": ent.Type, @@ -641,9 +642,12 @@ func (idx *Indexer) batchCreateEntities(entities []entityRecord) error { } f.Close() - // Bulk load via COPY FROM JSON + // Bulk load via COPY FROM JSON. pk is the project-namespaced primary key + // (first column so it leads the node table's key); the selective clear + // already removed this project's code-derived rows, so within one project + // every pk in the batch is new and the load cannot collide. query := fmt.Sprintf( - `COPY Entity(id, name, type, project_id, created_at, updated_at, visibility) FROM '%s'`, + `COPY Entity(pk, id, name, type, project_id, created_at, updated_at, visibility) FROM '%s'`, ndjsonPath, ) result, err := idx.store.Query(query) @@ -653,20 +657,29 @@ func (idx *Indexer) batchCreateEntities(entities []entityRecord) error { } // COPY FROM JSON not supported by this Kuzu version — fall back to - // individual parameterized inserts (slower but universally compatible). + // individual parameterized upserts (slower but universally compatible). + // + // MERGE on pk, not CREATE: pk is now project-safe, so last-write-wins on a + // re-indexed symbol is correct and idempotent — a second index of the same + // project refreshes the row instead of raising a duplicate-primary-key + // warning the way CREATE did. fmt.Printf("Note: COPY FROM JSON unavailable (%v); falling back to row-by-row insert\n", err) for _, ent := range entities { r, err := idx.store.QueryParams(` - CREATE (e:Entity { - id: $id, - name: $name, - type: $type, - project_id: $project_id, - created_at: $created_at, - updated_at: $updated_at, - visibility: $visibility - }) + MERGE (e:Entity {pk: $pk}) + ON CREATE SET e.id = $id, + e.name = $name, + e.type = $type, + e.project_id = $project_id, + e.created_at = $created_at, + e.updated_at = $updated_at, + e.visibility = $visibility + ON MATCH SET e.name = $name, + e.type = $type, + e.updated_at = $updated_at, + e.visibility = $visibility `, map[string]any{ + "pk": PK(ent.ProjectID, ent.ID), "id": ent.ID, "name": ent.Name, "type": ent.Type, @@ -828,9 +841,15 @@ func (idx *Indexer) bulkLoadRelations(relType string, rels []relationRecord) err if err != nil { return fmt.Errorf("create ndjson: %w", err) } + // COPY into a REL table resolves each endpoint against the node table's + // PRIMARY KEY, which is now pk, not id. Every relation in one index run is + // within idx.projectID, so both ends namespace by that project. enc := json.NewEncoder(f) for _, r := range rels { - if err := enc.Encode(map[string]string{"from": r.FromID, "to": r.ToID}); err != nil { + if err := enc.Encode(map[string]string{ + "from": PK(idx.projectID, r.FromID), + "to": PK(idx.projectID, r.ToID), + }); err != nil { f.Close() return fmt.Errorf("encode relation json: %w", err) } diff --git a/src/kg/internal/knowledge/indexer_crossproject_test.go b/src/kg/internal/knowledge/indexer_crossproject_test.go new file mode 100644 index 0000000..f68d9e0 --- /dev/null +++ b/src/kg/internal/knowledge/indexer_crossproject_test.go @@ -0,0 +1,363 @@ +package knowledge + +import ( + "fmt" + "io" + "os" + "path/filepath" + "strings" + "testing" +) + +// writeCommonProject creates a one-file Go project whose source-derived IDs are +// identical regardless of which directory it lives in: "file:common.go" and +// "function:common.go:Shared". Two such projects indexed into one database share +// every entity ID, which is exactly the collision the project-namespaced pk must +// keep from fusing them. +func writeCommonProject(t *testing.T, dir string) { + t.Helper() + if err := os.MkdirAll(dir, 0o755); err != nil { + t.Fatalf("MkdirAll: %v", err) + } + src := "package common\n\nimport \"logging\"\n\nfunc Shared() { logging.Init() }\n" + if err := os.WriteFile(filepath.Join(dir, "common.go"), []byte(src), 0o644); err != nil { + t.Fatalf("write source: %v", err) + } +} + +// indexInto runs a fresh indexer for projectID over srcDir against store. +func indexInto(t *testing.T, store *Store, projectID, srcDir string) *IndexStats { + t.Helper() + idx, err := NewIndexer(store, projectID, srcDir) + if err != nil { + t.Fatalf("NewIndexer(%s): %v", projectID, err) + } + stats, err := idx.Index() + if err != nil { + t.Fatalf("Index(%s): %v", projectID, err) + } + return stats +} + +// Two projects sharing one database file and sharing entity IDs must each keep +// their own entities. Before the pk fix, the second project's write collided +// with the first's identically-id'd (but project-scoped-clear-surviving) row and +// was silently dropped; the project under the live project_id then under-counted. +func TestIndexCrossProjectSharedIDsRetainsBoth(t *testing.T) { + const projA, projB = "project-a", "project-b" + + tmp := t.TempDir() + dirA := filepath.Join(tmp, "a") + dirB := filepath.Join(tmp, "b") + writeCommonProject(t, dirA) + writeCommonProject(t, dirB) + + store, err := OpenStore(filepath.Join(tmp, "knowledge.db")) + if err != nil { + t.Fatalf("OpenStore: %v", err) + } + defer store.Close() + + // Index A, then B — B shares every id A just wrote. + indexInto(t, store, projA, dirA) + indexInto(t, store, projB, dirB) + + countA, err := store.CountEntities(projA) + if err != nil { + t.Fatalf("CountEntities(A): %v", err) + } + countB, err := store.CountEntities(projB) + if err != nil { + t.Fatalf("CountEntities(B): %v", err) + } + if countA == 0 || countB == 0 { + t.Fatalf("a project lost all its entities: A=%d B=%d", countA, countB) + } + if countA != countB { + t.Errorf("projects sharing ids got different counts: A=%d B=%d (cross-project drop)", countA, countB) + } + + // The shared ids resolve to each project's OWN row, not the other's. + for _, proj := range []string{projA, projB} { + got, err := store.GetEntity("file:common.go", proj) + if err != nil { + t.Fatalf("GetEntity(file:common.go, %s): %v", proj, err) + } + if got.ProjectID != proj { + t.Errorf("file:common.go under %s resolved to project_id %q", proj, got.ProjectID) + } + if _, err := store.GetEntity("import:logging", proj); err != nil { + t.Errorf("import:logging missing for %s: %v", proj, err) + } + } + + // Re-indexing A must not disturb B — the selective clear is project-scoped and + // the pk keeps B's identically-id'd rows out of A's COPY collision set. + indexInto(t, store, projA, dirA) + reCountB, err := store.CountEntities(projB) + if err != nil { + t.Fatalf("CountEntities(B) after re-index A: %v", err) + } + if reCountB != countB { + t.Errorf("re-indexing A changed B's entity count: before=%d after=%d", countB, reCountB) + } + + // Relations must not bleed across projects: A's edges stay A's. + relsB, err := store.GetRelations("file:common.go", projB) + if err != nil { + t.Fatalf("GetRelations(file:common.go, B): %v", err) + } + for _, r := range relsB { + // Every endpoint GetRelations returns is addressed by local id; confirm it + // resolves within B rather than pointing at A's graph. + if _, err := store.GetEntity(r.ToID, projB); err != nil { + t.Errorf("relation from B points at %q which is not in B: %v", r.ToID, err) + } + } +} + +// captureStdout runs fn with os.Stdout redirected to a pipe and returns what was +// printed. The indexer reports fallbacks and collisions via fmt.Printf to stdout, +// so this is how a test sees whether the warning path was taken. +func captureStdout(t *testing.T, fn func()) string { + t.Helper() + orig := os.Stdout + r, w, err := os.Pipe() + if err != nil { + t.Fatalf("pipe: %v", err) + } + os.Stdout = w + done := make(chan string, 1) + go func() { + b, _ := io.ReadAll(r) + done <- string(b) + }() + fn() + w.Close() + os.Stdout = orig + return <-done +} + +// Re-indexing an already-indexed project must be clean: the pk makes the COPY +// idempotent (the selective clear empties this project's code-derived rows +// first, and no other project's rows share a pk), so neither the duplicate-key +// warning nor the COPY-unavailable fallback note should appear. +func TestReindexEmitsNoDuplicateKeyWarnings(t *testing.T) { + tmp := t.TempDir() + dir := filepath.Join(tmp, "src") + writeCommonProject(t, dir) + + store, err := OpenStore(filepath.Join(tmp, "knowledge.db")) + if err != nil { + t.Fatalf("OpenStore: %v", err) + } + defer store.Close() + + indexInto(t, store, "proj", dir) // first pass primes the graph + + out := captureStdout(t, func() { + indexInto(t, store, "proj", dir) // the re-index under test + }) + + for _, bad := range []string{"duplicated primary key", "COPY FROM JSON unavailable"} { + if strings.Contains(out, bad) { + t.Errorf("re-index emitted %q; output:\n%s", bad, out) + } + } +} + +// Observation has no project_id column, so an observation read that filters only +// on entity_id pulls in rows from every project sharing that (code-derived) +// entity id. A per-project search must return only that project's observations. +// This exercises the search observation-fetch path (batchGetObservations), which +// before the fix matched o.entity_id with no project scope. +func TestSearchObservationsDoNotBleedAcrossProjects(t *testing.T) { + const projA, projB = "obs-project-a", "obs-project-b" + + tmp := t.TempDir() + dirA := filepath.Join(tmp, "a") + dirB := filepath.Join(tmp, "b") + writeCommonProject(t, dirA) + writeCommonProject(t, dirB) + + store, err := OpenStore(filepath.Join(tmp, "knowledge.db")) + if err != nil { + t.Fatalf("OpenStore: %v", err) + } + defer store.Close() + + indexInto(t, store, projA, dirA) + indexInto(t, store, projB, dirB) + + // Hand-write an observation onto the SAME shared entity id in each project. + const idShared = "file:common.go" + if _, err := store.CreateObservation(idShared, "A-only note", projA); err != nil { + t.Fatalf("CreateObservation A: %v", err) + } + if _, err := store.CreateObservation(idShared, "B-only note", projB); err != nil { + t.Fatalf("CreateObservation B: %v", err) + } + + // A search scoped to project A, matching the shared entity by name, must + // surface only A's observation — never B's. + results, err := store.KeywordSearch(projA, "common", 10) + if err != nil { + t.Fatalf("KeywordSearch(A): %v", err) + } + var checked bool + for _, r := range results { + if r.Entity.ID != idShared { + continue + } + checked = true + for _, o := range r.Observations { + if o.Content == "B-only note" { + t.Errorf("project A search returned project B's observation %q (cross-project bleed)", o.Content) + } + } + var sawA bool + for _, o := range r.Observations { + if o.Content == "A-only note" { + sawA = true + } + } + if !sawA { + t.Errorf("project A search missing its own observation; got %+v", r.Observations) + } + } + if !checked { + t.Fatalf("search did not return the shared entity %q, so the bleed assertion never ran", idShared) + } +} + +// rawInsertEntity writes an entity directly, the way an indexer bulk path does, +// so a test can place a project-B-only neighbour with a chosen id. +func rawInsertEntity(t *testing.T, store *Store, projectID, id, name, typ string) { + t.Helper() + r, err := store.QueryParams(` + CREATE (e:Entity {pk: $pk, id: $id, name: $name, type: $type, project_id: $project_id, + created_at: timestamp('2020-01-01'), updated_at: timestamp('2020-01-01')}) + `, map[string]any{"pk": PK(projectID, id), "id": id, "name": name, "type": typ, "project_id": projectID}) + if err != nil { + t.Fatalf("rawInsertEntity %s/%s: %v", projectID, id, err) + } + r.Close() +} + +// rawCreateCrossEdge creates an edge between two nodes addressed by pk, with no +// project guard — simulating exactly the future/legacy buggy write path the +// reviewer warns about: a cross-project edge that the project-scoped reads must +// refuse to surface or delete. +func rawCreateCrossEdge(t *testing.T, store *Store, fromProject, fromID, toProject, toID, relType string) { + t.Helper() + r, err := store.QueryParams(fmt.Sprintf(` + MATCH (from:Entity {pk: $from_pk}), (to:Entity {pk: $to_pk}) + CREATE (from)-[:%s]->(to) + `, relType), map[string]any{"from_pk": PK(fromProject, fromID), "to_pk": PK(toProject, toID)}) + if err != nil { + t.Fatalf("rawCreateCrossEdge: %v", err) + } + r.Close() +} + +// countCrossEdge returns how many relType edges run from fromProject's fromID to +// toProject's toID, addressed by pk so it sees a cross-project edge the scoped +// APIs hide. +func countCrossEdge(t *testing.T, store *Store, fromProject, fromID, toProject, toID, relType string) int { + t.Helper() + r, err := store.QueryParams(fmt.Sprintf(` + MATCH (from:Entity {pk: $from_pk})-[e:%s]->(to:Entity {pk: $to_pk}) + RETURN count(e) + `, relType), map[string]any{"from_pk": PK(fromProject, fromID), "to_pk": PK(toProject, toID)}) + if err != nil { + t.Fatalf("countCrossEdge: %v", err) + } + defer r.Close() + if !r.HasNext() { + return 0 + } + row, err := r.Next() + if err != nil { + t.Fatalf("countCrossEdge next: %v", err) + } + v, _ := row.GetValue(0) + n, _ := v.(int64) + return int(n) +} + +// The read/delete siblings of CreateRelation (GetRelations, TraverseRelations, +// DeleteRelation) pinned only `from` by project_id, leaving the `to`/neighbour +// endpoint project-blind. That is masked while every edge is same-project, but a +// single cross-project edge (constructed here directly) must not leak through a +// project-scoped read, nor be deleted by a project-scoped delete. +func TestRelationReadsAndDeletesAreProjectScopedOnBothEnds(t *testing.T) { + const projA, projB = "rel-project-a", "rel-project-b" + + tmp := t.TempDir() + dirA := filepath.Join(tmp, "a") + dirB := filepath.Join(tmp, "b") + writeCommonProject(t, dirA) + writeCommonProject(t, dirB) + + store, err := OpenStore(filepath.Join(tmp, "knowledge.db")) + if err != nil { + t.Fatalf("OpenStore: %v", err) + } + defer store.Close() + + indexInto(t, store, projA, dirA) // both get file:common.go and import:logging + indexInto(t, store, projB, dirB) + + const shared = "file:common.go" + + // A project-B-only neighbour, reachable only by a cross-project edge from A. + rawInsertEntity(t, store, projB, "topic:bneighbor", "bneighbor", "topic") + rawCreateCrossEdge(t, store, projA, shared, projB, "topic:bneighbor", RelRelatesTo) + + // --- GetRelations: must not surface A→B.bneighbor ------------------------ + relsA, err := store.GetRelations(shared, projA) + if err != nil { + t.Fatalf("GetRelations(A): %v", err) + } + for _, r := range relsA { + if r.ToID == "topic:bneighbor" { + t.Errorf("GetRelations(A) returned project B's neighbour %q (project-blind `to`)", r.ToID) + } + } + + // --- TraverseRelations: must not cross into B --------------------------- + neighbours, err := store.TraverseRelations(shared, RelRelatesTo, projA) + if err != nil { + t.Fatalf("TraverseRelations(A): %v", err) + } + for _, n := range neighbours { + if n.ProjectID != projA { + t.Errorf("TraverseRelations(A) crossed into project %q via neighbour %q", n.ProjectID, n.ID) + } + } + + // --- DeleteRelation: a project-A delete must not remove an edge whose far + // end is project B's node. Give A both a same-project edge and a cross edge + // to B's import:logging (same id in both projects), then delete in A. ------ + if err := store.CreateRelation(shared, "import:logging", RelDependsOn, projA); err != nil { + t.Fatalf("CreateRelation A→A: %v", err) + } + rawCreateCrossEdge(t, store, projA, shared, projB, "import:logging", RelDependsOn) + if before := countCrossEdge(t, store, projA, shared, projB, "import:logging", RelDependsOn); before != 1 { + t.Fatalf("setup: expected 1 cross edge before delete, got %d", before) + } + + if err := store.DeleteRelation(shared, "import:logging", RelDependsOn, projA); err != nil { + t.Fatalf("DeleteRelation(A): %v", err) + } + + // The cross edge terminates at project B's node, so a project-A delete must + // leave it intact (pre-fix it was swept because `to` was project-blind). + if after := countCrossEdge(t, store, projA, shared, projB, "import:logging", RelDependsOn); after != 1 { + t.Errorf("DeleteRelation(A) touched the edge into project B's node: cross edges after = %d, want 1", after) + } + // A's own same-project edge was the intended target and should be gone. + if n := countCrossEdge(t, store, projA, shared, projA, "import:logging", RelDependsOn); n != 0 { + t.Errorf("DeleteRelation(A) did not remove A's own edge: count = %d, want 0", n) + } +} diff --git a/src/kg/internal/knowledge/indexer_pdf.go b/src/kg/internal/knowledge/indexer_pdf.go index c9660bd..2e6de55 100644 --- a/src/kg/internal/knowledge/indexer_pdf.go +++ b/src/kg/internal/knowledge/indexer_pdf.go @@ -161,10 +161,11 @@ func (idx *Indexer) processPDFFile( continue } *observations = append(*observations, obsRecord{ - id: uuid.NewString(), - entityID: entityID, - content: chunk, - created: now, + id: uuid.NewString(), + entityID: entityID, + content: chunk, + created: now, + projectID: idx.projectID, }) } diff --git a/src/kg/internal/knowledge/log_app_indexer.go b/src/kg/internal/knowledge/log_app_indexer.go index ce47709..21551f8 100644 --- a/src/kg/internal/knowledge/log_app_indexer.go +++ b/src/kg/internal/knowledge/log_app_indexer.go @@ -190,10 +190,11 @@ func IndexApplicationLogs(store *Store, projectID, projectRoot string, scopeFilt obs := buildLogEntityObservations(logEntity, relPath) for _, content := range obs { newObs = append(newObs, obsRecord{ - id: uuid.New().String(), - entityID: entityID, - content: content, - created: now, + id: uuid.New().String(), + entityID: entityID, + content: content, + created: now, + projectID: projectID, }) } } diff --git a/src/kg/internal/knowledge/log_indexer.go b/src/kg/internal/knowledge/log_indexer.go index 6094895..c0f7470 100644 --- a/src/kg/internal/knowledge/log_indexer.go +++ b/src/kg/internal/knowledge/log_indexer.go @@ -189,6 +189,11 @@ type obsRecord struct { entityID string content string created time.Time + // projectID namespaces both the observation's own pk and the pk of the + // parent entity the HAS_OBSERVATION edge resolves against (see + // bulkLoadExecObservations). Every observation in one run shares it, but it + // is carried per-row to keep the bulk loader free of an out-of-band argument. + projectID string } // LogIndexStats holds counters returned by IndexExecutionLogs. @@ -263,10 +268,11 @@ func IndexExecutionLogs(store *Store, projectID, projectRoot string) (LogIndexSt }) for _, o := range buildExecObservations(parsed, meta, folderName) { newObs = append(newObs, obsRecord{ - id: uuid.New().String(), - entityID: entityID, - content: o, - created: now, + id: uuid.New().String(), + entityID: entityID, + content: o, + created: now, + projectID: projectID, }) } } @@ -308,6 +314,7 @@ func bulkLoadExecEntities(store *Store, entities []entityRecord) error { enc := json.NewEncoder(f) for _, e := range entities { if err := enc.Encode(map[string]string{ + "pk": PK(e.ProjectID, e.ID), "id": e.ID, "name": e.Name, "type": e.Type, @@ -322,17 +329,24 @@ func bulkLoadExecEntities(store *Store, entities []entityRecord) error { f.Close() result, err := store.Query(fmt.Sprintf( - `COPY Entity(id, name, type, project_id, created_at, updated_at) FROM '%s'`, path, + `COPY Entity(pk, id, name, type, project_id, created_at, updated_at) FROM '%s'`, path, )) if err != nil { - // Fall back to row-by-row if COPY FROM JSON is unavailable. + // Fall back to row-by-row MERGE (idempotent on pk) if COPY FROM JSON is + // unavailable. These entities were already filtered for prior existence + // by name, so MERGE only refreshes a genuine re-run rather than colliding. for _, e := range entities { + // No visibility here is intentional: exec-log entities carry no + // visibility attribute today, unlike code-indexer entities. This is + // not a second instance of the ON CREATE/ON MATCH gap fixed in + // indexer.go — there is nothing to set. r, err2 := store.QueryParams(` - CREATE (n:Entity { - id: $id, name: $name, type: $type, - project_id: $project_id, - created_at: $created_at, updated_at: $updated_at - })`, map[string]any{ + MERGE (n:Entity {pk: $pk}) + ON CREATE SET n.id = $id, n.name = $name, n.type = $type, + n.project_id = $project_id, + n.created_at = $created_at, n.updated_at = $updated_at + ON MATCH SET n.name = $name, n.type = $type, n.updated_at = $updated_at`, map[string]any{ + "pk": PK(e.ProjectID, e.ID), "id": e.ID, "name": e.Name, "type": e.Type, "project_id": e.ProjectID, "created_at": e.CreatedAt, "updated_at": e.UpdatedAt, @@ -366,6 +380,7 @@ func bulkLoadExecObservations(store *Store, obs []obsRecord) error { eenc := json.NewEncoder(ef) for _, o := range obs { if err := nenc.Encode(map[string]string{ + "pk": PK(o.projectID, o.id), "id": o.id, "entity_id": o.entityID, "content": o.content, @@ -375,9 +390,12 @@ func bulkLoadExecObservations(store *Store, obs []obsRecord) error { ef.Close() return err } + // Both endpoints of HAS_OBSERVATION resolve against node primary keys, + // which are now pk: the parent Entity's pk and this Observation's pk, + // both namespaced by the observation's project. if err := eenc.Encode(map[string]string{ - "from": o.entityID, - "to": o.id, + "from": PK(o.projectID, o.entityID), + "to": PK(o.projectID, o.id), }); err != nil { nf.Close() ef.Close() @@ -388,7 +406,7 @@ func bulkLoadExecObservations(store *Store, obs []obsRecord) error { ef.Close() r, err := store.Query(fmt.Sprintf( - `COPY Observation(id, entity_id, content, created_at) FROM '%s'`, nodesPath, + `COPY Observation(pk, id, entity_id, content, created_at) FROM '%s'`, nodesPath, )) if err != nil { return fmt.Errorf("COPY Observation: %w", err) diff --git a/src/kg/internal/knowledge/store_wrappers.go b/src/kg/internal/knowledge/store_wrappers.go index e9885f8..20a0f73 100644 --- a/src/kg/internal/knowledge/store_wrappers.go +++ b/src/kg/internal/knowledge/store_wrappers.go @@ -28,6 +28,13 @@ func OpenStoreReadOnly(dbPath string) (*kglib.Store, error) { return kglib.OpenStoreReadOnly(dbPath) } +// PK derives the project-namespaced primary key for an entity or observation. +// Re-exported from kglib so the indexer's bulk-load (COPY/CREATE) paths derive +// pk through the exact same helper the Store's own writes use — the +// 0x1F-separator format must match byte-for-byte or endpoints and nodes would +// not resolve. +var PK = kglib.PK + // NewEmbedderFromEnv creates an embedder from environment variables var NewEmbedderFromEnv = kglib.NewEmbedderFromEnv diff --git a/src/kg/migrate.go b/src/kg/migrate.go index 99c79ea..fcbf049 100644 --- a/src/kg/migrate.go +++ b/src/kg/migrate.go @@ -25,25 +25,69 @@ import ( // this code — replay reads the journal, not the database — but a rebuild that // turns out worse than what it replaced should be recoverable. -// archiveIfMismatch moves a database aside when this build cannot read its -// storage format, returning the path it was archived to (empty when nothing was -// done). The check happens before any open, because a database in the wrong -// format cannot be opened to be asked about itself. +// archiveIfMismatch moves a database aside when this build cannot read or use +// its on-disk shape, returning the path it was archived to (empty when nothing +// was done). The stamp check happens before any open, because a database in the +// wrong storage format cannot be opened to be asked about itself. // -// An unstamped database — one written before stamping shipped — is left alone. -// There is no evidence against it, and rebuilding every pre-existing graph on -// first run would be precisely the forced breakage this exists to avoid. -func archiveIfMismatch(dbPath string, out io.Writer) (string, error) { +// An unstamped database — one written before stamping shipped — is normally left +// alone: there is no evidence against it, and rebuilding every pre-existing +// graph on first run would be the forced breakage this exists to avoid. The one +// exception is an unstamped database that ALSO predates the pk column: it cannot +// be opened read-write (the pk gate in OpenStore hard-errors) and neither `kg +// index` nor `kg migrate` would otherwise repair it, so it is a dead end unless +// it is archived and rebuilt here. That case is detected by a read-only open — +// a pre-pk schema is still readable — so a current unstamped database (which has +// pk) is never falsely archived; it simply gets stamped on the next write open. +// +// missingPKHint lets a caller that has already run databaseMissingPK for this +// database pass the result in, so a `kg migrate` run does not re-derive it with a +// second read-only open. It is nil on the standalone `kg index` path, where the +// value is computed here on demand. +func archiveIfMismatch(dbPath string, out io.Writer, missingPKHint *bool) (string, error) { status, stamp, err := kglib.CheckFormat(dbPath) if err != nil { return "", err } - if status != kglib.FormatMismatch { + + var reason string + switch status { + case kglib.FormatMismatch: + // A mismatch is either a storage-format (Kuzu) bump or a kg schema bump; + // the message names whichever actually differs so a same-Kuzu schema + // change does not print the confusing "written by kuzu X; this build uses X". + if stamp.KuzuVersion != kglib.KuzuVersion() { + reason = fmt.Sprintf("was written by kuzu %s; this build uses %s", + stamp.KuzuVersion, kglib.KuzuVersion()) + } else { + reason = fmt.Sprintf("uses kg schema v%d; this build needs v%d", + stamp.SchemaVersion, kglib.CurrentSchemaVersion) + } + case kglib.FormatUnstamped: + var missing bool + if missingPKHint != nil { + missing = *missingPKHint + } else { + m, err := databaseMissingPK(dbPath) + if err != nil { + // Cannot introspect the columns (e.g. the database will not even open + // read-only). Fall back to the historical leave-alone behavior rather + // than archiving on a guess. + return "", nil + } + missing = m + } + if !missing { + // Unstamped but already current — leave it alone; the next write open + // stamps it. + return "", nil + } + reason = "is unstamped and predates the project-namespaced primary key" + default: return "", nil } - fmt.Fprintf(out, "🔄 %s was written by kuzu %s; this build uses %s. Rebuilding.\n", - dbPath, stamp.KuzuVersion, kglib.KuzuVersion()) + fmt.Fprintf(out, "🔄 %s %s. Rebuilding.\n", dbPath, reason) aside, err := kglib.ArchiveDatabase(dbPath) if err != nil { @@ -53,11 +97,25 @@ func archiveIfMismatch(dbPath string, out io.Writer) (string, error) { return aside, nil } +// databaseMissingPK reports whether an existing database lacks the pk column, +// i.e. predates the project-namespaced primary key. It opens the database +// read-only purely to introspect its columns — a pre-pk schema is still readable, +// and a read-only open neither migrates the schema nor trips the write-open pk +// gate, so this is a safe question to ask of any database. +func databaseMissingPK(dbPath string) (bool, error) { + store, err := knowledge.OpenStoreReadOnly(dbPath) + if err != nil { + return false, err + } + defer store.Close() + return !store.HasPK(), nil +} + // rebuildHandWritesOnly repairs a database that has no source tree behind it — // the personal store. Archive, create an empty replacement, replay. There is // nothing to index, so the ordering hazard above does not apply. func rebuildHandWritesOnly(dbPath string, out io.Writer) (bool, error) { - aside, err := archiveIfMismatch(dbPath, out) + aside, err := archiveIfMismatch(dbPath, out, nil) if err != nil || aside == "" { return false, err } @@ -139,10 +197,36 @@ every scope and the personal store in one pass, or to see what would happen.`, // Counted only once the format is actually known: "checked" is a // claim about evidence, not about how many paths were visited. checked++ - if status != kglib.FormatMismatch { + // A stamped mismatch always rebuilds. An unstamped database is left + // alone UNLESS it also predates the pk column, which is a dead end + // (write-open refuses it) that only an archive+rebuild can repair — + // the same exception archiveIfMismatch makes. A current unstamped + // database (has pk) is still skipped. + needsRebuild := status == kglib.FormatMismatch + // missingHint carries the single databaseMissingPK result for this + // database into archiveIfMismatch (reached via indexScopeDB below), so + // the read-only open it does is not repeated there. + var missingHint *bool + if !needsRebuild && status == kglib.FormatUnstamped { + missing, mErr := databaseMissingPK(dbPath) + if mErr != nil { + // Mirror the CheckFormat-error path above: an introspection we + // could not complete is not evidence that this build can read + // the database, so it must count as unchecked (failf) rather + // than fall through with needsRebuild=false and reach the + // all-clear summary for a database we never resolved. + failf("scope %s: cannot determine whether %s predates the primary key: %v", label, dbPath, mErr) + continue + } + if missing { + needsRebuild = true + missingHint = &missing + } + } + if !needsRebuild { continue } - if err := indexScopeDB(root, aiDir, scope); err != nil { + if err := indexScopeDB(root, aiDir, scope, missingHint); err != nil { failf("scope %s: rebuild failed, %s is still unreadable: %v", label, dbPath, err) continue } diff --git a/src/kg/migrate_scopes_test.go b/src/kg/migrate_scopes_test.go index e9d6ca3..3e1346c 100644 --- a/src/kg/migrate_scopes_test.go +++ b/src/kg/migrate_scopes_test.go @@ -6,6 +6,8 @@ import ( "path/filepath" "strings" "testing" + + "github.com/cortexa-llc/mcp/kglib" ) // runMigrateInDir runs `kg migrate` with dir as the working directory and @@ -89,3 +91,59 @@ func TestMigrateDoesNotCountUncheckableScopeDatabases(t *testing.T) { t.Errorf("kg migrate reported an all-clear for a database it could not read:\n%s", out) } } + +// An unstamped scope database whose pk-column introspection fails is likewise +// unchecked. CheckFormat only reads the sidecar stamp, so it still reports +// FormatUnstamped for a database it never opened; databaseMissingPK then opens +// it read-only and fails. A regression made that error fall through with +// needsRebuild=false, so the loop continued and the run could still print an +// all-clear — a false yes for a database it could not resolve. +func TestMigrateDoesNotCountScopeDatabasesWhosePKCannotBeDetermined(t *testing.T) { + withTempPersonalStore(t) + + root := t.TempDir() + scopeDir := filepath.Join(root, ".ai", "scope") + if err := os.MkdirAll(scopeDir, 0o755); err != nil { + t.Fatalf("MkdirAll: %v", err) + } + cfg := `{"name":"platform","database":"platform.db"}` + if err := os.WriteFile(filepath.Join(scopeDir, "platform.json"), []byte(cfg), 0o600); err != nil { + t.Fatalf("write scope config: %v", err) + } + // The database exists (so CheckFormat stats it) and carries no stamp sidecar + // (so CheckFormat returns FormatUnstamped, reaching the pk check), but is + // unreadable, so the read-only open inside databaseMissingPK fails. + dbPath := filepath.Join(root, ".ai", "platform.db") + if err := os.MkdirAll(dbPath, 0o755); err != nil { + t.Fatalf("MkdirAll db: %v", err) + } + if err := os.Chmod(dbPath, 0o000); err != nil { + t.Fatalf("chmod db: %v", err) + } + t.Cleanup(func() { _ = os.Chmod(dbPath, 0o755) }) + + // Precondition: CheckFormat must see this as unstamped (not an error of its + // own), or the test would exercise the CheckFormat-error path instead of the + // databaseMissingPK-error path this regression is about. + status, _, err := kglib.CheckFormat(dbPath) + if err != nil { + t.Fatalf("CheckFormat precondition failed: %v", err) + } + if status != kglib.FormatUnstamped { + t.Fatalf("fixture is not FormatUnstamped (got %v); test would not reach the pk check", status) + } + + out, err := runMigrateInDir(t, root) + + if err == nil { + t.Error("kg migrate exited 0 for a database whose pk column it could not introspect") + } + if strings.Contains(out, "readable by this build") { + t.Errorf("kg migrate reported an all-clear for a database it could not resolve:\n%s", out) + } + // Prove it was the databaseMissingPK path, not the CheckFormat path, that + // flagged it — the new failf is the only source of this phrasing. + if !strings.Contains(out, "predates the primary key") { + t.Errorf("output does not show the pk-introspection failure was the reason it was flagged:\n%s", out) + } +} diff --git a/src/kg/migrate_test.go b/src/kg/migrate_test.go index 13be0f9..2d290bb 100644 --- a/src/kg/migrate_test.go +++ b/src/kg/migrate_test.go @@ -3,6 +3,7 @@ package main import ( "bytes" "encoding/json" + "fmt" "os" "path/filepath" "strings" @@ -10,8 +11,68 @@ import ( "github.com/cortexa-llc/mcp/kg/internal/knowledge" "github.com/cortexa-llc/mcp/kglib" + kuzu "github.com/kuzudb/go-kuzu" ) +// writePrePKDatabase builds a database with the pre-pk schema (Entity/Observation +// keyed on the global id, no pk column) and no format stamp, and drops a journal +// beside it carrying one hand-written entity. This is the legacy artifact the +// migration must still rescue: unstamped AND predating the pk column. +func writePrePKDatabase(t *testing.T, dbPath, projectID string, hand kglib.EntityRef) { + t.Helper() + + db, err := kuzu.OpenDatabase(dbPath, kuzu.DefaultSystemConfig()) + if err != nil { + t.Fatalf("OpenDatabase: %v", err) + } + conn, err := kuzu.OpenConnection(db) + if err != nil { + db.Close() + t.Fatalf("OpenConnection: %v", err) + } + + ddl := []string{ + `CREATE NODE TABLE Entity(id STRING PRIMARY KEY, name STRING, type STRING, + project_id STRING, created_at TIMESTAMP, updated_at TIMESTAMP, + embedding FLOAT[1536], visibility STRING)`, + `CREATE NODE TABLE Observation(id STRING PRIMARY KEY, entity_id STRING, + content STRING, created_at TIMESTAMP, embedding FLOAT[1536])`, + `CREATE REL TABLE HAS_OBSERVATION(FROM Entity TO Observation)`, + `CREATE NODE TABLE KGMeta(project_id STRING PRIMARY KEY, repo_url STRING, + commit_hash STRING, dirty BOOLEAN, indexed_at TIMESTAMP, + embed_model STRING, kg_version STRING)`, + } + for _, rt := range knowledge.AllowedRelTypes { + ddl = append(ddl, fmt.Sprintf("CREATE REL TABLE %s(FROM Entity TO Entity)", rt)) + } + for _, stmt := range ddl { + res, err := conn.Query(stmt) + if err != nil { + conn.Close() + db.Close() + t.Fatalf("pre-pk DDL %q: %v", stmt, err) + } + res.Close() + } + conn.Close() + db.Close() + + // The journal is the only copy of the hand-write across a rebuild. + rec := kglib.JournalRecord{ + Version: kglib.JournalVersion, + Op: kglib.OpCreateEntity, + ProjectID: projectID, + Entity: &hand, + } + data, err := json.Marshal(rec) + if err != nil { + t.Fatalf("marshal journal record: %v", err) + } + if err := os.WriteFile(kglib.JournalPath(dbPath), append(data, '\n'), 0600); err != nil { + t.Fatalf("write journal: %v", err) + } +} + // stampAs rewrites a database's format sidecar to claim a different engine // build wrote it, which is how a future go-kuzu bump is simulated without one. func stampAs(t *testing.T, dbPath, kuzuVersion string) { @@ -25,7 +86,170 @@ func stampAs(t *testing.T, dbPath, kuzuVersion string) { } } +// stampSchema rewrites the sidecar to claim an older kg schema version while +// keeping this build's Kuzu version — the way an on-disk schema bump (e.g. the +// pk primary key) presents on a database written by an otherwise-current build. +func stampSchema(t *testing.T, dbPath string, schemaVersion int) { + t.Helper() + data, err := json.Marshal(kglib.FormatStamp{ + KuzuVersion: kglib.KuzuVersion(), + KGVersion: "test", + SchemaVersion: schemaVersion, + }) + if err != nil { + t.Fatalf("marshal stamp: %v", err) + } + if err := os.WriteFile(kglib.FormatStampPath(dbPath), data, 0600); err != nil { + t.Fatalf("write stamp: %v", err) + } +} + +// A kg schema-version bump (not a Kuzu bump) must also trigger archive + rebuild +// and carry hand-written knowledge across via journal replay. This is the +// migration path the pk primary key rides: Kuzu cannot ALTER … ADD PRIMARY KEY, +// so an older-schema database is rebuilt fresh and its journal replayed. +func TestSchemaVersionBumpRebuildsAndRestoresHandWrites(t *testing.T) { + dbPath := filepath.Join(t.TempDir(), "knowledge.db") + + store, err := knowledge.OpenStore(dbPath) + if err != nil { + t.Fatalf("OpenStore: %v", err) + } + store.EnableJournal() + entity, err := store.CreateEntity("retry-policy", "decision", "proj") + if err != nil { + t.Fatalf("CreateEntity: %v", err) + } + if _, err := store.CreateObservation(entity.ID, "three attempts, exponential backoff", "proj"); err != nil { + t.Fatalf("CreateObservation: %v", err) + } + store.Close() + + // Same Kuzu build, but an older schema than this build writes. + stampSchema(t, dbPath, kglib.CurrentSchemaVersion-1) + + status, _, err := kglib.CheckFormat(dbPath) + if err != nil { + t.Fatalf("CheckFormat: %v", err) + } + if status != kglib.FormatMismatch { + t.Fatalf("an older schema_version was not reported as a mismatch: got %v", status) + } + + var out bytes.Buffer + rebuilt, err := rebuildHandWritesOnly(dbPath, &out) + if err != nil { + t.Fatalf("rebuildHandWritesOnly: %v", err) + } + if !rebuilt { + t.Fatal("a database with an older kg schema was not rebuilt") + } + + fresh, err := knowledge.OpenStore(dbPath) + if err != nil { + t.Fatalf("open rebuilt store: %v", err) + } + defer fresh.Close() + + restored, err := fresh.GetEntityByNameAndType("retry-policy", "decision", "proj") + if err != nil { + t.Fatalf("GetEntityByNameAndType: %v", err) + } + if restored == nil { + t.Fatal("hand-written entity did not survive the schema rebuild") + } + observations, err := fresh.GetObservations(restored.ID, "proj") + if err != nil { + t.Fatalf("GetObservations: %v", err) + } + if len(observations) != 1 || observations[0].Content != "three attempts, exponential backoff" { + t.Errorf("observations = %+v, want the hand-written note", observations) + } + + // The rebuilt database is stamped at the current schema, so it is FormatOK. + status, _, err = kglib.CheckFormat(dbPath) + if err != nil { + t.Fatalf("CheckFormat after rebuild: %v", err) + } + if status != kglib.FormatOK { + t.Errorf("status after rebuild = %v, want %v", status, kglib.FormatOK) + } +} + // The whole point of Phase 4, end to end: hand-written knowledge survives a +// A database that is unstamped AND predates the pk column has no other recovery +// path: a read-write open refuses it (the pk gate), `kg migrate`'s loop used to +// skip anything that was not a stamped mismatch, and `kg index` hit the same +// gate — a dead end. archiveIfMismatch must now treat that specific case as a +// mismatch, archive it, and replay the journal into the rebuilt database. +func TestUnstampedPrePKDatabaseRebuildsAndRestoresHandWrites(t *testing.T) { + dbPath := filepath.Join(t.TempDir(), "knowledge.db") + writePrePKDatabase(t, dbPath, "proj", kglib.EntityRef{Name: "retry-policy", Type: "decision"}) + + // Preconditions: no stamp (unstamped) and no pk column (pre-pk). + status, _, err := kglib.CheckFormat(dbPath) + if err != nil { + t.Fatalf("CheckFormat: %v", err) + } + if status != kglib.FormatUnstamped { + t.Fatalf("fixture is not unstamped: got %v", status) + } + missing, err := databaseMissingPK(dbPath) + if err != nil { + t.Fatalf("databaseMissingPK: %v", err) + } + if !missing { + t.Fatal("fixture unexpectedly already has the pk column") + } + + // Pre-fix this returns (false, nil) — archiveIfMismatch did nothing for an + // unstamped database, leaving the dead end. Post-fix it archives + rebuilds. + var out bytes.Buffer + rebuilt, err := rebuildHandWritesOnly(dbPath, &out) + if err != nil { + t.Fatalf("rebuildHandWritesOnly: %v", err) + } + if !rebuilt { + t.Fatal("an unstamped pre-pk database was not rebuilt (the dead-end path)") + } + + // The old database is archived, not deleted. + matches, err := filepath.Glob(dbPath + ".old-*") + if err != nil { + t.Fatalf("Glob: %v", err) + } + if len(matches) == 0 { + t.Error("no archived copy of the pre-pk database") + } + + // The rebuilt database has the pk column and the journal's hand-write. + fresh, err := knowledge.OpenStore(dbPath) + if err != nil { + t.Fatalf("open rebuilt store: %v", err) + } + defer fresh.Close() + if !fresh.HasPK() { + t.Error("rebuilt database still lacks the pk column") + } + restored, err := fresh.GetEntityByNameAndType("retry-policy", "decision", "proj") + if err != nil { + t.Fatalf("GetEntityByNameAndType: %v", err) + } + if restored == nil { + t.Fatal("hand-written entity did not replay from the journal into the rebuild") + } + + // And the rebuilt database is now stamped and current, so it is not rebuilt + // again on the next run. + status, _, err = kglib.CheckFormat(dbPath) + if err != nil { + t.Fatalf("CheckFormat after rebuild: %v", err) + } + if status != kglib.FormatOK { + t.Errorf("status after rebuild = %v, want %v", status, kglib.FormatOK) + } +} + // storage-format change with no old binary involved and no user action. func TestEnsureFormatRebuildsAndRestoresHandWrites(t *testing.T) { dbPath := filepath.Join(t.TempDir(), "knowledge.db") @@ -156,7 +380,7 @@ func TestRebuildKeepsTheOldDatabase(t *testing.T) { stampAs(t, dbPath, "v0.0.1-ancient") var out bytes.Buffer - aside, err := archiveIfMismatch(dbPath, &out) + aside, err := archiveIfMismatch(dbPath, &out, nil) if err != nil { t.Fatalf("archiveIfMismatch: %v", err) } diff --git a/src/kglib/embeddings.go b/src/kglib/embeddings.go index 17935a5..9087acb 100644 --- a/src/kglib/embeddings.go +++ b/src/kglib/embeddings.go @@ -36,9 +36,11 @@ func (s *Store) BatchEmbed(ctx context.Context, projectID string, embedder Embed "results are matched by position, so a mismatched count cannot be used", len(embeddings), len(texts)) } - // Store embeddings + // Store embeddings. entity.ProjectID equals the projectID being embedded + // (GetUnembeddedEntities filters on it); passing it keeps SetEmbedding on + // this project's node and invalidates this project's index. for i, entity := range entities { - if err := s.SetEmbedding(entity.ID, embeddings[i]); err != nil { + if err := s.SetEmbedding(entity.ID, entity.ProjectID, embeddings[i]); err != nil { return fmt.Errorf("set embedding for entity %s: %w", entity.ID, err) } } @@ -70,9 +72,11 @@ func (s *Store) BatchEmbed(ctx context.Context, projectID string, embedder Embed "results are matched by position, so a mismatched count cannot be used", len(obsEmbeddings), len(obsTexts)) } - // Store observation embeddings + // Store observation embeddings. projectID is the project BatchEmbed is + // embedding (GetUnembeddedObservations filtered on it), so it scopes the + // write to this project's observation rows. for i, obs := range observations { - if err := s.SetObservationEmbedding(obs.ID, obsEmbeddings[i]); err != nil { + if err := s.SetObservationEmbedding(obs.ID, projectID, obsEmbeddings[i]); err != nil { return fmt.Errorf("set embedding for observation %s: %w", obs.ID, err) } } @@ -156,17 +160,17 @@ func (s *Store) GetUnembeddedObservations(projectID string) ([]Observation, erro // SetEmbedding stores an embedding vector for an entity and invalidates the // in-memory HNSW index for the entity's project so that the next VectorSearch // call rebuilds the index with the new vector. -func (s *Store) SetEmbedding(entityID string, embedding []float32) error { - // Look up the project_id so we can invalidate the correct HNSW index. - projectID, err := s.entityProjectID(entityID) - if err != nil { - return fmt.Errorf("set embedding (lookup project): %w", err) - } - +// +// The entity is matched by its project-namespaced pk, not by id alone: under +// multi-project-per-DB a bare-id MATCH would set the embedding on every +// project's node sharing that id (and the old project_id lookup returned an +// arbitrary one of them, invalidating the wrong project's HNSW index). projectID +// is required and is the project whose index is invalidated. +func (s *Store) SetEmbedding(entityID, projectID string, embedding []float32) error { result, err := s.QueryParams(` - MATCH (e:Entity {id: $id}) + MATCH (e:Entity {pk: $pk}) SET e.embedding = $embedding - `, map[string]any{"id": entityID, "embedding": embedding}) + `, map[string]any{"pk": PK(projectID, entityID), "embedding": embedding}) if err != nil { return fmt.Errorf("set embedding: %w", err) } @@ -180,43 +184,19 @@ func (s *Store) SetEmbedding(entityID string, embedding []float32) error { return nil } -// entityProjectID returns the project_id for entityID, or "" if not found. -func (s *Store) entityProjectID(entityID string) (string, error) { - result, err := s.QueryParams(` - MATCH (e:Entity {id: $id}) - RETURN e.project_id - `, map[string]any{"id": entityID}) - if err != nil { - return "", fmt.Errorf("query entity project_id: %w", err) - } - defer result.Close() - - if result.HasNext() { - tuple, err := result.Next() - if err != nil { - return "", fmt.Errorf("next: %w", err) - } - row, err := tuple.GetAsSlice() - tuple.Close() - if err != nil { - return "", fmt.Errorf("get slice: %w", err) - } - if row[0] == nil { - return "", nil - } - if pid, ok := row[0].(string); ok { - return pid, nil - } - } - return "", nil -} - -// SetObservationEmbedding stores an embedding vector for an observation -func (s *Store) SetObservationEmbedding(observationID string, embedding []float32) error { +// SetObservationEmbedding stores an embedding vector for an observation. +// +// Matched by the project-namespaced pk, consistent with SetEmbedding and every +// other write: an observation id is a UUID so a cross-project collision is +// unlikely, but a bare-id match is the same project-blind pattern this change +// removes elsewhere, and there is no reason to leave one write path on it. No +// !hasPK fallback is needed — writes only happen on a read-write open, which +// refuses a pre-pk database outright (see openStoreWithConfig). +func (s *Store) SetObservationEmbedding(observationID, projectID string, embedding []float32) error { result, err := s.QueryParams(` - MATCH (o:Observation {id: $id}) + MATCH (o:Observation {pk: $pk}) SET o.embedding = $embedding - `, map[string]any{"id": observationID, "embedding": embedding}) + `, map[string]any{"pk": PK(projectID, observationID), "embedding": embedding}) if err != nil { return fmt.Errorf("set observation embedding: %w", err) } diff --git a/src/kglib/embeddings_test.go b/src/kglib/embeddings_test.go index c280cd4..e48b49e 100644 --- a/src/kglib/embeddings_test.go +++ b/src/kglib/embeddings_test.go @@ -167,7 +167,7 @@ func TestSetEmbedding_HNSWInvalidation(t *testing.T) { for i := range embedding { embedding[i] = float32(i) * 0.1 } - if err := store.SetEmbedding(entity.ID, embedding); err != nil { + if err := store.SetEmbedding(entity.ID, projectID, embedding); err != nil { t.Fatalf("SetEmbedding: %v", err) } @@ -178,3 +178,52 @@ func TestSetEmbedding_HNSWInvalidation(t *testing.T) { t.Error("expected HNSW cache to be invalidated after SetEmbedding") } } + +// SetObservationEmbedding must be project-scoped: a write addressed with the +// wrong project id must not reach an observation in another project. Proven via +// GetUnembeddedObservations, which lists only observations whose embedding is +// still NULL — a write that matched would drop the observation from the list. +func TestSetObservationEmbedding_ProjectScoped(t *testing.T) { + store := setupTestStore(t) + defer store.Close() + + const projectID = "obs-embed-scope" + entity, err := store.CreateEntity("auth.go", "file", projectID) + if err != nil { + t.Fatalf("CreateEntity: %v", err) + } + obs, err := store.CreateObservation(entity.ID, "token TTL is 15m", projectID) + if err != nil { + t.Fatalf("CreateObservation: %v", err) + } + + embedding := make([]float32, embeddingDim) + for i := range embedding { + embedding[i] = 0.1 + } + + // Wrong project: the pk PK("other", obs.ID) exists nowhere, so the match is + // empty and the observation stays un-embedded. + if err := store.SetObservationEmbedding(obs.ID, "other-project", embedding); err != nil { + t.Fatalf("SetObservationEmbedding (wrong project): %v", err) + } + unembedded, err := store.GetUnembeddedObservations(projectID) + if err != nil { + t.Fatalf("GetUnembeddedObservations: %v", err) + } + if len(unembedded) != 1 { + t.Fatalf("a wrong-project write changed the embedding: got %d un-embedded, want 1", len(unembedded)) + } + + // Correct project: the write lands and the observation is no longer un-embedded. + if err := store.SetObservationEmbedding(obs.ID, projectID, embedding); err != nil { + t.Fatalf("SetObservationEmbedding (correct project): %v", err) + } + unembedded, err = store.GetUnembeddedObservations(projectID) + if err != nil { + t.Fatalf("GetUnembeddedObservations after correct write: %v", err) + } + if len(unembedded) != 0 { + t.Errorf("correct-project write did not set the embedding: got %d un-embedded, want 0", len(unembedded)) + } +} diff --git a/src/kglib/entity.go b/src/kglib/entity.go index 97f1013..c43985b 100644 --- a/src/kglib/entity.go +++ b/src/kglib/entity.go @@ -53,10 +53,11 @@ func (s *Store) createEntityWithID(id, name, entityType, projectID string) (*Ent result, err := s.QueryParams(` CREATE (e:Entity { - id: $id, name: $name, type: $type, project_id: $project_id, + pk: $pk, id: $id, name: $name, type: $type, project_id: $project_id, created_at: $created_at, updated_at: $updated_at }) `, map[string]any{ + "pk": PK(projectID, id), "id": id, "name": name, "type": entityType, "project_id": projectID, "created_at": entity.CreatedAt, "updated_at": entity.UpdatedAt, }) @@ -120,9 +121,16 @@ func (s *Store) detectColumns() { if err != nil { return } - if len(row) > 1 && stringOrEmpty(row[1]) == "visibility" { + // table_info's column name is row[1]; skip any row too short to carry it + // rather than indexing unconditionally. + if len(row) <= 1 { + continue + } + switch name := stringOrEmpty(row[1]); name { + case "visibility": s.hasVisibility = true - return + case "pk": + s.hasPK = true } } } @@ -157,6 +165,7 @@ func (s *Store) CreateEntity(name, entityType, projectID string) (*Entity, error result, err := s.QueryParams(` CREATE (e:Entity { + pk: $pk, id: $id, name: $name, type: $type, @@ -165,6 +174,7 @@ func (s *Store) CreateEntity(name, entityType, projectID string) (*Entity, error updated_at: $updated_at }) `, map[string]any{ + "pk": PK(projectID, entity.ID), "id": entity.ID, "name": name, "type": entityType, @@ -188,13 +198,30 @@ func (s *Store) CreateEntity(name, entityType, projectID string) (*Entity, error return entity, nil } -// GetEntity retrieves an entity by ID for a specific project +// GetEntity retrieves an entity by ID for a specific project. +// +// It matches on the project-namespaced pk, which is both the fast path (pk is +// the PRIMARY KEY) and the project-safe one: a bare id can collide across +// projects sharing a database, and resolving it here by id alone is exactly the +// cross-project confusion pk exists to prevent. func (s *Store) GetEntity(id, projectID string) (*Entity, error) { + // A pre-pk database — only reachable on a read-only open, since a read-write + // open refuses to proceed without the column — has no pk to match on, and + // Kuzu rejects a query that names a missing property outright. Fall back to + // the original (id, project_id) match so reads keep working on a legacy graph + // that has not yet been rebuilt. This branch is project-scoped and so still + // correct; it is merely not a primary-key lookup. + where := "e.pk = $pk" + params := map[string]any{"pk": PK(projectID, id)} + if !s.hasPK { + where = "e.id = $id AND e.project_id = $project_id" + params = map[string]any{"id": id, "project_id": projectID} + } result, err := s.QueryParams(` MATCH (e:Entity) - WHERE e.id = $id AND e.project_id = $project_id + WHERE `+where+` RETURN `+s.entityColumns()+` - `, map[string]any{"id": id, "project_id": projectID}) + `, params) if err != nil { return nil, fmt.Errorf("query entity: %w", err) } diff --git a/src/kglib/federated_test.go b/src/kglib/federated_test.go index 953cae7..675c6c3 100644 --- a/src/kglib/federated_test.go +++ b/src/kglib/federated_test.go @@ -37,10 +37,11 @@ func insertEntityWithID(t *testing.T, s *Store, id, name, entityType, projectID now := time.Now().UTC() result, err := s.QueryParams(` CREATE (e:Entity { - id: $id, name: $name, type: $type, project_id: $project_id, + pk: $pk, id: $id, name: $name, type: $type, project_id: $project_id, created_at: $created_at, updated_at: $updated_at }) `, map[string]any{ + "pk": PK(projectID, id), "id": id, "name": name, "type": entityType, "project_id": projectID, "created_at": now, "updated_at": now, }) diff --git a/src/kglib/format.go b/src/kglib/format.go index 6e25550..5ed3212 100644 --- a/src/kglib/format.go +++ b/src/kglib/format.go @@ -45,13 +45,31 @@ import ( // FormatStampFile is the sidecar naming the engine build that wrote a database. const formatStampSuffix = ".format.json" +// CurrentSchemaVersion is the kg knowledge-graph schema version this build +// writes. Unlike KGVersion (the build string, which changes every release and +// means nothing for compatibility), this is a monotonic integer bumped ONLY on +// a breaking on-disk schema change that Kuzu cannot migrate in place — which, +// because Kuzu has no ALTER … ADD PRIMARY KEY and rel tables are bound to their +// node tables, means a change to a node table's primary key or columns. A +// database stamped with a lower version (or an older stamp with no version +// field, which decodes as 0) is reported as FormatMismatch so index/migrate +// archive and rebuild it. +// +// 1 — original schema: Entity/Observation keyed on global `id`. +// 2 — project-namespaced `pk` PRIMARY KEY added; `id` demoted to a plain column. +const CurrentSchemaVersion = 2 + // FormatStamp is the content of that sidecar. type FormatStamp struct { // KuzuVersion is the pinned go-kuzu module version, e.g. "v0.11.3". KuzuVersion string `json:"kuzu_version"` // KGVersion is the kg build that wrote the stamp, for diagnostics only. - KGVersion string `json:"kg_version,omitempty"` - StampedAt time.Time `json:"stamped_at"` + KGVersion string `json:"kg_version,omitempty"` + // SchemaVersion is the kg schema version (see CurrentSchemaVersion). Absent + // in stamps written before this field existed, where it decodes as 0 and is + // therefore treated as older than any real schema. + SchemaVersion int `json:"schema_version,omitempty"` + StampedAt time.Time `json:"stamped_at"` } // FormatStatus is the verdict of comparing a database's stamp to this build. @@ -196,9 +214,10 @@ func ReadFormatStamp(dbPath string) (*FormatStamp, error) { // read as corrupt and send a perfectly good database down the rebuild path. func WriteFormatStamp(dbPath string, kgVersion string) error { stamp := FormatStamp{ - KuzuVersion: KuzuVersion(), - KGVersion: kgVersion, - StampedAt: time.Now().UTC(), + KuzuVersion: KuzuVersion(), + KGVersion: kgVersion, + SchemaVersion: CurrentSchemaVersion, + StampedAt: time.Now().UTC(), } data, err := json.Marshal(stamp) if err != nil { @@ -250,7 +269,12 @@ func CheckFormat(dbPath string) (FormatStatus, *FormatStamp, error) { if stamp == nil { return FormatUnstamped, nil, nil } - if stamp.KuzuVersion != KuzuVersion() { + // A storage-format change (Kuzu bump) and a kg schema change (e.g. the pk + // primary key) both make the running build unable to use the database as-is; + // either one is a mismatch that index/migrate repair by archiving and + // rebuilding. SchemaVersion is absent (0) in stamps written before it + // existed, which is correctly older than any real schema. + if stamp.KuzuVersion != KuzuVersion() || stamp.SchemaVersion < CurrentSchemaVersion { return FormatMismatch, stamp, nil } return FormatOK, stamp, nil diff --git a/src/kglib/journal_test.go b/src/kglib/journal_test.go index d76794c..45041ed 100644 --- a/src/kglib/journal_test.go +++ b/src/kglib/journal_test.go @@ -633,9 +633,10 @@ func TestJournalOmitsUnstableIDs(t *testing.T) { func mustCreateWithID(t *testing.T, store *Store, id, name, entityType, projectID string) { t.Helper() result, err := store.QueryParams(` - CREATE (e:Entity {id: $id, name: $name, type: $type, project_id: $project_id, + CREATE (e:Entity {pk: $pk, id: $id, name: $name, type: $type, project_id: $project_id, created_at: $ts, updated_at: $ts}) `, map[string]any{ + "pk": PK(projectID, id), "id": id, "name": name, "type": entityType, "project_id": projectID, "ts": time.Now().UTC(), }) diff --git a/src/kglib/null_columns_test.go b/src/kglib/null_columns_test.go index 868b5cd..bbcc90b 100644 --- a/src/kglib/null_columns_test.go +++ b/src/kglib/null_columns_test.go @@ -211,7 +211,7 @@ func TestBatchEmbed_RejectsShortObservationResponse(t *testing.T) { } // Give the single entity its embedding so BatchEmbed reaches the observation // stage, then under-deliver there. - if err := store.SetEmbedding(entity.ID, make([]float32, embeddingDim)); err != nil { + if err := store.SetEmbedding(entity.ID, projectID, make([]float32, embeddingDim)); err != nil { t.Fatalf("SetEmbedding: %v", err) } diff --git a/src/kglib/observation.go b/src/kglib/observation.go index 3514f7b..83ad3e7 100644 --- a/src/kglib/observation.go +++ b/src/kglib/observation.go @@ -31,8 +31,9 @@ func (s *Store) CreateObservation(entityID, content, projectID string) (*Observa // statement there is no risk of an orphaned Observation node if the edge // creation were to fail. result, err := s.QueryParams(` - MATCH (e:Entity {id: $entity_id, project_id: $project_id}) + MATCH (e:Entity {pk: $entity_pk}) CREATE (o:Observation { + pk: $pk, id: $id, entity_id: $entity_id, content: $content, @@ -40,8 +41,9 @@ func (s *Store) CreateObservation(entityID, content, projectID string) (*Observa }) CREATE (e)-[:HAS_OBSERVATION]->(o) `, map[string]any{ + "entity_pk": PK(projectID, entityID), "entity_id": entityID, - "project_id": projectID, + "pk": PK(projectID, obs.ID), "id": obs.ID, "content": content, "created_at": obs.CreatedAt, diff --git a/src/kglib/pk.go b/src/kglib/pk.go new file mode 100644 index 0000000..f477403 --- /dev/null +++ b/src/kglib/pk.go @@ -0,0 +1,38 @@ +package kglib + +// Project-namespaced primary keys. +// +// Entity and Observation node tables carry a single PRIMARY KEY column, `pk`, +// that namespaces the human-facing `id` by the project it belongs to. The +// `id` column stays exactly what it always was — "file:…", "function:…:F", +// or a UUID — so search, CLI display, and the journal's stable-id hint are +// unchanged; `pk` exists only to make the primary key unique across projects +// that share one database file. +// +// Why this is necessary: Kuzu v0.11.3 node tables support exactly one PRIMARY +// KEY column, so a composite (project_id, id) key is not expressible. Before +// `pk`, the key was the global `id` alone, but every clear/delete and nearly +// every read is scoped by project_id — so a surviving row from a different +// project_id with the same `id` would collide with the current run's entity and +// be silently dropped. Namespacing the key fixes that at the source. +// +// Separator choice: the ADR specified NUL, but go-kuzu marshals strings across +// cgo as C strings and truncates them at the first NUL byte — a NUL-joined pk +// would collapse to just the project_id in storage, re-introducing the very +// collisions this fixes (verified: every pk reduced to its project prefix). The +// Unit Separator (0x1F) is used instead: it is a non-printable control byte that +// survives the round trip and that neither a project_id (a repo directory base +// name) nor a code-derived id ("::") nor a UUID ever contains. +const pkSeparator = "\x1f" + +// PK derives the project-namespaced primary key for an entity or observation +// from its project_id and its local id. +// +// The Unit Separator byte is collision-free and unambiguously reversible given +// the invariant above (neither operand contains it), which is what lets a single +// `pk` column stand in for the composite (project_id, id) key Kuzu cannot +// express. Every write derives `pk` here, never ad hoc, so the format lives in +// exactly one place. +func PK(projectID, id string) string { + return projectID + pkSeparator + id +} diff --git a/src/kglib/pk_test.go b/src/kglib/pk_test.go new file mode 100644 index 0000000..25d1e73 --- /dev/null +++ b/src/kglib/pk_test.go @@ -0,0 +1,99 @@ +package kglib + +import ( + "path/filepath" + "strings" + "testing" +) + +// TestPKSeparatorSurvivesCGoRoundTrip pins the invariant the whole +// project-namespaced primary key depends on: the Unit Separator (0x1F) joining +// project_id and id must survive go-kuzu's cgo string marshalling intact, where +// a NUL would truncate it. The comment in pk.go asserts this; this test proves +// it against the real Store write+read path, not the Go PK() function alone. +func TestPKSeparatorSurvivesCGoRoundTrip(t *testing.T) { + dbPath := filepath.Join(t.TempDir(), "pk.db") + + store, err := OpenStore(dbPath, testSchemaConfig()) + if err != nil { + t.Fatalf("OpenStore: %v", err) + } + + const projectID = "proj-alpha" + entity, err := store.CreateEntity("main.go", "file", projectID) + if err != nil { + store.Close() + t.Fatalf("CreateEntity: %v", err) + } + wantPK := PK(projectID, entity.ID) + + // Sanity: the value we expect to be stored actually contains the separator, + // so a passing round-trip is meaningful rather than vacuous. + if !strings.Contains(wantPK, pkSeparator) { + t.Fatalf("derived pk %q does not contain the 0x1F separator", wantPK) + } + store.Close() + + // Reopen (read-only, a fresh C database handle) so the assertions exercise a + // genuine marshal-out / marshal-in across the cgo boundary, not an in-memory + // cache from the writing connection. + ro, err := OpenStoreReadOnly(dbPath) + if err != nil { + t.Fatalf("OpenStoreReadOnly: %v", err) + } + defer ro.Close() + + // (a) The raw stored pk is byte-identical — separator present, not truncated + // to just the project prefix the way a NUL join would collapse to. + gotPK := queryOneString(t, ro, + `MATCH (e:Entity) WHERE e.id = $id RETURN e.pk`, + map[string]any{"id": entity.ID}) + if gotPK != wantPK { + t.Fatalf("stored pk mismatch:\n got %q (len %d)\n want %q (len %d)", + gotPK, len(gotPK), wantPK, len(wantPK)) + } + if !strings.Contains(gotPK, pkSeparator) { + t.Errorf("stored pk %q lost the 0x1F separator across the cgo round trip", gotPK) + } + if gotPK == projectID { + t.Errorf("stored pk collapsed to the project prefix %q (NUL-style truncation)", projectID) + } + + // (b) A MATCH by the exact pk string — separator byte and all — resolves the + // row, confirming the key is usable as the primary key it stands in for. + gotID := queryOneString(t, ro, + `MATCH (e:Entity) WHERE e.pk = $pk RETURN e.id`, + map[string]any{"pk": wantPK}) + if gotID != entity.ID { + t.Fatalf("MATCH by exact pk resolved id %q, want %q", gotID, entity.ID) + } +} + +// queryOneString runs a parameterised query expected to yield exactly one row +// whose first column is a string, and returns that string. +func queryOneString(t *testing.T, s *Store, stmt string, params map[string]any) string { + t.Helper() + result, err := s.QueryParams(stmt, params) + if err != nil { + t.Fatalf("query %q: %v", stmt, err) + } + defer result.Close() + + if !result.HasNext() { + t.Fatalf("query %q returned no rows", stmt) + } + tuple, err := result.Next() + if err != nil { + t.Fatalf("next: %v", err) + } + defer tuple.Close() + + row, err := tuple.GetAsSlice() + if err != nil { + t.Fatalf("get row: %v", err) + } + if len(row) == 0 { + t.Fatalf("query %q returned an empty row", stmt) + } + return stringOrEmpty(row[0]) +} diff --git a/src/kglib/relation.go b/src/kglib/relation.go index 9d4b3d6..87884b4 100644 --- a/src/kglib/relation.go +++ b/src/kglib/relation.go @@ -24,15 +24,21 @@ func (s *Store) CreateRelation(fromID, toID, relType, projectID string) error { // relType is a relationship label (not a value) and cannot be parameterised // in Cypher. It is validated against AllowedRelTypes above before use. + // + // Endpoints are matched on pk, never on bare id. A bare-id MATCH is + // project-blind: with two projects in one database it can bind an endpoint to + // an identically-id'd node in the OTHER project, wiring the edge to the wrong + // graph. pk namespaces id by project, so this both targets this project's + // nodes and uses the indexed primary key. query := fmt.Sprintf(` MATCH (from:Entity), (to:Entity) - WHERE from.id = $from_id AND to.id = $to_id + WHERE from.pk = $from_pk AND to.pk = $to_pk CREATE (from)-[:%s]->(to) `, relType) result, err := s.QueryParams(query, map[string]any{ - "from_id": fromID, - "to_id": toID, + "from_pk": PK(projectID, fromID), + "to_pk": PK(projectID, toID), }) if err != nil { return fmt.Errorf("create relation: %w", err) @@ -60,9 +66,17 @@ func (s *Store) GetRelations(entityID, projectID string) ([]*Relation, error) { return nil, err } + // Both endpoints are pinned to this project, not just `from`. An edge whose + // far end belongs to another project (only possible from a legacy + // cross-project edge predating the pk fix) must not be returned with a + // neighbour this project cannot see — the same project-blind-endpoint bug + // CreateRelation was fixed for. project_id is used rather than pk because the + // surrounding match already keys `from` by id+project_id and that pattern is + // legacy-safe; the returned id stays the local display id. result, err := s.QueryParams(` MATCH (from:Entity)-[r]->(to:Entity) WHERE from.id = $entity_id AND from.project_id = $project_id + AND to.project_id = $project_id RETURN from.id, to.id, label(r) `, map[string]any{"entity_id": entityID, "project_id": projectID}) if err != nil { @@ -113,9 +127,12 @@ func (s *Store) DeleteRelation(fromID, toID, relType, projectID string) error { } // relType is a relationship label validated against AllowedRelTypes above. + // Both endpoints are pinned to this project so a project-A delete can never + // reach into project B's identically-id'd edge (see GetRelations). query := fmt.Sprintf(` MATCH (from:Entity)-[r:%s]->(to:Entity) - WHERE from.id = $from_id AND to.id = $to_id AND from.project_id = $project_id + WHERE from.id = $from_id AND to.id = $to_id + AND from.project_id = $project_id AND to.project_id = $project_id DELETE r `, relType) @@ -155,9 +172,12 @@ func (s *Store) TraverseRelations(entityID, relType, projectID string) ([]*Entit } // relType is a relationship label validated against AllowedRelTypes above. + // The neighbour is pinned to this project so a traversal cannot cross into + // another project's graph via a shared entity id (see GetRelations). query := fmt.Sprintf(` MATCH (from:Entity)-[:%s]->(to:Entity) WHERE from.id = $entity_id AND from.project_id = $project_id + AND to.project_id = $project_id RETURN to.id, to.name, to.type, to.project_id, to.created_at, to.updated_at `, relType) diff --git a/src/kglib/schema.go b/src/kglib/schema.go index e11e58b..4619f3a 100644 --- a/src/kglib/schema.go +++ b/src/kglib/schema.go @@ -40,9 +40,15 @@ func (s *Store) initSchema(cfg *SchemaConfig) error { // Fixed node tables and the HAS_OBSERVATION edge staticStatements := []string{ - // Entity node table + // Entity node table. + // + // pk is the project-namespaced primary key (see pk.go); id stays the + // human-facing/journal-stable value ("file:…", a UUID, …) and is a plain + // indexed-by-nothing attribute. Kuzu allows only one PRIMARY KEY column, + // so the composite (project_id, id) key is carried by pk instead. `CREATE NODE TABLE IF NOT EXISTS Entity( - id STRING PRIMARY KEY, + pk STRING PRIMARY KEY, + id STRING, name STRING, type STRING, project_id STRING, @@ -52,9 +58,11 @@ func (s *Store) initSchema(cfg *SchemaConfig) error { visibility STRING )`, - // Observation node table + // Observation node table. pk namespaces id by project_id exactly as on + // Entity; id stays the UUID the rest of the code and the journal use. `CREATE NODE TABLE IF NOT EXISTS Observation( - id STRING PRIMARY KEY, + pk STRING PRIMARY KEY, + id STRING, entity_id STRING, content STRING, created_at TIMESTAMP, diff --git a/src/kglib/search.go b/src/kglib/search.go index 151397f..9431335 100644 --- a/src/kglib/search.go +++ b/src/kglib/search.go @@ -178,7 +178,7 @@ func (s *Store) KeywordSearchFiltered(projectID, query string, limit int, filter for i, e := range entities { entityIDs[i] = e.ID } - obsMap, err := s.batchGetObservations(entityIDs, 3) + obsMap, err := s.batchGetObservations(entityIDs, projectID, 3) if err != nil { // Degrade gracefully — empty observations rather than failing the search. obsMap = map[string][]*Observation{} @@ -212,16 +212,22 @@ func (s *Store) GetTopObservations(entityID, projectID string, limit int) ([]*Ob // Indexed rows share a timestamp routinely: the indexer stamps one `now` // per file, so every symbol in a file ties. // + // Scoped by project through the parent entity, not by o.entity_id alone: + // Observation has no project_id column, so matching the edge from an Entity + // filtered on project_id is what keeps a project-A search from pulling in a + // project-B observation that shares the entity_id. GetObservations takes the + // same route. + // // LIMIT takes an integer expression; using %d (not user input) is safe here. query := fmt.Sprintf(` - MATCH (o:Observation) - WHERE o.entity_id = $entity_id + MATCH (e:Entity)-[:HAS_OBSERVATION]->(o:Observation) + WHERE e.project_id = $project_id AND o.entity_id = $entity_id RETURN o.id, o.entity_id, o.content, o.created_at ORDER BY o.created_at DESC LIMIT %d `, limit) - result, err := s.QueryParams(query, map[string]any{"entity_id": entityID}) + result, err := s.QueryParams(query, map[string]any{"entity_id": entityID, "project_id": projectID}) if err != nil { return nil, fmt.Errorf("query observations: %w", err) } @@ -265,7 +271,12 @@ func (s *Store) GetTopObservations(entityID, projectID string, limit int) ([]*Ob // in entityIDs using a single Cypher query, eliminating N+1 round-trips. // It returns a map keyed by entity ID; entities with no observations are absent. // limit is applied per entity: only the newest `limit` rows are kept. -func (s *Store) batchGetObservations(entityIDs []string, limit int) (map[string][]*Observation, error) { +// +// projectID scopes the read: Observation carries no project_id, so under +// multi-project-per-DB two projects sharing an entity_id would otherwise cross- +// contaminate each other's search results. Matching the HAS_OBSERVATION edge +// from an Entity filtered on project_id confines the result to this project. +func (s *Store) batchGetObservations(entityIDs []string, projectID string, limit int) (map[string][]*Observation, error) { if len(entityIDs) == 0 { return map[string][]*Observation{}, nil } @@ -281,13 +292,13 @@ func (s *Store) batchGetObservations(entityIDs []string, limit int) (map[string] // counter instead. We fetch all rows ordered by (entity_id, created_at DESC) // and apply the per-entity limit in Go — this is still a single round-trip. query := ` - MATCH (o:Observation) - WHERE o.entity_id IN $entity_ids + MATCH (e:Entity)-[:HAS_OBSERVATION]->(o:Observation) + WHERE e.project_id = $project_id AND o.entity_id IN $entity_ids RETURN o.id, o.entity_id, o.content, o.created_at ORDER BY o.entity_id, o.created_at DESC ` - result, err := s.QueryParams(query, map[string]any{"entity_ids": ids}) + result, err := s.QueryParams(query, map[string]any{"entity_ids": ids, "project_id": projectID}) if err != nil { return nil, fmt.Errorf("batch query observations: %w", err) } @@ -379,7 +390,7 @@ func (s *Store) VectorSearch(projectID string, queryEmbedding []float32, limit i for i, c := range candidates { entityIDs[i] = c.entity.ID } - obsMap, err := s.batchGetObservations(entityIDs, 3) + obsMap, err := s.batchGetObservations(entityIDs, projectID, 3) if err != nil { obsMap = map[string][]*Observation{} } diff --git a/src/kglib/search_test.go b/src/kglib/search_test.go index 7ddd0f2..23f40b7 100644 --- a/src/kglib/search_test.go +++ b/src/kglib/search_test.go @@ -255,7 +255,7 @@ func TestBatchGetObservations(t *testing.T) { } t.Run("returns observations for multiple entities in one call", func(t *testing.T) { - obsMap, err := store.batchGetObservations([]string{entity1.ID, entity2.ID}, 3) + obsMap, err := store.batchGetObservations([]string{entity1.ID, entity2.ID}, projectID, 3) if err != nil { t.Fatalf("batchGetObservations failed: %v", err) } @@ -276,7 +276,7 @@ func TestBatchGetObservations(t *testing.T) { }) t.Run("empty entity list returns empty map", func(t *testing.T) { - obsMap, err := store.batchGetObservations([]string{}, 3) + obsMap, err := store.batchGetObservations([]string{}, projectID, 3) if err != nil { t.Fatalf("batchGetObservations with empty IDs failed: %v", err) } @@ -290,7 +290,7 @@ func TestBatchGetObservations(t *testing.T) { if err != nil { t.Fatalf("CreateEntity failed: %v", err) } - obsMap, err := store.batchGetObservations([]string{entityEmpty.ID}, 3) + obsMap, err := store.batchGetObservations([]string{entityEmpty.ID}, projectID, 3) if err != nil { t.Fatalf("batchGetObservations failed: %v", err) } @@ -350,10 +350,10 @@ func TestVectorSearch(t *testing.T) { query := make([]float32, dim) query[0] = 1 - if err := store.SetEmbedding(e1.ID, e1Emb); err != nil { + if err := store.SetEmbedding(e1.ID, pid, e1Emb); err != nil { t.Fatalf("SetEmbedding failed: %v", err) } - if err := store.SetEmbedding(e2.ID, e2Emb); err != nil { + if err := store.SetEmbedding(e2.ID, pid, e2Emb); err != nil { t.Fatalf("SetEmbedding failed: %v", err) } @@ -391,7 +391,7 @@ func TestVectorSearch(t *testing.T) { } emb := make([]float32, dim) emb[0] = float32(i) + 1 - if err := store.SetEmbedding(e.ID, emb); err != nil { + if err := store.SetEmbedding(e.ID, pid, emb); err != nil { t.Fatalf("SetEmbedding failed: %v", err) } } @@ -754,7 +754,7 @@ func TestHNSWScalability(t *testing.T) { raw[j] = float32(rng.Float64()*2 - 1) } emb := normF32(raw) - if err := store.SetEmbedding(ent.ID, emb); err != nil { + if err := store.SetEmbedding(ent.ID, projectID, emb); err != nil { t.Fatalf("SetEmbedding[%d] failed: %v", i, err) } } diff --git a/src/kglib/store.go b/src/kglib/store.go index 534ec10..600e9a2 100644 --- a/src/kglib/store.go +++ b/src/kglib/store.go @@ -33,6 +33,7 @@ type Store struct { allowedRelTypes []string // configured relation types for validation journal *Journal // hand-write journal; nil unless EnableJournal was called hasVisibility bool // Entity.visibility exists in this database (see entityColumns) + hasPK bool // Entity.pk (project-namespaced primary key) exists; false on pre-pk databases } // OpenStore opens or creates a Kuzu database in read-write mode with the given schema configuration. @@ -104,22 +105,41 @@ func openStoreWithConfig(dbPath string, readOnly bool, cfg *SchemaConfig) (*Stor store.Close() return nil, fmt.Errorf("initialize schema: %w", err) } - - // Record which engine build wrote this database. Reaching this point - // proves the running engine can read the format, so the stamp is - // truthful by construction. It is a sidecar rather than a row in the - // database because its whole job is to be readable when the database is - // not. A failure to stamp is not fatal — the database is fine, it just - // falls back to the unstamped path on a later version check. - _ = WriteFormatStamp(dbPath, buildVersion) } // Which optional columns this database actually has. After initSchema on a - // write open, so a freshly-migrated database is detected correctly; and on + // write open, so a freshly-created database is detected correctly; and on // read-only opens too, which never migrate and so may be reading a graph - // written before a column existed. + // written before a column existed. Must run before the pk gate and the + // stamp below, which both depend on hasPK. store.detectColumns() + if !readOnly { + // `CREATE NODE TABLE IF NOT EXISTS` cannot add the pk PRIMARY KEY column + // to an Entity table that predates it, and Kuzu has no ALTER … ADD + // PRIMARY KEY, so a pre-pk database opened read-write would otherwise let + // every pk-referencing write fail with an opaque "Cannot find property + // pk". Refuse clearly instead and point at the rebuild path. `kg index` + // and `kg migrate` archive + rebuild such a database before opening it + // (CheckFormat reports FormatMismatch on its lower schema_version), so + // they never reach this; a bare `kg add` against an un-rebuilt legacy + // database is what this catches. + if !store.hasPK { + store.Close() + return nil, fmt.Errorf( + "knowledge graph database at %s predates the project-namespaced primary key "+ + "and must be rebuilt: run 'kg index' (or 'kg migrate' to rebuild every graph)", dbPath) + } + + // Record which engine build and schema version wrote this database. + // Reaching this point proves the running engine can read the format, so + // the stamp is truthful by construction. It is a sidecar rather than a + // row in the database because its whole job is to be readable when the + // database is not. A failure to stamp is not fatal — the database is + // fine, it just falls back to the unstamped path on a later version check. + _ = WriteFormatStamp(dbPath, buildVersion) + } + return store, nil } @@ -153,10 +173,16 @@ func openDiagnostic(dbPath string, readOnly bool, err error) error { "(create one with 'kg index'): %w", dbPath, err) } case FormatMismatch: - return fmt.Errorf("knowledge graph database at %s was written by kuzu %s "+ - "but this build uses %s; it needs rebuilding "+ + if stamp.KuzuVersion != KuzuVersion() { + return fmt.Errorf("knowledge graph database at %s was written by kuzu %s "+ + "but this build uses %s; it needs rebuilding "+ + "(run 'kg index' to rebuild it, or 'kg migrate' to rebuild every graph): %w", + dbPath, stamp.KuzuVersion, KuzuVersion(), err) + } + return fmt.Errorf("knowledge graph database at %s uses kg schema v%d "+ + "but this build needs v%d; it needs rebuilding "+ "(run 'kg index' to rebuild it, or 'kg migrate' to rebuild every graph): %w", - dbPath, stamp.KuzuVersion, KuzuVersion(), err) + dbPath, stamp.SchemaVersion, CurrentSchemaVersion, err) } } @@ -168,6 +194,13 @@ func openDiagnostic(dbPath string, readOnly bool, err error) error { return fmt.Errorf("open kuzu database: %w", err) } +// HasPK reports whether this database carries the project-namespaced pk primary +// key. It is false for a pre-pk database — one written before the pk column +// existed — which a read-only open can still read but a read-write open refuses. +// Exposed so the migration path can distinguish an unstamped-but-current +// database (leave alone) from an unstamped pre-pk one (archive and rebuild). +func (s *Store) HasPK() bool { return s.hasPK } + // Close closes the database connection. // It acquires mu to ensure no query is in flight while closing. func (s *Store) Close() error { diff --git a/src/kglib/visibility_test.go b/src/kglib/visibility_test.go index 13d6c45..9ff4a19 100644 --- a/src/kglib/visibility_test.go +++ b/src/kglib/visibility_test.go @@ -47,10 +47,11 @@ func TestVisibilityRoundTrips(t *testing.T) { // Write directly, the way the indexer's bulk path does. result, err := store.QueryParams(` CREATE (e:Entity { - id: $id, name: $name, type: $type, project_id: $project_id, + pk: $pk, id: $id, name: $name, type: $type, project_id: $project_id, created_at: $ts, updated_at: $ts, visibility: $visibility }) `, map[string]any{ + "pk": PK("p", "function:main.go:helper"), "id": "function:main.go:helper", "name": "helper", "type": "function", "project_id": "p", "ts": time.Now().UTC(), "visibility": VisibilityPrivate, })