From 95be206e9f99dbb10a44a6b68e788e5e3739178d Mon Sep 17 00:00:00 2001 From: Bryan Woodruff Date: Tue, 13 Oct 2026 11:39:37 +0100 Subject: [PATCH 1/7] fix(kg): project-namespaced pk primary key; scope observation/embedding paths per project MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The Entity/Observation primary key was global `id` alone, but every clear/delete and most reads are scoped by `project_id`. When two projects shared a DB file, a prior project's rows survived the project-scoped clear and collided with the current run's identically-id'd entities on CREATE, which were then silently dropped — cross-project data loss, surfaced as the "duplicated primary key" / "COPY FROM JSON unavailable" warning flood. Relations bled too, because the CreateRelation endpoint MATCH was project-blind. Implements ADR docs/kg-entity-key-fix-design.md (Option A1): - Add `pk STRING PRIMARY KEY = project_id <0x1F> id` to Entity and Observation via a single central helper (src/kglib/pk.go); `id` stays a plain, human-facing, journal-stable column. Separator is 0x1F (Unit Separator), not NUL: cgo truncates C strings at NUL, which would collapse every pk to its project_id. - Qualify endpoint/lookup MATCHes by project: CreateRelation matches on pk (closes the cross-project bleed); CreateObservation binds its parent by pk; GetEntity matches on pk with an id+project_id fallback for un-migrated DBs. - Scope observation READ paths (GetTopObservations, batchGetObservations) through the parent entity's project, matching GetObservations. - Scope the embeddings write/invalidation path by project (SetEmbedding now takes projectID and matches on pk; removed the project-blind entityProjectID lookup). - Switch entity writes CREATE -> MERGE on pk (idempotent; ON CREATE populates the full row, ON MATCH refreshes only name/type/updated_at). - Migration: bump FormatStamp.SchemaVersion to 2 so CheckFormat trips the existing archiveIfMismatch rebuild + journal replay (Kuzu v0.11.3 has no ADD PRIMARY KEY). The journal is the only copy of hand-writes — the old DB is archived, never deleted. - Federated cross-graph joins continue to key on local id/name, never pk. Tests: cross-project shared-id retention, zero-warning re-index, observation non-bleed across projects, and schema-version rebuild + journal replay. Co-Authored-By: Claude --- docs/kg-entity-key-fix-design.md | 219 +++++++++++++++++ .../knowledge/graph_federated_test.go | 4 +- src/kg/internal/knowledge/indexer.go | 44 +++- .../knowledge/indexer_crossproject_test.go | 230 ++++++++++++++++++ src/kg/internal/knowledge/indexer_pdf.go | 9 +- src/kg/internal/knowledge/log_app_indexer.go | 9 +- src/kg/internal/knowledge/log_indexer.go | 42 ++-- src/kg/internal/knowledge/store_wrappers.go | 6 + src/kg/migrate.go | 12 +- src/kg/migrate_test.go | 90 +++++++ src/kglib/embeddings.go | 55 +---- src/kglib/embeddings_test.go | 2 +- src/kglib/entity.go | 34 ++- src/kglib/federated_test.go | 3 +- src/kglib/format.go | 36 ++- src/kglib/journal_test.go | 3 +- src/kglib/null_columns_test.go | 2 +- src/kglib/observation.go | 6 +- src/kglib/pk.go | 38 +++ src/kglib/relation.go | 12 +- src/kglib/schema.go | 16 +- src/kglib/search.go | 29 ++- src/kglib/search_test.go | 14 +- src/kglib/store.go | 52 +++- src/kglib/visibility_test.go | 3 +- 25 files changed, 834 insertions(+), 136 deletions(-) create mode 100644 docs/kg-entity-key-fix-design.md create mode 100644 src/kg/internal/knowledge/indexer_crossproject_test.go create mode 100644 src/kglib/pk.go diff --git a/docs/kg-entity-key-fix-design.md b/docs/kg-entity-key-fix-design.md new file mode 100644 index 0000000..cc75deb --- /dev/null +++ b/docs/kg-entity-key-fix-design.md @@ -0,0 +1,219 @@ +# ADR: Fix Entity primary-key / project-scope mismatch in the kg indexer + +Status: **Accepted — Option A (A1 variant)**. Multi-project-per-DB is a supported +scenario (product decision, 2026-10-09), so the global-PK bug is fixed by a +namespaced primary key, not by foreclosing multi-project DBs. +Date: 2026-10-09 + +## Context + +`kg index` emits large volumes of warnings of the form: + +``` +Note: COPY FROM JSON unavailable (... Found duplicated primary key value ...); falling back to row-by-row insert +Warning: insert entity :: ... Found duplicated primary key value , which violates the uniqueness constraint ... +``` + +These are **not benign**. Investigation (source-verified) shows: + +1. **The `Entity` primary key is global `id` alone** — `src/kglib/schema.go:45` + (`id STRING PRIMARY KEY`). Same for `Observation` (`schema.go:57`). +2. **Every clear/delete and nearly every read is scoped by `project_id`**: + - `clearCodeDerivedData` — `src/kg/internal/knowledge/indexer.go:280-288` + - `clearProjectData` — `indexer.go:373-399` + - `deleteProjectRelations` — `indexer.go:209-226` + - reads in `graph.go:136-178`, `ListEntities` (`src/kglib/entity.go:326`) +3. **In-run entity dedup is airtight**: `writeEntityVis` (`indexer.go:584`) is the + *only* append site and guards every append with `if seen[id] { return false }`. + So a batch never contains duplicate ids — collisions are strictly against rows + **already in the DB**. +4. **Relations match endpoints by bare `id` with no project filter in the MATCH** — + `CreateRelation`, `src/kglib/relation.go:24-28` + (`WHERE from.id = $from_id AND to.id = $to_id`). + +### Failure mechanism + +Because the PK is global but the clear is project-scoped, any pre-existing +code-derived row under a **different `project_id`** survives the clear. When the +current run writes the identically-id'd entity it collides and is **silently +dropped** (first/old row wins; `CREATE` at `indexer.go:659-677` and the COPY path +at `indexer.go:644-653`; the log indexer repeats the pattern at +`log_indexer.go:278-335`). Downstream: + +- The current project's entity, plus its freshly-computed observations/relations + (which attach **by id**), never land. +- Relations can bind to the *other* project's surviving node (relation.go MATCH is + project-blind), or be dropped as dangling (`batchCreateRelations`, indexer.go:763). +- `kg stats` / graph reads scoped to the live `project_id` under-return. + +A mismatched-`project_id` partition arises from any of: legacy `project_id = ""` +rows, a changed `projectIDFromCwd` (`src/kg/index.go:140`) across versions, a +moved/renamed repo, or two scope configs whose `cfg.Database` point at the same DB +file (`indexScopeDB`, `src/kg/index.go:103-118`). The *presence* of the warnings +proves such a partition exists — a same-project prior run would have been emptied +by the clear, producing no collision. + +### Hard constraint + +Kuzu **v0.11.3** (`src/kg*/go.mod`) node tables support exactly **one** PRIMARY KEY +column. A literal composite `(project_id, id)` node PK is **not expressible**. The +realistic options are a namespaced single key, or a structural one-project-per-DB +guarantee. + +## Gating open question (needs a human decision) + +**Is "multiple `project_id`s in one DB file" a supported scenario, or an accident?** + +- The deployment path opens **one DB per scope** and derives `project_id` from cwd + (`index.go:103-140`), i.e. one project per file in practice. +- But `KGMeta` keys by `project_id` (`schema.go:69`), and reads filter by + `project_id` — the schema *looks* built for many projects per file. +- Federated linking (`graph_federated.go`) links across **separate** graph files, + not multiple projects in one file. + +The answer selects the recommended option: + +- **One project per DB is the real model → Option B (+ C).** Minimal, correct. +- **Multi-project-per-DB must be supported → Option A.** Larger, fully correct. + +Recommendation absent other input: **one project per DB is the real model → Option B +primary, C as hardening.** Multi-project-per-file appears to be the accident that +creates stale partitions. + +## Accepted design — Option A1 (namespaced `pk`, `id` stays local) + +Decision: multi-project-per-DB **must** be supported, so Option B is rejected and +the key is made project-safe via a namespaced primary key. Within Option A, the +**A1** variant is chosen: keep the existing `id` column human-facing and unchanged +(`file:…`, `function:…`, UUIDs), and add a separate namespaced primary-key column. +This preserves the id contract that search, CLI display, and the journal +`stableIDHint` all depend on; A2 (rewriting `id` itself) is rejected as needlessly +breaking those. + +Concrete decisions: + +1. **Schema** (`src/kglib/schema.go`): Entity and Observation gain + `pk STRING PRIMARY KEY`; the current `id` becomes a plain (non-unique) STRING + attribute. `entity_id` on Observation stays, but observation→entity joins key on + the pair `(project_id, id)` (see 4). +2. **Key derivation** — one central helper: + `pk(projectID, id) = projectID + "\x00" + id`. NUL is the separator because + neither a `project_id` nor a code-derived/UUID `id` ever contains one; document + that invariant next to the helper. All writes derive `pk` here, never ad hoc. +3. **Writes populate `pk`**: COPY column lists (`indexer.go:646`, the log-indexer + COPY in `log_indexer.go`, the observation COPY) and every `CREATE` + (`indexer.go:659-677`, `entity.go:148`, `log_indexer.go` fallback) add `pk`. +4. **Lookups/endpoint matches become project-qualified** (the real correctness + fix): every `{id: …}` / `WHERE e.id = …` that is not already project-scoped must + match on `(project_id, id)` (equivalently on `pk`): + - `CreateRelation` MATCH — `relation.go:24-28` (currently project-blind — this is + the cross-project-bleed site). Add `from.project_id = $pid AND to.project_id = + $pid`, or match on `from.pk`/`to.pk`. + - `GetEntity` — `entity.go` (confirm it filters project_id; if it keys on id + alone, qualify it). + - Observation joins / `HAS_OBSERVATION` creation keyed on `entity_id`. + - `clearCodeDerivedData`, `clearProjectData`, `deleteProjectRelations` already + filter `project_id`; with a project-unique `pk` the DETACH DELETE now both + targets only this project's rows **and** stops colliding — no logic change + beyond confirming they read the new column where needed. +5. **Upsert**: with `pk` project-safe, switch entity writes `CREATE` → `MERGE` on + `pk` with last-write-wins refresh of `name`/`type`/`updated_at`. This makes + re-index idempotent and eliminates the warnings outright. (`MERGE` was unsafe + under the old global PK; it is correct now.) +6. **Federated linking** — `graph_federated.go`: cross-graph joins must key on the + **local `id`** (and/or name), never on `pk`, since `pk` embeds a per-graph + `project_id` that differs across files. Audit every cross-graph comparison. +7. **Migration**: Kuzu has no `ALTER TABLE … ADD PRIMARY KEY`, so the new `pk` + column forces a table recreate → rebuild-from-source. Bump `KGMeta.kg_version` + (`StampMeta`, `index.go:216`); on version mismatch, ride `archiveIfMismatch` + (`index.go:124`) to archive the old DB and rebuild fresh, then replay hand-writes + via `restoreHandWrites` (`index.go:210`). The journal is the only copy of + hand-writes — archive, never delete. + +Open sub-question for the engineer to resolve during implementation: whether Kuzu +rel tables in v0.11.3 let `MERGE`/`MATCH` resolve endpoints efficiently by a +non-PK `(project_id, id)` pair, or whether endpoint matching should go through the +`pk` column directly (likely faster, since `pk` is the indexed PK). Prefer `pk`. + +## Options (for the record) + +### Option A — Namespaced single primary key (supports multi-project-per-DB) + +Store PK as `pk = project_id + "\x00" + localID`; keep `id` (the current +`file:…`/`function:…`/UUID value) as a **non-PK indexed** attribute for display, +search, and the journal hint. + +- Rewrite all MATCH/CREATE/COPY on Entity/Observation to key on `pk`, including + relation endpoint matching (`relation.go:24-28` must match on `pk`, not `id`). +- Blast radius (each must change): + - id construction sites: `indexer_treesitter.go:198` (`qualifiedSymbolID`), + and `fmt.Sprintf("file:%s"|"type:%s:%s"|"function:%s:%s"|"import:%s")` across + `indexer_yaml.go`, `indexer_makefile.go`, `indexer_asm.go`, + `indexer_jsonschema.go`, `indexer_graphql.go`, `indexer_html.go`, + `indexer_markdown.go`, `indexer_pdf.go`, `indexer_cmake.go` — these now feed + `localID`, with `pk` derived centrally in `writeEntityVis`. + - `codeDerivedIDClause` / `codeDerivedIDPrefixes` — `kg_types.go:34-52` + (prefix test must run against `localID`, not `pk`). + - COPY column lists — `indexer.go:646`, `log_indexer.go` COPY, observation COPY. + - reads — `graph.go`, `entity.go` (`ListEntities`, `GetEntity`), + `src/kglib/search.go`. + - journal `stableIDHint` and replay — `relation.go` note, `index.go:210` + (`restoreHandWrites`); the hint must remain the *localID* so replay still works. + - federated cross-graph linking — `graph_federated.go` (joins by id across files). +- Pro: preserves multi-project-per-DB; fully correct. Con: wide, touches the + human-facing/stable id contract and the journal. + +### Option B — Enforce one `project_id` per DB file (recommended primary) + +At index time, read the DB's existing `KGMeta.project_id`. If it differs from the +current run's `project_id` (or legacy `""` rows exist), treat it exactly like a +storage-format mismatch: **archive the old DB and rebuild fresh** — reusing the +existing `archiveIfMismatch` lever (`index.go:124`). With one project per file, the +global `id` PK is project-scoped by construction and collisions cannot occur. + +- Change surface is small and localized to `index.go` (a project_id check beside + `archiveIfMismatch`) plus a KGMeta read helper. +- Journal safety: archived DB keeps its journal (`index.go:128-132`); hand-writes + replay into the fresh DB via `restoreHandWrites` (`index.go:210`), which is + idempotent. +- Pro: minimal, correct, matches real deployment. Con: forecloses + multi-project-per-DB (acceptable iff the gating question resolves that way). + +### Option C — Global code-derived clear + MERGE upsert (hardening, pairs with B) + +Under the one-project-per-DB premise, code-derived ids are globally unique within a +DB, so: + +- Drop the `{project_id: $project_id}` filter from the **code-derived** DETACH DELETE + in `clearCodeDerivedData` (indexer.go:280-288) so a rebuild clears *all* + code-derived rows regardless of a stale project_id — making re-index idempotent and + warning-free even on a legacy DB. +- Switch entity writes `CREATE` → Kuzu **`MERGE`** (idempotent upsert) with + last-write-wins refresh of `name`/`type`/`updated_at`. + - **MERGE is unsafe under the current global PK** (one project would overwrite + another's row). It is correct only after B (or A) makes the key project-safe. + +## Migration & schema versioning + +- Bump the schema/kg version stamp (`KGMeta.kg_version`, written by `StampMeta`, + `index.go:216`) so older DBs are recognized. +- Option B needs no table rewrite (fresh rebuild on project_id mismatch). +- Option A requires a one-time rebuild (new PK column); ride `archiveIfMismatch`: + on version mismatch, archive + rebuild-from-source (code-derived data is + regenerable), then journal-replay hand-writes. +- In both, the journal is the **only** copy of hand-written entities — migration must + archive (not delete) the prior DB and run `restoreHandWrites` against the fresh one. + +## Acceptance criteria + +- [ ] `cd src/kg && go build ./... && go test ./...` exits 0 +- [ ] `cd src/kglib && go build ./... && go test ./...` exits 0 +- [ ] New regression test: index project A, then project B that shares entity ids + (e.g. both `import:logging`, `file:common.py`); assert **both** projects retain + their own entities and `kg stats` per project is correct. +- [ ] Re-indexing an already-indexed project emits **zero** `duplicated primary key` + warnings and **no** `COPY FROM JSON unavailable` note. +- [ ] Hand-written entities/relations (`kg add`) survive a re-index and a migration + rebuild (journal replay verified). +- [ ] `MATCH (e:Entity) RETURN DISTINCT e.project_id` on a migrated DB shows exactly + one project_id (Option B) — a field diagnostic to confirm the fix landed. diff --git a/src/kg/internal/knowledge/graph_federated_test.go b/src/kg/internal/knowledge/graph_federated_test.go index 6e6dff4..6404522 100644 --- a/src/kg/internal/knowledge/graph_federated_test.go +++ b/src/kg/internal/knowledge/graph_federated_test.go @@ -269,8 +269,8 @@ func seedEntityWithID(t *testing.T, aiDir, layer, id, name, entityType string) { defer store.Close() result, err := store.QueryParams(` - CREATE (e:Entity {id: $id, name: $name, type: $type, project_id: $project_id}) - `, map[string]any{"id": id, "name": name, "type": entityType, "project_id": fedProject}) + CREATE (e:Entity {pk: $pk, id: $id, name: $name, type: $type, project_id: $project_id}) + `, map[string]any{"pk": PK(fedProject, id), "id": id, "name": name, "type": entityType, "project_id": fedProject}) if err != nil { t.Fatalf("seed entity %s in %s: %v", id, layer, err) } diff --git a/src/kg/internal/knowledge/indexer.go b/src/kg/internal/knowledge/indexer.go index bc5e996..de7bada 100644 --- a/src/kg/internal/knowledge/indexer.go +++ b/src/kg/internal/knowledge/indexer.go @@ -626,6 +626,7 @@ func (idx *Indexer) batchCreateEntities(entities []entityRecord) error { enc := json.NewEncoder(f) for _, ent := range entities { row := map[string]string{ + "pk": PK(ent.ProjectID, ent.ID), "id": ent.ID, "name": ent.Name, "type": ent.Type, @@ -641,9 +642,12 @@ func (idx *Indexer) batchCreateEntities(entities []entityRecord) error { } f.Close() - // Bulk load via COPY FROM JSON + // Bulk load via COPY FROM JSON. pk is the project-namespaced primary key + // (first column so it leads the node table's key); the selective clear + // already removed this project's code-derived rows, so within one project + // every pk in the batch is new and the load cannot collide. query := fmt.Sprintf( - `COPY Entity(id, name, type, project_id, created_at, updated_at, visibility) FROM '%s'`, + `COPY Entity(pk, id, name, type, project_id, created_at, updated_at, visibility) FROM '%s'`, ndjsonPath, ) result, err := idx.store.Query(query) @@ -653,20 +657,28 @@ func (idx *Indexer) batchCreateEntities(entities []entityRecord) error { } // COPY FROM JSON not supported by this Kuzu version — fall back to - // individual parameterized inserts (slower but universally compatible). + // individual parameterized upserts (slower but universally compatible). + // + // MERGE on pk, not CREATE: pk is now project-safe, so last-write-wins on a + // re-indexed symbol is correct and idempotent — a second index of the same + // project refreshes the row instead of raising a duplicate-primary-key + // warning the way CREATE did. fmt.Printf("Note: COPY FROM JSON unavailable (%v); falling back to row-by-row insert\n", err) for _, ent := range entities { r, err := idx.store.QueryParams(` - CREATE (e:Entity { - id: $id, - name: $name, - type: $type, - project_id: $project_id, - created_at: $created_at, - updated_at: $updated_at, - visibility: $visibility - }) + MERGE (e:Entity {pk: $pk}) + ON CREATE SET e.id = $id, + e.name = $name, + e.type = $type, + e.project_id = $project_id, + e.created_at = $created_at, + e.updated_at = $updated_at, + e.visibility = $visibility + ON MATCH SET e.name = $name, + e.type = $type, + e.updated_at = $updated_at `, map[string]any{ + "pk": PK(ent.ProjectID, ent.ID), "id": ent.ID, "name": ent.Name, "type": ent.Type, @@ -828,9 +840,15 @@ func (idx *Indexer) bulkLoadRelations(relType string, rels []relationRecord) err if err != nil { return fmt.Errorf("create ndjson: %w", err) } + // COPY into a REL table resolves each endpoint against the node table's + // PRIMARY KEY, which is now pk, not id. Every relation in one index run is + // within idx.projectID, so both ends namespace by that project. enc := json.NewEncoder(f) for _, r := range rels { - if err := enc.Encode(map[string]string{"from": r.FromID, "to": r.ToID}); err != nil { + if err := enc.Encode(map[string]string{ + "from": PK(idx.projectID, r.FromID), + "to": PK(idx.projectID, r.ToID), + }); err != nil { f.Close() return fmt.Errorf("encode relation json: %w", err) } diff --git a/src/kg/internal/knowledge/indexer_crossproject_test.go b/src/kg/internal/knowledge/indexer_crossproject_test.go new file mode 100644 index 0000000..f253c9f --- /dev/null +++ b/src/kg/internal/knowledge/indexer_crossproject_test.go @@ -0,0 +1,230 @@ +package knowledge + +import ( + "io" + "os" + "path/filepath" + "strings" + "testing" +) + +// writeCommonProject creates a one-file Go project whose source-derived IDs are +// identical regardless of which directory it lives in: "file:common.go" and +// "function:common.go:Shared". Two such projects indexed into one database share +// every entity ID, which is exactly the collision the project-namespaced pk must +// keep from fusing them. +func writeCommonProject(t *testing.T, dir string) { + t.Helper() + if err := os.MkdirAll(dir, 0o755); err != nil { + t.Fatalf("MkdirAll: %v", err) + } + src := "package common\n\nimport \"logging\"\n\nfunc Shared() { logging.Init() }\n" + if err := os.WriteFile(filepath.Join(dir, "common.go"), []byte(src), 0o644); err != nil { + t.Fatalf("write source: %v", err) + } +} + +// indexInto runs a fresh indexer for projectID over srcDir against store. +func indexInto(t *testing.T, store *Store, projectID, srcDir string) *IndexStats { + t.Helper() + idx, err := NewIndexer(store, projectID, srcDir) + if err != nil { + t.Fatalf("NewIndexer(%s): %v", projectID, err) + } + stats, err := idx.Index() + if err != nil { + t.Fatalf("Index(%s): %v", projectID, err) + } + return stats +} + +// Two projects sharing one database file and sharing entity IDs must each keep +// their own entities. Before the pk fix, the second project's write collided +// with the first's identically-id'd (but project-scoped-clear-surviving) row and +// was silently dropped; the project under the live project_id then under-counted. +func TestIndexCrossProjectSharedIDsRetainsBoth(t *testing.T) { + const projA, projB = "project-a", "project-b" + + tmp := t.TempDir() + dirA := filepath.Join(tmp, "a") + dirB := filepath.Join(tmp, "b") + writeCommonProject(t, dirA) + writeCommonProject(t, dirB) + + store, err := OpenStore(filepath.Join(tmp, "knowledge.db")) + if err != nil { + t.Fatalf("OpenStore: %v", err) + } + defer store.Close() + + // Index A, then B — B shares every id A just wrote. + indexInto(t, store, projA, dirA) + indexInto(t, store, projB, dirB) + + countA, err := store.CountEntities(projA) + if err != nil { + t.Fatalf("CountEntities(A): %v", err) + } + countB, err := store.CountEntities(projB) + if err != nil { + t.Fatalf("CountEntities(B): %v", err) + } + if countA == 0 || countB == 0 { + t.Fatalf("a project lost all its entities: A=%d B=%d", countA, countB) + } + if countA != countB { + t.Errorf("projects sharing ids got different counts: A=%d B=%d (cross-project drop)", countA, countB) + } + + // The shared ids resolve to each project's OWN row, not the other's. + for _, proj := range []string{projA, projB} { + got, err := store.GetEntity("file:common.go", proj) + if err != nil { + t.Fatalf("GetEntity(file:common.go, %s): %v", proj, err) + } + if got.ProjectID != proj { + t.Errorf("file:common.go under %s resolved to project_id %q", proj, got.ProjectID) + } + if _, err := store.GetEntity("import:logging", proj); err != nil { + t.Errorf("import:logging missing for %s: %v", proj, err) + } + } + + // Re-indexing A must not disturb B — the selective clear is project-scoped and + // the pk keeps B's identically-id'd rows out of A's COPY collision set. + indexInto(t, store, projA, dirA) + reCountB, err := store.CountEntities(projB) + if err != nil { + t.Fatalf("CountEntities(B) after re-index A: %v", err) + } + if reCountB != countB { + t.Errorf("re-indexing A changed B's entity count: before=%d after=%d", countB, reCountB) + } + + // Relations must not bleed across projects: A's edges stay A's. + relsB, err := store.GetRelations("file:common.go", projB) + if err != nil { + t.Fatalf("GetRelations(file:common.go, B): %v", err) + } + for _, r := range relsB { + // Every endpoint GetRelations returns is addressed by local id; confirm it + // resolves within B rather than pointing at A's graph. + if _, err := store.GetEntity(r.ToID, projB); err != nil { + t.Errorf("relation from B points at %q which is not in B: %v", r.ToID, err) + } + } +} + +// captureStdout runs fn with os.Stdout redirected to a pipe and returns what was +// printed. The indexer reports fallbacks and collisions via fmt.Printf to stdout, +// so this is how a test sees whether the warning path was taken. +func captureStdout(t *testing.T, fn func()) string { + t.Helper() + orig := os.Stdout + r, w, err := os.Pipe() + if err != nil { + t.Fatalf("pipe: %v", err) + } + os.Stdout = w + done := make(chan string, 1) + go func() { + b, _ := io.ReadAll(r) + done <- string(b) + }() + fn() + w.Close() + os.Stdout = orig + return <-done +} + +// Re-indexing an already-indexed project must be clean: the pk makes the COPY +// idempotent (the selective clear empties this project's code-derived rows +// first, and no other project's rows share a pk), so neither the duplicate-key +// warning nor the COPY-unavailable fallback note should appear. +func TestReindexEmitsNoDuplicateKeyWarnings(t *testing.T) { + tmp := t.TempDir() + dir := filepath.Join(tmp, "src") + writeCommonProject(t, dir) + + store, err := OpenStore(filepath.Join(tmp, "knowledge.db")) + if err != nil { + t.Fatalf("OpenStore: %v", err) + } + defer store.Close() + + indexInto(t, store, "proj", dir) // first pass primes the graph + + out := captureStdout(t, func() { + indexInto(t, store, "proj", dir) // the re-index under test + }) + + for _, bad := range []string{"duplicated primary key", "COPY FROM JSON unavailable"} { + if strings.Contains(out, bad) { + t.Errorf("re-index emitted %q; output:\n%s", bad, out) + } + } +} + +// Observation has no project_id column, so an observation read that filters only +// on entity_id pulls in rows from every project sharing that (code-derived) +// entity id. A per-project search must return only that project's observations. +// This exercises the search observation-fetch path (batchGetObservations), which +// before the fix matched o.entity_id with no project scope. +func TestSearchObservationsDoNotBleedAcrossProjects(t *testing.T) { + const projA, projB = "obs-project-a", "obs-project-b" + + tmp := t.TempDir() + dirA := filepath.Join(tmp, "a") + dirB := filepath.Join(tmp, "b") + writeCommonProject(t, dirA) + writeCommonProject(t, dirB) + + store, err := OpenStore(filepath.Join(tmp, "knowledge.db")) + if err != nil { + t.Fatalf("OpenStore: %v", err) + } + defer store.Close() + + indexInto(t, store, projA, dirA) + indexInto(t, store, projB, dirB) + + // Hand-write an observation onto the SAME shared entity id in each project. + const idShared = "file:common.go" + if _, err := store.CreateObservation(idShared, "A-only note", projA); err != nil { + t.Fatalf("CreateObservation A: %v", err) + } + if _, err := store.CreateObservation(idShared, "B-only note", projB); err != nil { + t.Fatalf("CreateObservation B: %v", err) + } + + // A search scoped to project A, matching the shared entity by name, must + // surface only A's observation — never B's. + results, err := store.KeywordSearch(projA, "common", 10) + if err != nil { + t.Fatalf("KeywordSearch(A): %v", err) + } + var checked bool + for _, r := range results { + if r.Entity.ID != idShared { + continue + } + checked = true + for _, o := range r.Observations { + if o.Content == "B-only note" { + t.Errorf("project A search returned project B's observation %q (cross-project bleed)", o.Content) + } + } + var sawA bool + for _, o := range r.Observations { + if o.Content == "A-only note" { + sawA = true + } + } + if !sawA { + t.Errorf("project A search missing its own observation; got %+v", r.Observations) + } + } + if !checked { + t.Fatalf("search did not return the shared entity %q, so the bleed assertion never ran", idShared) + } +} diff --git a/src/kg/internal/knowledge/indexer_pdf.go b/src/kg/internal/knowledge/indexer_pdf.go index c9660bd..2e6de55 100644 --- a/src/kg/internal/knowledge/indexer_pdf.go +++ b/src/kg/internal/knowledge/indexer_pdf.go @@ -161,10 +161,11 @@ func (idx *Indexer) processPDFFile( continue } *observations = append(*observations, obsRecord{ - id: uuid.NewString(), - entityID: entityID, - content: chunk, - created: now, + id: uuid.NewString(), + entityID: entityID, + content: chunk, + created: now, + projectID: idx.projectID, }) } diff --git a/src/kg/internal/knowledge/log_app_indexer.go b/src/kg/internal/knowledge/log_app_indexer.go index ce47709..21551f8 100644 --- a/src/kg/internal/knowledge/log_app_indexer.go +++ b/src/kg/internal/knowledge/log_app_indexer.go @@ -190,10 +190,11 @@ func IndexApplicationLogs(store *Store, projectID, projectRoot string, scopeFilt obs := buildLogEntityObservations(logEntity, relPath) for _, content := range obs { newObs = append(newObs, obsRecord{ - id: uuid.New().String(), - entityID: entityID, - content: content, - created: now, + id: uuid.New().String(), + entityID: entityID, + content: content, + created: now, + projectID: projectID, }) } } diff --git a/src/kg/internal/knowledge/log_indexer.go b/src/kg/internal/knowledge/log_indexer.go index 6094895..06b54c3 100644 --- a/src/kg/internal/knowledge/log_indexer.go +++ b/src/kg/internal/knowledge/log_indexer.go @@ -189,6 +189,11 @@ type obsRecord struct { entityID string content string created time.Time + // projectID namespaces both the observation's own pk and the pk of the + // parent entity the HAS_OBSERVATION edge resolves against (see + // bulkLoadExecObservations). Every observation in one run shares it, but it + // is carried per-row to keep the bulk loader free of an out-of-band argument. + projectID string } // LogIndexStats holds counters returned by IndexExecutionLogs. @@ -263,10 +268,11 @@ func IndexExecutionLogs(store *Store, projectID, projectRoot string) (LogIndexSt }) for _, o := range buildExecObservations(parsed, meta, folderName) { newObs = append(newObs, obsRecord{ - id: uuid.New().String(), - entityID: entityID, - content: o, - created: now, + id: uuid.New().String(), + entityID: entityID, + content: o, + created: now, + projectID: projectID, }) } } @@ -308,6 +314,7 @@ func bulkLoadExecEntities(store *Store, entities []entityRecord) error { enc := json.NewEncoder(f) for _, e := range entities { if err := enc.Encode(map[string]string{ + "pk": PK(e.ProjectID, e.ID), "id": e.ID, "name": e.Name, "type": e.Type, @@ -322,17 +329,20 @@ func bulkLoadExecEntities(store *Store, entities []entityRecord) error { f.Close() result, err := store.Query(fmt.Sprintf( - `COPY Entity(id, name, type, project_id, created_at, updated_at) FROM '%s'`, path, + `COPY Entity(pk, id, name, type, project_id, created_at, updated_at) FROM '%s'`, path, )) if err != nil { - // Fall back to row-by-row if COPY FROM JSON is unavailable. + // Fall back to row-by-row MERGE (idempotent on pk) if COPY FROM JSON is + // unavailable. These entities were already filtered for prior existence + // by name, so MERGE only refreshes a genuine re-run rather than colliding. for _, e := range entities { r, err2 := store.QueryParams(` - CREATE (n:Entity { - id: $id, name: $name, type: $type, - project_id: $project_id, - created_at: $created_at, updated_at: $updated_at - })`, map[string]any{ + MERGE (n:Entity {pk: $pk}) + ON CREATE SET n.id = $id, n.name = $name, n.type = $type, + n.project_id = $project_id, + n.created_at = $created_at, n.updated_at = $updated_at + ON MATCH SET n.name = $name, n.type = $type, n.updated_at = $updated_at`, map[string]any{ + "pk": PK(e.ProjectID, e.ID), "id": e.ID, "name": e.Name, "type": e.Type, "project_id": e.ProjectID, "created_at": e.CreatedAt, "updated_at": e.UpdatedAt, @@ -366,6 +376,7 @@ func bulkLoadExecObservations(store *Store, obs []obsRecord) error { eenc := json.NewEncoder(ef) for _, o := range obs { if err := nenc.Encode(map[string]string{ + "pk": PK(o.projectID, o.id), "id": o.id, "entity_id": o.entityID, "content": o.content, @@ -375,9 +386,12 @@ func bulkLoadExecObservations(store *Store, obs []obsRecord) error { ef.Close() return err } + // Both endpoints of HAS_OBSERVATION resolve against node primary keys, + // which are now pk: the parent Entity's pk and this Observation's pk, + // both namespaced by the observation's project. if err := eenc.Encode(map[string]string{ - "from": o.entityID, - "to": o.id, + "from": PK(o.projectID, o.entityID), + "to": PK(o.projectID, o.id), }); err != nil { nf.Close() ef.Close() @@ -388,7 +402,7 @@ func bulkLoadExecObservations(store *Store, obs []obsRecord) error { ef.Close() r, err := store.Query(fmt.Sprintf( - `COPY Observation(id, entity_id, content, created_at) FROM '%s'`, nodesPath, + `COPY Observation(pk, id, entity_id, content, created_at) FROM '%s'`, nodesPath, )) if err != nil { return fmt.Errorf("COPY Observation: %w", err) diff --git a/src/kg/internal/knowledge/store_wrappers.go b/src/kg/internal/knowledge/store_wrappers.go index e9885f8..61ecee2 100644 --- a/src/kg/internal/knowledge/store_wrappers.go +++ b/src/kg/internal/knowledge/store_wrappers.go @@ -28,6 +28,12 @@ func OpenStoreReadOnly(dbPath string) (*kglib.Store, error) { return kglib.OpenStoreReadOnly(dbPath) } +// PK derives the project-namespaced primary key for an entity or observation. +// Re-exported from kglib so the indexer's bulk-load (COPY/CREATE) paths derive +// pk through the exact same helper the Store's own writes use — the NUL-join +// format must match byte-for-byte or endpoints and nodes would not resolve. +var PK = kglib.PK + // NewEmbedderFromEnv creates an embedder from environment variables var NewEmbedderFromEnv = kglib.NewEmbedderFromEnv diff --git a/src/kg/migrate.go b/src/kg/migrate.go index 99c79ea..ac90588 100644 --- a/src/kg/migrate.go +++ b/src/kg/migrate.go @@ -42,8 +42,16 @@ func archiveIfMismatch(dbPath string, out io.Writer) (string, error) { return "", nil } - fmt.Fprintf(out, "🔄 %s was written by kuzu %s; this build uses %s. Rebuilding.\n", - dbPath, stamp.KuzuVersion, kglib.KuzuVersion()) + // A mismatch is either a storage-format (Kuzu) bump or a kg schema bump; the + // message names whichever actually differs so a same-Kuzu schema change does + // not print the confusing "written by kuzu X; this build uses X". + if stamp.KuzuVersion != kglib.KuzuVersion() { + fmt.Fprintf(out, "🔄 %s was written by kuzu %s; this build uses %s. Rebuilding.\n", + dbPath, stamp.KuzuVersion, kglib.KuzuVersion()) + } else { + fmt.Fprintf(out, "🔄 %s uses kg schema v%d; this build needs v%d. Rebuilding.\n", + dbPath, stamp.SchemaVersion, kglib.CurrentSchemaVersion) + } aside, err := kglib.ArchiveDatabase(dbPath) if err != nil { diff --git a/src/kg/migrate_test.go b/src/kg/migrate_test.go index 13be0f9..587d1ab 100644 --- a/src/kg/migrate_test.go +++ b/src/kg/migrate_test.go @@ -25,6 +25,96 @@ func stampAs(t *testing.T, dbPath, kuzuVersion string) { } } +// stampSchema rewrites the sidecar to claim an older kg schema version while +// keeping this build's Kuzu version — the way an on-disk schema bump (e.g. the +// pk primary key) presents on a database written by an otherwise-current build. +func stampSchema(t *testing.T, dbPath string, schemaVersion int) { + t.Helper() + data, err := json.Marshal(kglib.FormatStamp{ + KuzuVersion: kglib.KuzuVersion(), + KGVersion: "test", + SchemaVersion: schemaVersion, + }) + if err != nil { + t.Fatalf("marshal stamp: %v", err) + } + if err := os.WriteFile(kglib.FormatStampPath(dbPath), data, 0600); err != nil { + t.Fatalf("write stamp: %v", err) + } +} + +// A kg schema-version bump (not a Kuzu bump) must also trigger archive + rebuild +// and carry hand-written knowledge across via journal replay. This is the +// migration path the pk primary key rides: Kuzu cannot ALTER … ADD PRIMARY KEY, +// so an older-schema database is rebuilt fresh and its journal replayed. +func TestSchemaVersionBumpRebuildsAndRestoresHandWrites(t *testing.T) { + dbPath := filepath.Join(t.TempDir(), "knowledge.db") + + store, err := knowledge.OpenStore(dbPath) + if err != nil { + t.Fatalf("OpenStore: %v", err) + } + store.EnableJournal() + entity, err := store.CreateEntity("retry-policy", "decision", "proj") + if err != nil { + t.Fatalf("CreateEntity: %v", err) + } + if _, err := store.CreateObservation(entity.ID, "three attempts, exponential backoff", "proj"); err != nil { + t.Fatalf("CreateObservation: %v", err) + } + store.Close() + + // Same Kuzu build, but an older schema than this build writes. + stampSchema(t, dbPath, kglib.CurrentSchemaVersion-1) + + status, _, err := kglib.CheckFormat(dbPath) + if err != nil { + t.Fatalf("CheckFormat: %v", err) + } + if status != kglib.FormatMismatch { + t.Fatalf("an older schema_version was not reported as a mismatch: got %v", status) + } + + var out bytes.Buffer + rebuilt, err := rebuildHandWritesOnly(dbPath, &out) + if err != nil { + t.Fatalf("rebuildHandWritesOnly: %v", err) + } + if !rebuilt { + t.Fatal("a database with an older kg schema was not rebuilt") + } + + fresh, err := knowledge.OpenStore(dbPath) + if err != nil { + t.Fatalf("open rebuilt store: %v", err) + } + defer fresh.Close() + + restored, err := fresh.GetEntityByNameAndType("retry-policy", "decision", "proj") + if err != nil { + t.Fatalf("GetEntityByNameAndType: %v", err) + } + if restored == nil { + t.Fatal("hand-written entity did not survive the schema rebuild") + } + observations, err := fresh.GetObservations(restored.ID, "proj") + if err != nil { + t.Fatalf("GetObservations: %v", err) + } + if len(observations) != 1 || observations[0].Content != "three attempts, exponential backoff" { + t.Errorf("observations = %+v, want the hand-written note", observations) + } + + // The rebuilt database is stamped at the current schema, so it is FormatOK. + status, _, err = kglib.CheckFormat(dbPath) + if err != nil { + t.Fatalf("CheckFormat after rebuild: %v", err) + } + if status != kglib.FormatOK { + t.Errorf("status after rebuild = %v, want %v", status, kglib.FormatOK) + } +} + // The whole point of Phase 4, end to end: hand-written knowledge survives a // storage-format change with no old binary involved and no user action. func TestEnsureFormatRebuildsAndRestoresHandWrites(t *testing.T) { diff --git a/src/kglib/embeddings.go b/src/kglib/embeddings.go index 17935a5..3c699ff 100644 --- a/src/kglib/embeddings.go +++ b/src/kglib/embeddings.go @@ -36,9 +36,11 @@ func (s *Store) BatchEmbed(ctx context.Context, projectID string, embedder Embed "results are matched by position, so a mismatched count cannot be used", len(embeddings), len(texts)) } - // Store embeddings + // Store embeddings. entity.ProjectID equals the projectID being embedded + // (GetUnembeddedEntities filters on it); passing it keeps SetEmbedding on + // this project's node and invalidates this project's index. for i, entity := range entities { - if err := s.SetEmbedding(entity.ID, embeddings[i]); err != nil { + if err := s.SetEmbedding(entity.ID, entity.ProjectID, embeddings[i]); err != nil { return fmt.Errorf("set embedding for entity %s: %w", entity.ID, err) } } @@ -156,17 +158,17 @@ func (s *Store) GetUnembeddedObservations(projectID string) ([]Observation, erro // SetEmbedding stores an embedding vector for an entity and invalidates the // in-memory HNSW index for the entity's project so that the next VectorSearch // call rebuilds the index with the new vector. -func (s *Store) SetEmbedding(entityID string, embedding []float32) error { - // Look up the project_id so we can invalidate the correct HNSW index. - projectID, err := s.entityProjectID(entityID) - if err != nil { - return fmt.Errorf("set embedding (lookup project): %w", err) - } - +// +// The entity is matched by its project-namespaced pk, not by id alone: under +// multi-project-per-DB a bare-id MATCH would set the embedding on every +// project's node sharing that id (and the old project_id lookup returned an +// arbitrary one of them, invalidating the wrong project's HNSW index). projectID +// is required and is the project whose index is invalidated. +func (s *Store) SetEmbedding(entityID, projectID string, embedding []float32) error { result, err := s.QueryParams(` - MATCH (e:Entity {id: $id}) + MATCH (e:Entity {pk: $pk}) SET e.embedding = $embedding - `, map[string]any{"id": entityID, "embedding": embedding}) + `, map[string]any{"pk": PK(projectID, entityID), "embedding": embedding}) if err != nil { return fmt.Errorf("set embedding: %w", err) } @@ -180,37 +182,6 @@ func (s *Store) SetEmbedding(entityID string, embedding []float32) error { return nil } -// entityProjectID returns the project_id for entityID, or "" if not found. -func (s *Store) entityProjectID(entityID string) (string, error) { - result, err := s.QueryParams(` - MATCH (e:Entity {id: $id}) - RETURN e.project_id - `, map[string]any{"id": entityID}) - if err != nil { - return "", fmt.Errorf("query entity project_id: %w", err) - } - defer result.Close() - - if result.HasNext() { - tuple, err := result.Next() - if err != nil { - return "", fmt.Errorf("next: %w", err) - } - row, err := tuple.GetAsSlice() - tuple.Close() - if err != nil { - return "", fmt.Errorf("get slice: %w", err) - } - if row[0] == nil { - return "", nil - } - if pid, ok := row[0].(string); ok { - return pid, nil - } - } - return "", nil -} - // SetObservationEmbedding stores an embedding vector for an observation func (s *Store) SetObservationEmbedding(observationID string, embedding []float32) error { result, err := s.QueryParams(` diff --git a/src/kglib/embeddings_test.go b/src/kglib/embeddings_test.go index c280cd4..c82d837 100644 --- a/src/kglib/embeddings_test.go +++ b/src/kglib/embeddings_test.go @@ -167,7 +167,7 @@ func TestSetEmbedding_HNSWInvalidation(t *testing.T) { for i := range embedding { embedding[i] = float32(i) * 0.1 } - if err := store.SetEmbedding(entity.ID, embedding); err != nil { + if err := store.SetEmbedding(entity.ID, projectID, embedding); err != nil { t.Fatalf("SetEmbedding: %v", err) } diff --git a/src/kglib/entity.go b/src/kglib/entity.go index 97f1013..104838d 100644 --- a/src/kglib/entity.go +++ b/src/kglib/entity.go @@ -53,10 +53,11 @@ func (s *Store) createEntityWithID(id, name, entityType, projectID string) (*Ent result, err := s.QueryParams(` CREATE (e:Entity { - id: $id, name: $name, type: $type, project_id: $project_id, + pk: $pk, id: $id, name: $name, type: $type, project_id: $project_id, created_at: $created_at, updated_at: $updated_at }) `, map[string]any{ + "pk": PK(projectID, id), "id": id, "name": name, "type": entityType, "project_id": projectID, "created_at": entity.CreatedAt, "updated_at": entity.UpdatedAt, }) @@ -120,9 +121,11 @@ func (s *Store) detectColumns() { if err != nil { return } - if len(row) > 1 && stringOrEmpty(row[1]) == "visibility" { + switch name := stringOrEmpty(row[1]); name { + case "visibility": s.hasVisibility = true - return + case "pk": + s.hasPK = true } } } @@ -157,6 +160,7 @@ func (s *Store) CreateEntity(name, entityType, projectID string) (*Entity, error result, err := s.QueryParams(` CREATE (e:Entity { + pk: $pk, id: $id, name: $name, type: $type, @@ -165,6 +169,7 @@ func (s *Store) CreateEntity(name, entityType, projectID string) (*Entity, error updated_at: $updated_at }) `, map[string]any{ + "pk": PK(projectID, entity.ID), "id": entity.ID, "name": name, "type": entityType, @@ -188,13 +193,30 @@ func (s *Store) CreateEntity(name, entityType, projectID string) (*Entity, error return entity, nil } -// GetEntity retrieves an entity by ID for a specific project +// GetEntity retrieves an entity by ID for a specific project. +// +// It matches on the project-namespaced pk, which is both the fast path (pk is +// the PRIMARY KEY) and the project-safe one: a bare id can collide across +// projects sharing a database, and resolving it here by id alone is exactly the +// cross-project confusion pk exists to prevent. func (s *Store) GetEntity(id, projectID string) (*Entity, error) { + // A pre-pk database — only reachable on a read-only open, since a read-write + // open refuses to proceed without the column — has no pk to match on, and + // Kuzu rejects a query that names a missing property outright. Fall back to + // the original (id, project_id) match so reads keep working on a legacy graph + // that has not yet been rebuilt. This branch is project-scoped and so still + // correct; it is merely not a primary-key lookup. + where := "e.pk = $pk" + params := map[string]any{"pk": PK(projectID, id)} + if !s.hasPK { + where = "e.id = $id AND e.project_id = $project_id" + params = map[string]any{"id": id, "project_id": projectID} + } result, err := s.QueryParams(` MATCH (e:Entity) - WHERE e.id = $id AND e.project_id = $project_id + WHERE `+where+` RETURN `+s.entityColumns()+` - `, map[string]any{"id": id, "project_id": projectID}) + `, params) if err != nil { return nil, fmt.Errorf("query entity: %w", err) } diff --git a/src/kglib/federated_test.go b/src/kglib/federated_test.go index 953cae7..675c6c3 100644 --- a/src/kglib/federated_test.go +++ b/src/kglib/federated_test.go @@ -37,10 +37,11 @@ func insertEntityWithID(t *testing.T, s *Store, id, name, entityType, projectID now := time.Now().UTC() result, err := s.QueryParams(` CREATE (e:Entity { - id: $id, name: $name, type: $type, project_id: $project_id, + pk: $pk, id: $id, name: $name, type: $type, project_id: $project_id, created_at: $created_at, updated_at: $updated_at }) `, map[string]any{ + "pk": PK(projectID, id), "id": id, "name": name, "type": entityType, "project_id": projectID, "created_at": now, "updated_at": now, }) diff --git a/src/kglib/format.go b/src/kglib/format.go index 6e25550..5ed3212 100644 --- a/src/kglib/format.go +++ b/src/kglib/format.go @@ -45,13 +45,31 @@ import ( // FormatStampFile is the sidecar naming the engine build that wrote a database. const formatStampSuffix = ".format.json" +// CurrentSchemaVersion is the kg knowledge-graph schema version this build +// writes. Unlike KGVersion (the build string, which changes every release and +// means nothing for compatibility), this is a monotonic integer bumped ONLY on +// a breaking on-disk schema change that Kuzu cannot migrate in place — which, +// because Kuzu has no ALTER … ADD PRIMARY KEY and rel tables are bound to their +// node tables, means a change to a node table's primary key or columns. A +// database stamped with a lower version (or an older stamp with no version +// field, which decodes as 0) is reported as FormatMismatch so index/migrate +// archive and rebuild it. +// +// 1 — original schema: Entity/Observation keyed on global `id`. +// 2 — project-namespaced `pk` PRIMARY KEY added; `id` demoted to a plain column. +const CurrentSchemaVersion = 2 + // FormatStamp is the content of that sidecar. type FormatStamp struct { // KuzuVersion is the pinned go-kuzu module version, e.g. "v0.11.3". KuzuVersion string `json:"kuzu_version"` // KGVersion is the kg build that wrote the stamp, for diagnostics only. - KGVersion string `json:"kg_version,omitempty"` - StampedAt time.Time `json:"stamped_at"` + KGVersion string `json:"kg_version,omitempty"` + // SchemaVersion is the kg schema version (see CurrentSchemaVersion). Absent + // in stamps written before this field existed, where it decodes as 0 and is + // therefore treated as older than any real schema. + SchemaVersion int `json:"schema_version,omitempty"` + StampedAt time.Time `json:"stamped_at"` } // FormatStatus is the verdict of comparing a database's stamp to this build. @@ -196,9 +214,10 @@ func ReadFormatStamp(dbPath string) (*FormatStamp, error) { // read as corrupt and send a perfectly good database down the rebuild path. func WriteFormatStamp(dbPath string, kgVersion string) error { stamp := FormatStamp{ - KuzuVersion: KuzuVersion(), - KGVersion: kgVersion, - StampedAt: time.Now().UTC(), + KuzuVersion: KuzuVersion(), + KGVersion: kgVersion, + SchemaVersion: CurrentSchemaVersion, + StampedAt: time.Now().UTC(), } data, err := json.Marshal(stamp) if err != nil { @@ -250,7 +269,12 @@ func CheckFormat(dbPath string) (FormatStatus, *FormatStamp, error) { if stamp == nil { return FormatUnstamped, nil, nil } - if stamp.KuzuVersion != KuzuVersion() { + // A storage-format change (Kuzu bump) and a kg schema change (e.g. the pk + // primary key) both make the running build unable to use the database as-is; + // either one is a mismatch that index/migrate repair by archiving and + // rebuilding. SchemaVersion is absent (0) in stamps written before it + // existed, which is correctly older than any real schema. + if stamp.KuzuVersion != KuzuVersion() || stamp.SchemaVersion < CurrentSchemaVersion { return FormatMismatch, stamp, nil } return FormatOK, stamp, nil diff --git a/src/kglib/journal_test.go b/src/kglib/journal_test.go index d76794c..45041ed 100644 --- a/src/kglib/journal_test.go +++ b/src/kglib/journal_test.go @@ -633,9 +633,10 @@ func TestJournalOmitsUnstableIDs(t *testing.T) { func mustCreateWithID(t *testing.T, store *Store, id, name, entityType, projectID string) { t.Helper() result, err := store.QueryParams(` - CREATE (e:Entity {id: $id, name: $name, type: $type, project_id: $project_id, + CREATE (e:Entity {pk: $pk, id: $id, name: $name, type: $type, project_id: $project_id, created_at: $ts, updated_at: $ts}) `, map[string]any{ + "pk": PK(projectID, id), "id": id, "name": name, "type": entityType, "project_id": projectID, "ts": time.Now().UTC(), }) diff --git a/src/kglib/null_columns_test.go b/src/kglib/null_columns_test.go index 868b5cd..bbcc90b 100644 --- a/src/kglib/null_columns_test.go +++ b/src/kglib/null_columns_test.go @@ -211,7 +211,7 @@ func TestBatchEmbed_RejectsShortObservationResponse(t *testing.T) { } // Give the single entity its embedding so BatchEmbed reaches the observation // stage, then under-deliver there. - if err := store.SetEmbedding(entity.ID, make([]float32, embeddingDim)); err != nil { + if err := store.SetEmbedding(entity.ID, projectID, make([]float32, embeddingDim)); err != nil { t.Fatalf("SetEmbedding: %v", err) } diff --git a/src/kglib/observation.go b/src/kglib/observation.go index 3514f7b..83ad3e7 100644 --- a/src/kglib/observation.go +++ b/src/kglib/observation.go @@ -31,8 +31,9 @@ func (s *Store) CreateObservation(entityID, content, projectID string) (*Observa // statement there is no risk of an orphaned Observation node if the edge // creation were to fail. result, err := s.QueryParams(` - MATCH (e:Entity {id: $entity_id, project_id: $project_id}) + MATCH (e:Entity {pk: $entity_pk}) CREATE (o:Observation { + pk: $pk, id: $id, entity_id: $entity_id, content: $content, @@ -40,8 +41,9 @@ func (s *Store) CreateObservation(entityID, content, projectID string) (*Observa }) CREATE (e)-[:HAS_OBSERVATION]->(o) `, map[string]any{ + "entity_pk": PK(projectID, entityID), "entity_id": entityID, - "project_id": projectID, + "pk": PK(projectID, obs.ID), "id": obs.ID, "content": content, "created_at": obs.CreatedAt, diff --git a/src/kglib/pk.go b/src/kglib/pk.go new file mode 100644 index 0000000..f477403 --- /dev/null +++ b/src/kglib/pk.go @@ -0,0 +1,38 @@ +package kglib + +// Project-namespaced primary keys. +// +// Entity and Observation node tables carry a single PRIMARY KEY column, `pk`, +// that namespaces the human-facing `id` by the project it belongs to. The +// `id` column stays exactly what it always was — "file:…", "function:…:F", +// or a UUID — so search, CLI display, and the journal's stable-id hint are +// unchanged; `pk` exists only to make the primary key unique across projects +// that share one database file. +// +// Why this is necessary: Kuzu v0.11.3 node tables support exactly one PRIMARY +// KEY column, so a composite (project_id, id) key is not expressible. Before +// `pk`, the key was the global `id` alone, but every clear/delete and nearly +// every read is scoped by project_id — so a surviving row from a different +// project_id with the same `id` would collide with the current run's entity and +// be silently dropped. Namespacing the key fixes that at the source. +// +// Separator choice: the ADR specified NUL, but go-kuzu marshals strings across +// cgo as C strings and truncates them at the first NUL byte — a NUL-joined pk +// would collapse to just the project_id in storage, re-introducing the very +// collisions this fixes (verified: every pk reduced to its project prefix). The +// Unit Separator (0x1F) is used instead: it is a non-printable control byte that +// survives the round trip and that neither a project_id (a repo directory base +// name) nor a code-derived id ("::") nor a UUID ever contains. +const pkSeparator = "\x1f" + +// PK derives the project-namespaced primary key for an entity or observation +// from its project_id and its local id. +// +// The Unit Separator byte is collision-free and unambiguously reversible given +// the invariant above (neither operand contains it), which is what lets a single +// `pk` column stand in for the composite (project_id, id) key Kuzu cannot +// express. Every write derives `pk` here, never ad hoc, so the format lives in +// exactly one place. +func PK(projectID, id string) string { + return projectID + pkSeparator + id +} diff --git a/src/kglib/relation.go b/src/kglib/relation.go index 9d4b3d6..4043c08 100644 --- a/src/kglib/relation.go +++ b/src/kglib/relation.go @@ -24,15 +24,21 @@ func (s *Store) CreateRelation(fromID, toID, relType, projectID string) error { // relType is a relationship label (not a value) and cannot be parameterised // in Cypher. It is validated against AllowedRelTypes above before use. + // + // Endpoints are matched on pk, never on bare id. A bare-id MATCH is + // project-blind: with two projects in one database it can bind an endpoint to + // an identically-id'd node in the OTHER project, wiring the edge to the wrong + // graph. pk namespaces id by project, so this both targets this project's + // nodes and uses the indexed primary key. query := fmt.Sprintf(` MATCH (from:Entity), (to:Entity) - WHERE from.id = $from_id AND to.id = $to_id + WHERE from.pk = $from_pk AND to.pk = $to_pk CREATE (from)-[:%s]->(to) `, relType) result, err := s.QueryParams(query, map[string]any{ - "from_id": fromID, - "to_id": toID, + "from_pk": PK(projectID, fromID), + "to_pk": PK(projectID, toID), }) if err != nil { return fmt.Errorf("create relation: %w", err) diff --git a/src/kglib/schema.go b/src/kglib/schema.go index e11e58b..4619f3a 100644 --- a/src/kglib/schema.go +++ b/src/kglib/schema.go @@ -40,9 +40,15 @@ func (s *Store) initSchema(cfg *SchemaConfig) error { // Fixed node tables and the HAS_OBSERVATION edge staticStatements := []string{ - // Entity node table + // Entity node table. + // + // pk is the project-namespaced primary key (see pk.go); id stays the + // human-facing/journal-stable value ("file:…", a UUID, …) and is a plain + // indexed-by-nothing attribute. Kuzu allows only one PRIMARY KEY column, + // so the composite (project_id, id) key is carried by pk instead. `CREATE NODE TABLE IF NOT EXISTS Entity( - id STRING PRIMARY KEY, + pk STRING PRIMARY KEY, + id STRING, name STRING, type STRING, project_id STRING, @@ -52,9 +58,11 @@ func (s *Store) initSchema(cfg *SchemaConfig) error { visibility STRING )`, - // Observation node table + // Observation node table. pk namespaces id by project_id exactly as on + // Entity; id stays the UUID the rest of the code and the journal use. `CREATE NODE TABLE IF NOT EXISTS Observation( - id STRING PRIMARY KEY, + pk STRING PRIMARY KEY, + id STRING, entity_id STRING, content STRING, created_at TIMESTAMP, diff --git a/src/kglib/search.go b/src/kglib/search.go index 151397f..9431335 100644 --- a/src/kglib/search.go +++ b/src/kglib/search.go @@ -178,7 +178,7 @@ func (s *Store) KeywordSearchFiltered(projectID, query string, limit int, filter for i, e := range entities { entityIDs[i] = e.ID } - obsMap, err := s.batchGetObservations(entityIDs, 3) + obsMap, err := s.batchGetObservations(entityIDs, projectID, 3) if err != nil { // Degrade gracefully — empty observations rather than failing the search. obsMap = map[string][]*Observation{} @@ -212,16 +212,22 @@ func (s *Store) GetTopObservations(entityID, projectID string, limit int) ([]*Ob // Indexed rows share a timestamp routinely: the indexer stamps one `now` // per file, so every symbol in a file ties. // + // Scoped by project through the parent entity, not by o.entity_id alone: + // Observation has no project_id column, so matching the edge from an Entity + // filtered on project_id is what keeps a project-A search from pulling in a + // project-B observation that shares the entity_id. GetObservations takes the + // same route. + // // LIMIT takes an integer expression; using %d (not user input) is safe here. query := fmt.Sprintf(` - MATCH (o:Observation) - WHERE o.entity_id = $entity_id + MATCH (e:Entity)-[:HAS_OBSERVATION]->(o:Observation) + WHERE e.project_id = $project_id AND o.entity_id = $entity_id RETURN o.id, o.entity_id, o.content, o.created_at ORDER BY o.created_at DESC LIMIT %d `, limit) - result, err := s.QueryParams(query, map[string]any{"entity_id": entityID}) + result, err := s.QueryParams(query, map[string]any{"entity_id": entityID, "project_id": projectID}) if err != nil { return nil, fmt.Errorf("query observations: %w", err) } @@ -265,7 +271,12 @@ func (s *Store) GetTopObservations(entityID, projectID string, limit int) ([]*Ob // in entityIDs using a single Cypher query, eliminating N+1 round-trips. // It returns a map keyed by entity ID; entities with no observations are absent. // limit is applied per entity: only the newest `limit` rows are kept. -func (s *Store) batchGetObservations(entityIDs []string, limit int) (map[string][]*Observation, error) { +// +// projectID scopes the read: Observation carries no project_id, so under +// multi-project-per-DB two projects sharing an entity_id would otherwise cross- +// contaminate each other's search results. Matching the HAS_OBSERVATION edge +// from an Entity filtered on project_id confines the result to this project. +func (s *Store) batchGetObservations(entityIDs []string, projectID string, limit int) (map[string][]*Observation, error) { if len(entityIDs) == 0 { return map[string][]*Observation{}, nil } @@ -281,13 +292,13 @@ func (s *Store) batchGetObservations(entityIDs []string, limit int) (map[string] // counter instead. We fetch all rows ordered by (entity_id, created_at DESC) // and apply the per-entity limit in Go — this is still a single round-trip. query := ` - MATCH (o:Observation) - WHERE o.entity_id IN $entity_ids + MATCH (e:Entity)-[:HAS_OBSERVATION]->(o:Observation) + WHERE e.project_id = $project_id AND o.entity_id IN $entity_ids RETURN o.id, o.entity_id, o.content, o.created_at ORDER BY o.entity_id, o.created_at DESC ` - result, err := s.QueryParams(query, map[string]any{"entity_ids": ids}) + result, err := s.QueryParams(query, map[string]any{"entity_ids": ids, "project_id": projectID}) if err != nil { return nil, fmt.Errorf("batch query observations: %w", err) } @@ -379,7 +390,7 @@ func (s *Store) VectorSearch(projectID string, queryEmbedding []float32, limit i for i, c := range candidates { entityIDs[i] = c.entity.ID } - obsMap, err := s.batchGetObservations(entityIDs, 3) + obsMap, err := s.batchGetObservations(entityIDs, projectID, 3) if err != nil { obsMap = map[string][]*Observation{} } diff --git a/src/kglib/search_test.go b/src/kglib/search_test.go index 7ddd0f2..23f40b7 100644 --- a/src/kglib/search_test.go +++ b/src/kglib/search_test.go @@ -255,7 +255,7 @@ func TestBatchGetObservations(t *testing.T) { } t.Run("returns observations for multiple entities in one call", func(t *testing.T) { - obsMap, err := store.batchGetObservations([]string{entity1.ID, entity2.ID}, 3) + obsMap, err := store.batchGetObservations([]string{entity1.ID, entity2.ID}, projectID, 3) if err != nil { t.Fatalf("batchGetObservations failed: %v", err) } @@ -276,7 +276,7 @@ func TestBatchGetObservations(t *testing.T) { }) t.Run("empty entity list returns empty map", func(t *testing.T) { - obsMap, err := store.batchGetObservations([]string{}, 3) + obsMap, err := store.batchGetObservations([]string{}, projectID, 3) if err != nil { t.Fatalf("batchGetObservations with empty IDs failed: %v", err) } @@ -290,7 +290,7 @@ func TestBatchGetObservations(t *testing.T) { if err != nil { t.Fatalf("CreateEntity failed: %v", err) } - obsMap, err := store.batchGetObservations([]string{entityEmpty.ID}, 3) + obsMap, err := store.batchGetObservations([]string{entityEmpty.ID}, projectID, 3) if err != nil { t.Fatalf("batchGetObservations failed: %v", err) } @@ -350,10 +350,10 @@ func TestVectorSearch(t *testing.T) { query := make([]float32, dim) query[0] = 1 - if err := store.SetEmbedding(e1.ID, e1Emb); err != nil { + if err := store.SetEmbedding(e1.ID, pid, e1Emb); err != nil { t.Fatalf("SetEmbedding failed: %v", err) } - if err := store.SetEmbedding(e2.ID, e2Emb); err != nil { + if err := store.SetEmbedding(e2.ID, pid, e2Emb); err != nil { t.Fatalf("SetEmbedding failed: %v", err) } @@ -391,7 +391,7 @@ func TestVectorSearch(t *testing.T) { } emb := make([]float32, dim) emb[0] = float32(i) + 1 - if err := store.SetEmbedding(e.ID, emb); err != nil { + if err := store.SetEmbedding(e.ID, pid, emb); err != nil { t.Fatalf("SetEmbedding failed: %v", err) } } @@ -754,7 +754,7 @@ func TestHNSWScalability(t *testing.T) { raw[j] = float32(rng.Float64()*2 - 1) } emb := normF32(raw) - if err := store.SetEmbedding(ent.ID, emb); err != nil { + if err := store.SetEmbedding(ent.ID, projectID, emb); err != nil { t.Fatalf("SetEmbedding[%d] failed: %v", i, err) } } diff --git a/src/kglib/store.go b/src/kglib/store.go index 534ec10..1eeaed5 100644 --- a/src/kglib/store.go +++ b/src/kglib/store.go @@ -33,6 +33,7 @@ type Store struct { allowedRelTypes []string // configured relation types for validation journal *Journal // hand-write journal; nil unless EnableJournal was called hasVisibility bool // Entity.visibility exists in this database (see entityColumns) + hasPK bool // Entity.pk (project-namespaced primary key) exists; false on pre-pk databases } // OpenStore opens or creates a Kuzu database in read-write mode with the given schema configuration. @@ -104,22 +105,41 @@ func openStoreWithConfig(dbPath string, readOnly bool, cfg *SchemaConfig) (*Stor store.Close() return nil, fmt.Errorf("initialize schema: %w", err) } - - // Record which engine build wrote this database. Reaching this point - // proves the running engine can read the format, so the stamp is - // truthful by construction. It is a sidecar rather than a row in the - // database because its whole job is to be readable when the database is - // not. A failure to stamp is not fatal — the database is fine, it just - // falls back to the unstamped path on a later version check. - _ = WriteFormatStamp(dbPath, buildVersion) } // Which optional columns this database actually has. After initSchema on a - // write open, so a freshly-migrated database is detected correctly; and on + // write open, so a freshly-created database is detected correctly; and on // read-only opens too, which never migrate and so may be reading a graph - // written before a column existed. + // written before a column existed. Must run before the pk gate and the + // stamp below, which both depend on hasPK. store.detectColumns() + if !readOnly { + // `CREATE NODE TABLE IF NOT EXISTS` cannot add the pk PRIMARY KEY column + // to an Entity table that predates it, and Kuzu has no ALTER … ADD + // PRIMARY KEY, so a pre-pk database opened read-write would otherwise let + // every pk-referencing write fail with an opaque "Cannot find property + // pk". Refuse clearly instead and point at the rebuild path. `kg index` + // and `kg migrate` archive + rebuild such a database before opening it + // (CheckFormat reports FormatMismatch on its lower schema_version), so + // they never reach this; a bare `kg add` against an un-rebuilt legacy + // database is what this catches. + if !store.hasPK { + store.Close() + return nil, fmt.Errorf( + "knowledge graph database at %s predates the project-namespaced primary key "+ + "and must be rebuilt: run 'kg index' (or 'kg migrate' to rebuild every graph)", dbPath) + } + + // Record which engine build and schema version wrote this database. + // Reaching this point proves the running engine can read the format, so + // the stamp is truthful by construction. It is a sidecar rather than a + // row in the database because its whole job is to be readable when the + // database is not. A failure to stamp is not fatal — the database is + // fine, it just falls back to the unstamped path on a later version check. + _ = WriteFormatStamp(dbPath, buildVersion) + } + return store, nil } @@ -153,10 +173,16 @@ func openDiagnostic(dbPath string, readOnly bool, err error) error { "(create one with 'kg index'): %w", dbPath, err) } case FormatMismatch: - return fmt.Errorf("knowledge graph database at %s was written by kuzu %s "+ - "but this build uses %s; it needs rebuilding "+ + if stamp.KuzuVersion != KuzuVersion() { + return fmt.Errorf("knowledge graph database at %s was written by kuzu %s "+ + "but this build uses %s; it needs rebuilding "+ + "(run 'kg index' to rebuild it, or 'kg migrate' to rebuild every graph): %w", + dbPath, stamp.KuzuVersion, KuzuVersion(), err) + } + return fmt.Errorf("knowledge graph database at %s uses kg schema v%d "+ + "but this build needs v%d; it needs rebuilding "+ "(run 'kg index' to rebuild it, or 'kg migrate' to rebuild every graph): %w", - dbPath, stamp.KuzuVersion, KuzuVersion(), err) + dbPath, stamp.SchemaVersion, CurrentSchemaVersion, err) } } diff --git a/src/kglib/visibility_test.go b/src/kglib/visibility_test.go index 13d6c45..9ff4a19 100644 --- a/src/kglib/visibility_test.go +++ b/src/kglib/visibility_test.go @@ -47,10 +47,11 @@ func TestVisibilityRoundTrips(t *testing.T) { // Write directly, the way the indexer's bulk path does. result, err := store.QueryParams(` CREATE (e:Entity { - id: $id, name: $name, type: $type, project_id: $project_id, + pk: $pk, id: $id, name: $name, type: $type, project_id: $project_id, created_at: $ts, updated_at: $ts, visibility: $visibility }) `, map[string]any{ + "pk": PK("p", "function:main.go:helper"), "id": "function:main.go:helper", "name": "helper", "type": "function", "project_id": "p", "ts": time.Now().UTC(), "visibility": VisibilityPrivate, }) From 20b5bdd2b0861e46a98f57fbeb469f5d055ef920 Mon Sep 17 00:00:00 2001 From: Bryan Woodruff Date: Tue, 13 Oct 2026 12:21:33 +0100 Subject: [PATCH 2/7] fix(kg): project-qualify relation to-endpoint in Get/Delete/TraverseRelations MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Addresses PR #17 reviewer Major: GetRelations, DeleteRelation, and TraverseRelations pinned only the `from` endpoint by project_id, leaving the `to`/neighbour match project-blind — the same pattern CreateRelation was fixed to avoid. Currently masked (no write path can create a cross-project edge), but untested and a latent reintroduction of the cross-project bleed the moment any future path does. Add `AND to.project_id = $project_id` to all three (project_id column exists on every schema version, so no !hasPK branch; return contract stays to.id, never pk). New regression test TestRelationReadsAndDeletesAreProjectScopedOnBothEnds constructs a cross-project edge via raw Cypher and asserts Get/Traverse never cross and Delete scoped to one project leaves the other's edge intact (verified fail-before/pass-after). Co-Authored-By: Claude --- .../knowledge/indexer_crossproject_test.go | 133 ++++++++++++++++++ src/kglib/relation.go | 16 ++- 2 files changed, 148 insertions(+), 1 deletion(-) diff --git a/src/kg/internal/knowledge/indexer_crossproject_test.go b/src/kg/internal/knowledge/indexer_crossproject_test.go index f253c9f..f68d9e0 100644 --- a/src/kg/internal/knowledge/indexer_crossproject_test.go +++ b/src/kg/internal/knowledge/indexer_crossproject_test.go @@ -1,6 +1,7 @@ package knowledge import ( + "fmt" "io" "os" "path/filepath" @@ -228,3 +229,135 @@ func TestSearchObservationsDoNotBleedAcrossProjects(t *testing.T) { t.Fatalf("search did not return the shared entity %q, so the bleed assertion never ran", idShared) } } + +// rawInsertEntity writes an entity directly, the way an indexer bulk path does, +// so a test can place a project-B-only neighbour with a chosen id. +func rawInsertEntity(t *testing.T, store *Store, projectID, id, name, typ string) { + t.Helper() + r, err := store.QueryParams(` + CREATE (e:Entity {pk: $pk, id: $id, name: $name, type: $type, project_id: $project_id, + created_at: timestamp('2020-01-01'), updated_at: timestamp('2020-01-01')}) + `, map[string]any{"pk": PK(projectID, id), "id": id, "name": name, "type": typ, "project_id": projectID}) + if err != nil { + t.Fatalf("rawInsertEntity %s/%s: %v", projectID, id, err) + } + r.Close() +} + +// rawCreateCrossEdge creates an edge between two nodes addressed by pk, with no +// project guard — simulating exactly the future/legacy buggy write path the +// reviewer warns about: a cross-project edge that the project-scoped reads must +// refuse to surface or delete. +func rawCreateCrossEdge(t *testing.T, store *Store, fromProject, fromID, toProject, toID, relType string) { + t.Helper() + r, err := store.QueryParams(fmt.Sprintf(` + MATCH (from:Entity {pk: $from_pk}), (to:Entity {pk: $to_pk}) + CREATE (from)-[:%s]->(to) + `, relType), map[string]any{"from_pk": PK(fromProject, fromID), "to_pk": PK(toProject, toID)}) + if err != nil { + t.Fatalf("rawCreateCrossEdge: %v", err) + } + r.Close() +} + +// countCrossEdge returns how many relType edges run from fromProject's fromID to +// toProject's toID, addressed by pk so it sees a cross-project edge the scoped +// APIs hide. +func countCrossEdge(t *testing.T, store *Store, fromProject, fromID, toProject, toID, relType string) int { + t.Helper() + r, err := store.QueryParams(fmt.Sprintf(` + MATCH (from:Entity {pk: $from_pk})-[e:%s]->(to:Entity {pk: $to_pk}) + RETURN count(e) + `, relType), map[string]any{"from_pk": PK(fromProject, fromID), "to_pk": PK(toProject, toID)}) + if err != nil { + t.Fatalf("countCrossEdge: %v", err) + } + defer r.Close() + if !r.HasNext() { + return 0 + } + row, err := r.Next() + if err != nil { + t.Fatalf("countCrossEdge next: %v", err) + } + v, _ := row.GetValue(0) + n, _ := v.(int64) + return int(n) +} + +// The read/delete siblings of CreateRelation (GetRelations, TraverseRelations, +// DeleteRelation) pinned only `from` by project_id, leaving the `to`/neighbour +// endpoint project-blind. That is masked while every edge is same-project, but a +// single cross-project edge (constructed here directly) must not leak through a +// project-scoped read, nor be deleted by a project-scoped delete. +func TestRelationReadsAndDeletesAreProjectScopedOnBothEnds(t *testing.T) { + const projA, projB = "rel-project-a", "rel-project-b" + + tmp := t.TempDir() + dirA := filepath.Join(tmp, "a") + dirB := filepath.Join(tmp, "b") + writeCommonProject(t, dirA) + writeCommonProject(t, dirB) + + store, err := OpenStore(filepath.Join(tmp, "knowledge.db")) + if err != nil { + t.Fatalf("OpenStore: %v", err) + } + defer store.Close() + + indexInto(t, store, projA, dirA) // both get file:common.go and import:logging + indexInto(t, store, projB, dirB) + + const shared = "file:common.go" + + // A project-B-only neighbour, reachable only by a cross-project edge from A. + rawInsertEntity(t, store, projB, "topic:bneighbor", "bneighbor", "topic") + rawCreateCrossEdge(t, store, projA, shared, projB, "topic:bneighbor", RelRelatesTo) + + // --- GetRelations: must not surface A→B.bneighbor ------------------------ + relsA, err := store.GetRelations(shared, projA) + if err != nil { + t.Fatalf("GetRelations(A): %v", err) + } + for _, r := range relsA { + if r.ToID == "topic:bneighbor" { + t.Errorf("GetRelations(A) returned project B's neighbour %q (project-blind `to`)", r.ToID) + } + } + + // --- TraverseRelations: must not cross into B --------------------------- + neighbours, err := store.TraverseRelations(shared, RelRelatesTo, projA) + if err != nil { + t.Fatalf("TraverseRelations(A): %v", err) + } + for _, n := range neighbours { + if n.ProjectID != projA { + t.Errorf("TraverseRelations(A) crossed into project %q via neighbour %q", n.ProjectID, n.ID) + } + } + + // --- DeleteRelation: a project-A delete must not remove an edge whose far + // end is project B's node. Give A both a same-project edge and a cross edge + // to B's import:logging (same id in both projects), then delete in A. ------ + if err := store.CreateRelation(shared, "import:logging", RelDependsOn, projA); err != nil { + t.Fatalf("CreateRelation A→A: %v", err) + } + rawCreateCrossEdge(t, store, projA, shared, projB, "import:logging", RelDependsOn) + if before := countCrossEdge(t, store, projA, shared, projB, "import:logging", RelDependsOn); before != 1 { + t.Fatalf("setup: expected 1 cross edge before delete, got %d", before) + } + + if err := store.DeleteRelation(shared, "import:logging", RelDependsOn, projA); err != nil { + t.Fatalf("DeleteRelation(A): %v", err) + } + + // The cross edge terminates at project B's node, so a project-A delete must + // leave it intact (pre-fix it was swept because `to` was project-blind). + if after := countCrossEdge(t, store, projA, shared, projB, "import:logging", RelDependsOn); after != 1 { + t.Errorf("DeleteRelation(A) touched the edge into project B's node: cross edges after = %d, want 1", after) + } + // A's own same-project edge was the intended target and should be gone. + if n := countCrossEdge(t, store, projA, shared, projA, "import:logging", RelDependsOn); n != 0 { + t.Errorf("DeleteRelation(A) did not remove A's own edge: count = %d, want 0", n) + } +} diff --git a/src/kglib/relation.go b/src/kglib/relation.go index 4043c08..87884b4 100644 --- a/src/kglib/relation.go +++ b/src/kglib/relation.go @@ -66,9 +66,17 @@ func (s *Store) GetRelations(entityID, projectID string) ([]*Relation, error) { return nil, err } + // Both endpoints are pinned to this project, not just `from`. An edge whose + // far end belongs to another project (only possible from a legacy + // cross-project edge predating the pk fix) must not be returned with a + // neighbour this project cannot see — the same project-blind-endpoint bug + // CreateRelation was fixed for. project_id is used rather than pk because the + // surrounding match already keys `from` by id+project_id and that pattern is + // legacy-safe; the returned id stays the local display id. result, err := s.QueryParams(` MATCH (from:Entity)-[r]->(to:Entity) WHERE from.id = $entity_id AND from.project_id = $project_id + AND to.project_id = $project_id RETURN from.id, to.id, label(r) `, map[string]any{"entity_id": entityID, "project_id": projectID}) if err != nil { @@ -119,9 +127,12 @@ func (s *Store) DeleteRelation(fromID, toID, relType, projectID string) error { } // relType is a relationship label validated against AllowedRelTypes above. + // Both endpoints are pinned to this project so a project-A delete can never + // reach into project B's identically-id'd edge (see GetRelations). query := fmt.Sprintf(` MATCH (from:Entity)-[r:%s]->(to:Entity) - WHERE from.id = $from_id AND to.id = $to_id AND from.project_id = $project_id + WHERE from.id = $from_id AND to.id = $to_id + AND from.project_id = $project_id AND to.project_id = $project_id DELETE r `, relType) @@ -161,9 +172,12 @@ func (s *Store) TraverseRelations(entityID, relType, projectID string) ([]*Entit } // relType is a relationship label validated against AllowedRelTypes above. + // The neighbour is pinned to this project so a traversal cannot cross into + // another project's graph via a shared entity id (see GetRelations). query := fmt.Sprintf(` MATCH (from:Entity)-[:%s]->(to:Entity) WHERE from.id = $entity_id AND from.project_id = $project_id + AND to.project_id = $project_id RETURN to.id, to.name, to.type, to.project_id, to.created_at, to.updated_at `, relType) From caa19b2a4291172c097cec4448cc034df61fa3f6 Mon Sep 17 00:00:00 2001 From: Bryan Woodruff Date: Tue, 13 Oct 2026 14:15:28 +0100 Subject: [PATCH 3/7] fix(kg): project-scope observation embeddings; rebuild unstamped pre-pk DBs MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Addresses the two Major findings from PR #17's second review round. 1. SetObservationEmbedding was project-blind (`MATCH (o:Observation {id:$id})`), the one write path still on bare-id matching. Thread projectID through from BatchEmbed (which already scopes the batch via GetUnembeddedObservations) and match on the project-namespaced pk, consistent with SetEmbedding. Collision risk was low (observation ids are UUIDs) but the inconsistency was real and untested. 2. An unstamped database that also predates the pk column had no recovery path: CheckFormat returns FormatUnstamped (not FormatMismatch), so archiveIfMismatch did nothing, then the read-write open hit the !hasPK gate and hard-errored pointing at `kg index`/`kg migrate` — but index had just failed and migrate skipped unstamped DBs too. Both documented remedies dead-ended, contradicting migrate.go's "never an error, never manual recovery" promise. Fix: detect the unstamped+missing-pk case via a read-only open (databaseMissingPK / Store.HasPK) and route it through the existing archive+rebuild+journal-replay machinery, in both archiveIfMismatch and the migrate scope loop. An unstamped-but-current DB (has pk) is still left alone and simply stamped on next write. Adds TestUnstampedPrePKDatabaseRebuildsAndRestoresHandWrites. Co-Authored-By: Claude --- src/kg/migrate.go | 88 ++++++++++++++++++++------ src/kg/migrate_test.go | 134 ++++++++++++++++++++++++++++++++++++++++ src/kglib/embeddings.go | 21 +++++-- src/kglib/store.go | 7 +++ 4 files changed, 225 insertions(+), 25 deletions(-) diff --git a/src/kg/migrate.go b/src/kg/migrate.go index ac90588..0f2551f 100644 --- a/src/kg/migrate.go +++ b/src/kg/migrate.go @@ -25,33 +25,58 @@ import ( // this code — replay reads the journal, not the database — but a rebuild that // turns out worse than what it replaced should be recoverable. -// archiveIfMismatch moves a database aside when this build cannot read its -// storage format, returning the path it was archived to (empty when nothing was -// done). The check happens before any open, because a database in the wrong -// format cannot be opened to be asked about itself. +// archiveIfMismatch moves a database aside when this build cannot read or use +// its on-disk shape, returning the path it was archived to (empty when nothing +// was done). The stamp check happens before any open, because a database in the +// wrong storage format cannot be opened to be asked about itself. // -// An unstamped database — one written before stamping shipped — is left alone. -// There is no evidence against it, and rebuilding every pre-existing graph on -// first run would be precisely the forced breakage this exists to avoid. +// An unstamped database — one written before stamping shipped — is normally left +// alone: there is no evidence against it, and rebuilding every pre-existing +// graph on first run would be the forced breakage this exists to avoid. The one +// exception is an unstamped database that ALSO predates the pk column: it cannot +// be opened read-write (the pk gate in OpenStore hard-errors) and neither `kg +// index` nor `kg migrate` would otherwise repair it, so it is a dead end unless +// it is archived and rebuilt here. That case is detected by a read-only open — +// a pre-pk schema is still readable — so a current unstamped database (which has +// pk) is never falsely archived; it simply gets stamped on the next write open. func archiveIfMismatch(dbPath string, out io.Writer) (string, error) { status, stamp, err := kglib.CheckFormat(dbPath) if err != nil { return "", err } - if status != kglib.FormatMismatch { + + var reason string + switch status { + case kglib.FormatMismatch: + // A mismatch is either a storage-format (Kuzu) bump or a kg schema bump; + // the message names whichever actually differs so a same-Kuzu schema + // change does not print the confusing "written by kuzu X; this build uses X". + if stamp.KuzuVersion != kglib.KuzuVersion() { + reason = fmt.Sprintf("was written by kuzu %s; this build uses %s", + stamp.KuzuVersion, kglib.KuzuVersion()) + } else { + reason = fmt.Sprintf("uses kg schema v%d; this build needs v%d", + stamp.SchemaVersion, kglib.CurrentSchemaVersion) + } + case kglib.FormatUnstamped: + missing, err := databaseMissingPK(dbPath) + if err != nil { + // Cannot introspect the columns (e.g. the database will not even open + // read-only). Fall back to the historical leave-alone behavior rather + // than archiving on a guess. + return "", nil + } + if !missing { + // Unstamped but already current — leave it alone; the next write open + // stamps it. + return "", nil + } + reason = "is unstamped and predates the project-namespaced primary key" + default: return "", nil } - // A mismatch is either a storage-format (Kuzu) bump or a kg schema bump; the - // message names whichever actually differs so a same-Kuzu schema change does - // not print the confusing "written by kuzu X; this build uses X". - if stamp.KuzuVersion != kglib.KuzuVersion() { - fmt.Fprintf(out, "🔄 %s was written by kuzu %s; this build uses %s. Rebuilding.\n", - dbPath, stamp.KuzuVersion, kglib.KuzuVersion()) - } else { - fmt.Fprintf(out, "🔄 %s uses kg schema v%d; this build needs v%d. Rebuilding.\n", - dbPath, stamp.SchemaVersion, kglib.CurrentSchemaVersion) - } + fmt.Fprintf(out, "🔄 %s %s. Rebuilding.\n", dbPath, reason) aside, err := kglib.ArchiveDatabase(dbPath) if err != nil { @@ -61,6 +86,20 @@ func archiveIfMismatch(dbPath string, out io.Writer) (string, error) { return aside, nil } +// databaseMissingPK reports whether an existing database lacks the pk column, +// i.e. predates the project-namespaced primary key. It opens the database +// read-only purely to introspect its columns — a pre-pk schema is still readable, +// and a read-only open neither migrates the schema nor trips the write-open pk +// gate, so this is a safe question to ask of any database. +func databaseMissingPK(dbPath string) (bool, error) { + store, err := knowledge.OpenStoreReadOnly(dbPath) + if err != nil { + return false, err + } + defer store.Close() + return !store.HasPK(), nil +} + // rebuildHandWritesOnly repairs a database that has no source tree behind it — // the personal store. Archive, create an empty replacement, replay. There is // nothing to index, so the ordering hazard above does not apply. @@ -147,7 +186,18 @@ every scope and the personal store in one pass, or to see what would happen.`, // Counted only once the format is actually known: "checked" is a // claim about evidence, not about how many paths were visited. checked++ - if status != kglib.FormatMismatch { + // A stamped mismatch always rebuilds. An unstamped database is left + // alone UNLESS it also predates the pk column, which is a dead end + // (write-open refuses it) that only an archive+rebuild can repair — + // the same exception archiveIfMismatch makes. A current unstamped + // database (has pk) is still skipped. + needsRebuild := status == kglib.FormatMismatch + if !needsRebuild && status == kglib.FormatUnstamped { + if missing, mErr := databaseMissingPK(dbPath); mErr == nil && missing { + needsRebuild = true + } + } + if !needsRebuild { continue } if err := indexScopeDB(root, aiDir, scope); err != nil { diff --git a/src/kg/migrate_test.go b/src/kg/migrate_test.go index 587d1ab..6ced8bb 100644 --- a/src/kg/migrate_test.go +++ b/src/kg/migrate_test.go @@ -3,6 +3,7 @@ package main import ( "bytes" "encoding/json" + "fmt" "os" "path/filepath" "strings" @@ -10,8 +11,68 @@ import ( "github.com/cortexa-llc/mcp/kg/internal/knowledge" "github.com/cortexa-llc/mcp/kglib" + kuzu "github.com/kuzudb/go-kuzu" ) +// writePrePKDatabase builds a database with the pre-pk schema (Entity/Observation +// keyed on the global id, no pk column) and no format stamp, and drops a journal +// beside it carrying one hand-written entity. This is the legacy artifact the +// migration must still rescue: unstamped AND predating the pk column. +func writePrePKDatabase(t *testing.T, dbPath, projectID string, hand kglib.EntityRef) { + t.Helper() + + db, err := kuzu.OpenDatabase(dbPath, kuzu.DefaultSystemConfig()) + if err != nil { + t.Fatalf("OpenDatabase: %v", err) + } + conn, err := kuzu.OpenConnection(db) + if err != nil { + db.Close() + t.Fatalf("OpenConnection: %v", err) + } + + ddl := []string{ + `CREATE NODE TABLE Entity(id STRING PRIMARY KEY, name STRING, type STRING, + project_id STRING, created_at TIMESTAMP, updated_at TIMESTAMP, + embedding FLOAT[1536], visibility STRING)`, + `CREATE NODE TABLE Observation(id STRING PRIMARY KEY, entity_id STRING, + content STRING, created_at TIMESTAMP, embedding FLOAT[1536])`, + `CREATE REL TABLE HAS_OBSERVATION(FROM Entity TO Observation)`, + `CREATE NODE TABLE KGMeta(project_id STRING PRIMARY KEY, repo_url STRING, + commit_hash STRING, dirty BOOLEAN, indexed_at TIMESTAMP, + embed_model STRING, kg_version STRING)`, + } + for _, rt := range knowledge.AllowedRelTypes { + ddl = append(ddl, fmt.Sprintf("CREATE REL TABLE %s(FROM Entity TO Entity)", rt)) + } + for _, stmt := range ddl { + res, err := conn.Query(stmt) + if err != nil { + conn.Close() + db.Close() + t.Fatalf("pre-pk DDL %q: %v", stmt, err) + } + res.Close() + } + conn.Close() + db.Close() + + // The journal is the only copy of the hand-write across a rebuild. + rec := kglib.JournalRecord{ + Version: kglib.JournalVersion, + Op: kglib.OpCreateEntity, + ProjectID: projectID, + Entity: &hand, + } + data, err := json.Marshal(rec) + if err != nil { + t.Fatalf("marshal journal record: %v", err) + } + if err := os.WriteFile(kglib.JournalPath(dbPath), append(data, '\n'), 0600); err != nil { + t.Fatalf("write journal: %v", err) + } +} + // stampAs rewrites a database's format sidecar to claim a different engine // build wrote it, which is how a future go-kuzu bump is simulated without one. func stampAs(t *testing.T, dbPath, kuzuVersion string) { @@ -116,6 +177,79 @@ func TestSchemaVersionBumpRebuildsAndRestoresHandWrites(t *testing.T) { } // The whole point of Phase 4, end to end: hand-written knowledge survives a +// A database that is unstamped AND predates the pk column has no other recovery +// path: a read-write open refuses it (the pk gate), `kg migrate`'s loop used to +// skip anything that was not a stamped mismatch, and `kg index` hit the same +// gate — a dead end. archiveIfMismatch must now treat that specific case as a +// mismatch, archive it, and replay the journal into the rebuilt database. +func TestUnstampedPrePKDatabaseRebuildsAndRestoresHandWrites(t *testing.T) { + dbPath := filepath.Join(t.TempDir(), "knowledge.db") + writePrePKDatabase(t, dbPath, "proj", kglib.EntityRef{Name: "retry-policy", Type: "decision"}) + + // Preconditions: no stamp (unstamped) and no pk column (pre-pk). + status, _, err := kglib.CheckFormat(dbPath) + if err != nil { + t.Fatalf("CheckFormat: %v", err) + } + if status != kglib.FormatUnstamped { + t.Fatalf("fixture is not unstamped: got %v", status) + } + missing, err := databaseMissingPK(dbPath) + if err != nil { + t.Fatalf("databaseMissingPK: %v", err) + } + if !missing { + t.Fatal("fixture unexpectedly already has the pk column") + } + + // Pre-fix this returns (false, nil) — archiveIfMismatch did nothing for an + // unstamped database, leaving the dead end. Post-fix it archives + rebuilds. + var out bytes.Buffer + rebuilt, err := rebuildHandWritesOnly(dbPath, &out) + if err != nil { + t.Fatalf("rebuildHandWritesOnly: %v", err) + } + if !rebuilt { + t.Fatal("an unstamped pre-pk database was not rebuilt (the dead-end path)") + } + + // The old database is archived, not deleted. + matches, err := filepath.Glob(dbPath + ".old-*") + if err != nil { + t.Fatalf("Glob: %v", err) + } + if len(matches) == 0 { + t.Error("no archived copy of the pre-pk database") + } + + // The rebuilt database has the pk column and the journal's hand-write. + fresh, err := knowledge.OpenStore(dbPath) + if err != nil { + t.Fatalf("open rebuilt store: %v", err) + } + defer fresh.Close() + if !fresh.HasPK() { + t.Error("rebuilt database still lacks the pk column") + } + restored, err := fresh.GetEntityByNameAndType("retry-policy", "decision", "proj") + if err != nil { + t.Fatalf("GetEntityByNameAndType: %v", err) + } + if restored == nil { + t.Fatal("hand-written entity did not replay from the journal into the rebuild") + } + + // And the rebuilt database is now stamped and current, so it is not rebuilt + // again on the next run. + status, _, err = kglib.CheckFormat(dbPath) + if err != nil { + t.Fatalf("CheckFormat after rebuild: %v", err) + } + if status != kglib.FormatOK { + t.Errorf("status after rebuild = %v, want %v", status, kglib.FormatOK) + } +} + // storage-format change with no old binary involved and no user action. func TestEnsureFormatRebuildsAndRestoresHandWrites(t *testing.T) { dbPath := filepath.Join(t.TempDir(), "knowledge.db") diff --git a/src/kglib/embeddings.go b/src/kglib/embeddings.go index 3c699ff..9087acb 100644 --- a/src/kglib/embeddings.go +++ b/src/kglib/embeddings.go @@ -72,9 +72,11 @@ func (s *Store) BatchEmbed(ctx context.Context, projectID string, embedder Embed "results are matched by position, so a mismatched count cannot be used", len(obsEmbeddings), len(obsTexts)) } - // Store observation embeddings + // Store observation embeddings. projectID is the project BatchEmbed is + // embedding (GetUnembeddedObservations filtered on it), so it scopes the + // write to this project's observation rows. for i, obs := range observations { - if err := s.SetObservationEmbedding(obs.ID, obsEmbeddings[i]); err != nil { + if err := s.SetObservationEmbedding(obs.ID, projectID, obsEmbeddings[i]); err != nil { return fmt.Errorf("set embedding for observation %s: %w", obs.ID, err) } } @@ -182,12 +184,19 @@ func (s *Store) SetEmbedding(entityID, projectID string, embedding []float32) er return nil } -// SetObservationEmbedding stores an embedding vector for an observation -func (s *Store) SetObservationEmbedding(observationID string, embedding []float32) error { +// SetObservationEmbedding stores an embedding vector for an observation. +// +// Matched by the project-namespaced pk, consistent with SetEmbedding and every +// other write: an observation id is a UUID so a cross-project collision is +// unlikely, but a bare-id match is the same project-blind pattern this change +// removes elsewhere, and there is no reason to leave one write path on it. No +// !hasPK fallback is needed — writes only happen on a read-write open, which +// refuses a pre-pk database outright (see openStoreWithConfig). +func (s *Store) SetObservationEmbedding(observationID, projectID string, embedding []float32) error { result, err := s.QueryParams(` - MATCH (o:Observation {id: $id}) + MATCH (o:Observation {pk: $pk}) SET o.embedding = $embedding - `, map[string]any{"id": observationID, "embedding": embedding}) + `, map[string]any{"pk": PK(projectID, observationID), "embedding": embedding}) if err != nil { return fmt.Errorf("set observation embedding: %w", err) } diff --git a/src/kglib/store.go b/src/kglib/store.go index 1eeaed5..600e9a2 100644 --- a/src/kglib/store.go +++ b/src/kglib/store.go @@ -194,6 +194,13 @@ func openDiagnostic(dbPath string, readOnly bool, err error) error { return fmt.Errorf("open kuzu database: %w", err) } +// HasPK reports whether this database carries the project-namespaced pk primary +// key. It is false for a pre-pk database — one written before the pk column +// existed — which a read-only open can still read but a read-write open refuses. +// Exposed so the migration path can distinguish an unstamped-but-current +// database (leave alone) from an unstamped pre-pk one (archive and rebuild). +func (s *Store) HasPK() bool { return s.hasPK } + // Close closes the database connection. // It acquires mu to ensure no query is in flight while closing. func (s *Store) Close() error { From 1b6ae6672afdb490f740439aea6fac04e410e312 Mon Sep 17 00:00:00 2001 From: Bryan Woodruff Date: Tue, 13 Oct 2026 23:16:36 +0100 Subject: [PATCH 4/7] fix(kg): address PR #17 minor review findings; add orphaned obs-embed scope test MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - kglib/entity.go: restore len(row)>1 guard in detectColumns before indexing row[1] (defensive bounds check dropped in an earlier refactor). - kg/migrate.go,index.go: thread a precomputed missing-pk hint into archiveIfMismatch so `kg migrate` does one read-only open per unstamped-pre-pk database instead of two; standalone `kg index` passes nil and computes on demand. - knowledge/indexer.go: refresh e.visibility on the MERGE ON MATCH fallback so a re-indexed symbol whose visibility flips isn't left stale (log_indexer has no visibility attribute, left as-is). - kglib/pk_test.go: pin the 0x1F pk-separator cgo round-trip invariant with a real write + reopen + query test (asserts the stored pk is byte-identical, not truncated as a NUL join would be, and resolves by exact pk). - kglib/embeddings_test.go: TestSetObservationEmbedding_ProjectScoped — the project-scoping regression test for the SetObservationEmbedding fix in caa19b2, orphaned when that commit was staged file-by-file. Co-Authored-By: Claude --- src/kg/index.go | 9 ++- src/kg/internal/knowledge/indexer.go | 3 +- src/kg/migrate.go | 34 +++++++--- src/kg/migrate_test.go | 2 +- src/kglib/embeddings_test.go | 49 ++++++++++++++ src/kglib/entity.go | 5 ++ src/kglib/pk_test.go | 99 ++++++++++++++++++++++++++++ 7 files changed, 187 insertions(+), 14 deletions(-) create mode 100644 src/kglib/pk_test.go diff --git a/src/kg/index.go b/src/kg/index.go index 8c77660..281a290 100644 --- a/src/kg/index.go +++ b/src/kg/index.go @@ -45,7 +45,7 @@ Examples: // Index each scope totalStart := time.Now() for _, scopeName := range scopes { - if err := indexScopeDB(root, aiDir, scopeName); err != nil { + if err := indexScopeDB(root, aiDir, scopeName, nil); err != nil { return fmt.Errorf("index scope %s: %w", scopeName, err) } } @@ -100,7 +100,10 @@ func determineScopesToIndex(aiDir, requestedScope string, all bool) ([]string, e return names, nil } -func indexScopeDB(root, aiDir, scopeName string) error { +// missingPKHint is forwarded to archiveIfMismatch: the `kg migrate` caller has +// already computed databaseMissingPK for this database and passes it in to avoid +// a second read-only open; the standalone `kg index` caller passes nil. +func indexScopeDB(root, aiDir, scopeName string, missingPKHint *bool) error { var dbPath string var scopeConfig *knowledge.ScopeConfig @@ -121,7 +124,7 @@ func indexScopeDB(root, aiDir, scopeName string) error { // all, so the check has to happen before the open, not in response to its // error. Indexing is exactly the operation that repairs one: the derived // content is about to be regenerated from the tree anyway. - aside, err := archiveIfMismatch(dbPath, os.Stdout) + aside, err := archiveIfMismatch(dbPath, os.Stdout, missingPKHint) if err != nil { return fmt.Errorf("rebuild %s: %w", dbPath, err) } diff --git a/src/kg/internal/knowledge/indexer.go b/src/kg/internal/knowledge/indexer.go index de7bada..1483775 100644 --- a/src/kg/internal/knowledge/indexer.go +++ b/src/kg/internal/knowledge/indexer.go @@ -676,7 +676,8 @@ func (idx *Indexer) batchCreateEntities(entities []entityRecord) error { e.visibility = $visibility ON MATCH SET e.name = $name, e.type = $type, - e.updated_at = $updated_at + e.updated_at = $updated_at, + e.visibility = $visibility `, map[string]any{ "pk": PK(ent.ProjectID, ent.ID), "id": ent.ID, diff --git a/src/kg/migrate.go b/src/kg/migrate.go index 0f2551f..66ff299 100644 --- a/src/kg/migrate.go +++ b/src/kg/migrate.go @@ -39,7 +39,12 @@ import ( // it is archived and rebuilt here. That case is detected by a read-only open — // a pre-pk schema is still readable — so a current unstamped database (which has // pk) is never falsely archived; it simply gets stamped on the next write open. -func archiveIfMismatch(dbPath string, out io.Writer) (string, error) { +// +// missingPKHint lets a caller that has already run databaseMissingPK for this +// database pass the result in, so a `kg migrate` run does not re-derive it with a +// second read-only open. It is nil on the standalone `kg index` path, where the +// value is computed here on demand. +func archiveIfMismatch(dbPath string, out io.Writer, missingPKHint *bool) (string, error) { status, stamp, err := kglib.CheckFormat(dbPath) if err != nil { return "", err @@ -59,12 +64,18 @@ func archiveIfMismatch(dbPath string, out io.Writer) (string, error) { stamp.SchemaVersion, kglib.CurrentSchemaVersion) } case kglib.FormatUnstamped: - missing, err := databaseMissingPK(dbPath) - if err != nil { - // Cannot introspect the columns (e.g. the database will not even open - // read-only). Fall back to the historical leave-alone behavior rather - // than archiving on a guess. - return "", nil + var missing bool + if missingPKHint != nil { + missing = *missingPKHint + } else { + m, err := databaseMissingPK(dbPath) + if err != nil { + // Cannot introspect the columns (e.g. the database will not even open + // read-only). Fall back to the historical leave-alone behavior rather + // than archiving on a guess. + return "", nil + } + missing = m } if !missing { // Unstamped but already current — leave it alone; the next write open @@ -104,7 +115,7 @@ func databaseMissingPK(dbPath string) (bool, error) { // the personal store. Archive, create an empty replacement, replay. There is // nothing to index, so the ordering hazard above does not apply. func rebuildHandWritesOnly(dbPath string, out io.Writer) (bool, error) { - aside, err := archiveIfMismatch(dbPath, out) + aside, err := archiveIfMismatch(dbPath, out, nil) if err != nil || aside == "" { return false, err } @@ -192,15 +203,20 @@ every scope and the personal store in one pass, or to see what would happen.`, // the same exception archiveIfMismatch makes. A current unstamped // database (has pk) is still skipped. needsRebuild := status == kglib.FormatMismatch + // missingHint carries the single databaseMissingPK result for this + // database into archiveIfMismatch (reached via indexScopeDB below), so + // the read-only open it does is not repeated there. + var missingHint *bool if !needsRebuild && status == kglib.FormatUnstamped { if missing, mErr := databaseMissingPK(dbPath); mErr == nil && missing { needsRebuild = true + missingHint = &missing } } if !needsRebuild { continue } - if err := indexScopeDB(root, aiDir, scope); err != nil { + if err := indexScopeDB(root, aiDir, scope, missingHint); err != nil { failf("scope %s: rebuild failed, %s is still unreadable: %v", label, dbPath, err) continue } diff --git a/src/kg/migrate_test.go b/src/kg/migrate_test.go index 6ced8bb..2d290bb 100644 --- a/src/kg/migrate_test.go +++ b/src/kg/migrate_test.go @@ -380,7 +380,7 @@ func TestRebuildKeepsTheOldDatabase(t *testing.T) { stampAs(t, dbPath, "v0.0.1-ancient") var out bytes.Buffer - aside, err := archiveIfMismatch(dbPath, &out) + aside, err := archiveIfMismatch(dbPath, &out, nil) if err != nil { t.Fatalf("archiveIfMismatch: %v", err) } diff --git a/src/kglib/embeddings_test.go b/src/kglib/embeddings_test.go index c82d837..e48b49e 100644 --- a/src/kglib/embeddings_test.go +++ b/src/kglib/embeddings_test.go @@ -178,3 +178,52 @@ func TestSetEmbedding_HNSWInvalidation(t *testing.T) { t.Error("expected HNSW cache to be invalidated after SetEmbedding") } } + +// SetObservationEmbedding must be project-scoped: a write addressed with the +// wrong project id must not reach an observation in another project. Proven via +// GetUnembeddedObservations, which lists only observations whose embedding is +// still NULL — a write that matched would drop the observation from the list. +func TestSetObservationEmbedding_ProjectScoped(t *testing.T) { + store := setupTestStore(t) + defer store.Close() + + const projectID = "obs-embed-scope" + entity, err := store.CreateEntity("auth.go", "file", projectID) + if err != nil { + t.Fatalf("CreateEntity: %v", err) + } + obs, err := store.CreateObservation(entity.ID, "token TTL is 15m", projectID) + if err != nil { + t.Fatalf("CreateObservation: %v", err) + } + + embedding := make([]float32, embeddingDim) + for i := range embedding { + embedding[i] = 0.1 + } + + // Wrong project: the pk PK("other", obs.ID) exists nowhere, so the match is + // empty and the observation stays un-embedded. + if err := store.SetObservationEmbedding(obs.ID, "other-project", embedding); err != nil { + t.Fatalf("SetObservationEmbedding (wrong project): %v", err) + } + unembedded, err := store.GetUnembeddedObservations(projectID) + if err != nil { + t.Fatalf("GetUnembeddedObservations: %v", err) + } + if len(unembedded) != 1 { + t.Fatalf("a wrong-project write changed the embedding: got %d un-embedded, want 1", len(unembedded)) + } + + // Correct project: the write lands and the observation is no longer un-embedded. + if err := store.SetObservationEmbedding(obs.ID, projectID, embedding); err != nil { + t.Fatalf("SetObservationEmbedding (correct project): %v", err) + } + unembedded, err = store.GetUnembeddedObservations(projectID) + if err != nil { + t.Fatalf("GetUnembeddedObservations after correct write: %v", err) + } + if len(unembedded) != 0 { + t.Errorf("correct-project write did not set the embedding: got %d un-embedded, want 0", len(unembedded)) + } +} diff --git a/src/kglib/entity.go b/src/kglib/entity.go index 104838d..c43985b 100644 --- a/src/kglib/entity.go +++ b/src/kglib/entity.go @@ -121,6 +121,11 @@ func (s *Store) detectColumns() { if err != nil { return } + // table_info's column name is row[1]; skip any row too short to carry it + // rather than indexing unconditionally. + if len(row) <= 1 { + continue + } switch name := stringOrEmpty(row[1]); name { case "visibility": s.hasVisibility = true diff --git a/src/kglib/pk_test.go b/src/kglib/pk_test.go new file mode 100644 index 0000000..25d1e73 --- /dev/null +++ b/src/kglib/pk_test.go @@ -0,0 +1,99 @@ +package kglib + +import ( + "path/filepath" + "strings" + "testing" +) + +// TestPKSeparatorSurvivesCGoRoundTrip pins the invariant the whole +// project-namespaced primary key depends on: the Unit Separator (0x1F) joining +// project_id and id must survive go-kuzu's cgo string marshalling intact, where +// a NUL would truncate it. The comment in pk.go asserts this; this test proves +// it against the real Store write+read path, not the Go PK() function alone. +func TestPKSeparatorSurvivesCGoRoundTrip(t *testing.T) { + dbPath := filepath.Join(t.TempDir(), "pk.db") + + store, err := OpenStore(dbPath, testSchemaConfig()) + if err != nil { + t.Fatalf("OpenStore: %v", err) + } + + const projectID = "proj-alpha" + entity, err := store.CreateEntity("main.go", "file", projectID) + if err != nil { + store.Close() + t.Fatalf("CreateEntity: %v", err) + } + wantPK := PK(projectID, entity.ID) + + // Sanity: the value we expect to be stored actually contains the separator, + // so a passing round-trip is meaningful rather than vacuous. + if !strings.Contains(wantPK, pkSeparator) { + t.Fatalf("derived pk %q does not contain the 0x1F separator", wantPK) + } + store.Close() + + // Reopen (read-only, a fresh C database handle) so the assertions exercise a + // genuine marshal-out / marshal-in across the cgo boundary, not an in-memory + // cache from the writing connection. + ro, err := OpenStoreReadOnly(dbPath) + if err != nil { + t.Fatalf("OpenStoreReadOnly: %v", err) + } + defer ro.Close() + + // (a) The raw stored pk is byte-identical — separator present, not truncated + // to just the project prefix the way a NUL join would collapse to. + gotPK := queryOneString(t, ro, + `MATCH (e:Entity) WHERE e.id = $id RETURN e.pk`, + map[string]any{"id": entity.ID}) + if gotPK != wantPK { + t.Fatalf("stored pk mismatch:\n got %q (len %d)\n want %q (len %d)", + gotPK, len(gotPK), wantPK, len(wantPK)) + } + if !strings.Contains(gotPK, pkSeparator) { + t.Errorf("stored pk %q lost the 0x1F separator across the cgo round trip", gotPK) + } + if gotPK == projectID { + t.Errorf("stored pk collapsed to the project prefix %q (NUL-style truncation)", projectID) + } + + // (b) A MATCH by the exact pk string — separator byte and all — resolves the + // row, confirming the key is usable as the primary key it stands in for. + gotID := queryOneString(t, ro, + `MATCH (e:Entity) WHERE e.pk = $pk RETURN e.id`, + map[string]any{"pk": wantPK}) + if gotID != entity.ID { + t.Fatalf("MATCH by exact pk resolved id %q, want %q", gotID, entity.ID) + } +} + +// queryOneString runs a parameterised query expected to yield exactly one row +// whose first column is a string, and returns that string. +func queryOneString(t *testing.T, s *Store, stmt string, params map[string]any) string { + t.Helper() + result, err := s.QueryParams(stmt, params) + if err != nil { + t.Fatalf("query %q: %v", stmt, err) + } + defer result.Close() + + if !result.HasNext() { + t.Fatalf("query %q returned no rows", stmt) + } + tuple, err := result.Next() + if err != nil { + t.Fatalf("next: %v", err) + } + defer tuple.Close() + + row, err := tuple.GetAsSlice() + if err != nil { + t.Fatalf("get row: %v", err) + } + if len(row) == 0 { + t.Fatalf("query %q returned an empty row", stmt) + } + return stringOrEmpty(row[0]) +} From 74e69b34c30a29f2217018c9ef8505415ca935fe Mon Sep 17 00:00:00 2001 From: Bryan Woodruff Date: Wed, 14 Oct 2026 10:14:11 +0100 Subject: [PATCH 5/7] chore(kg): defend pk separator invariant at project_id derivation; clarify exec-log visibility omission MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Addresses the two remaining optional Minor findings on PR #17. - helpers.go: projectIDFromCwd now guards the invariant kglib.PK rests on — that project_id never contains the 0x1F Unit Separator. A repo directory basename holding 0x1F is effectively impossible, so this defends the invariant non-fatally at its single derivation site (replace the byte with '_' + a stderr warning) rather than panicking in hot-path PK() or adding an error return across projectIDFromCwd's many string-consuming callers. Adds a focused test. - log_indexer.go: one comment confirming exec-log entities intentionally carry no visibility attribute, so the MERGE's omission is deliberate and not a second instance of the ON CREATE/ON MATCH gap fixed in indexer.go. No behavior change. Co-Authored-By: Claude --- src/kg/helpers.go | 20 +++++++++++++++- src/kg/helpers_test.go | 29 ++++++++++++++++++++++++ src/kg/internal/knowledge/log_indexer.go | 4 ++++ 3 files changed, 52 insertions(+), 1 deletion(-) create mode 100644 src/kg/helpers_test.go diff --git a/src/kg/helpers.go b/src/kg/helpers.go index 9d05861..3f761aa 100644 --- a/src/kg/helpers.go +++ b/src/kg/helpers.go @@ -62,10 +62,28 @@ func findProjectRoot(dir string) string { return dir } +// pkSeparator mirrors the Unit Separator (0x1F) that kglib.PK joins project_id +// and id with to form a collision-free primary key. project_id must never +// contain it, or the pk for a whole project would be corrupted. +const pkSeparator = '\x1f' + // projectIDFromCwd returns the project ID for the given working directory. // The project ID is the base name of the project root. +// +// It also guards the one invariant kglib.PK rests on — that project_id never +// contains the 0x1F separator. A repo directory basename holding 0x1F is +// effectively impossible, so this is defending the invariant at its single +// derivation site rather than handling a real case: if one ever appeared we +// replace the byte (non-fatally, with a warning) instead of letting a raw +// separator reach PK and silently collapse every pk for the project. func projectIDFromCwd(cwd string) string { - return filepath.Base(findProjectRoot(cwd)) + id := filepath.Base(findProjectRoot(cwd)) + if strings.ContainsRune(id, pkSeparator) { + fmt.Fprintf(os.Stderr, + "Warning: project_id %q contains the reserved 0x1F separator; replacing with '_' to preserve primary-key uniqueness\n", id) + id = strings.ReplaceAll(id, string(pkSeparator), "_") + } + return id } // openStore locates the project root, opens the knowledge graph store in diff --git a/src/kg/helpers_test.go b/src/kg/helpers_test.go new file mode 100644 index 0000000..eb246f3 --- /dev/null +++ b/src/kg/helpers_test.go @@ -0,0 +1,29 @@ +package main + +import ( + "os" + "path/filepath" + "strings" + "testing" +) + +// A project_id derived from a repo directory basename must never contain the +// 0x1F Unit Separator that kglib.PK uses to join project_id and id, or the pk +// for the whole project would collapse. projectIDFromCwd guards that invariant +// at its single derivation site. +func TestProjectIDFromCwdReplacesSeparator(t *testing.T) { + base := t.TempDir() + dir := filepath.Join(base, "weird\x1fname") + if err := os.Mkdir(dir, 0o755); err != nil { + t.Fatalf("mkdir: %v", err) + } + + got := projectIDFromCwd(dir) + + if strings.ContainsRune(got, pkSeparator) { + t.Fatalf("project_id still contains reserved 0x1F separator: %q", got) + } + if got != "weird_name" { + t.Fatalf("project_id = %q, want %q", got, "weird_name") + } +} diff --git a/src/kg/internal/knowledge/log_indexer.go b/src/kg/internal/knowledge/log_indexer.go index 06b54c3..c0f7470 100644 --- a/src/kg/internal/knowledge/log_indexer.go +++ b/src/kg/internal/knowledge/log_indexer.go @@ -336,6 +336,10 @@ func bulkLoadExecEntities(store *Store, entities []entityRecord) error { // unavailable. These entities were already filtered for prior existence // by name, so MERGE only refreshes a genuine re-run rather than colliding. for _, e := range entities { + // No visibility here is intentional: exec-log entities carry no + // visibility attribute today, unlike code-indexer entities. This is + // not a second instance of the ON CREATE/ON MATCH gap fixed in + // indexer.go — there is nothing to set. r, err2 := store.QueryParams(` MERGE (n:Entity {pk: $pk}) ON CREATE SET n.id = $id, n.name = $name, n.type = $type, From a9ab07fe42f5032bc1024fcccbbc90106ae31009 Mon Sep 17 00:00:00 2001 From: Bryan Woodruff Date: Wed, 14 Oct 2026 10:53:05 +0100 Subject: [PATCH 6/7] fix(kg): fail-closed on pk-introspection error in migrate; funnel all project_id through sanitizer MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Addresses the Major + Minor from PR #17's latest review. Major (a regression from the earlier migrate double-open dedup): in the `kg migrate` scan loop, when databaseMissingPK errored (e.g. a corrupt .db that won't open read-only) the error was swallowed — needsRebuild stayed false and the loop continued, but checked++ had already run, so the run could print a false "all readable by this build" all-clear for a database it never resolved. Now failf + continue (mirroring the CheckFormat-error path), so failf's unchecked count gates the summary and no all-clear is emitted. Adds a regression test (TestMigrateDoesNotCountScopeDatabasesWhosePKCannotBeDetermined) that fails with the false all-clear before the fix. Minor + tree audit: graph.go (federated filter) derived project_id via filepath.Base(root), bypassing projectIDFromCwd's 0x1F sanitization — the federated filter matches project_id literally, so a sanitized index vs raw filter would mismatch. Auditing the tree surfaced two more raw sites: handle_server.go (the MCP server's project_id, feeding entity/observation/ relation writes — a second live bug) and helpers.go openStore's returned projectID. All three now route through projectIDFromCwd, making its "sole funnel" doc comment actually true. Co-Authored-By: Claude --- src/kg/graph.go | 2 +- src/kg/handle_server.go | 2 +- src/kg/helpers.go | 16 ++++++---- src/kg/migrate.go | 12 +++++++- src/kg/migrate_scopes_test.go | 58 +++++++++++++++++++++++++++++++++++ 5 files changed, 81 insertions(+), 9 deletions(-) diff --git a/src/kg/graph.go b/src/kg/graph.go index acf011a..8e70a31 100644 --- a/src/kg/graph.go +++ b/src/kg/graph.go @@ -302,7 +302,7 @@ func runFederatedGraph(cmd *cobra.Command, s graphSettings, out io.Writer) (know graph, report, err := knowledge.LoadFederatedGraph(knowledge.FederatedGraphOptions{ AIDir: aiDir, Scope: scope, - ProjectID: filepath.Base(root), + ProjectID: projectIDFromCwd(root), OnlyLayers: graphLayers, MaxJoinLayers: graphMaxJoin, JoinTypes: parseJoinTypes(cmd), diff --git a/src/kg/handle_server.go b/src/kg/handle_server.go index 5a96e26..d0a78d0 100644 --- a/src/kg/handle_server.go +++ b/src/kg/handle_server.go @@ -23,7 +23,7 @@ func handleServer(cmd *cobra.Command) { } projectRoot := findProjectRoot(cwd) aiDir := filepath.Join(projectRoot, ".ai") - projectID := filepath.Base(projectRoot) + projectID := projectIDFromCwd(projectRoot) // Determine which scope to use for the MCP server // Priority: default scope > legacy mode (knowledge.db) diff --git a/src/kg/helpers.go b/src/kg/helpers.go index 3f761aa..a1ef03c 100644 --- a/src/kg/helpers.go +++ b/src/kg/helpers.go @@ -71,11 +71,15 @@ const pkSeparator = '\x1f' // The project ID is the base name of the project root. // // It also guards the one invariant kglib.PK rests on — that project_id never -// contains the 0x1F separator. A repo directory basename holding 0x1F is -// effectively impossible, so this is defending the invariant at its single -// derivation site rather than handling a real case: if one ever appeared we -// replace the byte (non-fatally, with a warning) instead of letting a raw -// separator reach PK and silently collapse every pk for the project. +// contains the 0x1F separator. This is the derivation helper every path-derived +// project_id flows through (index, search, stats, show, meta, push, the MCP +// server, and the federated-graph filter all route through it), so guarding the +// byte here guards it everywhere the value is matched against the stored +// project_id column. A repo directory basename holding 0x1F is effectively +// impossible, so this defends the invariant rather than handling a real case: +// if one ever appeared we replace the byte (non-fatally, with a warning) +// instead of letting a raw separator reach PK and silently collapse every pk +// for the project. func projectIDFromCwd(cwd string) string { id := filepath.Base(findProjectRoot(cwd)) if strings.ContainsRune(id, pkSeparator) { @@ -138,5 +142,5 @@ func openStoreModeWithScope(readOnly bool, scopeName string) (*knowledge.Store, if err != nil { return nil, "", err } - return store, filepath.Base(root), nil + return store, projectIDFromCwd(root), nil } diff --git a/src/kg/migrate.go b/src/kg/migrate.go index 66ff299..fcbf049 100644 --- a/src/kg/migrate.go +++ b/src/kg/migrate.go @@ -208,7 +208,17 @@ every scope and the personal store in one pass, or to see what would happen.`, // the read-only open it does is not repeated there. var missingHint *bool if !needsRebuild && status == kglib.FormatUnstamped { - if missing, mErr := databaseMissingPK(dbPath); mErr == nil && missing { + missing, mErr := databaseMissingPK(dbPath) + if mErr != nil { + // Mirror the CheckFormat-error path above: an introspection we + // could not complete is not evidence that this build can read + // the database, so it must count as unchecked (failf) rather + // than fall through with needsRebuild=false and reach the + // all-clear summary for a database we never resolved. + failf("scope %s: cannot determine whether %s predates the primary key: %v", label, dbPath, mErr) + continue + } + if missing { needsRebuild = true missingHint = &missing } diff --git a/src/kg/migrate_scopes_test.go b/src/kg/migrate_scopes_test.go index e9d6ca3..3e1346c 100644 --- a/src/kg/migrate_scopes_test.go +++ b/src/kg/migrate_scopes_test.go @@ -6,6 +6,8 @@ import ( "path/filepath" "strings" "testing" + + "github.com/cortexa-llc/mcp/kglib" ) // runMigrateInDir runs `kg migrate` with dir as the working directory and @@ -89,3 +91,59 @@ func TestMigrateDoesNotCountUncheckableScopeDatabases(t *testing.T) { t.Errorf("kg migrate reported an all-clear for a database it could not read:\n%s", out) } } + +// An unstamped scope database whose pk-column introspection fails is likewise +// unchecked. CheckFormat only reads the sidecar stamp, so it still reports +// FormatUnstamped for a database it never opened; databaseMissingPK then opens +// it read-only and fails. A regression made that error fall through with +// needsRebuild=false, so the loop continued and the run could still print an +// all-clear — a false yes for a database it could not resolve. +func TestMigrateDoesNotCountScopeDatabasesWhosePKCannotBeDetermined(t *testing.T) { + withTempPersonalStore(t) + + root := t.TempDir() + scopeDir := filepath.Join(root, ".ai", "scope") + if err := os.MkdirAll(scopeDir, 0o755); err != nil { + t.Fatalf("MkdirAll: %v", err) + } + cfg := `{"name":"platform","database":"platform.db"}` + if err := os.WriteFile(filepath.Join(scopeDir, "platform.json"), []byte(cfg), 0o600); err != nil { + t.Fatalf("write scope config: %v", err) + } + // The database exists (so CheckFormat stats it) and carries no stamp sidecar + // (so CheckFormat returns FormatUnstamped, reaching the pk check), but is + // unreadable, so the read-only open inside databaseMissingPK fails. + dbPath := filepath.Join(root, ".ai", "platform.db") + if err := os.MkdirAll(dbPath, 0o755); err != nil { + t.Fatalf("MkdirAll db: %v", err) + } + if err := os.Chmod(dbPath, 0o000); err != nil { + t.Fatalf("chmod db: %v", err) + } + t.Cleanup(func() { _ = os.Chmod(dbPath, 0o755) }) + + // Precondition: CheckFormat must see this as unstamped (not an error of its + // own), or the test would exercise the CheckFormat-error path instead of the + // databaseMissingPK-error path this regression is about. + status, _, err := kglib.CheckFormat(dbPath) + if err != nil { + t.Fatalf("CheckFormat precondition failed: %v", err) + } + if status != kglib.FormatUnstamped { + t.Fatalf("fixture is not FormatUnstamped (got %v); test would not reach the pk check", status) + } + + out, err := runMigrateInDir(t, root) + + if err == nil { + t.Error("kg migrate exited 0 for a database whose pk column it could not introspect") + } + if strings.Contains(out, "readable by this build") { + t.Errorf("kg migrate reported an all-clear for a database it could not resolve:\n%s", out) + } + // Prove it was the databaseMissingPK path, not the CheckFormat path, that + // flagged it — the new failf is the only source of this phrasing. + if !strings.Contains(out, "predates the primary key") { + t.Errorf("output does not show the pk-introspection failure was the reason it was flagged:\n%s", out) + } +} From ecabb525547db40a0d340fddbfd307767f85aadb Mon Sep 17 00:00:00 2001 From: Bryan Woodruff Date: Wed, 14 Oct 2026 11:05:07 +0100 Subject: [PATCH 7/7] docs(kg): correct stale pk-separator comment (NUL-join -> 0x1F-separator) The re-exported PK comment in store_wrappers.go still described a NUL join, but pk.go rejected NUL (go-kuzu's cgo string marshalling truncates at NUL) and uses the 0x1F Unit Separator. Comment-only; the code just calls kglib.PK. Co-Authored-By: Claude --- src/kg/internal/knowledge/store_wrappers.go | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/src/kg/internal/knowledge/store_wrappers.go b/src/kg/internal/knowledge/store_wrappers.go index 61ecee2..20a0f73 100644 --- a/src/kg/internal/knowledge/store_wrappers.go +++ b/src/kg/internal/knowledge/store_wrappers.go @@ -30,8 +30,9 @@ func OpenStoreReadOnly(dbPath string) (*kglib.Store, error) { // PK derives the project-namespaced primary key for an entity or observation. // Re-exported from kglib so the indexer's bulk-load (COPY/CREATE) paths derive -// pk through the exact same helper the Store's own writes use — the NUL-join -// format must match byte-for-byte or endpoints and nodes would not resolve. +// pk through the exact same helper the Store's own writes use — the +// 0x1F-separator format must match byte-for-byte or endpoints and nodes would +// not resolve. var PK = kglib.PK // NewEmbedderFromEnv creates an embedder from environment variables