Skip to content
219 changes: 219 additions & 0 deletions docs/kg-entity-key-fix-design.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,219 @@
# ADR: Fix Entity primary-key / project-scope mismatch in the kg indexer

Status: **Accepted — Option A (A1 variant)**. Multi-project-per-DB is a supported
scenario (product decision, 2026-10-09), so the global-PK bug is fixed by a
namespaced primary key, not by foreclosing multi-project DBs.
Date: 2026-10-09

## Context

`kg index` emits large volumes of warnings of the form:

```
Note: COPY FROM JSON unavailable (... Found duplicated primary key value ...); falling back to row-by-row insert
Warning: insert entity <kind>:<id>: ... Found duplicated primary key value <id>, which violates the uniqueness constraint ...
```

These are **not benign**. Investigation (source-verified) shows:

1. **The `Entity` primary key is global `id` alone** — `src/kglib/schema.go:45`
(`id STRING PRIMARY KEY`). Same for `Observation` (`schema.go:57`).
2. **Every clear/delete and nearly every read is scoped by `project_id`**:
- `clearCodeDerivedData` — `src/kg/internal/knowledge/indexer.go:280-288`
- `clearProjectData` — `indexer.go:373-399`
- `deleteProjectRelations` — `indexer.go:209-226`
- reads in `graph.go:136-178`, `ListEntities` (`src/kglib/entity.go:326`)
3. **In-run entity dedup is airtight**: `writeEntityVis` (`indexer.go:584`) is the
*only* append site and guards every append with `if seen[id] { return false }`.
So a batch never contains duplicate ids — collisions are strictly against rows
**already in the DB**.
4. **Relations match endpoints by bare `id` with no project filter in the MATCH** —
`CreateRelation`, `src/kglib/relation.go:24-28`
(`WHERE from.id = $from_id AND to.id = $to_id`).

### Failure mechanism

Because the PK is global but the clear is project-scoped, any pre-existing
code-derived row under a **different `project_id`** survives the clear. When the
current run writes the identically-id'd entity it collides and is **silently
dropped** (first/old row wins; `CREATE` at `indexer.go:659-677` and the COPY path
at `indexer.go:644-653`; the log indexer repeats the pattern at
`log_indexer.go:278-335`). Downstream:

- The current project's entity, plus its freshly-computed observations/relations
(which attach **by id**), never land.
- Relations can bind to the *other* project's surviving node (relation.go MATCH is
project-blind), or be dropped as dangling (`batchCreateRelations`, indexer.go:763).
- `kg stats` / graph reads scoped to the live `project_id` under-return.

A mismatched-`project_id` partition arises from any of: legacy `project_id = ""`
rows, a changed `projectIDFromCwd` (`src/kg/index.go:140`) across versions, a
moved/renamed repo, or two scope configs whose `cfg.Database` point at the same DB
file (`indexScopeDB`, `src/kg/index.go:103-118`). The *presence* of the warnings
proves such a partition exists — a same-project prior run would have been emptied
by the clear, producing no collision.

### Hard constraint

Kuzu **v0.11.3** (`src/kg*/go.mod`) node tables support exactly **one** PRIMARY KEY
column. A literal composite `(project_id, id)` node PK is **not expressible**. The
realistic options are a namespaced single key, or a structural one-project-per-DB
guarantee.

## Gating open question (needs a human decision)

**Is "multiple `project_id`s in one DB file" a supported scenario, or an accident?**

- The deployment path opens **one DB per scope** and derives `project_id` from cwd
(`index.go:103-140`), i.e. one project per file in practice.
- But `KGMeta` keys by `project_id` (`schema.go:69`), and reads filter by
`project_id` — the schema *looks* built for many projects per file.
- Federated linking (`graph_federated.go`) links across **separate** graph files,
not multiple projects in one file.

The answer selects the recommended option:

- **One project per DB is the real model → Option B (+ C).** Minimal, correct.
- **Multi-project-per-DB must be supported → Option A.** Larger, fully correct.

Recommendation absent other input: **one project per DB is the real model → Option B
primary, C as hardening.** Multi-project-per-file appears to be the accident that
creates stale partitions.

## Accepted design — Option A1 (namespaced `pk`, `id` stays local)

Decision: multi-project-per-DB **must** be supported, so Option B is rejected and
the key is made project-safe via a namespaced primary key. Within Option A, the
**A1** variant is chosen: keep the existing `id` column human-facing and unchanged
(`file:…`, `function:…`, UUIDs), and add a separate namespaced primary-key column.
This preserves the id contract that search, CLI display, and the journal
`stableIDHint` all depend on; A2 (rewriting `id` itself) is rejected as needlessly
breaking those.

Concrete decisions:

1. **Schema** (`src/kglib/schema.go`): Entity and Observation gain
`pk STRING PRIMARY KEY`; the current `id` becomes a plain (non-unique) STRING
attribute. `entity_id` on Observation stays, but observation→entity joins key on
the pair `(project_id, id)` (see 4).
2. **Key derivation** — one central helper:
`pk(projectID, id) = projectID + "\x00" + id`. NUL is the separator because
neither a `project_id` nor a code-derived/UUID `id` ever contains one; document
that invariant next to the helper. All writes derive `pk` here, never ad hoc.
3. **Writes populate `pk`**: COPY column lists (`indexer.go:646`, the log-indexer
COPY in `log_indexer.go`, the observation COPY) and every `CREATE`
(`indexer.go:659-677`, `entity.go:148`, `log_indexer.go` fallback) add `pk`.
4. **Lookups/endpoint matches become project-qualified** (the real correctness
fix): every `{id: …}` / `WHERE e.id = …` that is not already project-scoped must
match on `(project_id, id)` (equivalently on `pk`):
- `CreateRelation` MATCH — `relation.go:24-28` (currently project-blind — this is
the cross-project-bleed site). Add `from.project_id = $pid AND to.project_id =
$pid`, or match on `from.pk`/`to.pk`.
- `GetEntity` — `entity.go` (confirm it filters project_id; if it keys on id
alone, qualify it).
- Observation joins / `HAS_OBSERVATION` creation keyed on `entity_id`.
- `clearCodeDerivedData`, `clearProjectData`, `deleteProjectRelations` already
filter `project_id`; with a project-unique `pk` the DETACH DELETE now both
targets only this project's rows **and** stops colliding — no logic change
beyond confirming they read the new column where needed.
5. **Upsert**: with `pk` project-safe, switch entity writes `CREATE` → `MERGE` on
`pk` with last-write-wins refresh of `name`/`type`/`updated_at`. This makes
re-index idempotent and eliminates the warnings outright. (`MERGE` was unsafe
under the old global PK; it is correct now.)
6. **Federated linking** — `graph_federated.go`: cross-graph joins must key on the
**local `id`** (and/or name), never on `pk`, since `pk` embeds a per-graph
`project_id` that differs across files. Audit every cross-graph comparison.
7. **Migration**: Kuzu has no `ALTER TABLE … ADD PRIMARY KEY`, so the new `pk`
column forces a table recreate → rebuild-from-source. Bump `KGMeta.kg_version`
(`StampMeta`, `index.go:216`); on version mismatch, ride `archiveIfMismatch`
(`index.go:124`) to archive the old DB and rebuild fresh, then replay hand-writes
via `restoreHandWrites` (`index.go:210`). The journal is the only copy of
hand-writes — archive, never delete.

Open sub-question for the engineer to resolve during implementation: whether Kuzu
rel tables in v0.11.3 let `MERGE`/`MATCH` resolve endpoints efficiently by a
non-PK `(project_id, id)` pair, or whether endpoint matching should go through the
`pk` column directly (likely faster, since `pk` is the indexed PK). Prefer `pk`.

## Options (for the record)

### Option A — Namespaced single primary key (supports multi-project-per-DB)

Store PK as `pk = project_id + "\x00" + localID`; keep `id` (the current
`file:…`/`function:…`/UUID value) as a **non-PK indexed** attribute for display,
search, and the journal hint.

- Rewrite all MATCH/CREATE/COPY on Entity/Observation to key on `pk`, including
relation endpoint matching (`relation.go:24-28` must match on `pk`, not `id`).
- Blast radius (each must change):
- id construction sites: `indexer_treesitter.go:198` (`qualifiedSymbolID`),
and `fmt.Sprintf("file:%s"|"type:%s:%s"|"function:%s:%s"|"import:%s")` across
`indexer_yaml.go`, `indexer_makefile.go`, `indexer_asm.go`,
`indexer_jsonschema.go`, `indexer_graphql.go`, `indexer_html.go`,
`indexer_markdown.go`, `indexer_pdf.go`, `indexer_cmake.go` — these now feed
`localID`, with `pk` derived centrally in `writeEntityVis`.
- `codeDerivedIDClause` / `codeDerivedIDPrefixes` — `kg_types.go:34-52`
(prefix test must run against `localID`, not `pk`).
- COPY column lists — `indexer.go:646`, `log_indexer.go` COPY, observation COPY.
- reads — `graph.go`, `entity.go` (`ListEntities`, `GetEntity`),
`src/kglib/search.go`.
- journal `stableIDHint` and replay — `relation.go` note, `index.go:210`
(`restoreHandWrites`); the hint must remain the *localID* so replay still works.
- federated cross-graph linking — `graph_federated.go` (joins by id across files).
- Pro: preserves multi-project-per-DB; fully correct. Con: wide, touches the
human-facing/stable id contract and the journal.

### Option B — Enforce one `project_id` per DB file (recommended primary)

At index time, read the DB's existing `KGMeta.project_id`. If it differs from the
current run's `project_id` (or legacy `""` rows exist), treat it exactly like a
storage-format mismatch: **archive the old DB and rebuild fresh** — reusing the
existing `archiveIfMismatch` lever (`index.go:124`). With one project per file, the
global `id` PK is project-scoped by construction and collisions cannot occur.

- Change surface is small and localized to `index.go` (a project_id check beside
`archiveIfMismatch`) plus a KGMeta read helper.
- Journal safety: archived DB keeps its journal (`index.go:128-132`); hand-writes
replay into the fresh DB via `restoreHandWrites` (`index.go:210`), which is
idempotent.
- Pro: minimal, correct, matches real deployment. Con: forecloses
multi-project-per-DB (acceptable iff the gating question resolves that way).

### Option C — Global code-derived clear + MERGE upsert (hardening, pairs with B)

Under the one-project-per-DB premise, code-derived ids are globally unique within a
DB, so:

- Drop the `{project_id: $project_id}` filter from the **code-derived** DETACH DELETE
in `clearCodeDerivedData` (indexer.go:280-288) so a rebuild clears *all*
code-derived rows regardless of a stale project_id — making re-index idempotent and
warning-free even on a legacy DB.
- Switch entity writes `CREATE` → Kuzu **`MERGE`** (idempotent upsert) with
last-write-wins refresh of `name`/`type`/`updated_at`.
- **MERGE is unsafe under the current global PK** (one project would overwrite
another's row). It is correct only after B (or A) makes the key project-safe.

## Migration & schema versioning

- Bump the schema/kg version stamp (`KGMeta.kg_version`, written by `StampMeta`,
`index.go:216`) so older DBs are recognized.
- Option B needs no table rewrite (fresh rebuild on project_id mismatch).
- Option A requires a one-time rebuild (new PK column); ride `archiveIfMismatch`:
on version mismatch, archive + rebuild-from-source (code-derived data is
regenerable), then journal-replay hand-writes.
- In both, the journal is the **only** copy of hand-written entities — migration must
archive (not delete) the prior DB and run `restoreHandWrites` against the fresh one.

## Acceptance criteria

- [ ] `cd src/kg && go build ./... && go test ./...` exits 0
- [ ] `cd src/kglib && go build ./... && go test ./...` exits 0
- [ ] New regression test: index project A, then project B that shares entity ids
(e.g. both `import:logging`, `file:common.py`); assert **both** projects retain
their own entities and `kg stats` per project is correct.
- [ ] Re-indexing an already-indexed project emits **zero** `duplicated primary key`
warnings and **no** `COPY FROM JSON unavailable` note.
- [ ] Hand-written entities/relations (`kg add`) survive a re-index and a migration
rebuild (journal replay verified).
- [ ] `MATCH (e:Entity) RETURN DISTINCT e.project_id` on a migrated DB shows exactly
one project_id (Option B) — a field diagnostic to confirm the fix landed.
2 changes: 1 addition & 1 deletion src/kg/graph.go
Original file line number Diff line number Diff line change
Expand Up @@ -302,7 +302,7 @@ func runFederatedGraph(cmd *cobra.Command, s graphSettings, out io.Writer) (know
graph, report, err := knowledge.LoadFederatedGraph(knowledge.FederatedGraphOptions{
AIDir: aiDir,
Scope: scope,
ProjectID: filepath.Base(root),
ProjectID: projectIDFromCwd(root),
OnlyLayers: graphLayers,
MaxJoinLayers: graphMaxJoin,
JoinTypes: parseJoinTypes(cmd),
Expand Down
2 changes: 1 addition & 1 deletion src/kg/handle_server.go
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ func handleServer(cmd *cobra.Command) {
}
projectRoot := findProjectRoot(cwd)
aiDir := filepath.Join(projectRoot, ".ai")
projectID := filepath.Base(projectRoot)
projectID := projectIDFromCwd(projectRoot)

// Determine which scope to use for the MCP server
// Priority: default scope > legacy mode (knowledge.db)
Expand Down
26 changes: 24 additions & 2 deletions src/kg/helpers.go
Original file line number Diff line number Diff line change
Expand Up @@ -62,10 +62,32 @@ func findProjectRoot(dir string) string {
return dir
}

// pkSeparator mirrors the Unit Separator (0x1F) that kglib.PK joins project_id
// and id with to form a collision-free primary key. project_id must never
// contain it, or the pk for a whole project would be corrupted.
const pkSeparator = '\x1f'

// projectIDFromCwd returns the project ID for the given working directory.
// The project ID is the base name of the project root.
//
// It also guards the one invariant kglib.PK rests on — that project_id never
// contains the 0x1F separator. This is the derivation helper every path-derived
// project_id flows through (index, search, stats, show, meta, push, the MCP
// server, and the federated-graph filter all route through it), so guarding the
// byte here guards it everywhere the value is matched against the stored
// project_id column. A repo directory basename holding 0x1F is effectively
// impossible, so this defends the invariant rather than handling a real case:
// if one ever appeared we replace the byte (non-fatally, with a warning)
// instead of letting a raw separator reach PK and silently collapse every pk
// for the project.
func projectIDFromCwd(cwd string) string {
return filepath.Base(findProjectRoot(cwd))
id := filepath.Base(findProjectRoot(cwd))
if strings.ContainsRune(id, pkSeparator) {
fmt.Fprintf(os.Stderr,
"Warning: project_id %q contains the reserved 0x1F separator; replacing with '_' to preserve primary-key uniqueness\n", id)
id = strings.ReplaceAll(id, string(pkSeparator), "_")
}
return id
}

// openStore locates the project root, opens the knowledge graph store in
Expand Down Expand Up @@ -120,5 +142,5 @@ func openStoreModeWithScope(readOnly bool, scopeName string) (*knowledge.Store,
if err != nil {
return nil, "", err
}
return store, filepath.Base(root), nil
return store, projectIDFromCwd(root), nil
}
29 changes: 29 additions & 0 deletions src/kg/helpers_test.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
package main

import (
"os"
"path/filepath"
"strings"
"testing"
)

// A project_id derived from a repo directory basename must never contain the
// 0x1F Unit Separator that kglib.PK uses to join project_id and id, or the pk
// for the whole project would collapse. projectIDFromCwd guards that invariant
// at its single derivation site.
func TestProjectIDFromCwdReplacesSeparator(t *testing.T) {
base := t.TempDir()
dir := filepath.Join(base, "weird\x1fname")
if err := os.Mkdir(dir, 0o755); err != nil {
t.Fatalf("mkdir: %v", err)
}

got := projectIDFromCwd(dir)

if strings.ContainsRune(got, pkSeparator) {
t.Fatalf("project_id still contains reserved 0x1F separator: %q", got)
}
if got != "weird_name" {
t.Fatalf("project_id = %q, want %q", got, "weird_name")
}
}
9 changes: 6 additions & 3 deletions src/kg/index.go
Original file line number Diff line number Diff line change
Expand Up @@ -45,7 +45,7 @@ Examples:
// Index each scope
totalStart := time.Now()
for _, scopeName := range scopes {
if err := indexScopeDB(root, aiDir, scopeName); err != nil {
if err := indexScopeDB(root, aiDir, scopeName, nil); err != nil {
return fmt.Errorf("index scope %s: %w", scopeName, err)
}
}
Expand Down Expand Up @@ -100,7 +100,10 @@ func determineScopesToIndex(aiDir, requestedScope string, all bool) ([]string, e
return names, nil
}

func indexScopeDB(root, aiDir, scopeName string) error {
// missingPKHint is forwarded to archiveIfMismatch: the `kg migrate` caller has
// already computed databaseMissingPK for this database and passes it in to avoid
// a second read-only open; the standalone `kg index` caller passes nil.
func indexScopeDB(root, aiDir, scopeName string, missingPKHint *bool) error {
var dbPath string
var scopeConfig *knowledge.ScopeConfig

Expand All @@ -121,7 +124,7 @@ func indexScopeDB(root, aiDir, scopeName string) error {
// all, so the check has to happen before the open, not in response to its
// error. Indexing is exactly the operation that repairs one: the derived
// content is about to be regenerated from the tree anyway.
aside, err := archiveIfMismatch(dbPath, os.Stdout)
aside, err := archiveIfMismatch(dbPath, os.Stdout, missingPKHint)
if err != nil {
return fmt.Errorf("rebuild %s: %w", dbPath, err)
}
Expand Down
4 changes: 2 additions & 2 deletions src/kg/internal/knowledge/graph_federated_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -269,8 +269,8 @@ func seedEntityWithID(t *testing.T, aiDir, layer, id, name, entityType string) {
defer store.Close()

result, err := store.QueryParams(`
CREATE (e:Entity {id: $id, name: $name, type: $type, project_id: $project_id})
`, map[string]any{"id": id, "name": name, "type": entityType, "project_id": fedProject})
CREATE (e:Entity {pk: $pk, id: $id, name: $name, type: $type, project_id: $project_id})
`, map[string]any{"pk": PK(fedProject, id), "id": id, "name": name, "type": entityType, "project_id": fedProject})
if err != nil {
t.Fatalf("seed entity %s in %s: %v", id, layer, err)
}
Expand Down
Loading
Loading