diff --git a/CHANGELOG.md b/CHANGELOG.md index 722b4c0..32e0756 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -12,14 +12,17 @@ everything below it is unreleased. ### Added - **`kg graph`** — render the knowledge graph as mermaid, DOT, or JSON, with `--root`, - `--depth`, `--type`, and `--limit`. `--federated` renders across every layer. (#5) + `--depth`, `--type`, and `--limit`. (#5) +- **`kg graph --federated`** — one graph across a scope and every local layer it + federates with, grouped into a subgraph per layer. (#6) - **Cross-layer entity linking** for `kg graph --federated`: derived `DEPENDS_ON` edges are resolved from imports to the packages they name, replacing the `(name, type)` join that manufactured edges nobody recorded. `--join-types` and `--no-derived` - control it. (#7, [design](docs/kg-graph-linking-design.md)) -- **Package declarations indexed for Java, Kotlin, and Scala**, giving JVM layers the - dotted namespaces cross-layer linking needs. Previously only Go minted package - entities, and a Go package name has no dot to match on. + control it. (#7 design, #10 implementation, [design](docs/kg-graph-linking-design.md)) +- **Package declarations indexed for Java and Kotlin**, giving those layers the dotted + namespaces cross-layer linking needs. Scala already had them: its `package_clause` is + the same tree-sitter node Go uses, and the indexer matched it generically. Go's own + package names are single identifiers with no dot to match on. - **Shared knowledge hub** — `kg hub serve` and `kg hub list` host read-only graphs; `kg push` seeds them. Hub graphs federate into local search as remote layers, queried in place rather than downloaded. diff --git a/docs/kg-cli-reference.md b/docs/kg-cli-reference.md index e81e06e..465457d 100644 --- a/docs/kg-cli-reference.md +++ b/docs/kg-cli-reference.md @@ -230,6 +230,8 @@ Error: "main" matches 2 entities — use an ID instead: | `--rel` | *(all)* | Only follow these relation types | | `--limit` | `200` | Maximum nodes; `0` for no limit | | `--output`, `-o` | *(stdout)* | Write to a file | +| `--scope` | *(default scope)* | Which scope's database to read | +| `--personal` | off | Read the personal store instead of this project's graph | | `--federated` | off | Render the scope together with every *local* layer it federates with | | `--layer` | *(all)* | With `--federated`, load only these scopes | | `--join-max-layers` | `3` | With `--federated`, the boilerplate guard (see below) | @@ -374,30 +376,32 @@ Resolution rules: node instead of a dependency map. - **Ambiguity is skipped, never guessed.** If the matched package is defined in more than one layer, no edge is drawn and the count is reported. On the estate - this discards 3,359 imports against 2,525 kept — the majority case, not an - edge case. + this discards 3,346 imports against 2,240 kept — the majority case, not an + edge case, and it grows as indexing coverage improves. - **Same-layer resolutions are dropped**; that structure is already in the layer's own graph. - `--no-derived` turns the whole thing off, leaving only recorded relations. -**Only dotted namespaces resolve.** Not one `package` entity in the measured -estate has a `/` in its name — the indexers do not mint package entities for npm -or Go module paths — so JVM, Scala and Kotlin layers get derived links and -JavaScript and Go layers get none. That is a gap in what is indexed rather than -in the matching, and closing it is an indexer change. +**Only dotted namespaces resolve**, and only for languages whose package +declarations are indexed: -> ⚠️ **The figures in this section predate JVM package indexing and are pending -> re-measurement** — including the 3,359/2,525 split above and the load cost below. -> They were measured when Go was the only source of package entities, which the -> specificity floor filters out. See -> [kg-graph-linking-design.md § Pending re-measurement](kg-graph-linking-design.md#pending-re-measurement) -> for the full list and what is expected to change. +| Language | Package entity | Linkable | +|---|---|---| +| Scala | dotted namespace (`com.depop.auth.client`) | yes | +| Java, Kotlin | dotted namespace | yes, on a graph indexed with kg ≥ `5211bc2` | +| Go | bare identifier (`auth`) | no — one segment, below the specificity floor | +| TypeScript, JavaScript, Python, C/C++ | none | no | + +Not one `package` entity in the measured estate has a `/` in its name, so npm and +Go module paths have nothing to resolve against. That is a gap in what is indexed +rather than in the matching, and closing it is an indexer change — see +[kg-graph-linking-design.md § Follow-up](kg-graph-linking-design.md#follow-up). **Cost.** Databases are read one at a time and released, so peak memory is the merged graph plus the largest single layer rather than all of them at once. -Measured on a 61-layer estate — 668k entities, 1.2M relations — a full load is -about 6 seconds and 1.2 GB resident. Use `--layer` to narrow it when you know -which layers you care about. +Measured on a 61-layer estate — 720k entities, 1.2M relations merged — a full +load is about 7 seconds and 1.3 GB resident. Use `--layer` to narrow it when you +know which layers you care about. --- diff --git a/docs/kg-graph-linking-design.md b/docs/kg-graph-linking-design.md index 228fe6a..5a32eab 100644 --- a/docs/kg-graph-linking-design.md +++ b/docs/kg-graph-linking-design.md @@ -2,39 +2,75 @@ **Status:** implemented · gate run and passed · **Date:** 2026-08-26 · **Follows:** PR #5 (`kg graph`), PR #6 (`--federated`) -## Pending re-measurement +## Re-measured — 2026-08-29 -> ⚠️ **Every estate figure in this document, and in the `--federated` section of -> [kg-cli-reference.md](kg-cli-reference.md#kg-graph), was measured before JVM package -> indexing shipped.** +The estate was re-indexed with `kg` at `3ac2f1a` and every figure re-derived. Two +corrections came out of it, one about history and one about what re-indexing does. -At the time, the only package entities `kg index` minted were Go's bare identifiers, -which this proposal's specificity floor filters out — so the derived edges reported below -cannot have come from an estate indexed by the binary that was current when they were -written. They are retained as the record of what was reasoned about, not as current -output. +### The warning this replaces was built on a wrong premise -Re-deriving them needs the estate itself, on a machine running `kg` at `5211bc2` or -later. This section is the single place tracking that: update the figures in place and -delete this section when they are reproduced. +An earlier revision said the figures could not be trusted because before `5211bc2` +the only package entities `kg index` minted were Go's bare identifiers, which the +three-segment floor discards. Two facts contradict that: -| Figure | Value on record | Where | +- `indexer_treesitter.go` at `5211bc2^` already matched `package_clause` generically, + and that tree-sitter node type is **Scala as well as Go**. `5211bc2` added Java + (`package_declaration`) and Kotlin (`package_header`); it did not add Scala. +- The pre-`5211bc2` databases held 4,300 package entities with three or more dotted + segments. Spot-checking `package:backend_driven.models.item` resolved it to `.scala` + files under `mobile-api-browse/app/backend_driven/models/item/`. + +So the original figures did come from indexed data — Scala's. + +### Re-indexing lowers the derived-edge count + +The expectation on record was that indexing more package declarations would raise the +number of derived edges. It halves it, and the reason is worth keeping. + +| Figure | Before re-index | After | |---|---|---| -| Estate size | 61 layers, 667,858 entities, 1,223,605 relations | this doc [§Problem](#problem); cli-ref (as 668k / 1.2M) | -| Distinct package names / import names | 5,138 / 50,993 | this doc [§Measurement gate](#measurement-gate--result) | -| Gate prediction — unambiguous cross-layer matches | 845 | this doc §Measurement gate, [§Acceptance](#acceptance), [§Open questions](#open-questions) | -| Prefix-match resolutions before the one-layer rule | 3,204 | this doc §Open questions | -| Derived `DEPENDS_ON` edges as implemented | **2,525** | this doc §Acceptance; cli-ref | -| Ambiguous imports discarded | 3,359 | cli-ref | -| Manufactured edges the rule replaced | 67,263 | this doc §Problem, §Acceptance | -| Full-load cost | ~6 s, 1.2 GB resident | cli-ref | - -The package-name count and the derived-edge count should both **rise**: Java, Kotlin, and -Scala layers contributed zero package entities to the numbers above and now contribute -dotted namespaces, which is exactly what this rule resolves against. Note also that -cli-ref's claim that “JVM, Scala and Kotlin layers get derived links” describes the -intended behaviour and only became true with that indexing change — it is not what the -measured binary did. +| Package entities | 5,138 | **7,031** | +| …linkable (3+ segments) | 4,300 | **6,140** | +| Derived `DEPENDS_ON` edges | 2,532 | **2,240** | +| Distinct import names resolving | 845 | **557** | +| Same-layer, left alone | 5,046 | **12,832** | +| Ambiguous, skipped | 3,355 | 3,346 | +| Federated merge | 717,633 nodes / 1,229,204 rel | 719,575 / 1,238,257 | +| Full-load cost | 7.6 s, 1.30 GB | 7.0 s, 1.33 GB | + +309 previously-derived links disappeared and 21 appeared. Sampling the ones that went +shows both mechanisms, and both are the rule behaving correctly on better data: + +- **The package is now defined in the importing layer too.** `com.depop.common` was + indexed only in `libraries`, so every `clients` import of it looked like a + cross-repo dependency. Kotlin indexing means `clients` now declares it as well, the + name resolves to two layers, and the one-layer rule abstains rather than guessing. +- **A more specific package now matches.** Longest-prefix resolution finds a local + `com.depop.backenddrivenui.models.parameter` where it previously had to settle for a + shorter name defined elsewhere — so the import is same-layer and no edge is drawn, + which is why that count more than doubled. + +**The earlier cross-layer links were partly artifacts of incomplete indexing.** A +dependency edge drawn because the importing repository's own copy of a package had not +been indexed is not a dependency. Fewer edges here is a more truthful graph, not a +regression — and it sharpens +[open question 3](#open-questions): completeness of indexing *increases* ambiguity, so +the one-layer rule discards more as coverage improves. + +### What the re-index cost, and an earlier miscount + +971 s for all 61 scopes, no failures, entities +1,901 and relations +8,709 with no +scope losing any. **All 420 observations survived** — the selective re-index preserves +hand-written knowledge as designed. + +An earlier draft of this section put the value of re-indexing at "seventy-five `.java` +files and no Kotlin". That was measured with `find -maxdepth 4`, which misses source +trees nested deeper than four levels — the whole Android app among them. The estate +actually holds **30,906 `.scala`, 7,591 `.kt` and 1,835 `.java`** files, so `5211bc2` +gave package declarations to roughly nine thousand files, not seventy-five. + +Still unindexed, and unchanged by `5211bc2`: Python, TypeScript and Go, which +[§Follow-up](#follow-up) covers. ## Problem @@ -233,18 +269,20 @@ Re-run the measurement. The success condition is not "more edges" — it is that **every surviving cross-layer relation can be traced to a package an import names**, and the count of name-coincidence edges is zero under default settings. -Yield on the estate, as implemented: **2,525 derived `DEPENDS_ON` edges**, -replacing 67,263 manufactured ones. +Yield on the estate, as implemented: **2,240 derived `DEPENDS_ON` edges** on the +re-indexed estate, replacing 67,263 manufactured ones. (2,525 as first shipped; +2,532 after the wildcard-import fix; see [§Re-measured](#re-measured--2026-08-29) +for why better indexing lowers the count.) -> These figures predate JVM package indexing — see -> [Pending re-measurement](#pending-re-measurement) at the top of this document. +> Re-derived 2026-08-29 against the estate — see +> [§Re-measured](#re-measured--2026-08-29) at the top of this document. The gate predicted 845, and both numbers are right: 845 is the count of distinct -import *names* that resolve, while 2,525 counts import *nodes* — the same name -appears as a separate node in each layer that imports it, since imports join only -across three layers or fewer. Verified by recomputing the rule independently -against the shipped output: 2,525 = 2,525, and the name count reproduces 845 -exactly. +import *names* that resolve, while the 2,525 shipped at the time counted import +*nodes* — the same name appears as a separate node in each layer that imports it, +since imports join only across three layers or fewer. Verified by recomputing the +rule independently against the shipped output: 2,525 = 2,525, and the name count +reproduced 845 exactly. ## Follow-up @@ -256,13 +294,13 @@ exactly. | Go | bare identifier (`auth`) | no — one segment, below the specificity floor | | TypeScript, JavaScript, Python, C/C++ | none | no | -An earlier draft of this section had that backwards, saying the indexers mint -package entities "only for dotted namespaces". The reverse was true when it was -written: Go's `package_clause` was the *only* source, and a Go package name never -contains a dot, so every package entity was filtered out and this proposal -derived nothing from indexed data. JVM package declarations were not indexed at -all until they were added separately (see the package-indexing change that -precedes this one), which is what makes the rule above work. +This table has been wrong in both directions. An early draft said the indexers +mint package entities "only for dotted namespaces", which understated Go. The +correction then overshot, asserting that Go's `package_clause` was the *only* +source and that every package entity was therefore filtered out — but +`package_clause` is Scala's node type too, so Scala has been minting dotted +package names since well before `5211bc2`. That change added Java and Kotlin. +See [§Re-measured](#re-measured--2026-08-29). The remaining gap is npm and Go. Reading `package.json` `name` fields and `go.mod` module paths would give the web, client and tooling layers a namespace @@ -278,16 +316,22 @@ and not first. ## Open questions 1. ~~Does package linking find anything?~~ **Answered by the gate:** yes, with - prefix matching (845 unambiguous cross-layer edges); no, with exact matching (71, - mostly generic). The matching rule in §2 changed accordingly. + prefix matching (845 unambiguous cross-layer resolutions at the time, 557 on the + re-indexed estate); no, with exact matching (71, mostly generic). The matching rule + in §2 changed accordingly. 2. ~~Naming conventions across ecosystems.~~ **Answered:** dotted namespaces resolve; npm and Go produce nothing because no package entity name contains a `/`. Fixing that is an indexer change — see [Follow-up](#follow-up). -3. **Is 74% ambiguity acceptable?** Prefix matching finds 3,204 cross-layer - resolutions but only 845 survive the "package defined in exactly one layer" rule. - The discarded ones are mostly packages genuinely duplicated across repos. Options: - accept the loss, draw them to all candidate layers with a marker, or rank by layer - priority. Proposal: accept the loss in v1 and report the count. +3. **Is the ambiguity rule still acceptable?** On the re-indexed estate it discards + 3,346 imports and keeps 2,240 — and the discarded share *grows as indexing coverage + improves*, because a package indexed in more repositories resolves to more layers. + The linking therefore gets weaker over time rather than stronger, which the + [re-measurement](#re-measured--2026-08-29) is the evidence for. The discarded ones + are mostly packages genuinely duplicated across repos. Options: accept the loss; + draw to all candidate layers with a marker; rank by layer priority; or prefer the + importing layer's own definition, which would reclassify most of these as + same-layer and draw nothing — the truthful answer in the `com.depop.common` case. + v1 accepts the loss and reports the count; that is worth revisiting first. 4. **Should `type` join by default?** `AddressClient` argues yes, `CodingKeys` argues no. Proposal: off, revisited once derived edges exist and can be compared against.