Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -28,4 +28,4 @@ jobs:

# Container suite is opt-in; ubuntu-latest ships Docker, so testcontainers works out of the box.
- name: Neo4j bolt container tests
run: RUN_CONTAINER_TESTS=1 bun test test/neo4j-bolt.test.ts
run: bun run test:container
30 changes: 28 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -153,6 +153,9 @@ Options:
Neo4j schema.json contract) (default: "json")
--app-name <name> logical application name for the graph
:Application anchor (default: input dir name)
--workspace-root <dir> directory file keys, module ids and artifact
ids are relative to (default: --input); must
contain --input
--neo4j-uri <uri> push the graph to a live Neo4j over Bolt
(incremental); omit to write graph.cypher (env:
NEO4J_URI)
Expand Down Expand Up @@ -229,12 +232,21 @@ Options:
cants --input ./my-ts-project --target-files src/a.ts src/b.ts
```

4. **Force a clean rebuild with a custom cache directory:**
4. **Repository-relative paths for a project inside a larger repo:**
```sh
cants --input ./monorepo/packages/web --workspace-root ./monorepo
# symbol_table key: packages/web/src/app.ts
```
Only the published keys and ids change: the application id (`can://<app-name>`) and
`signature`s stay as they are, entrypoint path rules still match from `--input`, and the cache
is reused either way.

5. **Force a clean rebuild with a custom cache directory:**
```sh
cants --input ./my-ts-project --eager --cache-dir /path/to/custom-cache
```

5. **Program graphs (level 3): CFG/PDG/SDG in `analysis.json`:**
6. **Program graphs (level 3): CFG/PDG/SDG in `analysis.json`:**
```sh
cants --input ./my-ts-project -a 3 # full program_graphs section
cants --input ./my-ts-project -a 3 --graphs cfg,pdg # scope the emitted graphs
Expand Down Expand Up @@ -336,6 +348,20 @@ statements scope on, and `:Application` is keyed on the id rather than on `--app
content hash changed are rewritten, and on a full run modules whose source file vanished are
pruned. Every graph carries a `schema_version` on its `:Application` node.

`--eager` purges this application's code nodes but not its `:Artifact`/`:ConfigKey` nodes, which
are shared with the sibling analyzers. An `--eager` full run (no `--target-files`) reconciles them
instead, so a deleted or renamed file, a removed dependency, or re-keying under
`--workspace-root` leaves nothing stale. TypeScript-owned artifact edges (`HAS_ARTIFACT`,
`DECLARES_DEPENDENCY`/`LOCKS` to `pkg:npm/…`, `TS_PROVIDES`, `TS_USES_CONFIG`,
`TS_READS_CONFIG_UNRESOLVED`, `TS_UNRESOLVED_IMPORT`) are rebuilt. An `:Artifact` or `:ConfigKey`
the run no longer emits loses its `:TSCanNode` label. An artifact is then deleted only if no other
analyzer's `HAS_ARTIFACT` still points at it; a config key only if no other analyzer's marker
(`:JCanNode`, `:PyCanNode`) claims it. Other analyzers' edges are never removed. An
`--eager --target-files` push reconciles nothing, so stale artifacts wait for the next full
`--eager` push. The `graph.cypher` snapshot is a full rebuild by design: its wipe deletes every
node this application wrote, shared artifacts included, and other analyzers restore their edges
on their next push.

The connection options also read the standard Neo4j environment variables — `NEO4J_URI`,
`NEO4J_USERNAME`, `NEO4J_PASSWORD`, `NEO4J_DATABASE` — when the corresponding flag is omitted (an
explicit flag wins). Prefer the env var for the password so it doesn't land in shell history or the
Expand Down
2 changes: 1 addition & 1 deletion package.json
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@
"build": "bun build ./src/main.ts ./src/dataflow/worker.ts --compile --outfile dist/cants",
"gen:schema": "bun run src/index.ts --emit schema > schema.neo4j.json",
"gen:readme": "bun run scripts/update-readme.ts",
"test:container": "RUN_CONTAINER_TESTS=1 bun test test/neo4j-bolt.test.ts",
"test:container": "RUN_CONTAINER_TESTS=1 bun test test/neo4j-bolt.test.ts test/neo4j-artifact-reconcile.test.ts",
"typecheck": "tsc --noEmit"
},
"license": "Apache-2.0",
Expand Down
70 changes: 67 additions & 3 deletions src/build/neo4j/bolt.ts
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,8 @@
* 4. upsert edges owned by changed modules (+ the shared edges).
* 5. on a FULL run only, prune modules whose source file vanished.
*
* A full `--eager` run also reconciles the app's artifacts (RELEASE_VANISHED_ARTIFACTS).
*
* Nodes are MERGE-upserted, never blindly deleted, so a declaration another (unchanged) module
* still references survives and its incoming edges stay valid. `:TSExternal` nodes are shared (no
* `_module`) and are MERGE-only.
Expand Down Expand Up @@ -58,7 +60,50 @@ export function shouldForceFullUpsert(dbVersion: string | null, producerVersion:
* `applicationPrefix`, never a bare app id — a bare id also matches `can://appXtra/...`.
*/
export const EAGER_PURGE =
`MATCH (n:${TS_MARKER}) WHERE n.id STARTS WITH $prefix CALL { WITH n DETACH DELETE n } IN TRANSACTIONS OF 5000 ROWS`;
`MATCH (n:${TS_MARKER}) WHERE n.id STARTS WITH $prefix AND NOT (n:Artifact OR n:ConfigKey) ` +
`CALL { WITH n DETACH DELETE n } IN TRANSACTIONS OF 5000 ROWS`;

/**
* Artifact reconciliation (`--eager` full runs). Artifacts and config keys are shared with the
* sibling analyzers, so EAGER_PURGE skips them and only TS-owned edges are dropped and rebuilt:
* HAS_ARTIFACT from a :TSApplication, DECLARES_DEPENDENCY/LOCKS to `pkg:npm/`, and incoming TS_
* edges. Not sha-diffed: siblings overwrite the shared `sha256`.
*/
export const PURGE_ARTIFACT_EDGES =
"UNWIND $pres AS pre " +
"MATCH (t) WHERE (t:Artifact OR t:ConfigKey) AND t.id STARTS WITH pre " +
"CALL { WITH t " +
"WITH [(t)<-[r]-(s) WHERE (type(r) = 'HAS_ARTIFACT' AND s:TSApplication) OR type(r) STARTS WITH 'TS_' | r] " +
"+ [(t)-[d:DECLARES_DEPENDENCY|LOCKS]->(p:Package) WHERE p.id STARTS WITH 'pkg:npm/' | d] AS rs " +
"FOREACH (r IN rs | DELETE r) } IN TRANSACTIONS OF 1000 ROWS";
/** The app root's unresolved-read/import edges, and dependency → import edges into its ghosts. */
export const PURGE_APP_IMPORT_EDGES =
"MATCH (:TSApplication {id: $appId})-[r:TS_UNRESOLVED_IMPORT|TS_READS_CONFIG_UNRESOLVED]->() " +
"CALL { WITH r DELETE r } IN TRANSACTIONS OF 1000 ROWS";
export const PURGE_PROVIDES =
"MATCH (:Package)-[r:TS_PROVIDES]->(g:TSExternal) WHERE g.id STARTS WITH $prefix " +
"CALL { WITH r DELETE r } IN TRANSACTIONS OF 1000 ROWS";
/**
* Release this analyzer's claim on artifacts/config keys the run no longer emits. An artifact is
* deleted once no HAS_ARTIFACT is left (every analyzer inventories with one, marked or not); a
* config key once no other analyzer's `*CanNode` claims it. Pre-1.5.1 artifacts carry no marker,
* so under the legacy prefix an unmarked node counts as ours.
*/
export const RELEASE_VANISHED_ARTIFACTS =
"UNWIND $pres AS pre " +
"MATCH (n) WHERE (n:Artifact OR n:ConfigKey) AND n.id STARTS WITH pre AND NOT n.id IN $keep " +
`AND (n:${TS_MARKER} OR (pre = $legacy AND none(l IN labels(n) WHERE l ENDS WITH 'CanNode'))) ` +
`CALL { WITH n REMOVE n:${TS_MARKER} ` +
"WITH n WHERE (n:Artifact AND NOT ()-[:HAS_ARTIFACT]->(n)) " +
"OR (n:ConfigKey AND none(l IN labels(n) WHERE l ENDS WITH 'CanNode')) " +
"DETACH DELETE n } IN TRANSACTIONS OF 1000 ROWS RETURN count(*) AS released";

/** `can://<app>/artifact/`, and `can://artifact/<app>/` for pre-1.5.1 pushes; both `/`-terminated. */
export function artifactPrefixes(prefix: string): [string, string] {
return [`${prefix}artifact/`, `can://artifact/${prefix.slice("can://".length)}`];
}

const isArtifactLabel = (label: string): boolean => label === "Artifact" || label === "ConfigKey";
/** The per-module purge (#140): the module by equality, its subtree by `/`-prefix; `$keys` survive. */
const PURGE_MODULE_EDGES =
`MATCH (x:${TS_MARKER}) WHERE x.id = $mid OR x.id STARTS WITH $pre MATCH (x)-[r]->() DELETE r`;
Expand Down Expand Up @@ -143,6 +188,24 @@ export async function boltWriter(
log.info(`neo4j(bolt): --eager, purged the existing graph under ${prefix}`);
}

// Artifacts survive EAGER_PURGE; a full eager run reconciles them instead. Auto-commit and
// batched, so a partial failure is finished by the next eager push.
const reconcile = fullRun && eager;
if (reconcile) {
const pres = artifactPrefixes(prefix);
const keep = rows.nodes.filter((n) => isArtifactLabel(n.labels[0]!)).map((n) => n.value);
const params = { pres, legacy: pres[1], keep, appId, prefix };
let released = 0;
await withSession(session, async (s) => {
await s.run(PURGE_ARTIFACT_EDGES, params);
await s.run(PURGE_APP_IMPORT_EDGES, params);
await s.run(PURGE_PROVIDES, params);
const res = await s.run(RELEASE_VANISHED_ARTIFACTS, params);
released = Number(res.records[0]?.get("released") ?? 0);
});
log.info(`neo4j(bolt): released ${released} vanished artifact/config node(s)`);
}

// 3. diff content_hash.
const dbHash = new Map<string, string | null>();
await withSession(session, async (s) => {
Expand Down Expand Up @@ -196,10 +259,11 @@ export async function boltWriter(
await upsertNodes(session, neo4j, nodes);
}

// 6. upsert edges owned by a changed module (owner = source node's module) or shared.
// 6. upsert edges owned by a changed module (owner = source node's module) or shared; after
// reconciliation also every edge into an artifact, which it purged.
const edges = rows.edges.filter((e) => {
const owner = moduleOf.get(e.from.value);
return owner === undefined || changed.has(owner);
return owner === undefined || changed.has(owner) || (reconcile && isArtifactLabel(e.to.label));
});
await upsertEdges(session, neo4j, edges);

Expand Down
2 changes: 2 additions & 0 deletions src/cli.ts
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@ export function buildProgram(): Command {
.option("-o, --output <dir>", "output directory (omit ⇒ compact output to stdout)")
.option("--emit <target>", "output target: json (analysis.json, default) | neo4j (graph.cypher or live push) | schema (the Neo4j schema.json contract)", "json")
.option("--app-name <name>", "logical application name for the graph :Application anchor (default: input dir name)")
.option("--workspace-root <dir>", "directory file keys, module ids and artifact ids are relative to (default: --input); must contain --input")
// The four Neo4j connection options also read the standard NEO4J_* environment variables when
// the flag is omitted (an explicit flag wins). Prefer NEO4J_PASSWORD over the flag — a flag
// value is visible in shell history / the process list. Commander renders the `(env: …)` hint.
Expand Down Expand Up @@ -146,6 +147,7 @@ export function parseArgs(argv: string[]): AnalysisOptions {
output: o.output ? path.resolve(String(o.output)) : null,
emit,
appName: o.appName ? String(o.appName) : null,
workspaceRoot: o.workspaceRoot ? path.resolve(String(o.workspaceRoot)) : null,
neo4jUri: o.neo4jUri ? String(o.neo4jUri) : null,
neo4jUser: String(o.neo4jUser),
neo4jPassword: String(o.neo4jPassword),
Expand Down
14 changes: 14 additions & 0 deletions src/core.ts
Original file line number Diff line number Diff line change
Expand Up @@ -51,8 +51,22 @@ function assertInputDir(input: string): void {
if (!stat.isDirectory()) throw new InputError(`input path is not a directory: ${input}`);
}

/**
* Published keys are relativized against the workspace root, so --input must be that root or inside
* it — otherwise a key would need `..`, which the schema forbids (relative, never absolute or `..`).
* Checked before any work, like the input itself.
*/
function assertInsideWorkspace(input: string, workspaceRoot: string | null | undefined): void {
if (!workspaceRoot) return;
const rel = path.relative(path.resolve(workspaceRoot), path.resolve(input));
if (rel.startsWith("..") || path.isAbsolute(rel)) {
throw new InputError(`--input ${input} is not inside --workspace-root ${workspaceRoot}`);
}
}

export async function analyze(opts: AnalysisOptions): Promise<AnalysisResult> {
assertInputDir(opts.input);
assertInsideWorkspace(opts.input, opts.workspaceRoot);
const log = new Logger(opts.verbosity);
log.info(`analyzing ${opts.input} (level ${opts.analysisLevel})`);
resetCheckerFailures();
Expand Down
6 changes: 6 additions & 0 deletions src/options/options.ts
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,12 @@ export interface AnalysisOptions {
emit: EmitTarget;
/** Logical application name for the graph's :Application anchor; null ⇒ derived from input. */
appName: string | null;
/**
* Directory the PUBLISHED file keys (and the module/artifact ids built from them) are relative to
* (absolute); null ⇒ `input`. Must contain `input`. Emission-only: signatures, entrypoint
* path rules, test detection and the cache keep the input-relative key.
*/
workspaceRoot?: string | null;
/** Bolt URI for a live Neo4j push (incremental). null ⇒ write a graph.cypher snapshot to -o. */
neo4jUri: string | null;
neo4jUser: string;
Expand Down
15 changes: 11 additions & 4 deletions src/schema/assignIds.ts
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,12 @@ export interface AssignedIds {
collisions: string[]; // signatures that mapped to two distinct ids (L1 id-uniqueness gate)
}

export function assignIds(app: AnalysisInternal, appName: string): AssignedIds {
/**
* @param keyPrefix prepended to every file key an id is minted from (`--workspace-root`: the
* input's path under the workspace, with a trailing `/`; `""` when unset). Ids only — the
* signature registry stays keyed on the input-relative module prefix every other pass joins on.
*/
export function assignIds(app: AnalysisInternal, appName: string, keyPrefix = ""): AssignedIds {
const appId = applicationIdOf(appName);
const idBySig = new Map<string, string>();
const typeIdBySig = new Map<string, string>();
Expand Down Expand Up @@ -70,7 +75,7 @@ export function assignIds(app: AnalysisInternal, appName: string): AssignedIds {
};

for (const [fileKey, mod] of Object.entries(app.symbol_table)) {
const moduleId = moduleIdOf(appName, fileKey);
const moduleId = moduleIdOf(appName, keyPrefix + fileKey);
const modulePrefix = modulePrefixOf(fileKey);
mod.id = moduleId;
// Module-scope execution is a call-graph SOURCE (python #131 parity: a call in module scope
Expand All @@ -86,7 +91,7 @@ export function assignIds(app: AnalysisInternal, appName: string): AssignedIds {
// language-NEUTRAL (`can://<app>/artifact/...`); dependency/import records are flat evidence
// rows with no node id of their own (the graph's :Package node is purl-keyed).
for (const [relPath, art] of Object.entries(app.artifacts ?? {})) {
art.id = artifactIdOf(appName, relPath);
art.id = artifactIdOf(appName, keyPrefix + relPath);
// Deployment-env id disambiguation (#101 unit D fix round 1, python v1.3.0 parity verbatim):
// the `key` FIELD always stays the bare variable name — env-namespace resolution still joins
// on a plain `key ===` match — but a bare name can collide across mints on the SAME artifact
Expand All @@ -106,7 +111,9 @@ export function assignIds(app: AnalysisInternal, appName: string): AssignedIds {
const artPath = dep.declared_in; // scanners record the REL PATH; re-stamp onto the id
// slice(4) drops `can:`, ``, `<app>`, `artifact` — the same depth the old `can://artifact/<app>/`
// shape had, so the reorder left this re-stamp guard alone.
dep.declared_in = artifactIdOf(appName, artPath.startsWith("can://") ? artPath.split("/").slice(4).join("/") : artPath);
dep.declared_in = artPath.startsWith("can://")
? artifactIdOf(appName, artPath.split("/").slice(4).join("/"))
: artifactIdOf(appName, keyPrefix + artPath);
}

return { appId, idBySig, typeIdBySig, callableBySig, collisions };
Expand Down
48 changes: 45 additions & 3 deletions src/schema/emit.ts
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@
import * as path from "node:path";
import type { Project } from "ts-morph";
import type { AnalysisOptions } from "../options";
import { relPosix } from "../utils/fs";
import { ANALYZER_VERSION } from "../utils/version";
import type { AnalysisInternal, TSAnalysis, TSApplication } from "./schema";
import type { ProgramGraphs } from "./graphs";
Expand Down Expand Up @@ -72,6 +73,37 @@ function stripInternal(root: TSApplication): void {
}
}

/**
* `--workspace-root`: the input's path under the workspace root, `/`-terminated, prepended to every
* PUBLISHED file key; `""` when unset or equal to the input. Emission-only — the live tree, the
* signatures, entrypoint path rules and the cache all keep the input-relative key.
*/
function workspaceKeyPrefix(opts: AnalysisOptions): string {
if (!opts.workspaceRoot || !opts.input) return "";
const rel = relPosix(opts.workspaceRoot, opts.input);
return rel === "" ? "" : `${rel}/`;
}

/**
* The wire's other path-keyed maps, re-keyed under `keyPrefix`: artifacts (map key + `path`) and the
* synthesized callables' owning-module `path`. Ids were already minted with the prefix by
* assignIds; this only renames. New objects, so the live tree keeps its input-relative paths.
*/
function rekeyRepoPaths(root: TSApplication, keyPrefix: string): Partial<TSApplication> {
if (!keyPrefix) return {};
const artifacts: TSApplication["artifacts"] = {};
for (const [key, art] of Object.entries(root.artifacts)) artifacts[keyPrefix + key] = { ...art, path: keyPrefix + art.path };
const out: Partial<TSApplication> = { artifacts };
if (root.synthesized_callables) {
const synthesized: NonNullable<TSApplication["synthesized_callables"]> = {};
for (const [id, sc] of Object.entries(root.synthesized_callables)) {
synthesized[id] = sc.path === undefined ? sc : { ...sc, path: keyPrefix + sc.path };
}
out.synthesized_callables = synthesized;
}
return out;
}

// ----------------------------------------------------------------------------------------------
// entry point
// ----------------------------------------------------------------------------------------------
Expand All @@ -95,9 +127,10 @@ export function finalizeAnalysis(
): AnalysisResult {
const level = opts.analysisLevel;
const appName = (opts.appName ?? (opts.input ? path.basename(opts.input) : "") ?? "").trim() || "app";
const keyPrefix = workspaceKeyPrefix(opts);

// L1 — stamp ids, derive body{}, project heritage (all overwrite-idempotent per-run passes).
const { appId, idBySig, typeIdBySig, callableBySig, collisions } = assignIds(app, appName);
const { appId, idBySig, typeIdBySig, callableBySig, collisions } = assignIds(app, appName, keyPrefix);
populateL1Body(app);
// `extends`/`implements` name TYPES: on a merged name (#177) the type facet's id wins over the
// value facet's, which is what `idBySig` holds for every other pass.
Expand Down Expand Up @@ -169,8 +202,17 @@ export function finalizeAnalysis(
// strip and shared by reference. A stringify roundtrip is out for the same reason: the string
// form OOMs at vscode-L4 scale.
const symbol_table: TSApplication["symbol_table"] = {};
for (const [key, mod] of Object.entries(root.symbol_table)) symbol_table[key] = structuredClone(mod);
const application: TSAnalysis = { ...envelope, application: { ...root, symbol_table } };
for (const [key, mod] of Object.entries(root.symbol_table)) {
const wire = structuredClone(mod);
// `resolved_module` names a symbol_table key, so it takes the same prefix (on the clone only).
if (keyPrefix) {
for (const b of [...(wire.imports ?? []), ...(wire.exports ?? [])]) {
if (b.resolved_module !== undefined) b.resolved_module = keyPrefix + b.resolved_module;
}
}
symbol_table[keyPrefix + key] = wire;
}
const application: TSAnalysis = { ...envelope, application: { ...root, symbol_table, ...rekeyRepoPaths(root, keyPrefix) } };
stripInternal(application.application);
return { application, internal: app, ...(pg ? { program_graphs: pg } : {}), idBySig, collisions, dangling };
}
Loading
Loading