Discipline (05-cost-model.md, whitepaper C6): a number is an assumption until measured. Every figure in the generated section below came from an actual run of
npm run benchon the machine recorded in the environment block — no projections, no targets, no numbers copied forward from a different machine. Re-runnpm run bench(≈10 s, node stdlib only) and the generated section is rewritten in place with your machine's numbers.
- Median of N runs after warmup. Each row states its own
runs; warmup runs (1–3 per row) are executed and discarded so JIT compilation and cold module state don't pollute the samples. p95 is nearest-rank — with fewer than 20 samples that is simply the max, reported as such rather than smoothed. - Timing is
performance.now()around the call, single process, no concurrency. - Repo-scale benchmarks (atlas, context, substrate, impact quality) run against a copy
of this repo in
os.tmpdir, excludingbench/(so the harness's own imports don't perturb the impact-quality eval) and dot-directories (.git,.forge— the copy starts cold). The copy is deleted afterwards. - Synthetic fixtures (ledger, reuse) are generated with a seeded PRNG (mulberry32,
fixed seeds) — byte-identical fixtures on every run and machine — built in
os.tmpdirand cleaned up.
- atlas / full build: directory walk + read + regex extraction + edge resolution +
writing
.forge/atlas.jsonand the per-file cache, on this repo's file count (recorded in the row's notes). Repo-shape-specific; not comparable across repos. - atlas / incremental rebuild (unchanged): by design the incremental path still reads and re-hashes every tracked file — the saving over a full build is the skipped regex extraction only. This row is that claim, measured.
- atlas / impact query: marginal per-query latency with the memoized reverse-adjacency
index already built (it is built once per atlas and cached —
ADJ_CACHEinsrc/atlas.js). The first query on a fresh atlas additionally pays that index build. - ledger / mint+put:
mintClaim+putClaimper claim (synchronous file write each), with one evidence append every 4th claim. Disk-bound; the throughput figure is claims/sec at the row's median. This is the noisiest row in the file: on a shared or virtualized disk, back-to-back invocations of the whole bench have produced medians from ~150 ms to ~830 ms for this row (4–6×) while the CPU-bound rows moved by percents — read it as "thousands of claims/sec, I/O-dominated", not as a stable constant of the code. - ledger / mergeDirs: replica directories are pre-copied outside the timing; the
number is
loadStateof both sides + semilattice union + idempotent re-puts + reindex. - ledger / val(): pure in-memory scoring; the fixture gives most claims 0–1 evidence records, and val() cost scales with evidence count — a heavily-evidenced ledger will be slower per claim.
- reuse / lookup: the memoized
_sketchcache is stripped before every timed run, so each run behaves like a fresh CLI process. The exact tier returns before any pool sketching (normalized-string compare); the near tier pays MinHash-sketching the whole candidate pool plus LSH banding — that difference is the point of reporting both. - context / assemble(): warm atlas, empty ledger (the repo copy has no
.forge), includes the real file reads for pinned items. Task: a three-symbol, one-file edit spec. - substrate / substrateCheck: the whole deterministic gate — preflight grounding,
routing rubric, up to 8 impact queries, reuse lookup, context assembly, scope
decomposition, lessons, minimality, goal anchor — with
llm: false. No model latency is included anywhere in this file; with LLM adjudication enabled, wall time is dominated by the model call, which is exactly why it ships opt-in.
Micro-medians below ~1 ms are subject to GC/JIT jitter even after warmup; treat them as order-of-magnitude, not three-significant-digit truths.
Cases live in bench/impact_cases.mjs, scored by
evalImpact() (src/eval.js). Labeling rule: expected = the defining file plus every
file with a direct reference (an import { X } — static, dynamic or aliased — with a
use, or a call site) — each one listed, per file and per line, in the fixture's comments.
The labels are ground truth, not the graph's own output: every one is re-derived with
git grep -n -w -F -e <symbol> -- 'src/*' 'test/*' and confirmed by reading each hit, and a
name that appears only in a comment or inside a string (an assertion message, a doc line) is
not a reference and is not labeled — the omissions are listed too, so a re-check can tell
"deliberate" from "missed".
Transitive dependents are not labeled, so the oracle's transitive predictions count
against precision — the same over-approximation penalty the paper's mutation-derived
scoring applied. That penalty is now most of the number. impact() walks reverse
dependencies transitively by default (maxHops: 6), so for contentHash it returns 89
files where 10 are directly labeled. Restricted to one hop it returns exactly the 10
labeled files plus one documentation edge — i.e. the precision figure below measures the
gap between "everything downstream" and "the direct referencers", not a graph that is
wrong about who calls what. Read precision here as how much wider than the direct set the
default answer is, and recall as does it ever miss a direct referencer (it does not).
contentHash used to carry a genuine false negative — src/atlas.js binds it to an alias,
const hash = contentHash;, with no call parentheses, and the old import regex captured
module paths rather than named bindings, so nothing reached atlas.js. Import specifiers
now resolve to the exact symbol (src/atlas.js:17 imports → src/util.js:contentHash:65) and
atlas.js is predicted at one hop; the case is kept for its fan-out, not for the miss.
What these numbers do not mean: n = 6 cases, one JavaScript repo, symbols chosen to be uniquely named (the atlas resolves ambiguous names to nothing — a separate, known limitation). They are not comparable to the paper's numbers, which came from mutation testing a Python codebase against a real test suite. The two appear side by side below, labeled, and are never blended.
History of this row. The precision 0.90 / F1 0.92 this file carried until 2026-09-21 came from a much smaller atlas (145 files) and a reverse walk that stopped at the direct referencers. Two things changed since:
impact()now walks reverse dependencies transitively by default, and four of the six label sets had gone stale against the source. Re-labelling every case fromgit grepand re-runningnpm run benchgives the generated table below — precision 0.17, recall 1.00, F1 0.29 (edited-file-only baseline recall 0.27), where the precision loss is the transitive closure being scored against direct-only labels, not a graph that is wrong about who calls what (at one hop the six cases return their labeled sets). Neither figure is field evidence: on nine real Python repositories the paper's prototype oracle reached recall 0.022 (research/empirical-refutation/), and its "recall 1.00" row below comes from five mutations of its own demo package. The generated block is rewritten only bynpm run bench.
{
"node": "v24.19.0",
"cpu": "AMD EPYC Processor (with IBPB)",
"cores": 4,
"memGB": 8,
"platform": "win32",
"arch": "x64",
"commit": "703da31d574c30d22bef019b1c8563ade0d0d6be",
"date": "2026-09-21T22:09:58.366Z"
}| suite | benchmark | median | p95 | runs | notes |
|---|---|---|---|---|---|
| atlas | full build (this repo) | 530 ms | 622 ms | 5 | 455 files, 10498 symbols, 29728 edges |
| atlas | incremental rebuild (unchanged) | 339 ms | 359 ms | 5 | per-file hash cache hit |
| atlas | impact("claimText") (warm adjacency) | 0.40 ms | 1.08 ms | 30 | 51 files impacted |
| ledger | mint+put 1000 claims | 1854 ms | 1986 ms | 5 | 539/s |
| ledger | loadClaims at 1000 claims | 213 ms | 230 ms | 5 | full state from disk |
| ledger | mergeDirs 2×500-claim replicas (250 shared) | 4308 ms | 4409 ms | 3 | +250 claims, +313 records |
| ledger | val() over 1000 claims | 0.076 ms | 0.16 ms | 20 | 13,140,604/s (mean val 0.51) |
| reuse | fingerprint 2000 specs | 116 ms | 156 ms | 5 | 17,171/s |
| reuse | lookup exact @ 100 artifacts | 5.71 ms | 10.5 ms | 10 | tier=miss |
| reuse | lookup near (LSH) @ 100 artifacts | 4.76 ms | 5.18 ms | 5 | tier=miss, j=- |
| reuse | lookup exact @ 1000 artifacts | 52.8 ms | 88.5 ms | 10 | tier=miss |
| reuse | lookup near (LSH) @ 1000 artifacts | 46.7 ms | 89.8 ms | 5 | tier=miss, j=- |
| context | assemble() (this repo, 3-symbol task) | 12.6 ms | 30.0 ms | 10 | 4070/6000 tokens, 9 required, complete |
| substrate | substrateCheck (allowBuild, llm off) | 886 ms | 908 ms | 3 | 99 impacted files, route simple |
| case (target) | precision | recall | F1 | predicted | truth |
|---|---|---|---|---|---|
| normalizeSpec | 0.12 | 1.00 | 0.21 | 17 | 2 |
| evalImpact | 0.29 | 1.00 | 0.44 | 7 | 2 |
| isStale | 0.20 | 1.00 | 0.33 | 30 | 6 |
| mergeStates | 0.16 | 1.00 | 0.28 | 25 | 4 |
| claimText | 0.16 | 1.00 | 0.27 | 51 | 8 |
| contentHash | 0.11 | 1.00 | 0.21 | 87 | 10 |
| mean of 6 | 0.17 | 1.00 | 0.29 |
Edited-file-only baseline recall over the same cases: 0.27.
Two methodologies, side by side — different codebases, different ground-truth derivations, so the rows are comparable in spirit only and are never blended:
| series | precision | recall | F1 | ground truth |
|---|---|---|---|---|
| paper prototype (Python, mutation-derived) | 0.63 | 1.00 | 0.75 | mutation testing against a real suite |
| this repo (regex atlas, hand-labeled) | 0.17 | 1.00 | 0.29 | 6 hand-labeled cases (bench/impact_cases.mjs) |
Snapshot boundary. The measured results above were generated at commit
eb68ea9and do not benchmark the optional embedding adapter now implemented insrc/embed.jsand exercised with a deterministic fake provider intest/embed.test.js. MinHash remains the zero-dependency default and failure fallback. The structural comparisons below describe the current source; they do not imply that Forgekit bundles an embedding model or vector database.
Adjacent tools per ecosystem_map.md. Every row is a structural claim checkable from the named spec/source — no adjectives, no multipliers. The last row of each table points the other way: what the adjacent tools do that forgekit structurally does not.
| structural property | forgekit PCM ledger | note stores |
|---|---|---|
| confidence moved only by independent oracles | yes — the ORACLES table in src/ledger.js is the closed set of who may move confidence; evidence without a verifiable ref is rejected (outcomeRecord) |
no — notes are stored as written; no oracle taxonomy exists |
| stored evidence weight distrusted | yes — val() re-reads weight from the ORACLES table, never the stored record; verify() flags a recorded weight that disagrees with the table |
n/a — there are no evidence records to forge |
| retrieval ranks by verified validity | yes — Eq. 3 score includes a g·val term (EQ3_WEIGHTS, score() in src/ledger.js) |
no — similarity and/or recency only |
| unreviewed knowledge decays toward uncertainty, not deletion | yes — time-decayed Beta posterior pulls val back to the 0.5 prior; below 0.35 a claim goes dormant but is kept for audit (DORMANT_VAL) |
no — a note persists unchanged until manually deleted or compacted |
| conflict-free team merge | yes — claim bytes are a pure function of (kind, body, scope); logs are hash-deduped unions; merge is a join-semilattice (mergeStates, property-tested), with a merge=union gitattributes rule |
no — per-machine SQLite or a hosted store; no CRDT merge contract |
| self-confirmation cannot buy trust | yes — authorTrust() excludes an author's own evidence on their own claims |
no equivalent mechanism |
| secrets refused at write time | yes — SECRET_RE enforced at both mintClaim and putClaim |
not a protocol invariant |
| what the note stores have that forgekit doesn't | — | hosted sync, web UI, a bundled embedding/index service, and LLM summarization pipelines; forgekit stores its ledger as files in git and can call an optional user-supplied embedding provider, with MinHash as the default and fallback |
| structural property | forgekit forge route |
LLM gateways |
|---|---|---|
| routing decision visible before dispatch | yes — returns band, signals, and per-signal reasons the user can read and override (src/route.js) |
decision is made inside the proxy at request time |
| rubric versioned in the repo | yes — deterministic scoring over src/model_tiers.json, diffable in PRs |
routing/cost logic lives in gateway config or the provider's service |
| same input ⇒ same route | yes — the exemplar k-NN rubric is deterministic (same text, same neighbors, same score); LLM adjudication is opt-in and clamped inside band rails (a proposal can never jump past them) | depends on gateway load/cost/failover state |
| what the gateways have that forgekit doesn't | — | they actually move traffic: proxying, failover, quotas, key management. forge route is advisory and at most emits a LiteLLM config exposing its tiers as aliases (src/route.js) — it is a transparency layer over gateways, not a replacement |
| structural property | forgekit reuse cache | plain RAG |
|---|---|---|
| serving gated on verification evidence | yes — SERVE_FLOOR = 0.6 (src/reuse.js); a fresh, unverified mint sits at the 0.5 prior and is not served, whatever its similarity |
serves on similarity alone |
| retrieved code revalidated against the current code graph | yes — revalidate() checks every declared dep still resolves in the atlas before a hit is served |
no dependency contract on retrieved chunks |
| cache demotes itself on ground truth | yes — a failed revalidation appends graph.reval contradict evidence to the ledger (reuseQuery), and the demotion reaches teammates through the merge |
index reflects content until re-embedded; retrieval outcomes don't feed back |
| explicit hit tiers with committed thresholds | yes — exact / near (J ≥ 0.8) / adapt (J ≥ 0.6) ladder with LSH banding (NEAR_J, ADAPT_J, bandKeys) |
top-k cosine; thresholds are informal per deployment |
| what RAG has that forgekit doesn't | — | document ingestion and chunking, a managed or persistent vector index, citation assembly, arbitrary-corpus retrieval, and hosted scaling; Forgekit's optional embedding adapter improves similarity for its curated claims and artifacts, but it does not provide those RAG pipeline components |
npm run bench # ≈10 s; prints the tables and rewrites the generated section above
npm test # includes a smoke test of the harness's pure helpers (test/bench.test.js)