Spec: Tracer Bullet 01 — document to cited answer #30

This is now the single tracker issue for Tracer Bullet 01. It was originally the /to-spec sibling of a hand-authored map issue, #15; the two restated the same architecture, assertions and decisions in two places, so #15 has been closed and its 14 sub-issues (NU-001…NU-014, #16–#28, #31) reparented here. #15's body and comment thread (including the fixture-corpus source text) are preserved for history, just no longer the entry point.

Synthesizes docs/tracer-bullet-01.md's design record into the /to-spec template shape, so the frontier (NU-001…NU-014, after reconciliation) has one spec to build against instead of thirteen independently-scoped tickets. Nothing here amends the PRD or docs/tracer-bullet-01.md — both stay authoritative; this is the same decisions, organized for build-and-test rather than for
rationale.

Problem Statement

Nuron is supposed to be a self-hosted "company brain": documents go in, a graph of Decisions and the entities they touch comes out, and a question gets answered with a citation someone can check. Nothing has been built yet, and the PRD that was supposed to found the build has three Tier-1-blocking contradictions in it (nuron-api's network exposure, sync-SSE-vs-async-RabbitMQ,
and which of two agents answers a query) plus a write-side hole where the persistence requirement
(FR-5) needs an author field that appears in neither of the two document schemas that precede
it. Before any of the thirteen production containers get built, there's a narrower question that
has to be answered first: does persisting Decisions into a property graph and retrieving through
it actually produce a cited answer that a flat chunk store could not have produced — and does
letting a human confirm entity merges actually make the system's matching get better over time, or
is that just a story people tell themselves about graphs?

If the answer turns out to be no — if a control run with graph traversal disabled passes just as
well — building five more containers on top of Neo4j would be effort spent proving nothing.

Solution

Build one true vertical slice, end to end, for a single user, against a small hand-built fixture
corpus designed so that the answer to the one query under test is only reachable by traversing the
graph — not by vector similarity alone. A document lands (watched directory or upload) → the
original is preserved byte-for-byte in content-addressed object storage → format extraction
converges every supported type on Markdown → a deterministic (non-LLM) parser extracts a Content
Header → every file blocks for a human review and approval, during which the reviewer may edit
body content, not just metadata → an LLM Compiler emits typed, schema-validated triples with an
explicit decisions[] → a second human gate confirms same-referent entity merges (auto-joins
are shown as information, never asked) → the result persists into a Neo4j property graph with
embeddings on the nodes and refcounted provenance → a query retrieves through HNSW-entry-then-
graph-expand and streams a cited answer over SSE.

Twelve assertions (A1–A12, reproduced in Implementation Decisions below) define what "done" means.
A3 is a deliberate negative control: the same query with graph expansion disabled must degrade or
fail, or the entire exercise has told us nothing. RabbitMQ is deliberately absent — the review
queue, backed by a Postgres row claimed with FOR UPDATE SKIP LOCKED, is the work queue, so a
broker would be a second copy of the same state that can disagree with the table about what stage
a document is in.

Six containers realize this: nuron-web (SvelteKit), nuron-api (Laravel), nuron-ai
(Python/LlamaIndex), Postgres (two schemas, two roles, no cross-schema grants), Neo4j (graph +
HNSW vectors), RustFS (S3-compatible object storage for originals).

User Stories

1. As a feeder, I want to drop a Markdown/text/docx/pdf file into a watched directory, so that it enters the pipeline without any manual trigger.
2. As a feeder, I want to upload a document directly instead of using the watched directory, so that I don't need filesystem access to the host running Nuron.
3. As a feeder, I want the same document to produce exactly one record whether I dropped it in the directory or uploaded it, so that I never end up with duplicate copies feeding the graph.
4. As a feeder, I want my original file bytes preserved unmodified in object storage, so that the source of truth survives even if every later processing step is wrong.
5. As a feeder, I want a write to object storage verified by reading it back and re-hashing it, so that a silent storage failure doesn't cost me the only copy of a file I uploaded.
6. As a feeder, I want the system to identify my document by a hash of its content rather than a random ID, so that re-adding the same document can never create a duplicate.
7. As a feeder, I want a file that's still being written to be left alone until its modification time is stable, so the pipeline never reviews a half-written file.
8. As a feeder, I want re-scanning a file I haven't touched to do nothing at all — no re-extraction, no re-parse, no review item, no LLM call — so the pipeline doesn't burn a human review slot or a model call on a no-op.
9. As a feeder dropping a .docx or .pdf, I want it converted to the same common Markdown form as a .md file, so one deterministic parser handles every format identically.
10. As a feeder, I want PDF extraction (a third-party cloud call) off by default, so my documents never leave the network to an external service unless I explicitly turn it on.
11. As a feeder whose PDF is a flat scanned image, I want extraction to fail loudly rather than silently produce a near-empty review item, so I notice the document was never actually read.
12. As a reviewer, I want every single file to stop for my review before anything downstream happens, so nothing reaches the graph without a human having looked at it.
13. As a reviewer, I want to edit the body content of a document, not just its header metadata, so I can fix extraction or parsing mistakes before they become permanent graph content.
14. As a reviewer, I want the deterministic parser to have already tried to fill in Subject, Author, Date and Tags — from frontmatter, then an in-prose signature, then a filename date — so I'm correcting instead of authoring from a blank page.
15. As a reviewer, I want to know whether an author name came from the document itself or from a configured fallback, so I can trust a citation that says "extracted" more than one that says "default".
16. As a reviewer, I want my approval to freeze an immutable, versioned copy of the document, so that every future citation resolves to content a human actually vouched for.
17. As a reviewer, I want the original raw file still reachable one hop back from my approved version, so I can check the true source when I need to, without it being what a query answer shows by default.
18. As a reviewer, when I re-review a changed version of a document I already approved, I want to see a diff against the raw version I approved before, alongside my own prior edits, so I can re-apply what I fixed instead of starting from scratch.
19. As a reviewer, I want the system to remember exactly which stage a document is at if I close my browser mid-review, so I can resume later without redoing or losing work.
20. As an asker, I want a query answer to cite a specific Decision node — its author, its timestamp, and the evidence it resolves to — so I can verify the answer instead of taking the system's word for it.
21. As an asker, I want a citation to resolve to the reviewer-approved version of a source, not the raw upload, so what I'm shown as evidence is content a person actually checked, with the raw original only a deliberate second click away.
22. As an asker, I want an answer that depends on information spread across more than one document to actually be reachable across documents, so the system is doing more than returning whichever single file is most similar to my question.
23. As an asker, when nothing in the graph matches my question, I want to be told that explicitly, along with the nearest related entities, so I never receive a confidently fabricated answer.
24. As an asker, I want my answer streamed to me as it's generated, so I'm not staring at a blank screen while the model runs.
25. As a reviewer, I want to be shown a short list of merge candidates when a new document might refer to something already in the graph under a different name, so I can confirm or reject the match myself.
26. As a reviewer, I want every merge candidate to default to "not the same" unless I actively confirm it, so a moment of fatigue can never silently merge two things that aren't the same.
27. As a reviewer, I want to see — as information, not as a question — when the system has already joined two documents' references to an identically-named thing, so I'm not asked to confirm what the name alone already made obvious.
28. As a reviewer, once I confirm that two differently-named things are the same, I want that decision remembered as an alias, so the next document using either name never asks me again.
29. As an operator, I want the persistence layer to keep a reference count of which approved documents contributed to each graph node, so removing one document's content can never delete something another document still depends on.
30. As an operator, I want re-approving an edited document to touch only the graph content that document actually contributed, so unrelated nodes are never re-embedded or re-touched.
31. As an operator, I want a startup check that the configured LLM endpoint actually returns valid structured output, so a bad endpoint fails loudly at boot instead of producing malformed triples at 3am.
32. As an operator, I want the embedding model's identifier stored on every node it produced, so a partial migration to a different model or dimension count is detectable instead of silently mixed into the same index.
33. As an operator, I want nuron-api and nuron-ai backed by separate database roles that cannot see each other's schema, so the one-way service boundary is enforced by the database, not just by convention.
34. As an operator, I want the review queue itself to be the work queue, with no separate broker, so restart-safety comes from one committed table rather than keeping two systems in sync.
35. As an evaluator of this tracer bullet, I want a control run with graph expansion disabled, so I can tell whether the graph actually contributed to the answer or vector search alone would have found it anyway.
36. As an evaluator, I want a fixture corpus specifically designed so the query under test is answerable only by traversing an edge, not by similarity to a single document, so a passing result means the graph did something a chunk store could not.
37. As an evaluator, I want a PDF fixture carrying the same Decision as the primary Markdown fixture, so the format-extraction seam is proven without letting a third-party service outage invalidate any of the graph-behavior assertions.

Implementation Decisions

Architecture

Six containers, arrows one-way: nuron-web → nuron-api → nuron-ai → {Postgres, Neo4j, RustFS}.
nuron-ai never calls upward. nuron-web and nuron-api bind to host loopback only
(127.0.0.1) — tightened from a plain host binding since this slice has no auth; nuron-ai,
Postgres, Neo4j and RustFS are internal-only. (This resolves the PRD's F-1: §5.4's "nuron-api is
internal-only" contradicted three other sections that assume external/direct API callers.)

Schema isolation between nuron_api and nuron_ai (both in one Postgres instance) is enforced by
separate DB login roles with no cross-schema grants — not by convention. If nuron-api could
SELECT from nuron_ai tables it eventually would, and the one-way layering becomes a comment.

Two agents in v1: the Compiler (LLM) and the query agent (LLM). The parser and format
extractors are deterministic code, not agents. The Curator is out of scope for this slice.
"Default Agent" is retired as a name for this slice — the ingest pipeline is a state machine, not
an agent loop. (Resolves PRD F-2/F-3: "reply" is redefined as v1.1 write-back, distinct from query
answering, which is synchronous end-to-end with no RabbitMQ in the path.)

Pipeline state machine

One Postgres table (nuron_ai schema) drives everything:

landed → extracted → parsed → awaiting_review → content_approved → compiled →
awaiting_merge_confirm → persisted

extracted is a distinct state because LlamaParse (the PDF path) is a network call with its own
failure/retry semantics. Both awaiting_review and awaiting_merge_confirm are real persisted
rows, not UI-only steps: a reviewer may close the browser between them, and the pipeline must
resume exactly where it paused. The row is claimed with SELECT ... FOR UPDATE SKIP LOCKED — this
table is the work queue, which is why there is no RabbitMQ.

Identity, naming, and deduplication

• Document identity is sha256(bytes); the digest is also the RustFS object key
  ({sha256[0:2]}/{sha256}), never a UUID — a UUID key would let the same content arrive by two
  paths and store twice.
• A watched-directory write and an uploaded write of the same bytes must converge on one Landing
  Zone record and one object (A12).
• Entity identity across compiles is a natural key: normalize(name) + label, extended by
  human-confirmed aliases. This deliberately under-merges — "session store" and "sessions table" stay distinct nodes until a human confirms they're the same. No embedding-similarity
  auto-merge threshold in this slice; that needs corpus volume to calibrate (PRD FR-3's
  near-duplicate consequence is partially satisfied: identical names auto-join, different names
  need a human).
• Content Header parsing precedence: frontmatter → in-prose signature → filename date. File
  mtime is never consulted for this — mtime is the file's date, not the decision's. Anything
  unmatched is left blank for the reviewer, never guessed.
• author_source is recorded as extracted | default | unknown alongside author, and the
  config value supplying the fallback is named source_owner — never author — so a citation
  can never silently claim a person said something a config file actually asserted.

Human review gate 1 (content approval)

• Every file blocks here, no exceptions. The reviewer may edit body content, not only metadata.
• On approval, a Reviewed Source version is frozen — immutable, versioned. This, not the raw
  file, is the Evidence root: evidence_span offsets resolve into it. If reviewers edit the
  body, offsets into the raw file would resolve intermittently — worse than never. The raw
  original stays reachable one hop back (two hops for .pdf/.docx, since extracted text sits
  between); "view original" is a deliberate second action, never the default.
• Re-ingest of changed content shows a raw-v1→raw-v2 diff alongside the reviewer's prior edits;
  they reapply manually. No three-way merge in this slice — flagged as the workflow most likely to
  determine whether reviewers actually engage or start rubber-stamping.

Compiler

• SchemaLLMPathExtractor, strict=True, Pydantic-validated typed triples:

  possible_entities   = Literal["DECISION", "ENTITY", "EVIDENCE"]
  possible_relations  = Literal["SUPERSEDES", "EVIDENCED_BY", "AFFECTS", "DEPENDS_ON", "PART_OF"]

• Output is structured, with an explicit decisions[] — not a Markdown document. (This
  resolves the write-side hole under PRD §3/FR-5: the four-section LLM-Wiki schema had no
  Decisions section, and author appeared in neither upstream schema.)

• The extractor does not carry author, author_source, timestamp, evidence_span, or
  provenance refs — those originate in the parser, which the extractor never sees. A custom
  extractor subclass or a post-extraction enrichment step must stamp them before insert_nodes().

• LLM is OpenAILike (any OpenAI-compatible endpoint). A startup check must verify structured
  output actually works on the configured endpoint before accepting traffic — fail loudly at boot.

• Do not use PropertyGraphIndex.from_documents() — it chunks→extracts→persists in one shot; this
  pipeline splits that across two human gates, so the extractor and insert_nodes() are driven
  directly.

Persistence & provenance

• Neo4jPropertyGraphStore, embed_kg_nodes=True — embeddings live on the graph nodes, not
  in pgvector. One store, one identifier, for both the HNSW entry point and the graph expansion.
• Provenance is a set of refs per node, refcounted. Re-approving a document replaces only that
  document's slice of the set; a node is deleted only when its last ref is released. (This is the
  reference-counting behavior a single-document fixture would never exercise — an entity shared
  by two source documents must survive either one's removal.)
• plan_delta(prior, current) → add | update | unchanged | drop_ref; unchanged content is never
  re-embedded — that's where embedding spend is saved.
• The embedding model id is stored on every node, so a partial migration is detectable.
• One-way door: embedding dimensions (1024, the PRD §5.2 floor) are baked into the Neo4j
  vector index. Changing model or dimensions later is a re-embed-and-rebuild migration, not a
  config edit.

Human merge gate 2 (entity resolution)

• Two candidate signals: normalized-name overlap auto-joins (shown as information, never
  asked — this is what makes the cross-document traversal path exist from compiler output alone,
  with no human link step); cosine similarity on differently-named entities is a question.
• The human confirms merges ("is X the same thing as Y?"), never typed links — link creation
  (a directed, typed edge vocabulary) is deferred; possible_relations is a closed Literal and
  getting it wrong from a five-document fixture would bake in a wrong vocabulary. Every "I want to
  link these but there's no type" moment should be logged to design that vocabulary later from
  evidence, not imagination.
• On confirm: rewrite every inbound/outbound edge of the losing node onto the survivor, union the
  two provenance ref-sets, then delete the losing node — retired, not left as a zero-edge orphan.
  Not one transaction — the alias lives in Postgres, the node/edge mutation in Neo4j, no
  cross-store transaction exists in this design. Write the Postgres alias first (ON CONFLICT DO NOTHING), then the Neo4j rewrite-and-delete: a crash between them leaves temporary graph
  clutter but a new document under the losing name already resolves correctly via the alias; the
  reverse order risks a genuine duplicate node instead. This is the one part of the system that
  improves monotonically from human input — no threshold, no tuning, no regression risk.
• Overload guards: cap candidates at 5, gate on a precision bar (§ Configuration), default every
  suggestion to "not the same," and suppress anything the natural key already auto-joined.

Retrieval & query

• VectorContextRetriever(store, similarity_top_k=k, path_depth=2) wired via
  index.as_retriever(sub_retrievers=[vector_retriever]) — HNSW entry, two hops of graph
  expansion (Decision → session store → rate limiter is two edges from the vector hit). Pin
  llama-index==0.14.23 (llama-index-core==0.14.23) so path_depth stays "hops after the
  vector hit." This single configured class is the retrieval design for this slice. Queries
  must filter to the currently configured embedding model id/dimension (ADR-0004) — a stale vector
  from an interrupted migration would otherwise silently corrupt the similarity score.
• No BM25/Tags, no RRF, no LLMRerank yet — RRF has nothing to fuse until a second genuine ranked
  list exists; LLMRerank can mask a broken retriever, so it's deferred until retrieval is proven
  sound by A1/A3.
• No matching Decision → an explicit "no matching decision" response citing the nearest entities.
  Never a fabricated answer.
• Query answering is synchronous, end-to-end: web → api → ai → retrieve → generate → stream
  over SSE. No broker, no async bridge — this is the resolution of PRD F-2/F-3 (sync SSE vs.
  async-only dispatch; two agents both claiming to realize the same journey).

Assertions (acceptance criteria for this slice)

Fixture: four Markdown files (A the decision, in-prose-attributed; B an entity doc that
never mentions the decision; C plausible noise; D the alias case) plus E, a one-page
PDF carrying A's decision. Query under test: "what did the auth change affect?" — answerable
only via Decision(A) → Entity(session store) → Entity(rate limiter), i.e. only through B,
reachable only by traversal.

A5, A6, A7 were previously claimed in full by two stories each with no qualifier distinguishing a
unit-level guarantee from the end-to-end one; A10's split was asymmetric. Reconciliation (tracked
against #15) relabels these so each has exactly one end-to-end owner plus explicitly-named
contributing stories, following the pattern A4/A12 already used correctly.

Configuration

PRD statements this supersedes (pointer)

Full table with rationale lives in docs/tracer-bullet-01.md §5. Highlights: §5.4's Docker-internal-only claim for nuron-api (F-1); FR-10's Default-Agent-handles-reply framing (F-2/F-3); FR-3/§3's LLM-Wiki-as-Markdown-document schema (the author/Decisions write-side hole); FR-4's RabbitMQ-is-required NFR; FR-7's Merkle-subtree conditionality (moot for the write path — provenance refs give the touched set directly; the Merkle question survives only for scheduled re-curation, out of scope here).

Testing Decisions

Seam: one black-box seam around the nuron-ai pipeline state machine. Drive it by dropping
fixture files into the watched directory or POSTing them to the upload endpoint; simulate both
human gates via direct calls, not a browser; assert against the Postgres documents/
reviewed_sources state, the Neo4j graph, and the SSE query response. nuron-web (the reviewer
and query UI) is exercised manually — it carries no automated assertions in this slice; its story
is ready-for-human because reviewer-fatigue affordances are judgment calls, not spec.

What a good test looks like here: asserts on the four things a caller can actually observe —
Postgres pipeline state, Neo4j node/edge/property content, the object stored in RustFS, and the
streamed query response — never on LlamaIndex internals or prompt text. A3 (the path_depth=0
control) is not optional polish; a suite that never runs it cannot distinguish "the graph worked"
from "vector search alone would have worked," which is the entire point of building this slice
before the other eight production containers.

Fixture corpus ownership: the corpus (fixtures/watched/{A,B,C,D}.md + PDF E) is a
dedicated deliverable (new story, inserted after NU-001 in the reconciled frontier) — not a
byproduct of any one story's unit tests — because nine of the twelve assertions depend on files
that don't exist anywhere yet except as text in a comment on #15, and the fixture's constraints
(B must never mention JWT/sessions/the decision; C must be plausibly similar, not obviously
unrelated; A's attribution must be in-prose, not frontmatter) are exactly what make A1 and A2 test
anything rather than passing by accident.

Unit-level tests (deterministic core: content_hash, parse_header, normalize,
natural_key, plan_delta, release_ref): pure functions, no network, no LLM, no fixture corpus
needed — these should be the fastest, most numerous tests in the slice, per NU-002/#17's own
"21 tests green" from its (deleted) earlier draft.

Prior art: none in this repo — it's empty. These are the first tests written under this
tracker; whatever pattern they establish (framework, fixture layout, how a human-gate is
simulated) is inherited by every later slice, so prefer the plainest structure that works over one
tuned for this slice's five-document fixture.

Out of Scope

Everything PRD/tracer-bullet-01.md already deferred, restated once: auth, users, roles, a config
UI, an audit log, rate limiting, multi-tenancy, Decision supersession (FR-6), Curator re-curation
(FR-7) beyond what write-time provenance refs already give for free, connectors (Confluence/Jira/
forums), write-back/"reply" (v1.1), BM25/Tags + RRF + LLMRerank (FR-8's remaining two-thirds),
OCR, .doc (legacy binary — no LlamaIndex reader), multipart/resumable upload, page-anchored
citations, automatic near-duplicate merge below a tuned threshold, typed/directed link creation
between entities, gate-on-low-confidence-only structuring, local (non-LlamaParse) PDF extraction
for production use (explicitly flagged as on the critical path to shipping, just not to this
slice), and Postgres-as-graph-store (revisit once PG19/SQL-PGQ is GA and this slice has shown which
graph queries actually get issued).

Further Notes

• docs/tracer-bullet-01.md remains the long-form rationale document — this spec is the
  build-and-test-shaped sibling, not a replacement. Where they'd ever disagree, that's a bug to
  fix in one of them, not a real fork in the design.
• CONTEXT.md and docs/adr/0001–0006 exist and are committed, covering the load-bearing,
  hard-to-reverse decisions this slice makes (RabbitMQ's removal, Neo4j over Postgres/AGE,
  embeddings-in-Neo4j over pgvector, the 1024-dimension one-way door, content-hash identity,
  merges-not-links) and the domain vocabulary this design record introduces beyond the PRD's own
  glossary.
• A11 is deliberately non-blocking everywhere in this spec: no assertion about graph behavior
  (A1–A10, A12) may be allowed to depend on LlamaCloud being reachable.
• docs/tracer-bullet-01.md was refined further after review (commit 2672aef): watched-directory
  scanning is recursive; the pipeline's work-queue rows are claimed with an explicit worker
  claim/lease and bounded retry (attempt_count/next_attempt_at, default limit 5, terminal
  failed state); the Reviewed-Source contract supersedes FR-2's four-section schema as the
  Compiler's actual input shape; and A1–A10 have an offline acceptance profile (local
  OpenAI-compatible LLM + local/deterministic 1024-dim embedder) so they never depend on a cloud
  LLM — only A11 may call LlamaCloud.
• A further pass (commits 8d53cf0, 6823a6c) added: next_attempt_at to the worker claim
  predicate (backoff was described but not enforced); a correction that the Postgres
  lease_token fences the state transition, not the RustFS/Neo4j writes directly (those are safe
  via content-hash idempotency and live-state delta computation instead — short of true
  cross-store fencing, accepted for a single-worker slice); OFFLINE=1 now allows a compose
  service name or the host-gateway hostname and rejects only public destinations, not a
  loopback-only check (which would have wrongly rejected a local LLM running as its own compose
  service); and the offline embedding test double must be golden/fixture-pinned, not
  hash-derived, since fixture C is deliberately designed to defeat a naive similarity formula.
• Two more passes (commits cd4855f, f1a7cec) fixed: upload_max_filesize was wrongly raised
  to match post_max_size (26M/26M), which moves the effective size check past PHP's own parsing —
  it's upload_max_filesize=25M/post_max_size=26M, MiB throughout since PHP's M and Laravel's
  KB are both already binary; the same-hash-different-filename fix now inserts a bare row before
  parsing so only the confirmed winner ever parses (the previous version had both concurrent
  arrivals parse and discarded a completed result); the merge confirmation's Postgres alias write
  and Neo4j rewrite-and-delete are ordered explicitly (alias first) since they aren't one
  transaction; and the offline profile pins similarity_top_k=3 with an explicit A-in/B-out/C-in
  ranking precondition, required for a real local embedding model exactly as much as the golden
  test double.

Frontier

Children are listed as sub-issues and ordered by dependency. A child is workable when every
blocker is closed.

Order and blockers

Every "half" split is intentional, not an ownership gap: each half is independently real risk
— a correct primitive in one story cannot make the observable behavior correct if the story that
exercises it end-to-end is wrong, and vice versa.

Open frontier: NU-001, NU-002, NU-014. Everything else has an open blocker.

Nothing is implemented. A first pass at NU-001 and (old) NU-002 was written before these stories
existed and has been deleted — code ahead of the tracker was the wrong way to start.

The fixture corpus text is preserved in a comment on the now-closed #15 as the source of truth for
exact wording; NU-002/#31 is the story that turns it into committed files. The constraints on it
(B must never mention JWT; C must be plausible noise; A must carry attribution in prose, not
frontmatter) are what make A1 and A2 test anything.