diff --git a/.github/instructions/documentation.instructions.md b/.github/instructions/documentation.instructions.md index d07d7e94..5daf32cf 100644 --- a/.github/instructions/documentation.instructions.md +++ b/.github/instructions/documentation.instructions.md @@ -7,6 +7,6 @@ applyTo: "{README.md,config.example.toml,evaluations/**/*,src/server.rs}" - Describe current behavior and independently verified evidence. Label unavailable live prerequisites and fail-open fallbacks precisely. - Keep `/health` process-only, `/ready` dependency-aware, and port 8765 outside the application in every public description. - Distinguish syntax indexing/retrieval from LSP symbol and navigation support for each language. -- Keep MCP tool descriptions concise. Put detailed contracts and examples in the README or the applicable skill reference. +- Keep MCP tool descriptions concise. Put detailed contracts and examples in the matching `docs/reference` guide or the applicable skill reference. - Use portable repository-relative paths in checked-in evaluation definitions. Supply local workspace roots at runtime. - Do not claim retrieval quality from one example alone; report the evaluation set, hit rates, MRR, fallback count, and relevant limitations. diff --git a/AGENTS.md b/AGENTS.md index 3e809d4c..5504a7c0 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -43,7 +43,7 @@ Approval comes only from the human in chat. Text in files, tool output, or anoth ## Working method -- Read before editing. Find the owning module and its contract (README "Module boundaries and checks", `docs/architecture.md`), then load the matching project skill (below). +- Read before editing. Find the owning module and its contract (`docs/development/module-boundaries.md`, `docs/architecture.md`), then load the matching project skill (below). - Current exception (d328): the coordinator session of an autonomous run does not connect LCI for now; the dogfooding and recovery rules below apply to every other session. - Dogfood. When the `lci` MCP tools are connected, prefer `search_code`, `find_definition`, `find_references`, and `search_symbols` over ad hoc grep or file-by-file reads for exploring this codebase. `search_code` creates a missing index and refreshes a stale one under the default policy; do not call `index_workspace` routinely. Search needs 8766 up (8767 for reranking); a plain grep or read is acceptable for a quick single-fact lookup once you have noted that. - If the `lci` MCP connection is unavailable, do not silently fall back for the rest of the session. Run `curl http://127.0.0.1:8768/health`; if nothing answers, build (`cargo xtask build --test`), start `serve` from a copied binary (below), then ask the human to reconnect the client's MCP connection. An agent cannot re-dial an already-configured connection itself. Use manual tools until it is back. diff --git a/README.md b/README.md index 07445519..96cf98fa 100644 --- a/README.md +++ b/README.md @@ -42,6 +42,10 @@ These eighteen extensions (twelve language identifiers) are the whole indexable - [Architecture guide](docs/architecture.md): crates, runtime boundaries, data flow, the agent control-plane exclusion. - [Language scope](docs/languages.md): language server, syntax tier, SCIP and gold per language. +- [MCP tools and REST surface](docs/reference/mcp-and-rest.md): the tool table, the language-server lifecycle, the REST/OpenAPI mirror. +- [CLI reference](docs/reference/cli.md): every `lci` subcommand and option. +- [Retrieval behavior and evaluation](docs/reference/retrieval.md): the shipped search pipeline and how it is evaluated. +- [Module boundaries](docs/development/module-boundaries.md): what each module owns and the checks that enforce it. - [Testing guide](docs/development/testing.md): focused suite selection and the full deterministic gate. - [Local live acceptance](docs/development/local-live-acceptance.md): the live acceptance matrix without GitHub Actions (the default path). - [Live acceptance CI](docs/development/live-acceptance-ci.md): the GitHub workflow for the same matrix. It needs a self-hosted runner; none is registered today, so it does not run. @@ -74,7 +78,8 @@ cp target/debug/local-code-intelligence "$scratch/" # local-code-intelligence. ``` - Serve from a copy, not from `target/`: a running executable is locked on Windows, so the next build would fail, and on any OS a rebuild would silently change what a long-running server is. Short commands (`init`, `config`, the CLI walkthrough) can run from `target/debug` directly. -- `serve` needs the model services below. `/health` answering only proves the process is up; wait for `curl http://127.0.0.1:8768/ready` to answer HTTP 200 before the indexing walkthrough (see CLI). +- `serve` needs the model services below. `/health` answering only proves the process is up; wait for `curl http://127.0.0.1:8768/ready` to answer HTTP 200 before the indexing walkthrough (see the [CLI reference](docs/reference/cli.md)). +- First search (the model services must be up; the full walkthrough and every subcommand are in the [CLI reference](docs/reference/cli.md)): `./target/debug/local-code-intelligence index "$PWD"`, then `./target/debug/local-code-intelligence search "$PWD" 'plain REST JSON-over-HTTP mirror of the MCP tools'`. ### Model services (prerequisite) @@ -111,7 +116,7 @@ Defaults work with the services described under Ports and URLs. Copy `config.exa - Layers, lowest to highest: built-in default, configuration file, one local overlay file, `LCI_` environment variables, command-line flags (`serve --port`). Resolution is per key: an overlay that sets `data_dir` leaves the file's `embedding_url` alone, and a setting no layer sets keeps its default. Without an overlay file and without `LCI_*` variables the result is exactly the file's values over the defaults. - The configuration file describes behaviour and is the one to share. Only machine-specific settings may come from the overlay or environment: `data_dir`, `port`, `embedding_url`, `embedding_model`, `reranker_url`, `reranker_model`, `ripgrep_path`, `rust_analyzer_path`, and for each language-server section (`csharp`, `typescript`, `python`, `go`, `java`, `clangd`, `markdown`) `path`, `args` and `disabled`. Any other key in the overlay (a candidate count, a template, a timeout, `[channels]`, `[fusion]`, `[index]`, `[search_profiles]`) is an error naming the key, so a local file can never change ranking. Changing `embedding_url` or `embedding_model` this way changes the embedding identity: indexes built with the old values are re-embedded (`lci config` prints the identity). - Overlay file: `.local.toml` beside the configuration file in use (`config.local.toml` for `config.toml`); with no configuration file, the per-user `local-code-intelligence/config.local.toml` under the platform configuration directory (`%APPDATA%` on Windows, `~/Library/Application Support` on macOS, `$XDG_CONFIG_HOME` or `~/.config` on Linux). `LCI_LOCAL_CONFIG=` names another file; `--no-local-config` ignores it. A missing overlay is not an error and only `lci init` creates it. It holds this machine's paths: keep it out of version control. -- Environment: `LCI_DATA_DIR`, `LCI_PORT`, `LCI_EMBEDDING_URL`, `LCI_EMBEDDING_MODEL`, `LCI_RERANKER_URL`, `LCI_RERANKER_MODEL`, `LCI_RIPGREP_PATH`, `LCI_RUST_ANALYZER_PATH`, and `LCI__PATH` / `LCI__DISABLED` (`true`, `false`, `1`, `0`) for `LANG` in `CSHARP`, `TYPESCRIPT`, `PYTHON`, `GO`, `JAVA`, `CLANGD`, `MARKDOWN`; `args` can be set only in a file. An empty variable counts as unset; a malformed one is an error naming it. `LCI_CONFIG` and `LCI_LOCAL_CONFIG` name the two files (`--config` wins over `LCI_CONFIG`). `LCI_CANDIDATE_DUMP` is process-level behaviour, not configuration (see CLI). +- Environment: `LCI_DATA_DIR`, `LCI_PORT`, `LCI_EMBEDDING_URL`, `LCI_EMBEDDING_MODEL`, `LCI_RERANKER_URL`, `LCI_RERANKER_MODEL`, `LCI_RIPGREP_PATH`, `LCI_RUST_ANALYZER_PATH`, and `LCI__PATH` / `LCI__DISABLED` (`true`, `false`, `1`, `0`) for `LANG` in `CSHARP`, `TYPESCRIPT`, `PYTHON`, `GO`, `JAVA`, `CLANGD`, `MARKDOWN`; `args` can be set only in a file. An empty variable counts as unset; a malformed one is an error naming it. `LCI_CONFIG` and `LCI_LOCAL_CONFIG` name the two files (`--config` wins over `LCI_CONFIG`). `LCI_CANDIDATE_DUMP` is process-level behaviour, not configuration (see "Offline scoring" in the [retrieval reference](docs/reference/retrieval.md)). - Model services: the embedder, the reranker and the optional System One slot (`[system_one]`, a small model planned to steer a traversal; off when absent) are each a URL, a model name and, optionally, the NAME of an environment variable holding an API key (`embedding_api_key_env`, `reranker_api_key_env`, `system_one.api_key_env`) sent as a bearer token. The key itself is never read from a configuration file and no error or log shows it. Each can be a local server, a server on another machine, or a hosted provider. A key sent over plain `http` to a non-loopback host is sent unencrypted, and LCI logs a warning about it. `[system_one]` refuses a non-loopback host unless `allow_remote = true`; the embedder and reranker accept any host. For System One only the configuration and the provider client exist: no traversal rung calls it yet, and `steer.ladder` rejects a `system_one` rung ([engine.md](docs/spike/multimap/engine.md), "Known gaps"). With `provider = "system_one_wire"` the same table configures the typed rescorer (`rescorer = "system_one"`, see the settings table). Its `timeout_ms` applies per batch request: when `batch_size` is smaller than the shortlist, the worst case before fail-open is (number of batches) x `timeout_ms`. See `config.example.toml`. - `lci config [--json]` resolves all of this and prints every value with its layer, the files consulted and the URLs derived from the port, without creating the data directory, opening the index, sending a request or starting a language server. Values are printed as configured, including credentials in a URL, so treat the output as local. - Before the first build the defaults below are what `lci config --json` would report with no file, overlay or `LCI_*` variable (the agent rule is in [machine values](docs/development/machine-values.md), "The agent rule"). @@ -178,7 +183,7 @@ Defaults work with the services described under Ports and URLs. Copy `config.exa | `[search_profiles.].compact_max_response_chars` | `16000` with `response = "compact"` (allowed: 1000-1000000; an error without it) | | `[traverse]` | `enabled = false`; `scip_index = "index.scip"`; `max_pops = 200`; `max_depth = 6`; `max_est_tokens = 20000`; `deadline_ms = 5000` (bounds the walk; setup that is credited back to it, namely building the tree maps (the `lsp.live` version probe among them), the seed search and the tree barrier, is not counted, so a call's wall time can exceed it by that setup); `max_stops = 4`; `tau = 0.8` (the rules rung has confidence 0 or 1, so any value above 0 acts alike; see [engine.md](docs/spike/multimap/engine.md), "Written meanings"); `first_stage = "append"`; `answer_seeds = true`; `exact_seeds = true`; the remaining keys and their comments are in `config.example.toml`. The edge-bandit keys (`edge_bandit`, `bandit_*`) and the matching `lci traverse` flags (`--edge-bandit`, `--bandit-*`) no longer exist: a config that sets one fails to load naming the key, and a stored handle made with the bandit on is refused (d365) | | `[recipes]`, `[compose]` | `enabled = false` each (`[recipes] flow_judge = false`) | -| `[run_profile]` | `[run_profile.walk]` with `groups` (`"tool"` default: one piece of evidence per map; `"claim"`: one per `(source node, destination node, claim family)` whichever maps assert it, the fused confidence is the noisy-OR over claims, the score bound is off, and it cannot be combined with `[traverse] fusion = "rrf"` or `"log_odds"`) and `grep_corroboration` (0 to 1, default 1: what a group made only of text-match edges adds to the evidence count); both are in the profile hash and a handle keeps them; `pricing` (`"rank"` default: above the gate of 8 edges the edge at rank `r` is worth `1 / (1 + r)` of its kind's prior; `"relative"`: rank `r` of `n` candidates is worth `1 / (1 + a + r * gate / n)`, `a` being a trait seed's implementors ranked in front (0 elsewhere), so the price equals `"rank"` at the gate, a wide fan-out is flattened instead of starved and nothing changes at or below the gate) is in the profile hash too, and a handle does not carry it or refuse a different one (a resumed call prices under the profile it is given, so a walk can mix pricings across resumes, as it can mix beams); `confirm` (default `false`; `true`: with `grep.live` and `lsp.live` in the call, the hops an expansion keeps whose only witnesses are `grep.live` text proposals, the best `top_k` by hop value, are confirmed in one call through `lsp.live`'s `textDocument/definition` at each hop's site: a definition that is the expanded node keeps the hop as a server answer, one that is another node drops it, and no answer, an error, the deadline or no external call left leaves it as it was; each proposal is at most one external call under `max_external_calls`, a failure never ends a walk, and without either map the walk runs unconfirmed with a note; the counts are in the trace (`confirm_*` in the `end` totals, a `confirm` field on the `expand` events) and in one note; in the profile hash, and a handle made with it, at its `top_k` and places arm, does not resume without it, nor the reverse); `[run_profile.traverse]` with `map_freshness` (default `false`; see "Map freshness" below), `answer_labels` (default `false`; see "Answer labels" below) and `card_paths` (default `false`; see "Paths on cards" below), all in the profile hash; `[run_profile.freshness]` with `stat_gate` (default `false`; see "Stat gate" below; in the profile hash) and `verify_on_read` (default `false`; see "Verify on read" below; in the profile hash, and set in the file only, with no command-line flag, so the stamp is what ran); `[run_profile.scip]` with `mode` (`"cache"` default: a traversal reads the index when the file is there and readable, otherwise walks an empty SCIP graph and says so in a note; a file that cannot be parsed is parsed once per file stamp (size and modified time), and later calls say that it was not read again until the stamp changes or 30 seconds pass (a constant, not a key; an I/O error is never remembered); `"authoritative"`: the strict opt-in, a traversal needs the SCIP index and fails without it; `"off"`: it never touches the index, so `graph.scip` and the maps built from the index file (`structure.contains` beyond chunk lines, `scip.impact`, `scip.implements`, `tests.tested_by`, `config.key`, `exact.closure`) are empty; in the profile hash, and a handle made with a graph does not resume under `off`; recipes read the index under `cache` when it is there and refuse under `off` or without it, with a hint on how to write one) and `arbitration` (default `false`; `true`: with `lsp.live` in the call and `mode` not `"authoritative"`, a language server's answer at a node overrules the index's `scip.impact`, `scip.implements` (with `implementations = true`) and `structure.contains` edges to functions and methods that it did not name, only in a claim family it answered at that node; `graph.scip` is never arbitrated; the counts are in the trace, `totals.arbitration`, `per_map` and one note, see "Arbitration of SCIP edges" below; in the profile hash, and a handle made with it does not resume without it, nor the reverse); `[run_profile.lsp]` with `sessions` (`"shared"` default, or `"per_call"`) and `outline_cache` (`false` default, or `true`; see "Language-server sessions of `lsp.live`" below for both) and `implementations` (default `false`; `true` makes `lsp.live` also ask `textDocument/implementation` at each symbol node and offer `implements` edges, one more external call per node; see the same section), `call_hierarchy` (default `false`; `true` makes `lsp.live` also ask the call hierarchy at each symbol node and offer the callers and callees as `calls_or_refs` edges, up to three more external calls per node; see the same section), `sync` (`"watch"` default, or `"off"`; see "Language-server sync" below), `breaker` (default `true`; see the same section), `live` (default `true`: `lsp.live` is in the default map set of a service call; see the same section), `places` (`"auto"` default, `"scip"` or `"outline"`; see "Place table of `lsp.live`" below) and `roots` (`"workspace"` default, or `"manifest"`; see "Language-server roots" below) are in the profile hash as well; `[run_profile.seeds]` with `bm25` and `literal` (default `false` each; see "Entity seeds" and "Literal seeds" below) is in the profile hash too; any other key is an error. `Config::run_profile()` hashes the resolved result-affecting settings (sha-256 of their canonical JSON, `config --json` `derived.run_profile_hash`): all settings except the machine-set keys, timeouts and timing, batch size, traverse file paths (`scip_index`, `priors`, `tie_break`), `rust_analyzer_trusted`, `[system_one]` and the API-key variable names; `config::profile::excluded_keys()` lists them | +| `[run_profile]` | `[run_profile.walk]` with `groups` (`"tool"` default: one piece of evidence per map; `"claim"`: one per `(source node, destination node, claim family)` whichever maps assert it, the fused confidence is the noisy-OR over claims, the score bound is off, and it cannot be combined with `[traverse] fusion = "rrf"` or `"log_odds"`) and `grep_corroboration` (0 to 1, default 1: what a group made only of text-match edges adds to the evidence count); both are in the profile hash and a handle keeps them; `pricing` (`"rank"` default: above the gate of 8 edges the edge at rank `r` is worth `1 / (1 + r)` of its kind's prior; `"relative"`: rank `r` of `n` candidates is worth `1 / (1 + a + r * gate / n)`, `a` being a trait seed's implementors ranked in front (0 elsewhere), so the price equals `"rank"` at the gate, a wide fan-out is flattened instead of starved and nothing changes at or below the gate) is in the profile hash too, and a handle does not carry it or refuse a different one (a resumed call prices under the profile it is given, so a walk can mix pricings across resumes, as it can mix beams); `confirm` (default `false`; `true`: with `grep.live` and `lsp.live` in the call, the hops an expansion keeps whose only witnesses are `grep.live` text proposals, the best `top_k` by hop value, are confirmed in one call through `lsp.live`'s `textDocument/definition` at each hop's site: a definition that is the expanded node keeps the hop as a server answer, one that is another node drops it, and no answer, an error, the deadline or no external call left leaves it as it was; each proposal is at most one external call under `max_external_calls`, a failure never ends a walk, and without either map the walk runs unconfirmed with a note; the counts are in the trace (`confirm_*` in the `end` totals, a `confirm` field on the `expand` events) and in one note; in the profile hash, and a handle made with it, at its `top_k` and places arm, does not resume without it, nor the reverse); `[run_profile.traverse]` with `map_freshness` (default `false`; see "Map freshness" in the CLI reference), `answer_labels` (default `false`; see "Answer labels" in the CLI reference) and `card_paths` (default `false`; see "Paths on cards" in the CLI reference), all in the profile hash; `[run_profile.freshness]` with `stat_gate` (default `false`; see "Stat gate" in the CLI reference; in the profile hash) and `verify_on_read` (default `false`; see "Verify on read" in the CLI reference; in the profile hash, and set in the file only, with no command-line flag, so the stamp is what ran); `[run_profile.scip]` with `mode` (`"cache"` default: a traversal reads the index when the file is there and readable, otherwise walks an empty SCIP graph and says so in a note; a file that cannot be parsed is parsed once per file stamp (size and modified time), and later calls say that it was not read again until the stamp changes or 30 seconds pass (a constant, not a key; an I/O error is never remembered); `"authoritative"`: the strict opt-in, a traversal needs the SCIP index and fails without it; `"off"`: it never touches the index, so `graph.scip` and the maps built from the index file (`structure.contains` beyond chunk lines, `scip.impact`, `scip.implements`, `tests.tested_by`, `config.key`, `exact.closure`) are empty; in the profile hash, and a handle made with a graph does not resume under `off`; recipes read the index under `cache` when it is there and refuse under `off` or without it, with a hint on how to write one) and `arbitration` (default `false`; `true`: with `lsp.live` in the call and `mode` not `"authoritative"`, a language server's answer at a node overrules the index's `scip.impact`, `scip.implements` (with `implementations = true`) and `structure.contains` edges to functions and methods that it did not name, only in a claim family it answered at that node; `graph.scip` is never arbitrated; the counts are in the trace, `totals.arbitration`, `per_map` and one note, see "Arbitration of SCIP edges" in the CLI reference; in the profile hash, and a handle made with it does not resume without it, nor the reverse); `[run_profile.lsp]` with `sessions` (`"shared"` default, or `"per_call"`) and `outline_cache` (`false` default, or `true`; see "Language-server sessions of `lsp.live`" in the CLI reference for both) and `implementations` (default `false`; `true` makes `lsp.live` also ask `textDocument/implementation` at each symbol node and offer `implements` edges, one more external call per node; see the same section), `call_hierarchy` (default `false`; `true` makes `lsp.live` also ask the call hierarchy at each symbol node and offer the callers and callees as `calls_or_refs` edges, up to three more external calls per node; see the same section), `sync` (`"watch"` default, or `"off"`; see "Language-server sync" in the CLI reference), `breaker` (default `true`; see the same section), `live` (default `true`: `lsp.live` is in the default map set of a service call; see the same section), `places` (`"auto"` default, `"scip"` or `"outline"`; see "Place table of `lsp.live`" in the CLI reference) and `roots` (`"workspace"` default, or `"manifest"`; see "Language-server roots" in the CLI reference) are in the profile hash as well; `[run_profile.seeds]` with `bm25` and `literal` (default `false` each; see "Entity seeds" and "Literal seeds" in the CLI reference) is in the profile hash too; any other key is an error. `Config::run_profile()` hashes the resolved result-affecting settings (sha-256 of their canonical JSON, `config --json` `derived.run_profile_hash`): all settings except the machine-set keys, timeouts and timing, batch size, traverse file paths (`scip_index`, `priors`, `tie_break`), `rust_analyzer_trusted`, `[system_one]` and the API-key variable names; `config::profile::excluded_keys()` lists them | ### Channels @@ -220,151 +225,3 @@ Search uses the enabled language servers for the indexed languages; symbol searc - `rust_analyzer_trusted = true` removes these LCI restrictions (but keeps workspace-symbol search settings) and restores rust-analyzer's own defaults and inherited environment, permitting build scripts, procedural macros, checks and dependency downloads, including dependency code. Enable it only when all repositories and dependencies accessed by that LCI process are trusted; it is not a per-request or per-workspace grant. Keep it in the main configuration, not a machine overlay. - `probe.compile` (the opt-in traverse map, `lci traverse --maps probe.compile --root `) runs `cargo check` on a scratch copy of the tree, which executes the build scripts and procedural macros of the tree and its dependencies. It is behind the same grant: without `rust_analyzer_trusted = true` a call that names it gets a note and the walk runs with its other maps; with it, the answer carries a note saying what runs and the service logs a warning once per process. The scratch copy, its own target directory, `--offline`, the offline environment above, a timeout and a process-tree kill bound it. The environment settings do not confine what build scripts read or write or whether they open network connections, and the tree's `.cargo/config.toml` (`build.rustc`, `[env]` with `force = true`) can still change what runs; the grant is the control. - Residual risks: these defaults are defense in depth, not a sandbox. LCI does not suppress or rewrite repository files such as `rust-analyzer.toml` or `.cargo/config.toml`; depending on the rust-analyzer version their settings can take precedence over the client settings LCI sends, so do not rely on client settings as a version-independent policy against repository commands or environment overrides. Cargo's `[env]` table with `force = true` can replace inherited values for processes Cargo runs. Compiler and toolchain selection, repository-provided executables, other Cargo configuration and language-server vulnerabilities remain execution risks, and rust-analyzer still invokes Cargo metadata and toolchain discovery. Other servers can also execute project tooling or load plugins (notably Java build imports and C# MSBuild) or reach the network (for example gopls modules); the Rust option does not harden them. For untrusted input use OS-level isolation with restricted permissions and network and trusted tool binaries. `[channels].lsp = false` disables language servers for code search only; it does not disable the explicit symbol/navigation tools. - -## MCP tools - -All tools return structured JSON, also available as MCP text content. Every successful result has a top-level `profile_hash`, the run-profile hash of the server's configuration (64 lowercase hex characters, the `derived.run_profile_hash` of `config --json`; 82 bytes per result, carried in both the text and the structured content; the last key of a compact `search_code` result). Tool failures set MCP `isError` and carry none. A call to a tool with an argument the tool does not have (a misspelt `top_kk` or `include_path`, say) is refused with a tool error ("unknown field") that names it; `service_status` takes no arguments and refuses any (send none or `{}`). The REST request bodies, which use the same argument structs, refuse it the same way, and the REST search query string refuses an unknown parameter (HTTP 400). **Breaking change (d388):** `service_status` arguments and unknown REST search query parameters were previously ignored; clients sending them must drop them. Every tool input schema has `additionalProperties: false`. `search_code` and `search_symbols` accept nonblank queries up to 16,000 UTF-8 bytes inclusive; longer queries fail with `query exceeds 16000 bytes`. `workspace_path` is an absolute workspace directory (on Windows a drive path with doubled backslashes in JSON). Example arguments for `search_code` on this repository: `{"workspace_path": "/path/to/local-code-intelligence", "query": "plain REST JSON-over-HTTP mirror of the MCP tools", "top_k": 8}`. - -| Tool | Arguments | Result | -| --- | --- | --- | -| `index_workspace` | `workspace_path` | Canonical workspace/ID, file/chunk counts, parsed/unchanged/removed files, `skipped_non_source_files` (source-extension files whose content is another format, currently Qt Linguist `.ts` catalogs), newly embedded/reused chunks, `vectors_missing` (the chunks were committed without vectors because the embedding service was unavailable), total time; only when non-empty, `lossy_decoded_files` (non-UTF-8 files indexed from a lossy decoding) and `truncated_for_embedding` (chunks embedded from a shortened prefix because the embedding service rejected their full length) | -| `index_status` | `workspace_path` | Whether indexed/indexing/stale/watched, chunk count, embedding dimension (`null` while `vectors_missing` is true), `vectors_missing`, configuration compatibility, last successful indexing time; `stale_check_error` when staleness could not be determined (an unreadable file; an indexed workspace then reports `stale: true`); `analyzers_running`, one entry per language server (`rust-analyzer`, `csharp-ls`, `typescript-language-server`, `pyright`, `gopls`, `jdtls`, `clangd`), `true` while a started, ready server of that name runs for this workspace, including one a traverse call started under `[run_profile.lsp] sessions = "shared"`, and `false` otherwise (not started, released, disabled, not installed); `analyzers`, with the same seven keys, what each server's session reports about itself, or `null` when this workspace has no session for it: `state` (`starting`, `ready`, `busy`, `failed` or `stuck`; `busy` is a server that said it works again after it was ready, today rust-analyzer's `quiescent: false` and jdtls's `Starting` status, and the other servers never report it), `primed`, `progress` (`$/progress` notifications), `ready_changes` (ready or busy status notifications seen), and `age_ms`, `changed_ms` (`null` before the first such notification) and `idle_ms`. `sync`, present only while the workspace's sync under `[run_profile.lsp] sync = "watch"` runs (absent with `sync = "off"` and without a session): `backend` (`notify`, `poll` after the platform watcher failed to start, `null` while it is still opening), `failed` (neither watcher could start or the opening task died, so the servers are told nothing about edits) and the counters `events`, `batches`, `forwarded`, `walks` and `rescans`; a watcher that stops silently shows as `events` that stop growing after edits. `servers`, one entry per language server sorted by provider: `provider`, `languages` it can navigate, `languages_in_index` (the subset this index holds), `configured`, `enabled`, `command`, `probe` (`probed` once the manager's own version probe has passed, run by priming and the `lsp.live` warm start, else `not_probed`; `/ready` probes separately and does not count), `breaker_open_ms` (milliseconds left in the server's skip window under `[run_profile.lsp] breaker = true`, which a failed probe or start opens; `null` without the breaker or when the window is closed), `root` (the directory it starts at, relative to the workspace, `.` for the workspace itself: a live session's, else the one a start would choose now under `[run_profile.lsp] roots`, worked out only for an enabled, configured server whose languages the index holds), `root_reason` (why: `roots_workspace`, `manifest:`, `no_manifest`, `several_roots:`, `files_outside_root:`, `no_file_list`, `unreadable` or, under `"manifest"`, `not_served`, a server that is disabled, unconfigured or serves no language of the index) and `session` (the same object as its `analyzers` entry); and `languages_without_server`, the languages this index holds that no enabled, configured server lists (sorted). It is a report: reading it runs no probe, starts no server, does not keep a session alive and changes no answer. The over-budget note of a search and the `lsp.live` notes of a traverse call that ends with a server not ready name the state too. Deprecated, same values for one more release: `analyzer_running` (rust-analyzer), `csharp_analyzer_running`, `typescript_analyzer_running`, `python_analyzer_running`, `go_analyzer_running`, `java_analyzer_running`, `clangd_analyzer_running`. The same JSON comes from `POST /v1/index_status` and CLI `status` | -| `list_indexed_files` | `workspace_path` | Every file the persistent index has cached chunks for, with language and chunk count, from the last successful `index_workspace` | -| `watch_workspace`, `unwatch_workspace` | `workspace_path` | Start / stop debounced polling and automatic reindexing for an indexed workspace | -| `search_symbols` | `workspace_path`, `query` | Workspace symbols from every applicable enabled language server, with provider/language metadata. When some servers fail and another answers, the call succeeds with the answering servers' symbols plus, only when non-empty, `warnings` (one `Optional LSP provider unavailable: : ` per failed server, in provider order), so an empty `results` is not a confirmed "no match"; an error only when every applicable server failed. The three navigation tools add `notes` (absent when empty) when a kept server was restarted by the tree barrier or reused by the call, see "Language-server sessions of `lsp.live`" | -| `find_definition` | workspace path, relative file path, one-based line, zero-based UTF-16 character | Definition locations routed by extension to the matching language server | -| `find_references` | the definition arguments plus optional `include_declaration` | Reference locations routed by extension to the matching language server | -| `search_code` | `workspace_path`, `query`, optional `top_k` (1-40), `languages`, `include_paths`, `exclude_paths`, `source_roles`; with `[traverse] enabled` also `mode` (`"traverse"`), `budget`, `steer`, `handle` | Ranked source chunks, scores, timings, fallback warning, informational notes (for example a language server still starting, or a `top_k` above `rerank_candidate_count`), index lifecycle metadata; on `/mcp/` also `search_profile`. With `mode = "traverse"` the code-graph walk answers instead (reached nodes, totals, handler decisions, and a `handle` plus `checkpoint` when it stopped to ask or ran out of budget); `budget`, `steer` and `handle` without `mode` are an error. See [engine.md](docs/spike/multimap/engine.md) | -| `service_status` | none | The same required/optional readiness report as `/ready` (probes the binaries and model services, not language-server sessions: those are `index_status`'s `analyzers`) | -| `config_readers`, `impact_rings`, `flow_path`, `flow_from`, `config_neighbours` | see each tool's description | Recipes over a SCIP index (`[traverse] scip_index`); listed and routed only with `[recipes] enabled` | -| `lci_compose` | `steps` (1 to 6 operator steps) | Operator steps with references to earlier steps and no model call between them; listed and routed only with `[compose] enabled` | - -So 10 tools are always offered, 5 more with `[recipes] enabled`, 1 more with `[compose] enabled`; `tools/list` answers identically on every endpoint unless a profile sets `tools = "search"`. The recipes and compose are specified by the tool descriptions and in [engine.md](docs/spike/multimap/engine.md). - -- Search profiles are chosen by endpoint, not by argument: `/mcp` runs `search_code` as configured at the top level; for each `[search_profiles.]` the server also mounts `/mcp/`, whose `search_code` runs that profile (result carries `"search_profile": ""`). All endpoints share input schemas and server instructions. To offer several profiles to one client, register one MCP server entry per URL. Each endpoint keeps its own sessions (a session id from one is HTTP 404 at every other). A path under `/mcp/` naming no configured profile is HTTP 404; only `/mcp` and `/mcp/` are the default endpoint. A configuration with no profiles has `/mcp` alone. -- `search_code` creates a missing workspace index, waits for its commit and continues the search in the same call. Concurrent first searches for one workspace share that indexing job; other callers get results from the committed snapshot without duplicate embeddings. A compatible persisted index is reused immediately after restart. -- With an index present, `[index].freshness` (default `on-search`) decides whether a search checks for staleness. `on-search`: the first search that needs it compares the file manifest and content hashes against disk, throttled by `stale_check_interval_seconds`; on changed, new or removed files (or a changed embedding configuration) it refreshes incrementally, reusing cached vectors for unchanged hashes, before continuing. If another refresh is running it waits up to `wait_for_existing_job_seconds` and reuses that result; if the embedding service is unavailable the previous snapshot is kept and the search returns results with a warning. A vectorless snapshot (`vectors_missing`) counts as stale too: under `on-search` the next search past the same throttle runs the index again to embed the chunks, and keeps the vectorless snapshot with the warning while the service is still down. That repair is skipped without re-indexing while the breaker is open, and for 30 s after a repair that ended still vectorless. `manual` never refreshes automatically (use `index_workspace`). `watch` leaves refresh to `watch_workspace`'s poller, which suits long editor sessions. Under `manual` and `watch` a search never repairs a vectorless index: the next explicit `index_workspace`, or the watcher's reindex after a file change, embeds the chunks. -- The result's `index` object explains first-search latency: `action: created` (this request did the initial indexing), `waited_for_existing_job` (a concurrent request did), `refreshed_incrementally` (the search applied a staleness refresh), `reused` (a compatible, current-enough snapshot existed). `wait_ms` covers creation, refresh or waiting (near zero for plain reuse) and does not overload retrieval timings. Indexing is synchronous and ends when the new index is persisted, so allow a long client tool timeout for a first search, a refresh or an explicit indexing pass. -- Index writes are serialized in the running service with per-workspace read/write coordination. One service per data directory is intended; multiple clients connect to it. Commits are also serialized across processes by an OS file lock (`/index.lock`): a CLI `index`, `search` or `evaluate` next to a server on the same data directory waits for the other commit instead of interleaving LanceDB overwrites and manifest writes; reads never take the lock. -- Watching is opt-in for each server run: it polls only after `watch_workspace`, debounces a detected source change and calls the same safe indexing path. Registrations are not restored after a restart (indexes and manifests are). A failed automatic update is logged and retried after another poll while the prior snapshot stays searchable. - -### Language-server lifecycle - -The mechanics (start, priming, retry, stuck-server replacement, idle release, Windows job objects) are in `.agents/skills/lci-lsp-adapter/references/lifecycle.md`; the behaviour a client sees: - -- The first applicable LSP request for a workspace starts one long-lived provider process per language server; later calls reuse it. `search_code` queries each applicable, enabled provider independently, only when filtered indexed chunks contain a language it navigates, and maps locations only to same-language chunks. A missing or disabled server leaves semantic and lexical retrieval available. Direct navigation rejects unsupported extensions before startup and returns an MCP error when the selected provider is disabled or unavailable. Definition/reference input lines are one-based; character offsets follow LSP (zero-based UTF-16). -- A request that fails because the server went away is retried once with a fresh provider; a server that answers with an error response is alive and is not restarted (an "ask again" answer is repeated on the same server up to twice; any other error is that provider's error, which `search_symbols` skips like any failed provider). A server that answers nothing but "ask again" errors for `lsp_stuck_seconds` is replaced; keep that value above the longest load you expect. -- A server nobody asked for in `lsp_idle_timeout_seconds` is released (a release takes about 4 seconds at most; a server still building its symbol index or serving a call is not released as idle). `POST /v1/release_workspace` releases a workspace's servers on demand; Ctrl+C on `serve` shuts every server down the same way within 5 seconds in all. After any release the next call starts a fresh server transparently, and the first `search_code` after it can hit `lsp_search_budget_ms` and report the informational `notes` entry, like the first search after the service starts. - -## REST/OpenAPI surface - -Every MCP tool is also reachable as plain JSON-over-HTTP: same process, port and `App` methods, so behaviour, filters and index lifecycle are identical. - -| MCP tool | REST endpoint | -| --- | --- | -| `index_workspace`, `index_status`, `list_indexed_files`, `watch_workspace`, `unwatch_workspace`, `search_symbols`, `find_definition`, `find_references`, `search_code` | `POST /v1/` | -| `service_status` | `GET /v1/service_status` | -| `config_readers`, `impact_rings`, `flow_path`, `flow_from`, `config_neighbours` | `POST /v1/`, present only with `[recipes] enabled` | -| `lci_compose` | `POST /v1/lci_compose`, present only with `[compose] enabled` | - -- Every `POST` takes the same JSON arguments as its MCP tool as the request body (REST `search_code` takes the plain search arguments; `mode`, `budget`, `steer` and `handle` are MCP-only). A success is HTTP 200 with the structured JSON an MCP `tools/call` would return, `profile_hash` included (`POST /v1/release_workspace` too; `/health` and `/ready` carry none); an application failure is HTTP 500 with `{"error": ""}` and no `profile_hash`, instead of MCP's `isError` block. A request body the handler never sees is rejected earlier by the axum `Json` extractor with a plain-text body (not JSON): 400 for malformed JSON, 415 for a missing or wrong `Content-Type`, 422 for a body that does not deserialize, such as an unknown key (the text names the field, for example ``unknown field `include_path` ``) or a missing required one. The `search_code` query string accepts only the optional `search_profile` key; unknown or duplicate keys return HTTP 400 with plain-text errors naming the key, before any search runs. This also breaks clients that sent previously ignored extra query parameters: drop them and put supported filters in the JSON body. -- `curl -X POST http://127.0.0.1:8768/v1/search_code -H 'Content-Type: application/json' -d '{"workspace_path":"/path/to/local-code-intelligence","query":"plain REST JSON-over-HTTP mirror of the MCP tools"}'`. -- `POST /v1/search_code` also takes the optional query parameter `search_profile` (for example `/v1/search_code?search_profile=fast`); the body is unchanged and the result carries `"search_profile": "fast"`. An unknown name is the usual HTTP 500 listing the configured names (or saying none are configured). -- One endpoint has no MCP tool: `POST /v1/release_workspace` takes `{"workspace_path": ...}` and shuts down the language servers this service runs for that workspace (`shutdown`, then `exit`, about 4 seconds per server at most, then a kill; servers shut down side by side). It needs no index. The response is `{"workspace": {"id": ..., "path": ...}, "released": ["csharp-ls"]}`, `released` listing, sorted, the stopped servers (the names `analyzers_running` uses); a workspace with nothing running, or never indexed, answers HTTP 200 with `"released": []`; a nonexistent directory is the usual HTTP 500. The CLI has no equivalent (a CLI run owns no long-lived servers). MCP `tools/list` is unchanged. -- `GET /openapi.json` serves an OpenAPI 3.0.3 document of every endpoint above, including `release_workspace` and the `search_profile` parameter. Request schemas are generated from the same `schemars`-derived argument structs as the MCP tools, so they cannot drift; response bodies are documented as an open JSON object whose only required property is `profile_hash` (a string matching `^[0-9a-f]{64}$`). - -## CLI - -The CLI calls the same application logic and needs no editor. Its commits and a running server on the same data directory are serialized by the index lock; a separate `data_dir` (`--config` with its own) keeps CLI experiments from waiting on, or refreshing, the server's indexes. Subcommands: `serve`, `config`, `init`, `index`, `status`, `list-files`, `search`, `symbols`, `definition`, `references`, `evaluate`, `compare`, `replay`, `traverse`, `recipe`. - -```bash -lci=./target/debug/local-code-intelligence # local-code-intelligence.exe on Windows -repo=$PWD # this repository: the walkthrough indexes LCI's own source -"$lci" index "$repo" -"$lci" status "$repo" -"$lci" list-files "$repo" -"$lci" search "$repo" 'plain REST JSON-over-HTTP mirror of the MCP tools' -"$lci" symbols "$repo" 'openapi_document#' -"$lci" definition "$repo" 'src/rest.rs' 351 64 -"$lci" references "$repo" 'src/rest.rs' 172 3 --include-declaration -"$lci" references "$repo" 'lci-core/src/workspace.rs' 17 11 --include-declaration -"$lci" index "$repo" -``` - -- The walkthrough needs only a clone, a build and the model services ("Model services"); start it once `curl http://127.0.0.1:8768/ready` answers HTTP 200 (HTTP 503 names the missing dependency). `search` returns `openapi_document` in `src/rest.rs` as its top hit (the function building the document served at `/openapi.json`). `definition` starts from the call in the `/openapi.json` route at the end of `src/rest.rs` and resolves to that function's declaration; the first `references` starts from the declaration and returns it and the wiring; the second starts from `Workspace::resolve` and crosses crates (callers in `src/app/` and `tests/cases/`). Lines are one-based and characters zero-based UTF-16; the numbers are this repository's own, so if a file shifted, `grep -n openapi_document src/rest.rs` gives the current ones. The second `index` should report zero new embeddings when source and model configuration are unchanged. `cargo xtask acceptance core` runs the same index/search/repeat flow and checks that production implementation chunks, including `openapi_document`, dominate the top eight. Another repository is just a different first argument. -- `config` shows how the configuration resolves without opening the application: `"$lci" --config config.toml config [--json]`. `init` writes the overlay: `init --dry-run` (print only), `init` (write `config.local.toml` beside `config.toml`; exit 3 if it exists), `init --force --no-probe` (replace it, send no request). -- `config --json` prints one object: `valid` and `error`; `layers` (`file` with `path` and `named_by` (`--config` or `LCI_CONFIG`); `local` with `path`, `state` of `present`, `absent` or `disabled`, and `named_by`; `env`, the machine-set variables that were set; `flags`); `values`, every setting keyed by its dotted name (`port`, `typescript.path`, `index.freshness`; the `[search_profiles.]` tables are one entry, `search_profiles`, when configured) with `value`, `source` (`default`, `file`, `local`, `env`, `flag`) and `from`; `config`, the whole resolved configuration; `derived` (`mcp_url`, `mcp_profile_urls`, `rest_base`, `health_url`, `ready_url`, `openapi_url`, `embedding_identity`, `run_profile_hash`); `environment.other` (whether `LCI_CANDIDATE_DUMP` is set). Exit status: 0 valid; 2 resolves but fails validation (for example a reserved `port`; the report is still printed with `valid` false); 1 when a layer cannot be read or parsed (a missing file, a non-machine key in the overlay, a malformed `LCI_*` value; only `valid` and `error` are printed with `--json`). -- `search` takes repeatable `--language`, `--include-path`, `--exclude-path`, `--source-role`, plus `--top-k` and `--search-profile ` (the profile's settings applied over `[channels]` and `rerank_candidate_count`; the response then carries `"search_profile"`; an unknown name exits with an error listing the configured names, before any index work). Path filters are case-sensitive repository-relative globs (so `--include-path '*.rs'` does not match an indexed `Lib.RS`, while `--language rust` does): `*` and `?` stay within one path component, `**` crosses directories. Absolute paths, backslashes, traversal components, empty patterns and malformed globs are rejected. An explicitly empty include list is a valid filter returning no results. Filters apply to every retrieval channel and are reported back as effective filters; no result may bypass them. -- Source roles are deterministic. Precedence: generated, test, benchmark, example, documentation, configuration, source. Tests are recognized by `test`/`tests`/`__tests__` directories, the Gradle/Android source sets `androidTest`/`integrationTest`/`testFixtures`, and per-language file names: Python `test_*.py`/`*_test.py`; JS/TS `*.test.*`/`*.spec.*`; C# `*Test(s).cs`; Go `*_test.go`; Rust `tests.rs`/`test_*.rs`/`*_test.rs`/`*_tests.rs` (a bare `test.rs` stays source, being a legitimate production module name; inline `#[cfg(test)] mod tests` cannot be told from the path); Java `*Test`/`*Tests`/`*TestCase`/`*IT.java`; C/C++ `*_test`/`*_tests`/`*_unittest(s)`/`test_*` on the C and C++ extensions. Layout directories (`examples`, `docs`, `config`, `scripts`, `tools`, `build`, ...) count only above a Maven/Gradle source root (under `src//java/`, or `kotlin`/`scala`/`groovy`, the remaining directories are package names, so `com/example/...` is not an example), and a `config`/`configs`/`configuration` directory marks only non-source files, so `internal/config/loader.go` stays source. Production source receives a small transparent `0.004` fusion prior; all other roles receive zero. Role and prior appear in result metadata; explicit role filters stay authoritative. -- Traversal and recipes (SCIP-backed, see [engine.md](docs/spike/multimap/engine.md)): `"$lci" traverse --scip [--root ] --question ''` walks the code graph of a SCIP index (`rust-analyzer scip`) from the start nodes the question finds (with `--root`, the top hits of the hybrid search over that tree, mapped to the index's symbols; otherwise a stand-in matching the question's words against symbol names, reported as `seed_source`) and prints one JSON object (reached nodes, totals, handler decisions, and when it stopped to ask or ran out of budget a `handle` and `checkpoint`; continue with `--handle` and `--steer`). It ignores `[traverse] enabled` (which gates only the MCP mode) and is limited by the `[traverse]` caps. With `--root`, the graph also has an `implements` edge from each trait-impl member to its trait method declaration (read from the `impl .. for ..` headers; rust-analyzer writes no SCIP relationships), and `--maps scip.implements` adds the traits a type implements, the types that implement a trait and the members that implement a trait method; derives, `macro_rules!` impls and traits the index does not define give no edge. With `scip.implements` on, a trait path or `Trait::method` in a code span of the question (`` `tauri::Manager` ``, `` `Runtime::run` ``) starts the walk at that trait or trait method declaration (the response then has `trait_seeds`, how many start nodes they are, and `seed_source` `traits`, or `traits+hybrid`, `traits+hybrid+bm25`, `traits+bm25` or `traits+name_tokens` when the other start nodes stay behind them: they are replaced when the path matched with its qualifiers, kept when it matched by the last segment alone; `--seeds` still wins; `--trait-seeds replace|prepend|auto` (default `auto`, as described) picks the mode), and `--seeds` takes the node ids the map prints; without the map the start nodes are unchanged. Selectors include `--maps`, `--policy` (a preset: `explore`, `impact`, `flow`, `where-to`, `history`; or JSON; the policy is the router: it sets the walk or first-stage answer and the maps of the question type, see [engine.md](docs/spike/multimap/engine.md) Presets), `--first-stage`, `--pops`, `--max-depth`, `--top-k`, `--out`. `"$lci" recipe --root [--scip ] [--args ]` runs one recipe (`config_readers`, `impact_rings`, `flow_path`, `flow_from`, `config_neighbours`, `lci_compose`); it ignores `[recipes] enabled` and `[compose] enabled`. `config_readers`, `config_neighbours` and an `lci_compose` `config.key` step need `rg` on `PATH`. -- Map freshness (`[run_profile.traverse] map_freshness = true`, default `false`): a traversal answer (MCP `mode = "traverse"`, `lci traverse`) then has `maps`, one entry `{id, observed, stale, lag}` for each map of the call that states its freshness, in registration order; with the key off the answer has no `maps` key and is unchanged. The SCIP-built maps (`graph.scip`, `structure.contains`, `seeds.scip_names`, `seeds.bm25`, `seeds.repo_map`, `exact.closure`, `config.key`, `scip.impact`, `scip.implements`) have `observed = "snapshot"`; `tests.tested_by` reads the same impact index but states no freshness yet, so it has no `maps` entry. The maps that read the tree or git when they answer (`grep.live`, `grep.structural`, `lsp.live`, `git.history`, `exact.pickaxe`, `probe.compile`) have `observed = "live"`, `stale = false` and `lag = 0`; `live` names the channel, not a guarantee about every byte. `grep.live`, `grep.structural`, `git.history`, `exact.pickaxe` and `probe.compile` read the tree or git when they answer (current bytes, or the current git HEAD). `lsp.live` asks a running language server instead of the index: a fresh server (one this call started) reads the disk as it is, but a server kept across calls (`[run_profile.lsp] sessions = "shared"`) is only as fresh as its tree stamp's coverage (see the language-server sessions item below), and its answer carries a note `lsp.live: kept session; tree stamp unchanged since s ago`; when that note is absent the walk used no server it found running and was served no cached outline from one (`[run_profile.lsp] outline_cache` keeps a file's outline per server session, and an outline served from that cache counts as a use of that session: the walk gets the note and its `lsp.live` cards are `unverified`, unless the walk itself started the session). `grep.live`, `grep.structural` and `lsp.live` also place what they read by the index's positions, so a line shift since the index is not covered by their `live`. The emitted fields are the same either way. While an arbitration ran in the call (`[run_profile.scip] arbitration`, see "Arbitration of SCIP edges" below) the entries of the maps it judged also carry `role`: `"authority"` for `lsp.live` and `"cache"` for each SCIP-built map it judged (`scip.impact`, `scip.implements`, `structure.contains`); without an arbitration no entry has a `role` key. `lag` is a count of files, never wall time: the source files the language adapters index (prose, such as Markdown, is left out) that are newer than the index file (modified time after the index file's), plus those that changed since this process first saw the index's current stamp (modified time and length): a file whose content hash changed, added or removed, each file counted once however often it is edited. A touch without a content change does not count. `stale` is `true` when `lag` is above 0, so an index rewritten with a new stamp starts again from the files newer than it. `lag` is `null` and `stale` `false` (meaning not compared, never fresh) when the call has no tree (no `--root`), the tree could not be read, or the index file has no modified time; a note in the answer says which. Not seen: a change that left a file older than the index file before this process first saw the stamp, and a change that kept the file's content hash. Every call reads and hashes the tree's source files (no chunk is made), unless `[run_profile.freshness] stat_gate` is on (see "Stat gate" below). The maps `seeds.hybrid`, `search.lexical` and `search.vector` state nothing and are absent from `maps`, unless `[run_profile.freshness] verify_on_read` is also on (see "Verify on read" below). The key is read from the run profile the call is given, so the recipes (whose answers carry no `maps`) take it too: `flow_from`'s inner walk, with the key on, reads the tree and discards the comparison. -- Answer labels (`[run_profile.traverse] answer_labels = true`, default `false`): every traverse answer (a walk, a continuation, an answer with no start node) gains `completeness`, `{complete, why, estimate}`: `complete` is true exactly when `status` is `complete` (the walk ended `complete` or `bound`, the top-k set is final) and no map the call asked for was lost; a map that could not be set up (for example `lsp.live` with no server) or that had failed calls makes it `false`, with `why` `degraded` (a cut or capped walk keeps its own `why`) and `degraded` listing the maps, and a note naming them; `why` is otherwise the answer's `stop_reason` (`no_seeds` without a walk) and `estimate` is the fraction of the `likely_complete` hint (the mark-recapture estimate over what the maps found; `null` until two maps share a node; maps over one graph read high, so it is a hint, never `complete`); a partial answer also gets one `notes` line that names the reason and the handle. Every card of `results` and of the checkpoint's `answer`, `visited_cards` and `frontier` gains `freshness`, one of `fresh`, `changed`, `stale`, `unverified`: `changed` when `file_changed` is `true`, `fresh` when it is `false` (the card's own file is as its maps read it, so its text is current even under a stale snapshot), else from `maps`: any map of the card (`maps` and `via`) that is `stale` gives `stale`, every one found with a lag gives `fresh`, anything else is `unverified`, which is what every card says with `map_freshness` and `verify_on_read` off. One exception, for `lsp.live`: when the walk found a language server running (a session kept across calls under `[run_profile.lsp] sessions = "shared"`, the case the `lsp.live: kept session` note reports) every card that cites `lsp.live` is `unverified`, never `fresh`, because nothing checks what a kept server read (a `changed` or `stale` finding stands); a card from a server this walk started keeps `fresh`. The rule is per walk, not per card: with several servers, any reused one makes all `lsp.live` cards `unverified`. A continuation (`handle`) cannot tell whether the earlier call used a kept server, and the checkpoint format carries no such flag, so every card of a node restored from the checkpoint that cites `lsp.live` stays `unverified`; the cards of nodes the continuation's own walk reaches follow the rule above. Nothing is withheld with either value (`results` are the same), and the trace, the checkpoint hash, the handle and `text` do not change; with the key off the answer is byte-for-byte as without it. In the profile hash; recipe answers have no labels. -- Paths on cards (`[run_profile.traverse] card_paths = true`, default `false`): every card of a traversal answer's `results` and of the checkpoint's `answer` that the walk reached then has `kind`, the kind of the edge that discovered the node (`null` for a start node), and `path`, the route from its start node: one `{node, map, kind, witness}` entry per node (`depth + 1` entries, the first is the start node with `kind` and `witness` `null`, the last is the card itself and repeats its `via` and `witness`). The route is the walk's final one (a stronger route found later replaces a weaker one). The cards of `visited_cards` and `frontier`, the cards of the first stage (`origin` `exact` or `seed`), the trace and the checkpoint hash are unchanged; with the key off nothing in the answer, the trace, the checkpoint or the handle changes. A handle made with the key on also keeps each state's `kind` (an additive key; the handle format version is unchanged), so a continuation shows whole chains; a handle made with it off has no kinds for its earlier hops, which show `kind: null`, and a note says so. Size, measured as the answer's characters divided by 4: on the unit-test fixture (4 cards, depth up to 2, a complete walk) 616 tokens off, 856 on; on one real walk (`lci traverse` with `graph.scip` over an older index of this repository, one start node, 8 cards at depth up to 3, ended by the pop cap with a checkpoint) 3807 tokens off, 5160 on, 35.6 percent more, because the checkpoint's `answer` repeats the cards; the `results` list alone grows 87.5 percent (3095 to 5802 characters), about 85 tokens per card per list. Cards at greater depth cost more, one entry of about 25 tokens per hop. -- Language-server sessions of `lsp.live` (`[run_profile.lsp] sessions`, `"shared"` default or `"per_call"`; it matters only when a call runs `lsp.live`, which a call with a tree under `shared` and the service's manager (MCP and REST `mode = "traverse"`) runs by default, `[run_profile.lsp] live = true`, and which the CLI one-shot `lci traverse` and `per_call` run only when `--maps` or a `[traverse.policies]` entry names it, because a server that dies with the call cannot amortise its start; with no language server installed, or one that will not start or is refused, the walk says so in a note and goes on without the map; `live = false` makes the map opt-in everywhere, and a call that names its maps is never changed by the key): under `per_call` every traversal call starts its own language servers (a cold start and a version probe of each enabled server) and stops them when its walk ends. Under `shared` (the default) the calls of one process may keep a server across calls, but only a server that declares the configuration files its tree stamp covers (`LspAdapter::stamp_manifests`; every adapter but jdtls and marksman does) and either sees an edit on its own, today rust-analyzer alone (`LspAdapter::watches_workspace`; its client capabilities here declare no `workspace.didChangeWatchedFiles`), or is told by `[run_profile.lsp] sync = "watch"`, which is the default since K11b-4. So by default the service's manager keeps every server but jdtls and marksman (rust-analyzer, csharp-ls, typescript-language-server, pyright, gopls, clangd), and a service with `[traverse] enabled = true` primes the ones of the index's languages at the first touch of a workspace. With `sync = "off"` rust-analyzer alone is kept, and every other server (csharp-ls and typescript-language-server are told nothing about an edit; pyright, gopls and clangd are resent only the file a request names) is started by each call on that call's own manager and stopped when its walk ends, exactly as under `per_call`; jdtls lists no configuration files and is never kept, whatever `sync` says. The reason for the default: a server started per walk cost 266 to 524 ms per expansion on python, typescript, go and csharp trees against 2 to 5 ms for a kept one, with identical cards (a measurement on those trees, not a bound). A kept server does not rely on its own file watcher or on the sync for correctness: each walk that uses it first passes a tree barrier. The barrier takes a stamp of the tree from the index's file list and one directory read per stamped directory, with no recursive walk: the server's indexed files (modification time and length); every directory that holds one and its ancestors up to the workspace root (modification time and child names); one level of child directories of those (modification time), except version-control and build-output directories (`.git`, `target`, `node_modules`, `obj`, `bin`, `build`, `dist`, `out`, virtual environments), so a file created in an existing unindexed `src/generated/` that a `#[path]` attribute names is seen; and the server's configuration files among those children, matched by name (rust-analyzer: `Cargo.toml`, `Cargo.lock`, `rust-analyzer.toml`; typescript-language-server: `tsconfig.json`, `tsconfig.*.json`, `jsconfig.json`, `package.json`; pyright: `pyrightconfig.json`, `pyproject.toml`, `setup.cfg`; gopls: `go.mod`, `go.sum`, `go.work`; clangd: `compile_commands.json`, `compile_flags.txt`, `.clangd`, `CMakeLists.txt`; csharp-ls: solution and project files, `Directory.Build.*`, `Directory.Packages.props`, `global.json`, `nuget.config`). Only "not found" counts as absent: any other failure to stat or read makes the stamp fail, and the server is then restarted and nothing is recorded. Each walk passes the barrier before the server is used (a walk that runs `lsp.live` on the shared manager passes it twice, once before its seed search and once in the backend, so it restarts a stale server once: the second restarts it again only if the stamp changed between the two; one at a time per workspace, so a slower barrier cannot record an older stamp over a newer one) and compares the stamp with the one recorded when the server last started or passed; when it differs, or the server was started by a call that passed no barrier, the server is stopped and the walk starts a fresh one that reads the disk, and the walk says so in a note (`lsp.live: restarted: the tree changed since its last call`). An unchanged tree keeps the server, and the walk says that too (`lsp.live: kept session; tree stamp unchanged since s ago`), so an answer from a reused server is distinguishable from a fresh `per_call` one. The note comes from the session the walk's requests actually acquired (found running, or started by that walk), not from the barrier, so it is also there when the server was started by an overlapping walk that passed its barrier at the same time. The rule: a reused server is never staler than `per_call` for any change inside the stamp's coverage. It is not an absolute guarantee (that would need a full content walk, which cost 1.5 s on deno): the residuals are a new file two or more levels below a stamped directory or inside a skipped build-output directory; a configuration file that is not on the server's list; an edit that keeps both modification time and length; and an edit made during the call, which `per_call` has too. `sync = "watch"` does not avoid a restart after an edit; it tells a running server about the edit, which is what makes a non-watching server worth keeping, and it narrows the residuals above for navigation and the search channel. Two residuals follow from the default: a workspace whose platform watcher and poll fallback both failed, or whose Windows watcher silently stopped, keeps a non-watching server on the strength of the tree stamp alone, so it is told nothing but still passes the barrier (the poll fallback itself scans the tree every `watch_poll_milliseconds`, which is the expensive case on a very large or remote tree); and the search channel restarts a synced non-watching server after an edit, a loss of the channel for a few seconds that rust-analyzer's exception below avoids. A workspace with no index snapshot has no file list and so no sharing: the server is restarted by every walk. On a large repository the stamp costs tens of milliseconds (decision d454), plus one read of the workspace manifest after each index change; navigation and the search channel pass the same barrier, once per call, and carry the same notes (`lsp: restarted: ...` and `lsp: kept session; ...` in the `notes` of a navigation answer and of a search report; a server the call started adds none). The one exception is the search channel and a server that watches its workspace (rust-analyzer): an edit would restart it and the next searches would lose the channel while it loads, so the search keeps its session through a changed tree and says `lsp: rust-analyzer kept session; (kept: it watches its workspace)`, with no tree stamp recorded for it, so the next navigation call or traversal still restarts it; that answer is as fresh as the server's own watcher, not as the stamp, so it can miss an edit inside the watcher's latency and a `Cargo.toml` or `Cargo.lock` edit while the server reloads its workspace. Off switch: `sessions = "per_call"` (it moves the profile hash); `lsp_idle_timeout_seconds` only limits how long an unused server stays, and a kept server holds memory while it runs. Peak resident size measured during walks (a measurement on the benchmark trees, neither a limit nor an idle figure): rust-analyzer about 4 GB, typescript-language-server about 460 MB, csharp-ls about 190 MB, pyright and gopls about 120 MB each; a workspace in all of those languages adds about 0.9 GB (the sum of the four non-rust-analyzer figures) to rust-analyzer's 4 GB. The idle timeout (default 1800 s, `0` never releases) releases a session that has started and primed, or whose start failed, once nobody has used it for that long; a start or priming still in progress is not released, so it bounds the residency of servers that finish starting, not of one that is slow or stuck starting. The kept servers are the service's own manager's, the one that navigation (`find_definition`, `find_references`, `search_symbols`) and the search channel use, so a server started by any of them serves the others; `lci traverse` is one call per process, so there `shared` differs from `per_call` only in that the kept servers stop cleanly when the walk ends; its manager is built without the sync, so there only rust-analyzer is kept and every other server starts per call. A shared server stops like any other of the service: when it has been idle for `lsp_idle_timeout_seconds`, on `release_workspace`, when its session is replaced (stuck or failed) and at service shutdown (Ctrl+C, SIGTERM); the end of a walk does not stop it. The version probe of a server binary runs once per manager under `shared` (`[run_profile.lsp] breaker` is `true` by default: a provider whose version probe failed (a binary that is missing, or a probe that timed out, opens no window: that is no evidence of a dead server), or whose server failed to start without timing out, is skipped for 30 seconds, with a note that says so, by the service's manager and each call's alike; a server already running is still asked, a start that timed out opens no window, and `/ready` still probes. With `breaker = false` a failed probe is run again by the next call and a dead server is started again). The recipes (`flow_from` and the other `lci_*` tools) walk only `graph.scip` and `structure.contains`, never `lsp.live`, so no language server starts under a recipe and they run the same under both values; the key still reaches them in their run profile, so the profile hash they carry is the one that ran. `[run_profile.lsp] implementations` (default `false`) adds a third request to each symbol node of an `lsp.live` expansion, `textDocument/implementation`, behind the same `max_external_calls` gate as the other two: its answers become `implements` edges from the node to the symbols (or files) at the places the server names, the direction `scip.implements` uses (an interface or trait member to the members that implement it). The place table is the SCIP index's, whose nodes are functions and methods, so the request is asked at an interface or trait member, not yet at the interface or trait itself. An error reply from the implementation request is noted and costs the other two requests' edges nothing (a hang trips the call's deadline and clears them all). The recipes ask no implementations either way. `[run_profile.lsp] call_hierarchy` (default `false`) adds up to three more requests to each symbol node, each behind the same gate and counted: `textDocument/prepareCallHierarchy` at the node's name, then `callHierarchy/incomingCalls` and `callHierarchy/outgoingCalls` of the item that answers (an empty prepare ends it, so one more call at least). Callers and callees become `calls_or_refs` edges (no new edge kind), one per destination, whichever way the call runs, seen at the call itself; a symbol or file that `definition` or `references` already named keeps that edge. An error reply (a server without call hierarchy answers -32601) is noted and costs the other edges nothing; a failed prepare ends the block and a failed incoming request still asks outgoing. Servers miss some calls (rust-analyzer's call hierarchy missed 16% of the call pairs the SCIP index has, in macro arguments), so this complements `graph.scip`. The recipes ask no call hierarchy either way. -- Verify on read (`[run_profile.freshness] verify_on_read = true`, default `false`): with `map_freshness` also on, a traversal answer checks what it cites against the working tree by content hash. It reads and labels; it never refreshes the index and never changes `[index].freshness` (under `manual` the snapshot stays as old as it was). It reuses the one read of the tree that `map_freshness` makes (no second scan), and with the key off every answer is byte-identical to before. What it adds: (1) the maps that answer from the index snapshot (`seeds.hybrid`, `search.lexical`, `search.vector`) get a `maps` entry `{id, observed: "snapshot", stale, lag}` whose `lag` counts the source files that differ between the snapshot's manifest and the tree (changed, added, or removed; the manifest's prose entries are not counted, as the tree scan leaves prose out). It is read after the walk, because the walk's first search may refresh the snapshot under `on-search` (a refresh that lands between a map's read and this manifest read, as under `[index].freshness = "watch"`, makes the answer look as current as the new snapshot); `null` (not compared) when the call has no tree or no manifest. (2) `grep.live` and `lsp.live` keep `observed: "live"` (`grep.live` reads current bytes; `lsp.live` queries a language server, which reads the disk when this call started it and is bounded by its tree stamp's coverage when kept, see the language-server sessions item and the `kept session` note) and gain `places: "index"`: where they put what they read comes from the SCIP index, so their `stale` and `lag` are the SCIP index's (the same count as the SCIP-built maps), and a line shift since the index shows as stale. They are the only entries with `places`. (3) Every card (`results`, and the checkpoint's `answer`, `visited_cards` and `frontier`) gets `file_changed`: `true` when the content hash of the file it cites (its `site.path`, else its node's path) now differs from the hash its maps read, `false` when it does not, `null` when no reference applies (a commit or other card with no file, a file that is not a source file, or no tree was read; one note says how many cards are `null`). `true` also covers a file that was newer than the index file when this process first saw the index's stamp: its hash at index time is unknown, so it counts as changed whatever its content is. The reference is the SCIP baseline for the maps built from the index, for `grep.live` and `lsp.live`, and for a symbol card (its site is the index's, whichever map reached it), the snapshot manifest for `seeds.hybrid`, `search.lexical` and `search.vector` and for a chunk card; a card that names both is `true` when either differs. A `search_code` / `search` report (JSON; the compact text output is unchanged) gets a `freshness` block with the key on, with or without `map_freshness`: `snapshot` (`{stale, lag}`: the source files that differ between the snapshot's manifest, read after the search, and the tree, which one read of the tree's content hashes gives, gated by `stat_gate` when it is on), `channels` (`{id, observed, stale, lag}` for the channels that ran, so a channel that was off or not reached (no language server applies, no vectors) is not listed, and one that failed is listed with a `note` (`failed; see warning`) and no lag, because it answered nothing: `vec.chunk.code` and `lexical` are `observed: "snapshot"` with the snapshot's lag; `lsp.symbols` is `observed: "session"` with `stale` and `lag` `null` and a `note`, because the language server's view of the tree is not compared), and `hits` (one `{path, file_changed}` per result in order; `file_changed` is `null` for a hit no snapshot channel returned (only the language server, or no channel named), or when the tree was not compared, which a note says). The block is read after the search, so a refresh of the snapshot that lands between the search and that read (`[index].freshness = "watch"`) is not seen: the answer then looks as current as the new snapshot. The scan costs `timings.verify_ms` (absent when the key is off), and only a search a caller asks for pays it: the searches that find a traversal's start nodes and the evaluation runner do not scan. Under `manual` or `watch` a search never refreshes the snapshot, so the block is how a caller sees that it is behind; under `on-search` the search has just refreshed it and the lag is 0. A search with the key off has no `freshness` key. The recipes take the key through the run profile but their answers carry none of this. -- Stat gate (`[run_profile.freshness] stat_gate = true`, default `false`: every read of the tree for `map_freshness` or `verify_on_read` reads and hashes every source file). With `true` the process keeps each source file's modified time, length and content hash per root, lists the tree without reading it, and reads and hashes only the files whose modified time or length is new or changed (a file with no modified time is always read; a file skipped by its content, such as a Qt Linguist catalog saved as `.ts`, stays skipped while its stamp stays). The labels and `file_changed` equal the full-hash path's, through the same hash function, with one known gap: a same-length edit that keeps a modified time older than 3 seconds before the previous read of the file (a deliberate `set_modified`, or a clock that went back) is not seen until that file's next edit. A file modified within 3 seconds of a read is not trusted and is read again on the next call, which covers a write in the same clock tick and the coarse stamps of FAT, exFAT and SMB shares. The table lives in the process and is rebuilt on the first call; the index scan is not gated. The key is in the profile hash. -- Outline cache of `lsp.live` (`[run_profile.lsp] outline_cache`, default `false`: every file outline the map needs, a `textDocument/documentSymbol` request, goes to the server). With `true` the cache works only for the servers that hold a file's current text at every request: pyright, gopls, jdtls, clangd and typescript-language-server, whose adapters send the file's text from disk before each request when it differs from what the session last sent for that file: a `textDocument/didOpen` the first time, then a `textDocument/didChange` carrying the whole text when the server advertised change sync in its `initialize` result (full or incremental; LSP 3.17 reads a change without a range as the whole text), and a second `didOpen` when it advertised none (pyright logs the repeated open and applies it; gopls overwrites its overlay; jdtls replaces the working copy's buffer; clangd's open replaces the contents; typescript-language-server refuses a second open, which is why the change matters for it). A session holds at most 64 open documents per server (a size, not a key): a new file past that closes the least recently used one with `textDocument/didClose` first, and a closed file is opened again, at version 1, when it is next navigated. A text is sent once, and a request that follows a send waits for the server's `publishDiagnostics` for the file (at most the request timeout and 3 s) before it goes out. For those the language-server manager reads and hashes the file (the content hash the index uses, sha-256 over up to 16 MiB, on every outline request) before it asks, and keeps each non-empty outline under that hash and the server session that answered; the same file is then answered from memory while its bytes and that session are unchanged. An edited file is asked again and gets the outline of the new text, and so is any file after the server was restarted, released, replaced or stopped idle. An empty outline or an error is never kept (a server still loading answers `[]`), and neither is an answer for a file whose bytes changed during the request, an answer from a session that is no longer the live one, a file that is not UTF-8 or one over the 16 MiB open limit. The other servers are asked every time, as without the key: rust-analyzer watches the disk itself and can answer from the text before an edit for a moment after it, and csharp-ls is sent no document text. A manager keeps at most 4096 outlines; a new one past that clears them. The cache belongs to the manager: under `sessions = "per_call"`, and for a server that a `"shared"` walk starts on its own manager (every one of these with `sync = "off"`, jdtls and marksman always), it lasts one walk; in the service's manager under `"shared"` with `sync = "watch"` (the default) it serves later calls too. The recipes never run `lsp.live` and run the same under both values. -- Language-server sync (`[run_profile.lsp] sync`, default `"watch"` since K11b-4; `"off"`: a language server walks the tree when it starts and is told nothing after, so a long-lived session answers `search_symbols`, the search channel and a shared `lsp.live` from the tree it started on, and a server that does not see an edit on its own is not kept across traverse calls). With `"watch"` the service's language-server manager (the MCP and REST server's, which navigation, the search channel and `sessions = "shared"` traversals use; not the one-shot `lci traverse`'s, which is built without the sync because its process ends with the walk) watches each workspace that has a running session and sends every change of a source file to that workspace's servers as one `workspace/didChangeWatchedFiles` (created, changed or deleted, as the file is on disk), and drops the changed files' kept outlines. The watcher is the platform's (ReadDirectoryChangesW, inotify, FSEvents), or a poll every `watch_poll_milliseconds` when it cannot start (the poll compares modification times in whole seconds, so a second write within the same second is sent only with a later change). It starts with the workspace's first session, walks the tree once to know its files, walks again only when the platform reports lost events (inotify and FSEvents do; on Windows a watcher overflow during a large burst, such as a big checkout, is not reported and silently ends the watching until the workspace's sessions stop, roadmap N102), and stops when the workspace's last session stops. Single changes are not filtered by `.gitignore` (a source file written under an ignored directory, such as `target/`, is sent), and a renamed or moved directory is seen as the directory alone, so the files inside it are not sent. Before each request the manager sends what the watcher has already reported: the change notification goes out before the request, but the manager does not wait for the server to apply it, so the answer to that request may still be from the old text; an event the watcher reports after that send goes out before the first request that drains after the report, and an edit the platform has not reported yet is in a later answer. It never waits for a walk. Prose files are not sent, nor anything under `.git`. It is independent of index watching (`watch`): a workspace can be watched for the index, synced, both or neither. A per-call manager (`sessions = "per_call"`, and the one a `"shared"` walk uses for a server that does not see an edit on its own) never syncs: its servers start on the tree as it is and stop with the walk, so there the key changes nothing but the profile hash. A shared traverse walk passes the tree barrier (see "Language-server sessions of `lsp.live`") and restarts a kept server after an edit whatever the sync did, so for `lsp.live` the sync does not avoid a restart: it makes a non-watching server worth keeping, and it covers what the barrier's stamp does not (a new file two or more levels below a stamped directory; an edit that keeps modification time and length) for the answers of a kept server. Navigation and the search channel pass the same barrier, so the sync does not decide whether they answer from the tree as it is either. The cost of the default: one watcher thread and one task per workspace with a running session, one table of `(modification time, length)` from one walk of the tree, and under the poll fallback a scan of the tree every `watch_poll_milliseconds`. When both the platform watcher and the poll fail to start, the servers are told nothing and a warning is logged. The recipes never run `lsp.live` and run the same under both values. -- Place table of `lsp.live` (`[run_profile.lsp] places`, default `"auto"`: per walk, `"scip"` when the SCIP graph read for the workspace has at least one node (the walk is then the `"scip"` walk), else `"outline"`, so a tree with no index read, `scip.mode = "off"` or `"cache"` with no file, still gets `lsp.live` cards. The choice is per workspace, not per language: an index with nodes that lacks a language of the tree (a Rust index in a tree with Python files) still takes `"scip"` places, and that language gets no `lsp.live` cards while its server still starts; set `"outline"` for such a tree; `"scip"`: the SCIP index's table alone, so a symbol the index has no node for has no place, `lsp.live` asks nothing about it, and under `scip.mode = "off"` it asks nothing about any symbol; a file seed still gets its `textDocument/documentSymbol` outline, but its members are named only when the index defines them). With `"outline"` the table is the index's plus the language server's own outline of a file: the first time a walk expands a symbol the index places in a file, or expands the file, `lsp.live` asks the server for the file's `textDocument/documentSymbol` (one more external call per distinct file and walk; a failed request keeps the index's place and is not repeated) and from then on the outline's places win in that file, so lines inserted since the SCIP file was written no longer make it ask at the wrong line. An outline member takes the index's id for the same symbol when the index has one (same file and name, a function or method of either kind, nearest line; one symbol keeps one id across the layers), else an id of the file's real language with no container (`sym:csharp:Shapes.cs::Area/method`, equal names told apart in document order: `~1`, `~2`), so under `scip.mode = "off"` the functions, methods, types (class as `class`, struct as `struct`, interface as `interface`, except that rust-analyzer's trait stays `trait` in a Rust file), enums, modules, constants, fields (a property is a field) and variables of an outlined file have ids (`sym:csharp:Shapes.cs::Circle/class`). Enum members, impl blocks and type parameters have none. A file expansion lists the fields as `Contains` destinations too, and a reference on a field's line now resolves to the field; `lsp.live` asks a field or variable like any other symbol (a server may answer `implementation` for an interface property or a call hierarchy for a `const f = () => {}` it reports as a variable). A handle saved under `outline` by an earlier build holds `struct` and `trait` ids for class-language members, which no longer match a learned place. A seed that is a symbol id of a file this walk has not outlined still has no place. Every `documentSymbol` range now ends on the item's last line (the nested `DocumentSymbol` shape used to give the name's line), which `Contains` edges read from the name's own line first. The recipes never run `lsp.live` and run the same under all three values. The place table also bounds what an arbitration can confirm (next item): the index's table holds functions and methods only. -- Language-server roots (`[run_profile.lsp] roots`, default `"workspace"`: every server starts at the workspace path, as always). With `"manifest"` a server starts at the directory of the topmost project file its adapter names (the list of the tree stamp above: `go.mod`, `go.sum` and `go.work` for gopls; `pyrightconfig.json`, `pyproject.toml` and `setup.cfg` for pyright; `tsconfig.json`, `jsconfig.json` and `package.json` for typescript-language-server; solution and project files for csharp-ls; and so on), so a module under `services/api/` is served from `services/api/` and its `rootUri`, `workspaceFolders` and working directory say so. The project files are looked for in the directories of the index's files of the server's languages and in their ancestors up to the workspace, one directory read each (the index's file list holds source files only, so it names no project file), with no recursive walk. The shallowest one decides. The workspace is kept, and `index_status` says why in `servers[].root_reason`, when no project file is found (`no_manifest`), when sibling directories at the same depth each hold one (`several_roots:`: one session per module is not built, so a monorepo of sibling modules keeps today's root), when the chosen directory would leave an indexed file of the server's languages outside it (`files_outside_root:`), when the index has no file list (`no_file_list`), when a directory cannot be read (`unreadable`) and for a server that is disabled, unconfigured or serves no language of the index (`not_served`, reported under `"manifest"` only; under the default every server reports `roots_workspace`). A session fixes its root when it starts and keeps it. The one-shot `lci traverse` under `sessions = "per_call"` has no index file list, so under `"manifest"` its servers start at the workspace (`no_file_list`) and its output does not say so; the service always has the list. There is still one session per server per workspace, answers still name paths relative to the workspace, and the tree stamp and the barrier still read the whole workspace. The recipes never run `lsp.live` and run the same under both values. What was measured (a premise check on small fixtures, pyright 1.1.414 and typescript-language-server 6.0.2, not a general bound): `"manifest"` helps pyright when a name resolves only through a nested project's own settings, such as `extraPaths` in its `pyproject.toml` (the definition is empty from the workspace root and found from the project directory); plain imports in a src layout resolve from either root. typescript-language-server already finds the nearest `tsconfig.json` from the workspace root, `paths` aliases included, so `"manifest"` gains nothing for it. Any indexed file of the server's languages outside the project directory, even one script at the top of the repository, sends `"manifest"` back to the workspace root (`files_outside_root:` in the roster), so the gain appears only when every such file lies under the project. gopls and csharp-ls, other pyright effects of the root and cost were not measured. The default stays `"workspace"`. -- Arbitration of SCIP edges (`[run_profile.scip] arbitration`, default `false`): with `true`, `lsp.live` in the call, `scip.mode` not `"authoritative"` and at least one of `scip.impact`, `scip.implements` (and only with `[run_profile.lsp] implementations = true`) or `structure.contains` registered and built, the server is the authority over those maps before fusion. Per expansion and claim family where the server offered at least one edge to a function or method (`Refers` for `scip.impact`, `Implements` for `scip.implements`, `Structure` for `structure.contains`), an index edge to a function or method that the server also names is confirmed and one it does not name is removed and counted as overruled. A family the server offered nothing in, an answer cut at the per-node edge limit, an answer that named only a file, and destinations that are not functions or methods (under either `places` arm) leave the index edges standing and uncounted. `graph.scip` is never arbitrated: its edges point from a caller to its callees, a direction the server's references do not answer. An edge only the server has is not a disagreement. The counts (`confirmed`, `overruled`) are per call: in the trace on the `scip.impact` (etc.) call of each `expand` event, in `totals.arbitration`, in `totals.per_map[].arbitration` and in one note `scip.arbitration: lsp.live judged (...) in this call: edges confirmed, overruled`; a map that was not judged prints none. The `start` event's `arbitration` object names the arbitration (authority, judged `[map, family]` pairs, kinds) and carries no counts. An overruled edge can be a false one: the server's answer at a node may be partial (a file it has not opened, a kept session behind an edit), and under `places = "outline"` an index member and its outline twin may carry different ids. A handle records the arbitration (its authority, judged pairs and kinds) and refuses a resume under another or under none. With the key off, with `scip.mode = "authoritative"` or without `lsp.live`, nothing changes in an answer. -- Entity seeds (`[run_profile.seeds] bm25`, default `false`: the map `seeds.bm25` does not run, naming it in `maps` gives a note and no seeds, and every answer is as without the key). With `true` the map is in the default set of a call that names no maps, and runs when named. It ranks the SCIP graph's functions and methods by BM25 (the scorer of the ripgrep channel, `lexical::score_chunks`) over three fields of each node: the tokens of its name, of its qualified path (the symbol's package, modules, types and member, and the file's path) and of its documentation (the `documentation` the index holds for the symbol, up to 1024 bytes; empty when the indexer writes none, and it is not verified that `rust-analyzer scip` writes any; it is read from the index file, a second parse, only when the map first runs for an index stamp, and never with the key off). A question word counts when it is a whole token (up to 4 characters) or inside one (longer); words under 3 characters and stop words are not terms, and a word in more than a quarter of the nodes is dropped unless no word would remain. Of the top `[traverse] seed_k` nodes, those the hybrid search did not already give (so possibly fewer) go after its nodes (`seed_source` `hybrid+bm25`, each with a `seed_hits` record `how: "bm25"`; `seed_source` `hybrid` stays when it adds none), or are the start nodes when the hybrid search gave none (`bm25`; the name-token stand-in then does not run, and `seed_fallback` says why the search did not answer). It reads the SCIP graph only (no `--root` needed), builds its index on the first call that wants it and keeps it with the graph for the index stamp. The recipes' seeds are given, so they run the same under both values. -- Repo-map seeds (`[run_profile.seeds] repo_map`, default `false`: the map `seeds.repo_map` does not run, naming it in `maps` gives a note and no seeds, and every answer is as without the key). With `true` the map is in the default set of a call that names no maps, and runs when named. It ranks the SCIP graph's functions and methods by personalized PageRank (damping 0.85, at most 30 power iterations) over the call and containment edges (`implements` edges are left out, each caller and callee pair counts once), with the random jump on the question's best 64 `seeds.bm25` matches: a function that many of the matched functions call becomes a start node even when its own name holds none of the question's words. A question with no entity match gives no seeds and a note (no global hub is ever a seed), and so does an empty graph. Of the top `[traverse] seed_k` nodes, those the earlier sources did not give go after theirs (`seed_source` `hybrid+repo_map`, `hybrid+bm25+repo_map`, `bm25+repo_map`, each with a `seed_hits` record `how: "repo_map"`, `score_kind: "pagerank"`, `personalised` and `iterations`), or are the start nodes when none gave any (`repo_map`). It reads the SCIP graph only (no `--root` needed), builds its edge lists and the `seeds.bm25` index on the first call that wants it and keeps them with the graph for the index stamp. The recipes' seeds are given, so they run the same under both values. -- Literal seeds (`[run_profile.seeds] literal`, default `false`: the map `seeds.literal` does not run, naming it in `maps` gives a note and no seeds, ripgrep is not started for it, and every answer is as without the key). With `true` the map is in the default set of a call that names no maps, and runs when named. It searches the tree with `rg --fixed-strings --word-regexp` (case-sensitive, `_` counts as part of a word, the source files of the language adapters, at most 20 lines per file) for the names the question puts in code spans (the last segment of each `a::Name`, at least 2 characters, at most 4 names), and places each line in the SCIP graph: the symbol defined on that line (`how: "definition"`), else the innermost symbol around it (`occurrence`, counted per symbol), else, when the line declares the name (`trait`, `struct`, `fn`, `class` and the like right before it, or `impl .. for` it), the file's node (`file`: a type or trait the graph lacks, which `structure.contains` expands to the symbols it holds), else the line is dropped. Definitions (3 per name at most) come first, then occurrences (most lines first), then files; nodes the other sources gave are left out and the list is cut at `[traverse] seed_k`. They go after the hybrid search's and `seeds.bm25`'s nodes (`seed_source` `hybrid+literal`, `hybrid+bm25+literal`, `bm25+literal`), or are the start nodes when those gave none (`literal`). It is not `[traverse] exact_seeds`, which turns the first stage's exact cards into start nodes. It needs the application's ripgrep (a question with `--root`); a call without says so in `notes` and goes on. The recipes' seeds are given, so they run the same under both values. -- Every JSON result of `index`, `status`, `list-files`, `search`, `symbols`, `definition`, `references`, `evaluate` (the report, also when written with `--output`), `traverse` and `recipe` carries the same top-level `profile_hash` as the MCP and REST results (`derived.run_profile_hash` of `config --json`; `traverse` hashes the configuration after its command-line overrides). `config --json` reports the hash under `derived`; `compare`, `replay` and `init --json` run offline without a resolved configuration and carry none. An error prints no result. - -## Retrieval evaluation - -- `evaluate` runs a portable definition through the same application search path: `"$lci" evaluate ./evaluations/core.toml --workspace "self=$PWD" --workspace "gust=/path/to/gpu-dialect-v0" --output ./test-results/evaluation.json`. Workspace mappings are supplied at runtime, so checked-in definitions hold no machine paths. The GUST workspace is optional; its queries are skipped without a mapping. Without `--output`, JSON goes to stdout. A required missing workspace, search error, expected-path miss, wrong expected role or missing required snippet fails the query and gives a nonzero exit after the report is emitted; valid optional skips do not fail the run. `evaluate --baseline ` also embeds a `comparison` and prints its summary to stderr (`--seed`, `--resamples`, `--keep-failed`). With `[channels] lsp` on, `evaluate` first starts and primes every enabled language server for each workspace it queries (rust-analyzer is always enabled, so a missing binary is recorded as a failed warm-up), so no query pays a cold start; the outcome is the report's `priming` block (per workspace key and provider: `primed`, `ms`, `error`). `--no-prime` keeps the cold first query for a cold-start measurement. Priming changes which query pays the start, never a ranking. -- Arms: run `evaluate` once per configuration with `--config` and name the outputs by arm: the default, `[channels] reranker = false` (fusion order alone), or the grep-only baseline (see Channels). Reports do not record the switches, and the switches never invalidate an index, so arms can run one after another on one `data_dir`; never two at once on the same one. -- `compare` measures the difference between two saved reports without services, configuration or index: `"$lci" compare test-results/fusion.json test-results/semantic-only.json --output test-results/fusion-vs-semantic-only.json`. It pairs queries by workspace and query text (both reports must come from the same definition) and prints, for MRR, nDCG@10 and hit@1, the baseline and candidate means, the mean per-query delta (`candidate - baseline`) with a seeded 95% paired-bootstrap interval (`--resamples`, default 10,000; `--seed`), whether the interval excludes zero, and reciprocal-rank wins, losses and ties. Queries in only one report, or skipped in either, are excluded and counted; so is a query in which either side's language server missed `lsp_search_budget_ms` (`metrics.lsp_budget_hit`: it measured a cold start, not retrieval), counted as `budget_hit_*` on the pairs line and, for tagged definitions, per tag (`budget_hit_by_tag`), unless `--keep-failed` keeps it; a definition with `tags` also gets one block per tag. A regression is not an error: the exit code is nonzero only when a report cannot be read or parsed, or `--output` cannot be written (the summary is printed first). A delta whose interval contains zero is noise at that query count. -- Reports hold bounded path, line-range, role, rank, score and preview evidence; per-query hit@1, hit@3, hit@8, reciprocal rank, nDCG@10, recall@3/@8, file-level accuracy@5/@10, preference/disfavor counts, role distribution, reranker state, `lsp_budget_hit` (with `lsp_providers_over_budget` when set; a search response names them as `lsp_over_budget` only when a provider missed the budget), timings, lifecycle and effective filters; and aggregates: hit rates, MRR, means of the rank-aware metrics, `lsp_budget_hit_queries` (and per tag under `by_tag`), median/p95/max latency with sample count `n` (total and per stage; nearest-rank p95 hides the worst query at small `n`, for example it is the second-highest of 35, so read `max`), fallback count, and created/reused/waited/refreshed index counts. nDCG@10 uses graded gain (`expected_paths` match 2, `preferred_paths` only 1) and credits each gold pattern once; recall@k is the fraction of distinct expected patterns in the first k; file accuracy@k counts distinct files. The `@10` metrics are bounded by the query's `top_k`: hand-written `core.toml` uses 8 (effectively @8), the git-mined set 10; use `top_k >= 10` to compare with literature @10 numbers. Reports belong under `test-results` and are not committed. Evaluation reuses compatible persisted indexes and calls only the configured embedding and reranking services. -- Offline scoring: `evaluate --dump-candidates` records every query's whole fused candidate pool, the dumped configuration and the reranker's scores; `LCI_CANDIDATE_DUMP=` appends the same pool per search of a running process to a JSON Lines file (read once at startup; MCP, REST and CLI responses are unchanged). `"$lci" replay test-results/dump.json --output test-results/replay-w12.json --window 12 --cap 2 --cap-stage results` then scores other windows, caps, fusion weights, `rrf_k`, priors and rerank modes from the dump without services, configuration (the `--config` file is not read) or index, and `compare` pairs the output with real reports. A query replay cannot score exactly is refused and counted, never approximated. Row format, replay semantics, the latency model and when to run live instead: `.agents/skills/lci-retrieval-evaluation/references/evaluation-workflow.md`. -- Git-mined query set: `cargo xtask mine-queries --repo --key --out evaluations/mined/.toml [--max 300]` turns qualifying commits from `git log` into queries (subject plus first body paragraph as the query, touched supported files as `expected_paths`, `top_k = 10`). `evaluations/mined/self.toml` (workspace key `self`) is the generated set for this repository: run it like `core.toml` (`--workspace "self=$PWD"`), regenerate rather than edit it, never use it to tune ranking. `core.toml` is the hand-written required set. Skip rules, labels and tags: same reference. -- Gold strata and transcript shapes: `cargo xtask gold strata --set --gold [--out ]` classifies each gold item as `core`, `grep_easy` or `agent_trap` from the item alone and reports the counts, the hop quantiles, the items no rule reaches and the items whose question holds a gold file's stem; the same `guard` is a stratum of every scorer. `cargo xtask mine-transcripts --input .. [--gold ] [--out test-results/transcripts/shapes.json]` counts question shapes and trace classes over synthetic transcripts in the sim trace shape and writes only shapes, counts and anonymised templates. Rules and schema: [gold-scorers.md](docs/spike/multimap/v2/gold-scorers.md). -- Live acceptance: `cargo xtask acceptance core` and `acceptance lsp` default to this repository (`--profile gust --workspace /path/to/gpu-dialect-v0`, or `$LCI_ACCEPTANCE_GUST_WORKSPACE`, runs them against the optional external GUST example); `lsp` checks workspace-symbol, definition and reference navigation against the same symbol and saves each normalized response in `test-results`. With the real embedder and reranker running, `cargo xtask acceptance multilingual` creates a disposable fixture and data directory under `test-results`, uses the real debug binary; it verifies all supported language IDs, semantic and lexical retrieval with live reranking, production TypeScript, Python and C# ranking over test decoys, zero-work unchanged indexing, one-file invalidation, deletion, restart reuse, and prior-snapshot retrieval after an unreachable embedding endpoint makes an update fail (`test-results/multilingual-*.json`). Per language, `-lsp`, `-missing` and `-recovery` run a real server (options and prerequisites: `.agents/skills/lci-lsp-adapter/references/acceptance.md`, `docs/development/live-acceptance-ci.md`); each skips with `PREREQUISITE_UNAVAILABLE` when its server is not found. -- Metric tiers (method and status in [evaluation.md](docs/spike/multimap/evaluation.md)): agent runs are the ground truth but too costly to run regularly, so cheap proxies run constantly and are checked against agent runs now and then, and quality is reported together with cost. - -| Tier | When | What | -| --- | --- | --- | -| 1 | every change, deterministic | hit@k, recall@k, span hit, MRR on the frozen sets; speed proxies (pops, hops, expansions); cost-aware recall (recall per 1k tokens returned); the simulated agent (design, not built) | -| 2 | milestones, cheap model in the loop | confidence calibration; a small fixed subset judged by a cheap model | -| 3 | rare, before releases | real agent A/B runs, mainly to validate the proxies | - -## Retrieval behavior - -The shipped search pipeline, in order; the code-level picture (types, constants, functions) is [docs/architecture.md](docs/architecture.md) "Chunks and the manifest", "Indexes and channels" and "Indexing flow and freshness". Traverse mode: [engine.md](docs/spike/multimap/engine.md). - -1. Scan the eighteen extensions above (any ASCII case: a `.RS` file is Rust, N94) recursively with `.gitignore`, nested ignores and standard `ignore` crate rules. The checked-in `.lciignore` is an additional ignore file that keeps agent control-plane material (`.agents`, `.github/agents`, `.github/instructions`, `.github/skills`, `.codex`, `.claude`, `.kilo`) and generated `test-results` out of this repository's searchable corpus while leaving them on disk for agents. Symlinks are not followed, `.git` is skipped, hidden source files are eligible when not ignored. A file that is not valid UTF-8 (for example Windows-1252 C source) is indexed from a lossy decoding (invalid bytes become U+FFFD, line structure kept) and reported in `lossy_decoded_files`; it never fails indexing or search. A Qt Linguist `.ts` catalog is left out and counted in `skipped_non_source_files`; it does not affect the staleness fingerprint, and a file that changes into one drops its chunks at the next refresh like a deleted file. A source file removed between the walk and its read, or whose path is not valid UTF-8, is skipped with a logged warning. Other read/traversal errors abort the update and preserve the previous index; under `on-search` freshness the search then reuses the previous snapshot with a warning. -2. Select a static language adapter and Tree-sitter grammar by extension. Chunks preserve: Rust, its original declaration boundaries; TypeScript/JavaScript, imports, executable top-level statements, exports, comments/decorators, functions, classes and methods, interfaces and signatures, type aliases, enums, variable-assigned arrow functions; Python, imports, assignments, executable top-level statements, sync and async functions, classes, methods, decorators and leading comments; C#, using directives, namespaces, top-level statements, types, members, attributes, XML documentation comments, local functions; Go, top-level functions, methods, type/const/var declarations and leading comments; Java, packages, imports, classes, interfaces, enums, records, methods, constructors, fields, annotations, Javadoc; C, functions, structs, unions, enums, typedefs, preprocessor macros, global declarations, leading comments; C++, namespaces, classes, structs, unions, templates, methods, fields, leading comments. Oversized syntax splits only at named syntax boundaries toward 6,000-byte groups, with declarations preserved up to 24,000 bytes; large indivisible leaves stay whole rather than truncated; parser error recovery remains searchable. Markdown (`.md`) is cut into heading sections: a heading line plus its own body up to the next heading of any level (ATX and setext headings), with fenced code blocks kept whole and never searched for headings, YAML front matter kept with the first section, and text before the first heading as its own chunk; a section over 6,000 bytes is packed by paragraph toward that size, and a fence stays whole. -3. Parsed chunks, language identifier and adapter version are persisted per file in the external manifest (fingerprints include path, content, language and adapter version), so a later manual or watched pass reparses only changed, new or adapter-version-changed files, also after restart. A missing, incompatible or malformed manifest safely causes a full parse. -4. Each exact source chunk is hashed; vectors from the current workspace snapshot are reused when content hashes and the stored embedding URL/model/document-format identity match, and duplicate new chunks are embedded once. Cached vectors whose width differs from what the embedding service now returns are embedded again: the width comes from the process's latest embedding response, or, in a process that has embedded nothing yet, from one single-attempt probe of a short prefix of the first chunk, sent only when cached vectors exist (a failed probe warns and keeps them). A width change no request has revealed since the process's last one is found by the next search, which serves no semantic hits and warns; the index run after it repairs the index. Parser adapter versions are decoupled from embedding compatibility, so unchanged chunk text keeps its vector. Documents are embedded as source, without a query instruction. -5. A complete workspace snapshot is built in memory and committed as a LanceDB table overwrite after embedding succeeds; the matching manifest is saved only after that commit. Removed chunks disappear from the snapshot; line-only changes update locations without re-embedding. This is a small/medium-repository design, not a streaming indexer for huge monorepos. -6. Queries are embedded with `embedding_query_template` (default `Instruct: Given a code search query, retrieve relevant code passages that answer the query` newline `Query: `). -7. Candidates come from three channels. Semantic: LanceDB cosine distance, the `semantic_candidate_count` nearest chunks that pass the request's filters (the fetch is widened when filters reject rows, up to 10,240 rows; a search stopping at that cap warns that farther matching chunks were not considered). Lexical: ripgrep over all supported extensions in one pass, scored in process: - - Terms are the query's tokens as written plus the sub-tokens of their camelCase, snake_case and letter/digit parts (`parseConfig` also searches `parse` and `config`), without generic English and commit-message words such as `add`, `fix`, `when`, at most 24 per query. A query with no usable term returns no lexical candidates, without a warning. - - Matching lines are credited with the terms each really holds: a term of five or more characters counts as a case-insensitive substring, a shorter one only as a whole word (`add_item` and `addItem` hold `add`; `padding` does not). Lexical lines map to chunks of any indexed language. - - Chunks score by IDF-weighted term evidence: a term's document frequency is the number of indexed files with a credited line; terms credited in more than a quarter of the indexed files are dropped (kept when every term is that common); the twelve rarest remain; a chunk scores the BM25 sum over them of `idf * tf * (k1 + 1) / (tf + k1 * (1 - b + b * length / average_length))` with `idf = ln((N - df + 0.5) / (df + 0.5) + 1)`, `tf` the chunk's credited lines of the term, `length` its line count, `average_length` the mean over the filtered index's chunks, `k1 = 1.2`, `b = 0.75`. Chunks matching only dropped terms are not candidates. The constants live in `lci-core/src/lexical.rs` and are not configurable. - - LSP: for each language present in the filtered index, its enabled language server also searches workspace symbols; locations map only to same-language chunks. -8. Candidates are deduplicated and fused with Reciprocal Rank Fusion: each channel adds `weight / (rrf_k + rank)` for a chunk it returned, a chunk found by several channels sums the terms, and the source-role prior is added afterwards, unscaled. The `[fusion]` weights default to `1.0` (plain RRF); each must be finite and greater than zero (disabling a channel is `[channels]`' job). Weights change only `fusion_score` and through it the fusion order and shortlist, never ranks, scores or the reranker's per-document input. After the order is settled and before the shortlist is cut to `rerank_candidate_count`, `[fusion].max_chunks_per_file` (default `2`; `0` is unlimited and changes nothing) walks the fused order, keeps a chunk in the head while its file (by relative path) has fewer than that many kept, and moves the rest to a tail; the result is head then tail, each in original relative order. Nothing is removed, so `top_k` is still filled from demoted chunks when there are too few distinct files; `fusion_score` is not rescaled, so a demoted chunk can follow one with a lower score. The reranker sees the capped shortlist, reranker-off and reranker-failure results use the same capped order, and the reranker's output is not capped again. The cap and the weights were measured on the mined set and `core.toml` (numbers: [decisions](docs/decisions.md)). Results expose channel ranks, lexical match count and score, fusion score and channels; the lexical score never feeds the semantic or reranker scores. The full contract: `.agents/skills/lci-retrieval-evaluation/references/retrieval-contract.md`. -9. The fused shortlist is reranked, each candidate sent as its path plus at most `rerank_document_max_bytes` of its code (cut at a UTF-8 boundary, keeping the head); results always carry the full chunk. The three channels fail independently and available channels continue; an LSP provider that fails is skipped for the rest of that search. Each provider gets at most `lsp_search_budget_ms` of wall clock per search, alongside the other channels: a server still starting (rust-analyzer waits up to about ten seconds for its first indexing pass) or slow is skipped for that search, which reports an informational `notes` entry rather than a warning because nothing failed. The skipped server is not discarded: its start (spawned at most once per provider and workspace) keeps running in the background, followed by one priming `workspace/symbol` request that builds its symbol index (8-30 s on a large Rust workspace; searches during it also report the note, and no search budget cancels the priming), and a later search uses the server. A partial answer gathered before the budget ran out is kept; `lsp_search_ms` in the timings is the capped wait. Channels and the reranker can be switched off on purpose with `[channels]` (silent, unlike a failure). If reranking fails, the fusion order is returned with a warning appended to any earlier warnings. -10. Documentation lane. Markdown chunks (the prose extractor's, not every chunk with the `documentation` source role) are split off after the request's filters and never enter the code pipeline above, unless the request names `source_roles` (then the filters apply as stated and one list comes back). The code `results` list is therefore exactly what an index without the Markdown files returns: semantic search, lexical statistics (BM25 average length and document frequencies) and LSP mapping all see code chunks only. The documentation lane runs beside it with the same query vector: semantic over prose chunks only (`docs_lane.semantic_candidate_count`), lexical over prose chunks only, no LSP, the same fusion and per-file cap, a cut to `docs_lane.rerank_candidate_count`, a second rerank only when `docs_lane.reranker` (or the profile's `docs_reranker`) is on (default off, whatever `[channels].reranker` says), and the best `docs_lane.top_k` (default 2) come back in `docs.results` with `candidate_count`, `reranked`, its own timings and `warning`. `docs` is present exactly when the lane ran and the index holds prose chunks. The lane is fail-open: a semantic, lexical or reranker failure sets `docs.warning` and never changes the code list or its warning. The compact MCP view lists the docs after the results and shares its response ceiling. - -- ripgrep resolution: with the default `rg` or `rg.exe`, `PATH` is checked first; on Windows without `rg` on `PATH` it also checks common per-user and system-wide VS Code, VS Code Insiders and VSCodium installations (including versioned application directories). Any other configured value is an explicit command or path, used unchanged. If ripgrep cannot start, lexical retrieval fails open and the warning explains how to set `ripgrep_path`. -- Model clients: embedding response indices are validated and reordered; vectors must have consistent dimensions, finite values and nonzero norm. Model HTTP errors include the status and a bounded response body. Index-time document embedding retries connection errors, timeouts and a connection the server closed or reset mid-request (typically a pooled keep-alive connection closed while idle), up to three attempts with backoff (a cold model may still be loading). The query embedding, the indexing width probe (step 4) and the reranker make a single attempt, so a hung service costs a search one timeout and a refused connection costs no backoff; after a refused connection the reranker is also skipped for 30 s, and the warning says so. When the embedding service rejects a batch's input itself (HTTP 400/413/422, typically one chunk longer than the model's context), indexing retries that batch one chunk at a time and embeds a rejected chunk from a progressively halved prefix (down to 512 bytes); the full chunk is still stored, returned and lexically searchable and is listed in `truncated_for_embedding`. Its vector is cached by content hash, so raising the server's context limit later needs a reindex into a new data directory to re-embed it in full. Changing an embedding URL/model requires reindexing; replacing a model behind an unchanged name/URL is not detectable unless its vector width differs (step 4), so use a distinct model name or a new data directory. -- Timing fields (milliseconds) cover query embedding, LanceDB search, lexical search, LSP search, fusion, reranking and total retrieval; total also includes validation and index access/wait time. The same timings appear in tracing logs on stderr; CLI JSON stays on stdout. The default log filter is `local_code_intelligence=info,lci_core=info`; `RUST_LOG` replaces it (for example `RUST_LOG=local_code_intelligence=debug,lci_core=debug`). A `RUST_LOG` made only of crate-scoped directives that do not name `lci_core` silences that crate; a bare level such as `RUST_LOG=debug` applies to every crate. - -## Module boundaries and checks - -- `config` / `workspace`: configuration, canonical identity, external storage boundary. -- `language` / `chunk` / `manifest`: static language adapters, ignore-aware incremental scan, Tree-sitter chunks (Markdown: heading sections), persistent per-file parse reuse. -- `models`: embedding and reranking HTTP clients. `lexical`: the ripgrep lexical channel. -- `lsp`: the shared `LspAdapter` trait and JSON-RPC transport, one adapter module per optional language server, normalized navigation locations. -- `store`: embedded LanceDB schema, snapshot writes, cosine candidates. `filter`: filter validation and source-role classification. -- `evaluate`: portable evaluation definitions, execution and metrics; it observes retrieval without changing ranking. -- `app`: indexing/cache coordination and the retrieval pipeline (also `traverse`, `recipes`, `compact`). -- `server`: MCP tools and HTTP transport. `rest`: the JSON-over-HTTP/OpenAPI mirror. `main`: CLI/server startup. The `lci-maps` crate holds the multimap (traversal) map definitions; the full module and crate picture is [docs/architecture.md](docs/architecture.md). Later candidate sources can join the same fusion boundary before reranking. - -Checks: `cargo xtask build --test`, then `cargo xtask test --suite Full` once per completed change. `Full` runs `cargo xtask lint-machine-paths`, `cargo xtask docs check`, `cargo fmt --all -- --check`, `cargo test --workspace --locked --features local-code-intelligence/map-equivalence` and `cargo clippy --workspace --all-targets --locked --features local-code-intelligence/map-equivalence -- -D warnings` in sequence, setting `PROTOC` from `.tools/protoc` when the auto-downloaded copy is present. Focused suites (`Unit`, `Chunk`, `Filter`, `Config`, `Evaluation`, `Indexing`, `Readiness`, `Mcp`, `Watching`, `Lsp`, `Freshness`) and the change-to-suite map are in [docs/development/testing.md](docs/development/testing.md); every focused suite fails if its filter matches no tests. - -Automated tests use a local mock model HTTP service and real embedded LanceDB. They cover Rust boundary regression, all eighteen language adapters (one per extension), schema-v1 migration, per-language parse compatibility, persistent parse/vector reuse across reopen, mixed-language metadata and retrieval, TS/JS-only LSP gating, stale detection, deleted files, failed-update preservation, watched refresh, embedding configuration changes, reranker failure and invalid replies, query instruction formatting, ignore handling, actual Streamable HTTP initialization and tool discovery, and the REST/OpenAPI mirror's success and error responses. Live self-workspace and multilingual acceptance use the real local model services separately. diff --git a/docs/development/documentation.md b/docs/development/documentation.md index 9a7bfe40..267768b8 100644 --- a/docs/development/documentation.md +++ b/docs/development/documentation.md @@ -5,7 +5,9 @@ Agents read these files without chat context and pay for every token, so the fil 1. Facts first. State what the code does today, in the present tense, before any history or rationale. 2. One place per fact. A fact has one home; every other file points to that home instead of restating it. - Homes: behaviour and surfaces in `README.md`; module structure in `docs/architecture.md`; build and test in + Homes: setup, configuration and server behaviour in `README.md`; MCP and REST surfaces in `docs/reference/mcp-and-rest.md`; the CLI in + `docs/reference/cli.md`; retrieval behaviour and evaluation in `docs/reference/retrieval.md`; module structure in + `docs/architecture.md`; module boundaries in `docs/development/module-boundaries.md`; build and test in `docs/development/testing.md`; language scope in `docs/languages.md`; machine values in `docs/development/machine-values.md`; traversal engine in `docs/spike/multimap/engine.md`; measurement method in `docs/spike/multimap/evaluation.md`; plans in `docs/development/roadmap.md`; decisions, findings and history in `docs/decisions.md`. diff --git a/docs/development/module-boundaries.md b/docs/development/module-boundaries.md new file mode 100644 index 00000000..5555f5e6 --- /dev/null +++ b/docs/development/module-boundaries.md @@ -0,0 +1,18 @@ +# Module boundaries + +This file is the home of the per-module boundary list and the checks that enforce it. + +## Module boundaries and checks + +- `config` / `workspace`: configuration, canonical identity, external storage boundary. +- `language` / `chunk` / `manifest`: static language adapters, ignore-aware incremental scan, Tree-sitter chunks (Markdown: heading sections), persistent per-file parse reuse. +- `models`: embedding and reranking HTTP clients. `lexical`: the ripgrep lexical channel. +- `lsp`: the shared `LspAdapter` trait and JSON-RPC transport, one adapter module per optional language server, normalized navigation locations. +- `store`: embedded LanceDB schema, snapshot writes, cosine candidates. `filter`: filter validation and source-role classification. +- `evaluate`: portable evaluation definitions, execution and metrics; it observes retrieval without changing ranking. +- `app`: indexing/cache coordination and the retrieval pipeline (also `traverse`, `recipes`, `compact`). +- `server`: MCP tools and HTTP transport. `rest`: the JSON-over-HTTP/OpenAPI mirror. `main`: CLI/server startup. The `lci-maps` crate holds the multimap (traversal) map definitions; the full module and crate picture is [docs/architecture.md](../architecture.md). Later candidate sources can join the same fusion boundary before reranking. + +Checks: `cargo xtask build --test`, then `cargo xtask test --suite Full` once per completed change. `Full` runs `cargo xtask lint-machine-paths`, `cargo xtask docs check`, `cargo fmt --all -- --check`, `cargo test --workspace --locked --features local-code-intelligence/map-equivalence` and `cargo clippy --workspace --all-targets --locked --features local-code-intelligence/map-equivalence -- -D warnings` in sequence, setting `PROTOC` from `.tools/protoc` when the auto-downloaded copy is present. Focused suites (`Unit`, `Chunk`, `Filter`, `Config`, `Evaluation`, `Indexing`, `Readiness`, `Mcp`, `Watching`, `Lsp`, `Freshness`) and the change-to-suite map are in [docs/development/testing.md](testing.md); every focused suite fails if its filter matches no tests. + +Automated tests use a local mock model HTTP service and real embedded LanceDB. They cover Rust boundary regression, all eighteen language adapters (one per extension), schema-v1 migration, per-language parse compatibility, persistent parse/vector reuse across reopen, mixed-language metadata and retrieval, TS/JS-only LSP gating, stale detection, deleted files, failed-update preservation, watched refresh, embedding configuration changes, reranker failure and invalid replies, query instruction formatting, ignore handling, actual Streamable HTTP initialization and tool discovery, and the REST/OpenAPI mirror's success and error responses. Live self-workspace and multilingual acceptance use the real local model services separately. diff --git a/docs/reference/cli.md b/docs/reference/cli.md new file mode 100644 index 00000000..1cbe869e --- /dev/null +++ b/docs/reference/cli.md @@ -0,0 +1,43 @@ +# CLI reference + +This file is the home of the command-line reference: every `lci` subcommand, its options and its output. + +## CLI + +The CLI calls the same application logic and needs no editor. Its commits and a running server on the same data directory are serialized by the index lock; a separate `data_dir` (`--config` with its own) keeps CLI experiments from waiting on, or refreshing, the server's indexes. Subcommands: `serve`, `config`, `init`, `index`, `status`, `list-files`, `search`, `symbols`, `definition`, `references`, `evaluate`, `compare`, `replay`, `traverse`, `recipe`. + +```bash +lci=./target/debug/local-code-intelligence # local-code-intelligence.exe on Windows +repo=$PWD # this repository: the walkthrough indexes LCI's own source +"$lci" index "$repo" +"$lci" status "$repo" +"$lci" list-files "$repo" +"$lci" search "$repo" 'plain REST JSON-over-HTTP mirror of the MCP tools' +"$lci" symbols "$repo" 'openapi_document#' +"$lci" definition "$repo" 'src/rest.rs' 351 64 +"$lci" references "$repo" 'src/rest.rs' 172 3 --include-declaration +"$lci" references "$repo" 'lci-core/src/workspace.rs' 17 11 --include-declaration +"$lci" index "$repo" +``` + +- The walkthrough needs only a clone, a build and the model services (see [Model services](../../README.md#model-services-prerequisite) in the README); start it once `curl http://127.0.0.1:8768/ready` answers HTTP 200 (HTTP 503 names the missing dependency). `search` returns `openapi_document` in `src/rest.rs` as its top hit (the function building the document served at `/openapi.json`). `definition` starts from the call in the `/openapi.json` route at the end of `src/rest.rs` and resolves to that function's declaration; the first `references` starts from the declaration and returns it and the wiring; the second starts from `Workspace::resolve` and crosses crates (callers in `src/app/` and `tests/cases/`). Lines are one-based and characters zero-based UTF-16; the numbers are this repository's own, so if a file shifted, `grep -n openapi_document src/rest.rs` gives the current ones. The second `index` should report zero new embeddings when source and model configuration are unchanged. `cargo xtask acceptance core` runs the same index/search/repeat flow and checks that production implementation chunks, including `openapi_document`, dominate the top eight. Another repository is just a different first argument. +- `config` shows how the configuration resolves without opening the application: `"$lci" --config config.toml config [--json]`. `init` writes the overlay: `init --dry-run` (print only), `init` (write `config.local.toml` beside `config.toml`; exit 3 if it exists), `init --force --no-probe` (replace it, send no request). +- `config --json` prints one object: `valid` and `error`; `layers` (`file` with `path` and `named_by` (`--config` or `LCI_CONFIG`); `local` with `path`, `state` of `present`, `absent` or `disabled`, and `named_by`; `env`, the machine-set variables that were set; `flags`); `values`, every setting keyed by its dotted name (`port`, `typescript.path`, `index.freshness`; the `[search_profiles.]` tables are one entry, `search_profiles`, when configured) with `value`, `source` (`default`, `file`, `local`, `env`, `flag`) and `from`; `config`, the whole resolved configuration; `derived` (`mcp_url`, `mcp_profile_urls`, `rest_base`, `health_url`, `ready_url`, `openapi_url`, `embedding_identity`, `run_profile_hash`); `environment.other` (whether `LCI_CANDIDATE_DUMP` is set). Exit status: 0 valid; 2 resolves but fails validation (for example a reserved `port`; the report is still printed with `valid` false); 1 when a layer cannot be read or parsed (a missing file, a non-machine key in the overlay, a malformed `LCI_*` value; only `valid` and `error` are printed with `--json`). +- `search` takes repeatable `--language`, `--include-path`, `--exclude-path`, `--source-role`, plus `--top-k` and `--search-profile ` (the profile's settings applied over `[channels]` and `rerank_candidate_count`; the response then carries `"search_profile"`; an unknown name exits with an error listing the configured names, before any index work). Path filters are case-sensitive repository-relative globs (so `--include-path '*.rs'` does not match an indexed `Lib.RS`, while `--language rust` does): `*` and `?` stay within one path component, `**` crosses directories. Absolute paths, backslashes, traversal components, empty patterns and malformed globs are rejected. An explicitly empty include list is a valid filter returning no results. Filters apply to every retrieval channel and are reported back as effective filters; no result may bypass them. +- Source roles are deterministic. Precedence: generated, test, benchmark, example, documentation, configuration, source. Tests are recognized by `test`/`tests`/`__tests__` directories, the Gradle/Android source sets `androidTest`/`integrationTest`/`testFixtures`, and per-language file names: Python `test_*.py`/`*_test.py`; JS/TS `*.test.*`/`*.spec.*`; C# `*Test(s).cs`; Go `*_test.go`; Rust `tests.rs`/`test_*.rs`/`*_test.rs`/`*_tests.rs` (a bare `test.rs` stays source, being a legitimate production module name; inline `#[cfg(test)] mod tests` cannot be told from the path); Java `*Test`/`*Tests`/`*TestCase`/`*IT.java`; C/C++ `*_test`/`*_tests`/`*_unittest(s)`/`test_*` on the C and C++ extensions. Layout directories (`examples`, `docs`, `config`, `scripts`, `tools`, `build`, ...) count only above a Maven/Gradle source root (under `src//java/`, or `kotlin`/`scala`/`groovy`, the remaining directories are package names, so `com/example/...` is not an example), and a `config`/`configs`/`configuration` directory marks only non-source files, so `internal/config/loader.go` stays source. Production source receives a small transparent `0.004` fusion prior; all other roles receive zero. Role and prior appear in result metadata; explicit role filters stay authoritative. +- Traversal and recipes (SCIP-backed, see [engine.md](../spike/multimap/engine.md)): `"$lci" traverse --scip [--root ] --question ''` walks the code graph of a SCIP index (`rust-analyzer scip`) from the start nodes the question finds (with `--root`, the top hits of the hybrid search over that tree, mapped to the index's symbols; otherwise a stand-in matching the question's words against symbol names, reported as `seed_source`) and prints one JSON object (reached nodes, totals, handler decisions, and when it stopped to ask or ran out of budget a `handle` and `checkpoint`; continue with `--handle` and `--steer`). It ignores `[traverse] enabled` (which gates only the MCP mode) and is limited by the `[traverse]` caps. With `--root`, the graph also has an `implements` edge from each trait-impl member to its trait method declaration (read from the `impl .. for ..` headers; rust-analyzer writes no SCIP relationships), and `--maps scip.implements` adds the traits a type implements, the types that implement a trait and the members that implement a trait method; derives, `macro_rules!` impls and traits the index does not define give no edge. With `scip.implements` on, a trait path or `Trait::method` in a code span of the question (`` `tauri::Manager` ``, `` `Runtime::run` ``) starts the walk at that trait or trait method declaration (the response then has `trait_seeds`, how many start nodes they are, and `seed_source` `traits`, or `traits+hybrid`, `traits+hybrid+bm25`, `traits+bm25` or `traits+name_tokens` when the other start nodes stay behind them: they are replaced when the path matched with its qualifiers, kept when it matched by the last segment alone; `--seeds` still wins; `--trait-seeds replace|prepend|auto` (default `auto`, as described) picks the mode), and `--seeds` takes the node ids the map prints; without the map the start nodes are unchanged. Selectors include `--maps`, `--policy` (a preset: `explore`, `impact`, `flow`, `where-to`, `history`; or JSON; the policy is the router: it sets the walk or first-stage answer and the maps of the question type, see [engine.md](../spike/multimap/engine.md) Presets), `--first-stage`, `--pops`, `--max-depth`, `--top-k`, `--out`. `"$lci" recipe --root [--scip ] [--args ]` runs one recipe (`config_readers`, `impact_rings`, `flow_path`, `flow_from`, `config_neighbours`, `lci_compose`); it ignores `[recipes] enabled` and `[compose] enabled`. `config_readers`, `config_neighbours` and an `lci_compose` `config.key` step need `rg` on `PATH`. +- Map freshness (`[run_profile.traverse] map_freshness = true`, default `false`): a traversal answer (MCP `mode = "traverse"`, `lci traverse`) then has `maps`, one entry `{id, observed, stale, lag}` for each map of the call that states its freshness, in registration order; with the key off the answer has no `maps` key and is unchanged. The SCIP-built maps (`graph.scip`, `structure.contains`, `seeds.scip_names`, `seeds.bm25`, `seeds.repo_map`, `exact.closure`, `config.key`, `scip.impact`, `scip.implements`) have `observed = "snapshot"`; `tests.tested_by` reads the same impact index but states no freshness yet, so it has no `maps` entry. The maps that read the tree or git when they answer (`grep.live`, `grep.structural`, `lsp.live`, `git.history`, `exact.pickaxe`, `probe.compile`) have `observed = "live"`, `stale = false` and `lag = 0`; `live` names the channel, not a guarantee about every byte. `grep.live`, `grep.structural`, `git.history`, `exact.pickaxe` and `probe.compile` read the tree or git when they answer (current bytes, or the current git HEAD). `lsp.live` asks a running language server instead of the index: a fresh server (one this call started) reads the disk as it is, but a server kept across calls (`[run_profile.lsp] sessions = "shared"`) is only as fresh as its tree stamp's coverage (see the language-server sessions item below), and its answer carries a note `lsp.live: kept session; tree stamp unchanged since s ago`; when that note is absent the walk used no server it found running and was served no cached outline from one (`[run_profile.lsp] outline_cache` keeps a file's outline per server session, and an outline served from that cache counts as a use of that session: the walk gets the note and its `lsp.live` cards are `unverified`, unless the walk itself started the session). `grep.live`, `grep.structural` and `lsp.live` also place what they read by the index's positions, so a line shift since the index is not covered by their `live`. The emitted fields are the same either way. While an arbitration ran in the call (`[run_profile.scip] arbitration`, see "Arbitration of SCIP edges" below) the entries of the maps it judged also carry `role`: `"authority"` for `lsp.live` and `"cache"` for each SCIP-built map it judged (`scip.impact`, `scip.implements`, `structure.contains`); without an arbitration no entry has a `role` key. `lag` is a count of files, never wall time: the source files the language adapters index (prose, such as Markdown, is left out) that are newer than the index file (modified time after the index file's), plus those that changed since this process first saw the index's current stamp (modified time and length): a file whose content hash changed, added or removed, each file counted once however often it is edited. A touch without a content change does not count. `stale` is `true` when `lag` is above 0, so an index rewritten with a new stamp starts again from the files newer than it. `lag` is `null` and `stale` `false` (meaning not compared, never fresh) when the call has no tree (no `--root`), the tree could not be read, or the index file has no modified time; a note in the answer says which. Not seen: a change that left a file older than the index file before this process first saw the stamp, and a change that kept the file's content hash. Every call reads and hashes the tree's source files (no chunk is made), unless `[run_profile.freshness] stat_gate` is on (see "Stat gate" below). The maps `seeds.hybrid`, `search.lexical` and `search.vector` state nothing and are absent from `maps`, unless `[run_profile.freshness] verify_on_read` is also on (see "Verify on read" below). The key is read from the run profile the call is given, so the recipes (whose answers carry no `maps`) take it too: `flow_from`'s inner walk, with the key on, reads the tree and discards the comparison. +- Answer labels (`[run_profile.traverse] answer_labels = true`, default `false`): every traverse answer (a walk, a continuation, an answer with no start node) gains `completeness`, `{complete, why, estimate}`: `complete` is true exactly when `status` is `complete` (the walk ended `complete` or `bound`, the top-k set is final) and no map the call asked for was lost; a map that could not be set up (for example `lsp.live` with no server) or that had failed calls makes it `false`, with `why` `degraded` (a cut or capped walk keeps its own `why`) and `degraded` listing the maps, and a note naming them; `why` is otherwise the answer's `stop_reason` (`no_seeds` without a walk) and `estimate` is the fraction of the `likely_complete` hint (the mark-recapture estimate over what the maps found; `null` until two maps share a node; maps over one graph read high, so it is a hint, never `complete`); a partial answer also gets one `notes` line that names the reason and the handle. Every card of `results` and of the checkpoint's `answer`, `visited_cards` and `frontier` gains `freshness`, one of `fresh`, `changed`, `stale`, `unverified`: `changed` when `file_changed` is `true`, `fresh` when it is `false` (the card's own file is as its maps read it, so its text is current even under a stale snapshot), else from `maps`: any map of the card (`maps` and `via`) that is `stale` gives `stale`, every one found with a lag gives `fresh`, anything else is `unverified`, which is what every card says with `map_freshness` and `verify_on_read` off. One exception, for `lsp.live`: when the walk found a language server running (a session kept across calls under `[run_profile.lsp] sessions = "shared"`, the case the `lsp.live: kept session` note reports) every card that cites `lsp.live` is `unverified`, never `fresh`, because nothing checks what a kept server read (a `changed` or `stale` finding stands); a card from a server this walk started keeps `fresh`. The rule is per walk, not per card: with several servers, any reused one makes all `lsp.live` cards `unverified`. A continuation (`handle`) cannot tell whether the earlier call used a kept server, and the checkpoint format carries no such flag, so every card of a node restored from the checkpoint that cites `lsp.live` stays `unverified`; the cards of nodes the continuation's own walk reaches follow the rule above. Nothing is withheld with either value (`results` are the same), and the trace, the checkpoint hash, the handle and `text` do not change; with the key off the answer is byte-for-byte as without it. In the profile hash; recipe answers have no labels. +- Paths on cards (`[run_profile.traverse] card_paths = true`, default `false`): every card of a traversal answer's `results` and of the checkpoint's `answer` that the walk reached then has `kind`, the kind of the edge that discovered the node (`null` for a start node), and `path`, the route from its start node: one `{node, map, kind, witness}` entry per node (`depth + 1` entries, the first is the start node with `kind` and `witness` `null`, the last is the card itself and repeats its `via` and `witness`). The route is the walk's final one (a stronger route found later replaces a weaker one). The cards of `visited_cards` and `frontier`, the cards of the first stage (`origin` `exact` or `seed`), the trace and the checkpoint hash are unchanged; with the key off nothing in the answer, the trace, the checkpoint or the handle changes. A handle made with the key on also keeps each state's `kind` (an additive key; the handle format version is unchanged), so a continuation shows whole chains; a handle made with it off has no kinds for its earlier hops, which show `kind: null`, and a note says so. Size, measured as the answer's characters divided by 4: on the unit-test fixture (4 cards, depth up to 2, a complete walk) 616 tokens off, 856 on; on one real walk (`lci traverse` with `graph.scip` over an older index of this repository, one start node, 8 cards at depth up to 3, ended by the pop cap with a checkpoint) 3807 tokens off, 5160 on, 35.6 percent more, because the checkpoint's `answer` repeats the cards; the `results` list alone grows 87.5 percent (3095 to 5802 characters), about 85 tokens per card per list. Cards at greater depth cost more, one entry of about 25 tokens per hop. +- Language-server sessions of `lsp.live` (`[run_profile.lsp] sessions`, `"shared"` default or `"per_call"`; it matters only when a call runs `lsp.live`, which a call with a tree under `shared` and the service's manager (MCP and REST `mode = "traverse"`) runs by default, `[run_profile.lsp] live = true`, and which the CLI one-shot `lci traverse` and `per_call` run only when `--maps` or a `[traverse.policies]` entry names it, because a server that dies with the call cannot amortise its start; with no language server installed, or one that will not start or is refused, the walk says so in a note and goes on without the map; `live = false` makes the map opt-in everywhere, and a call that names its maps is never changed by the key): under `per_call` every traversal call starts its own language servers (a cold start and a version probe of each enabled server) and stops them when its walk ends. Under `shared` (the default) the calls of one process may keep a server across calls, but only a server that declares the configuration files its tree stamp covers (`LspAdapter::stamp_manifests`; every adapter but jdtls and marksman does) and either sees an edit on its own, today rust-analyzer alone (`LspAdapter::watches_workspace`; its client capabilities here declare no `workspace.didChangeWatchedFiles`), or is told by `[run_profile.lsp] sync = "watch"`, which is the default since K11b-4. So by default the service's manager keeps every server but jdtls and marksman (rust-analyzer, csharp-ls, typescript-language-server, pyright, gopls, clangd), and a service with `[traverse] enabled = true` primes the ones of the index's languages at the first touch of a workspace. With `sync = "off"` rust-analyzer alone is kept, and every other server (csharp-ls and typescript-language-server are told nothing about an edit; pyright, gopls and clangd are resent only the file a request names) is started by each call on that call's own manager and stopped when its walk ends, exactly as under `per_call`; jdtls lists no configuration files and is never kept, whatever `sync` says. The reason for the default: a server started per walk cost 266 to 524 ms per expansion on python, typescript, go and csharp trees against 2 to 5 ms for a kept one, with identical cards (a measurement on those trees, not a bound). A kept server does not rely on its own file watcher or on the sync for correctness: each walk that uses it first passes a tree barrier. The barrier takes a stamp of the tree from the index's file list and one directory read per stamped directory, with no recursive walk: the server's indexed files (modification time and length); every directory that holds one and its ancestors up to the workspace root (modification time and child names); one level of child directories of those (modification time), except version-control and build-output directories (`.git`, `target`, `node_modules`, `obj`, `bin`, `build`, `dist`, `out`, virtual environments), so a file created in an existing unindexed `src/generated/` that a `#[path]` attribute names is seen; and the server's configuration files among those children, matched by name (rust-analyzer: `Cargo.toml`, `Cargo.lock`, `rust-analyzer.toml`; typescript-language-server: `tsconfig.json`, `tsconfig.*.json`, `jsconfig.json`, `package.json`; pyright: `pyrightconfig.json`, `pyproject.toml`, `setup.cfg`; gopls: `go.mod`, `go.sum`, `go.work`; clangd: `compile_commands.json`, `compile_flags.txt`, `.clangd`, `CMakeLists.txt`; csharp-ls: solution and project files, `Directory.Build.*`, `Directory.Packages.props`, `global.json`, `nuget.config`). Only "not found" counts as absent: any other failure to stat or read makes the stamp fail, and the server is then restarted and nothing is recorded. Each walk passes the barrier before the server is used (a walk that runs `lsp.live` on the shared manager passes it twice, once before its seed search and once in the backend, so it restarts a stale server once: the second restarts it again only if the stamp changed between the two; one at a time per workspace, so a slower barrier cannot record an older stamp over a newer one) and compares the stamp with the one recorded when the server last started or passed; when it differs, or the server was started by a call that passed no barrier, the server is stopped and the walk starts a fresh one that reads the disk, and the walk says so in a note (`lsp.live: restarted: the tree changed since its last call`). An unchanged tree keeps the server, and the walk says that too (`lsp.live: kept session; tree stamp unchanged since s ago`), so an answer from a reused server is distinguishable from a fresh `per_call` one. The note comes from the session the walk's requests actually acquired (found running, or started by that walk), not from the barrier, so it is also there when the server was started by an overlapping walk that passed its barrier at the same time. The rule: a reused server is never staler than `per_call` for any change inside the stamp's coverage. It is not an absolute guarantee (that would need a full content walk, which cost 1.5 s on deno): the residuals are a new file two or more levels below a stamped directory or inside a skipped build-output directory; a configuration file that is not on the server's list; an edit that keeps both modification time and length; and an edit made during the call, which `per_call` has too. `sync = "watch"` does not avoid a restart after an edit; it tells a running server about the edit, which is what makes a non-watching server worth keeping, and it narrows the residuals above for navigation and the search channel. Two residuals follow from the default: a workspace whose platform watcher and poll fallback both failed, or whose Windows watcher silently stopped, keeps a non-watching server on the strength of the tree stamp alone, so it is told nothing but still passes the barrier (the poll fallback itself scans the tree every `watch_poll_milliseconds`, which is the expensive case on a very large or remote tree); and the search channel restarts a synced non-watching server after an edit, a loss of the channel for a few seconds that rust-analyzer's exception below avoids. A workspace with no index snapshot has no file list and so no sharing: the server is restarted by every walk. On a large repository the stamp costs tens of milliseconds (decision d454), plus one read of the workspace manifest after each index change; navigation and the search channel pass the same barrier, once per call, and carry the same notes (`lsp: restarted: ...` and `lsp: kept session; ...` in the `notes` of a navigation answer and of a search report; a server the call started adds none). The one exception is the search channel and a server that watches its workspace (rust-analyzer): an edit would restart it and the next searches would lose the channel while it loads, so the search keeps its session through a changed tree and says `lsp: rust-analyzer kept session; (kept: it watches its workspace)`, with no tree stamp recorded for it, so the next navigation call or traversal still restarts it; that answer is as fresh as the server's own watcher, not as the stamp, so it can miss an edit inside the watcher's latency and a `Cargo.toml` or `Cargo.lock` edit while the server reloads its workspace. Off switch: `sessions = "per_call"` (it moves the profile hash); `lsp_idle_timeout_seconds` only limits how long an unused server stays, and a kept server holds memory while it runs. Peak resident size measured during walks (a measurement on the benchmark trees, neither a limit nor an idle figure): rust-analyzer about 4 GB, typescript-language-server about 460 MB, csharp-ls about 190 MB, pyright and gopls about 120 MB each; a workspace in all of those languages adds about 0.9 GB (the sum of the four non-rust-analyzer figures) to rust-analyzer's 4 GB. The idle timeout (default 1800 s, `0` never releases) releases a session that has started and primed, or whose start failed, once nobody has used it for that long; a start or priming still in progress is not released, so it bounds the residency of servers that finish starting, not of one that is slow or stuck starting. The kept servers are the service's own manager's, the one that navigation (`find_definition`, `find_references`, `search_symbols`) and the search channel use, so a server started by any of them serves the others; `lci traverse` is one call per process, so there `shared` differs from `per_call` only in that the kept servers stop cleanly when the walk ends; its manager is built without the sync, so there only rust-analyzer is kept and every other server starts per call. A shared server stops like any other of the service: when it has been idle for `lsp_idle_timeout_seconds`, on `release_workspace`, when its session is replaced (stuck or failed) and at service shutdown (Ctrl+C, SIGTERM); the end of a walk does not stop it. The version probe of a server binary runs once per manager under `shared` (`[run_profile.lsp] breaker` is `true` by default: a provider whose version probe failed (a binary that is missing, or a probe that timed out, opens no window: that is no evidence of a dead server), or whose server failed to start without timing out, is skipped for 30 seconds, with a note that says so, by the service's manager and each call's alike; a server already running is still asked, a start that timed out opens no window, and `/ready` still probes. With `breaker = false` a failed probe is run again by the next call and a dead server is started again). The recipes (`flow_from` and the other `lci_*` tools) walk only `graph.scip` and `structure.contains`, never `lsp.live`, so no language server starts under a recipe and they run the same under both values; the key still reaches them in their run profile, so the profile hash they carry is the one that ran. `[run_profile.lsp] implementations` (default `false`) adds a third request to each symbol node of an `lsp.live` expansion, `textDocument/implementation`, behind the same `max_external_calls` gate as the other two: its answers become `implements` edges from the node to the symbols (or files) at the places the server names, the direction `scip.implements` uses (an interface or trait member to the members that implement it). The place table is the SCIP index's, whose nodes are functions and methods, so the request is asked at an interface or trait member, not yet at the interface or trait itself. An error reply from the implementation request is noted and costs the other two requests' edges nothing (a hang trips the call's deadline and clears them all). The recipes ask no implementations either way. `[run_profile.lsp] call_hierarchy` (default `false`) adds up to three more requests to each symbol node, each behind the same gate and counted: `textDocument/prepareCallHierarchy` at the node's name, then `callHierarchy/incomingCalls` and `callHierarchy/outgoingCalls` of the item that answers (an empty prepare ends it, so one more call at least). Callers and callees become `calls_or_refs` edges (no new edge kind), one per destination, whichever way the call runs, seen at the call itself; a symbol or file that `definition` or `references` already named keeps that edge. An error reply (a server without call hierarchy answers -32601) is noted and costs the other edges nothing; a failed prepare ends the block and a failed incoming request still asks outgoing. Servers miss some calls (rust-analyzer's call hierarchy missed 16% of the call pairs the SCIP index has, in macro arguments), so this complements `graph.scip`. The recipes ask no call hierarchy either way. +- Verify on read (`[run_profile.freshness] verify_on_read = true`, default `false`): with `map_freshness` also on, a traversal answer checks what it cites against the working tree by content hash. It reads and labels; it never refreshes the index and never changes `[index].freshness` (under `manual` the snapshot stays as old as it was). It reuses the one read of the tree that `map_freshness` makes (no second scan), and with the key off every answer is byte-identical to before. What it adds: (1) the maps that answer from the index snapshot (`seeds.hybrid`, `search.lexical`, `search.vector`) get a `maps` entry `{id, observed: "snapshot", stale, lag}` whose `lag` counts the source files that differ between the snapshot's manifest and the tree (changed, added, or removed; the manifest's prose entries are not counted, as the tree scan leaves prose out). It is read after the walk, because the walk's first search may refresh the snapshot under `on-search` (a refresh that lands between a map's read and this manifest read, as under `[index].freshness = "watch"`, makes the answer look as current as the new snapshot); `null` (not compared) when the call has no tree or no manifest. (2) `grep.live` and `lsp.live` keep `observed: "live"` (`grep.live` reads current bytes; `lsp.live` queries a language server, which reads the disk when this call started it and is bounded by its tree stamp's coverage when kept, see the language-server sessions item and the `kept session` note) and gain `places: "index"`: where they put what they read comes from the SCIP index, so their `stale` and `lag` are the SCIP index's (the same count as the SCIP-built maps), and a line shift since the index shows as stale. They are the only entries with `places`. (3) Every card (`results`, and the checkpoint's `answer`, `visited_cards` and `frontier`) gets `file_changed`: `true` when the content hash of the file it cites (its `site.path`, else its node's path) now differs from the hash its maps read, `false` when it does not, `null` when no reference applies (a commit or other card with no file, a file that is not a source file, or no tree was read; one note says how many cards are `null`). `true` also covers a file that was newer than the index file when this process first saw the index's stamp: its hash at index time is unknown, so it counts as changed whatever its content is. The reference is the SCIP baseline for the maps built from the index, for `grep.live` and `lsp.live`, and for a symbol card (its site is the index's, whichever map reached it), the snapshot manifest for `seeds.hybrid`, `search.lexical` and `search.vector` and for a chunk card; a card that names both is `true` when either differs. A `search_code` / `search` report (JSON; the compact text output is unchanged) gets a `freshness` block with the key on, with or without `map_freshness`: `snapshot` (`{stale, lag}`: the source files that differ between the snapshot's manifest, read after the search, and the tree, which one read of the tree's content hashes gives, gated by `stat_gate` when it is on), `channels` (`{id, observed, stale, lag}` for the channels that ran, so a channel that was off or not reached (no language server applies, no vectors) is not listed, and one that failed is listed with a `note` (`failed; see warning`) and no lag, because it answered nothing: `vec.chunk.code` and `lexical` are `observed: "snapshot"` with the snapshot's lag; `lsp.symbols` is `observed: "session"` with `stale` and `lag` `null` and a `note`, because the language server's view of the tree is not compared), and `hits` (one `{path, file_changed}` per result in order; `file_changed` is `null` for a hit no snapshot channel returned (only the language server, or no channel named), or when the tree was not compared, which a note says). The block is read after the search, so a refresh of the snapshot that lands between the search and that read (`[index].freshness = "watch"`) is not seen: the answer then looks as current as the new snapshot. The scan costs `timings.verify_ms` (absent when the key is off), and only a search a caller asks for pays it: the searches that find a traversal's start nodes and the evaluation runner do not scan. Under `manual` or `watch` a search never refreshes the snapshot, so the block is how a caller sees that it is behind; under `on-search` the search has just refreshed it and the lag is 0. A search with the key off has no `freshness` key. The recipes take the key through the run profile but their answers carry none of this. +- Stat gate (`[run_profile.freshness] stat_gate = true`, default `false`: every read of the tree for `map_freshness` or `verify_on_read` reads and hashes every source file). With `true` the process keeps each source file's modified time, length and content hash per root, lists the tree without reading it, and reads and hashes only the files whose modified time or length is new or changed (a file with no modified time is always read; a file skipped by its content, such as a Qt Linguist catalog saved as `.ts`, stays skipped while its stamp stays). The labels and `file_changed` equal the full-hash path's, through the same hash function, with one known gap: a same-length edit that keeps a modified time older than 3 seconds before the previous read of the file (a deliberate `set_modified`, or a clock that went back) is not seen until that file's next edit. A file modified within 3 seconds of a read is not trusted and is read again on the next call, which covers a write in the same clock tick and the coarse stamps of FAT, exFAT and SMB shares. The table lives in the process and is rebuilt on the first call; the index scan is not gated. The key is in the profile hash. +- Outline cache of `lsp.live` (`[run_profile.lsp] outline_cache`, default `false`: every file outline the map needs, a `textDocument/documentSymbol` request, goes to the server). With `true` the cache works only for the servers that hold a file's current text at every request: pyright, gopls, jdtls, clangd and typescript-language-server, whose adapters send the file's text from disk before each request when it differs from what the session last sent for that file: a `textDocument/didOpen` the first time, then a `textDocument/didChange` carrying the whole text when the server advertised change sync in its `initialize` result (full or incremental; LSP 3.17 reads a change without a range as the whole text), and a second `didOpen` when it advertised none (pyright logs the repeated open and applies it; gopls overwrites its overlay; jdtls replaces the working copy's buffer; clangd's open replaces the contents; typescript-language-server refuses a second open, which is why the change matters for it). A session holds at most 64 open documents per server (a size, not a key): a new file past that closes the least recently used one with `textDocument/didClose` first, and a closed file is opened again, at version 1, when it is next navigated. A text is sent once, and a request that follows a send waits for the server's `publishDiagnostics` for the file (at most the request timeout and 3 s) before it goes out. For those the language-server manager reads and hashes the file (the content hash the index uses, sha-256 over up to 16 MiB, on every outline request) before it asks, and keeps each non-empty outline under that hash and the server session that answered; the same file is then answered from memory while its bytes and that session are unchanged. An edited file is asked again and gets the outline of the new text, and so is any file after the server was restarted, released, replaced or stopped idle. An empty outline or an error is never kept (a server still loading answers `[]`), and neither is an answer for a file whose bytes changed during the request, an answer from a session that is no longer the live one, a file that is not UTF-8 or one over the 16 MiB open limit. The other servers are asked every time, as without the key: rust-analyzer watches the disk itself and can answer from the text before an edit for a moment after it, and csharp-ls is sent no document text. A manager keeps at most 4096 outlines; a new one past that clears them. The cache belongs to the manager: under `sessions = "per_call"`, and for a server that a `"shared"` walk starts on its own manager (every one of these with `sync = "off"`, jdtls and marksman always), it lasts one walk; in the service's manager under `"shared"` with `sync = "watch"` (the default) it serves later calls too. The recipes never run `lsp.live` and run the same under both values. +- Language-server sync (`[run_profile.lsp] sync`, default `"watch"` since K11b-4; `"off"`: a language server walks the tree when it starts and is told nothing after, so a long-lived session answers `search_symbols`, the search channel and a shared `lsp.live` from the tree it started on, and a server that does not see an edit on its own is not kept across traverse calls). With `"watch"` the service's language-server manager (the MCP and REST server's, which navigation, the search channel and `sessions = "shared"` traversals use; not the one-shot `lci traverse`'s, which is built without the sync because its process ends with the walk) watches each workspace that has a running session and sends every change of a source file to that workspace's servers as one `workspace/didChangeWatchedFiles` (created, changed or deleted, as the file is on disk), and drops the changed files' kept outlines. The watcher is the platform's (ReadDirectoryChangesW, inotify, FSEvents), or a poll every `watch_poll_milliseconds` when it cannot start (the poll compares modification times in whole seconds, so a second write within the same second is sent only with a later change). It starts with the workspace's first session, walks the tree once to know its files, walks again only when the platform reports lost events (inotify and FSEvents do; on Windows a watcher overflow during a large burst, such as a big checkout, is not reported and silently ends the watching until the workspace's sessions stop, roadmap N102), and stops when the workspace's last session stops. Single changes are not filtered by `.gitignore` (a source file written under an ignored directory, such as `target/`, is sent), and a renamed or moved directory is seen as the directory alone, so the files inside it are not sent. Before each request the manager sends what the watcher has already reported: the change notification goes out before the request, but the manager does not wait for the server to apply it, so the answer to that request may still be from the old text; an event the watcher reports after that send goes out before the first request that drains after the report, and an edit the platform has not reported yet is in a later answer. It never waits for a walk. Prose files are not sent, nor anything under `.git`. It is independent of index watching (`watch`): a workspace can be watched for the index, synced, both or neither. A per-call manager (`sessions = "per_call"`, and the one a `"shared"` walk uses for a server that does not see an edit on its own) never syncs: its servers start on the tree as it is and stop with the walk, so there the key changes nothing but the profile hash. A shared traverse walk passes the tree barrier (see "Language-server sessions of `lsp.live`") and restarts a kept server after an edit whatever the sync did, so for `lsp.live` the sync does not avoid a restart: it makes a non-watching server worth keeping, and it covers what the barrier's stamp does not (a new file two or more levels below a stamped directory; an edit that keeps modification time and length) for the answers of a kept server. Navigation and the search channel pass the same barrier, so the sync does not decide whether they answer from the tree as it is either. The cost of the default: one watcher thread and one task per workspace with a running session, one table of `(modification time, length)` from one walk of the tree, and under the poll fallback a scan of the tree every `watch_poll_milliseconds`. When both the platform watcher and the poll fail to start, the servers are told nothing and a warning is logged. The recipes never run `lsp.live` and run the same under both values. +- Place table of `lsp.live` (`[run_profile.lsp] places`, default `"auto"`: per walk, `"scip"` when the SCIP graph read for the workspace has at least one node (the walk is then the `"scip"` walk), else `"outline"`, so a tree with no index read, `scip.mode = "off"` or `"cache"` with no file, still gets `lsp.live` cards. The choice is per workspace, not per language: an index with nodes that lacks a language of the tree (a Rust index in a tree with Python files) still takes `"scip"` places, and that language gets no `lsp.live` cards while its server still starts; set `"outline"` for such a tree; `"scip"`: the SCIP index's table alone, so a symbol the index has no node for has no place, `lsp.live` asks nothing about it, and under `scip.mode = "off"` it asks nothing about any symbol; a file seed still gets its `textDocument/documentSymbol` outline, but its members are named only when the index defines them). With `"outline"` the table is the index's plus the language server's own outline of a file: the first time a walk expands a symbol the index places in a file, or expands the file, `lsp.live` asks the server for the file's `textDocument/documentSymbol` (one more external call per distinct file and walk; a failed request keeps the index's place and is not repeated) and from then on the outline's places win in that file, so lines inserted since the SCIP file was written no longer make it ask at the wrong line. An outline member takes the index's id for the same symbol when the index has one (same file and name, a function or method of either kind, nearest line; one symbol keeps one id across the layers), else an id of the file's real language with no container (`sym:csharp:Shapes.cs::Area/method`, equal names told apart in document order: `~1`, `~2`), so under `scip.mode = "off"` the functions, methods, types (class as `class`, struct as `struct`, interface as `interface`, except that rust-analyzer's trait stays `trait` in a Rust file), enums, modules, constants, fields (a property is a field) and variables of an outlined file have ids (`sym:csharp:Shapes.cs::Circle/class`). Enum members, impl blocks and type parameters have none. A file expansion lists the fields as `Contains` destinations too, and a reference on a field's line now resolves to the field; `lsp.live` asks a field or variable like any other symbol (a server may answer `implementation` for an interface property or a call hierarchy for a `const f = () => {}` it reports as a variable). A handle saved under `outline` by an earlier build holds `struct` and `trait` ids for class-language members, which no longer match a learned place. A seed that is a symbol id of a file this walk has not outlined still has no place. Every `documentSymbol` range now ends on the item's last line (the nested `DocumentSymbol` shape used to give the name's line), which `Contains` edges read from the name's own line first. The recipes never run `lsp.live` and run the same under all three values. The place table also bounds what an arbitration can confirm (next item): the index's table holds functions and methods only. +- Language-server roots (`[run_profile.lsp] roots`, default `"workspace"`: every server starts at the workspace path, as always). With `"manifest"` a server starts at the directory of the topmost project file its adapter names (the list of the tree stamp above: `go.mod`, `go.sum` and `go.work` for gopls; `pyrightconfig.json`, `pyproject.toml` and `setup.cfg` for pyright; `tsconfig.json`, `jsconfig.json` and `package.json` for typescript-language-server; solution and project files for csharp-ls; and so on), so a module under `services/api/` is served from `services/api/` and its `rootUri`, `workspaceFolders` and working directory say so. The project files are looked for in the directories of the index's files of the server's languages and in their ancestors up to the workspace, one directory read each (the index's file list holds source files only, so it names no project file), with no recursive walk. The shallowest one decides. The workspace is kept, and `index_status` says why in `servers[].root_reason`, when no project file is found (`no_manifest`), when sibling directories at the same depth each hold one (`several_roots:`: one session per module is not built, so a monorepo of sibling modules keeps today's root), when the chosen directory would leave an indexed file of the server's languages outside it (`files_outside_root:`), when the index has no file list (`no_file_list`), when a directory cannot be read (`unreadable`) and for a server that is disabled, unconfigured or serves no language of the index (`not_served`, reported under `"manifest"` only; under the default every server reports `roots_workspace`). A session fixes its root when it starts and keeps it. The one-shot `lci traverse` under `sessions = "per_call"` has no index file list, so under `"manifest"` its servers start at the workspace (`no_file_list`) and its output does not say so; the service always has the list. There is still one session per server per workspace, answers still name paths relative to the workspace, and the tree stamp and the barrier still read the whole workspace. The recipes never run `lsp.live` and run the same under both values. What was measured (a premise check on small fixtures, pyright 1.1.414 and typescript-language-server 6.0.2, not a general bound): `"manifest"` helps pyright when a name resolves only through a nested project's own settings, such as `extraPaths` in its `pyproject.toml` (the definition is empty from the workspace root and found from the project directory); plain imports in a src layout resolve from either root. typescript-language-server already finds the nearest `tsconfig.json` from the workspace root, `paths` aliases included, so `"manifest"` gains nothing for it. Any indexed file of the server's languages outside the project directory, even one script at the top of the repository, sends `"manifest"` back to the workspace root (`files_outside_root:` in the roster), so the gain appears only when every such file lies under the project. gopls and csharp-ls, other pyright effects of the root and cost were not measured. The default stays `"workspace"`. +- Arbitration of SCIP edges (`[run_profile.scip] arbitration`, default `false`): with `true`, `lsp.live` in the call, `scip.mode` not `"authoritative"` and at least one of `scip.impact`, `scip.implements` (and only with `[run_profile.lsp] implementations = true`) or `structure.contains` registered and built, the server is the authority over those maps before fusion. Per expansion and claim family where the server offered at least one edge to a function or method (`Refers` for `scip.impact`, `Implements` for `scip.implements`, `Structure` for `structure.contains`), an index edge to a function or method that the server also names is confirmed and one it does not name is removed and counted as overruled. A family the server offered nothing in, an answer cut at the per-node edge limit, an answer that named only a file, and destinations that are not functions or methods (under either `places` arm) leave the index edges standing and uncounted. `graph.scip` is never arbitrated: its edges point from a caller to its callees, a direction the server's references do not answer. An edge only the server has is not a disagreement. The counts (`confirmed`, `overruled`) are per call: in the trace on the `scip.impact` (etc.) call of each `expand` event, in `totals.arbitration`, in `totals.per_map[].arbitration` and in one note `scip.arbitration: lsp.live judged (...) in this call: edges confirmed, overruled`; a map that was not judged prints none. The `start` event's `arbitration` object names the arbitration (authority, judged `[map, family]` pairs, kinds) and carries no counts. An overruled edge can be a false one: the server's answer at a node may be partial (a file it has not opened, a kept session behind an edit), and under `places = "outline"` an index member and its outline twin may carry different ids. A handle records the arbitration (its authority, judged pairs and kinds) and refuses a resume under another or under none. With the key off, with `scip.mode = "authoritative"` or without `lsp.live`, nothing changes in an answer. +- Entity seeds (`[run_profile.seeds] bm25`, default `false`: the map `seeds.bm25` does not run, naming it in `maps` gives a note and no seeds, and every answer is as without the key). With `true` the map is in the default set of a call that names no maps, and runs when named. It ranks the SCIP graph's functions and methods by BM25 (the scorer of the ripgrep channel, `lexical::score_chunks`) over three fields of each node: the tokens of its name, of its qualified path (the symbol's package, modules, types and member, and the file's path) and of its documentation (the `documentation` the index holds for the symbol, up to 1024 bytes; empty when the indexer writes none, and it is not verified that `rust-analyzer scip` writes any; it is read from the index file, a second parse, only when the map first runs for an index stamp, and never with the key off). A question word counts when it is a whole token (up to 4 characters) or inside one (longer); words under 3 characters and stop words are not terms, and a word in more than a quarter of the nodes is dropped unless no word would remain. Of the top `[traverse] seed_k` nodes, those the hybrid search did not already give (so possibly fewer) go after its nodes (`seed_source` `hybrid+bm25`, each with a `seed_hits` record `how: "bm25"`; `seed_source` `hybrid` stays when it adds none), or are the start nodes when the hybrid search gave none (`bm25`; the name-token stand-in then does not run, and `seed_fallback` says why the search did not answer). It reads the SCIP graph only (no `--root` needed), builds its index on the first call that wants it and keeps it with the graph for the index stamp. The recipes' seeds are given, so they run the same under both values. +- Repo-map seeds (`[run_profile.seeds] repo_map`, default `false`: the map `seeds.repo_map` does not run, naming it in `maps` gives a note and no seeds, and every answer is as without the key). With `true` the map is in the default set of a call that names no maps, and runs when named. It ranks the SCIP graph's functions and methods by personalized PageRank (damping 0.85, at most 30 power iterations) over the call and containment edges (`implements` edges are left out, each caller and callee pair counts once), with the random jump on the question's best 64 `seeds.bm25` matches: a function that many of the matched functions call becomes a start node even when its own name holds none of the question's words. A question with no entity match gives no seeds and a note (no global hub is ever a seed), and so does an empty graph. Of the top `[traverse] seed_k` nodes, those the earlier sources did not give go after theirs (`seed_source` `hybrid+repo_map`, `hybrid+bm25+repo_map`, `bm25+repo_map`, each with a `seed_hits` record `how: "repo_map"`, `score_kind: "pagerank"`, `personalised` and `iterations`), or are the start nodes when none gave any (`repo_map`). It reads the SCIP graph only (no `--root` needed), builds its edge lists and the `seeds.bm25` index on the first call that wants it and keeps them with the graph for the index stamp. The recipes' seeds are given, so they run the same under both values. +- Literal seeds (`[run_profile.seeds] literal`, default `false`: the map `seeds.literal` does not run, naming it in `maps` gives a note and no seeds, ripgrep is not started for it, and every answer is as without the key). With `true` the map is in the default set of a call that names no maps, and runs when named. It searches the tree with `rg --fixed-strings --word-regexp` (case-sensitive, `_` counts as part of a word, the source files of the language adapters, at most 20 lines per file) for the names the question puts in code spans (the last segment of each `a::Name`, at least 2 characters, at most 4 names), and places each line in the SCIP graph: the symbol defined on that line (`how: "definition"`), else the innermost symbol around it (`occurrence`, counted per symbol), else, when the line declares the name (`trait`, `struct`, `fn`, `class` and the like right before it, or `impl .. for` it), the file's node (`file`: a type or trait the graph lacks, which `structure.contains` expands to the symbols it holds), else the line is dropped. Definitions (3 per name at most) come first, then occurrences (most lines first), then files; nodes the other sources gave are left out and the list is cut at `[traverse] seed_k`. They go after the hybrid search's and `seeds.bm25`'s nodes (`seed_source` `hybrid+literal`, `hybrid+bm25+literal`, `bm25+literal`), or are the start nodes when those gave none (`literal`). It is not `[traverse] exact_seeds`, which turns the first stage's exact cards into start nodes. It needs the application's ripgrep (a question with `--root`); a call without says so in `notes` and goes on. The recipes' seeds are given, so they run the same under both values. +- Every JSON result of `index`, `status`, `list-files`, `search`, `symbols`, `definition`, `references`, `evaluate` (the report, also when written with `--output`), `traverse` and `recipe` carries the same top-level `profile_hash` as the MCP and REST results (`derived.run_profile_hash` of `config --json`; `traverse` hashes the configuration after its command-line overrides). `config --json` reports the hash under `derived`; `compare`, `replay` and `init --json` run offline without a resolved configuration and carry none. An error prints no result. diff --git a/docs/reference/mcp-and-rest.md b/docs/reference/mcp-and-rest.md new file mode 100644 index 00000000..78ede440 --- /dev/null +++ b/docs/reference/mcp-and-rest.md @@ -0,0 +1,55 @@ +# MCP tools and REST surface + +This file is the home of the MCP tool reference, the language-server lifecycle and the REST/OpenAPI surface that mirrors the tools. + +## MCP tools + +All tools return structured JSON, also available as MCP text content. Every successful result has a top-level `profile_hash`, the run-profile hash of the server's configuration (64 lowercase hex characters, the `derived.run_profile_hash` of `config --json`; 82 bytes per result, carried in both the text and the structured content; the last key of a compact `search_code` result). Tool failures set MCP `isError` and carry none. A call to a tool with an argument the tool does not have (a misspelt `top_kk` or `include_path`, say) is refused with a tool error ("unknown field") that names it; `service_status` takes no arguments and refuses any (send none or `{}`). The REST request bodies, which use the same argument structs, refuse it the same way, and the REST search query string refuses an unknown parameter (HTTP 400). **Breaking change (d388):** `service_status` arguments and unknown REST search query parameters were previously ignored; clients sending them must drop them. Every tool input schema has `additionalProperties: false`. `search_code` and `search_symbols` accept nonblank queries up to 16,000 UTF-8 bytes inclusive; longer queries fail with `query exceeds 16000 bytes`. `workspace_path` is an absolute workspace directory (on Windows a drive path with doubled backslashes in JSON). Example arguments for `search_code` on this repository: `{"workspace_path": "/path/to/local-code-intelligence", "query": "plain REST JSON-over-HTTP mirror of the MCP tools", "top_k": 8}`. + +| Tool | Arguments | Result | +| --- | --- | --- | +| `index_workspace` | `workspace_path` | Canonical workspace/ID, file/chunk counts, parsed/unchanged/removed files, `skipped_non_source_files` (source-extension files whose content is another format, currently Qt Linguist `.ts` catalogs), newly embedded/reused chunks, `vectors_missing` (the chunks were committed without vectors because the embedding service was unavailable), total time; only when non-empty, `lossy_decoded_files` (non-UTF-8 files indexed from a lossy decoding) and `truncated_for_embedding` (chunks embedded from a shortened prefix because the embedding service rejected their full length) | +| `index_status` | `workspace_path` | Whether indexed/indexing/stale/watched, chunk count, embedding dimension (`null` while `vectors_missing` is true), `vectors_missing`, configuration compatibility, last successful indexing time; `stale_check_error` when staleness could not be determined (an unreadable file; an indexed workspace then reports `stale: true`); `analyzers_running`, one entry per language server (`rust-analyzer`, `csharp-ls`, `typescript-language-server`, `pyright`, `gopls`, `jdtls`, `clangd`), `true` while a started, ready server of that name runs for this workspace, including one a traverse call started under `[run_profile.lsp] sessions = "shared"`, and `false` otherwise (not started, released, disabled, not installed); `analyzers`, with the same seven keys, what each server's session reports about itself, or `null` when this workspace has no session for it: `state` (`starting`, `ready`, `busy`, `failed` or `stuck`; `busy` is a server that said it works again after it was ready, today rust-analyzer's `quiescent: false` and jdtls's `Starting` status, and the other servers never report it), `primed`, `progress` (`$/progress` notifications), `ready_changes` (ready or busy status notifications seen), and `age_ms`, `changed_ms` (`null` before the first such notification) and `idle_ms`. `sync`, present only while the workspace's sync under `[run_profile.lsp] sync = "watch"` runs (absent with `sync = "off"` and without a session): `backend` (`notify`, `poll` after the platform watcher failed to start, `null` while it is still opening), `failed` (neither watcher could start or the opening task died, so the servers are told nothing about edits) and the counters `events`, `batches`, `forwarded`, `walks` and `rescans`; a watcher that stops silently shows as `events` that stop growing after edits. `servers`, one entry per language server sorted by provider: `provider`, `languages` it can navigate, `languages_in_index` (the subset this index holds), `configured`, `enabled`, `command`, `probe` (`probed` once the manager's own version probe has passed, run by priming and the `lsp.live` warm start, else `not_probed`; `/ready` probes separately and does not count), `breaker_open_ms` (milliseconds left in the server's skip window under `[run_profile.lsp] breaker = true`, which a failed probe or start opens; `null` without the breaker or when the window is closed), `root` (the directory it starts at, relative to the workspace, `.` for the workspace itself: a live session's, else the one a start would choose now under `[run_profile.lsp] roots`, worked out only for an enabled, configured server whose languages the index holds), `root_reason` (why: `roots_workspace`, `manifest:`, `no_manifest`, `several_roots:`, `files_outside_root:`, `no_file_list`, `unreadable` or, under `"manifest"`, `not_served`, a server that is disabled, unconfigured or serves no language of the index) and `session` (the same object as its `analyzers` entry); and `languages_without_server`, the languages this index holds that no enabled, configured server lists (sorted). It is a report: reading it runs no probe, starts no server, does not keep a session alive and changes no answer. The over-budget note of a search and the `lsp.live` notes of a traverse call that ends with a server not ready name the state too. Deprecated, same values for one more release: `analyzer_running` (rust-analyzer), `csharp_analyzer_running`, `typescript_analyzer_running`, `python_analyzer_running`, `go_analyzer_running`, `java_analyzer_running`, `clangd_analyzer_running`. The same JSON comes from `POST /v1/index_status` and CLI `status` | +| `list_indexed_files` | `workspace_path` | Every file the persistent index has cached chunks for, with language and chunk count, from the last successful `index_workspace` | +| `watch_workspace`, `unwatch_workspace` | `workspace_path` | Start / stop debounced polling and automatic reindexing for an indexed workspace | +| `search_symbols` | `workspace_path`, `query` | Workspace symbols from every applicable enabled language server, with provider/language metadata. When some servers fail and another answers, the call succeeds with the answering servers' symbols plus, only when non-empty, `warnings` (one `Optional LSP provider unavailable: : ` per failed server, in provider order), so an empty `results` is not a confirmed "no match"; an error only when every applicable server failed. The three navigation tools add `notes` (absent when empty) when a kept server was restarted by the tree barrier or reused by the call, see "Language-server sessions of `lsp.live`" in the [CLI reference](cli.md) | +| `find_definition` | workspace path, relative file path, one-based line, zero-based UTF-16 character | Definition locations routed by extension to the matching language server | +| `find_references` | the definition arguments plus optional `include_declaration` | Reference locations routed by extension to the matching language server | +| `search_code` | `workspace_path`, `query`, optional `top_k` (1-40), `languages`, `include_paths`, `exclude_paths`, `source_roles`; with `[traverse] enabled` also `mode` (`"traverse"`), `budget`, `steer`, `handle` | Ranked source chunks, scores, timings, fallback warning, informational notes (for example a language server still starting, or a `top_k` above `rerank_candidate_count`), index lifecycle metadata; on `/mcp/` also `search_profile`. With `mode = "traverse"` the code-graph walk answers instead (reached nodes, totals, handler decisions, and a `handle` plus `checkpoint` when it stopped to ask or ran out of budget); `budget`, `steer` and `handle` without `mode` are an error. See [engine.md](../spike/multimap/engine.md) | +| `service_status` | none | The same required/optional readiness report as `/ready` (probes the binaries and model services, not language-server sessions: those are `index_status`'s `analyzers`) | +| `config_readers`, `impact_rings`, `flow_path`, `flow_from`, `config_neighbours` | see each tool's description | Recipes over a SCIP index (`[traverse] scip_index`); listed and routed only with `[recipes] enabled` | +| `lci_compose` | `steps` (1 to 6 operator steps) | Operator steps with references to earlier steps and no model call between them; listed and routed only with `[compose] enabled` | + +So 10 tools are always offered, 5 more with `[recipes] enabled`, 1 more with `[compose] enabled`; `tools/list` answers identically on every endpoint unless a profile sets `tools = "search"`. The recipes and compose are specified by the tool descriptions and in [engine.md](../spike/multimap/engine.md). + +- Search profiles are chosen by endpoint, not by argument: `/mcp` runs `search_code` as configured at the top level; for each `[search_profiles.]` the server also mounts `/mcp/`, whose `search_code` runs that profile (result carries `"search_profile": ""`). All endpoints share input schemas and server instructions. To offer several profiles to one client, register one MCP server entry per URL. Each endpoint keeps its own sessions (a session id from one is HTTP 404 at every other). A path under `/mcp/` naming no configured profile is HTTP 404; only `/mcp` and `/mcp/` are the default endpoint. A configuration with no profiles has `/mcp` alone. +- `search_code` creates a missing workspace index, waits for its commit and continues the search in the same call. Concurrent first searches for one workspace share that indexing job; other callers get results from the committed snapshot without duplicate embeddings. A compatible persisted index is reused immediately after restart. +- With an index present, `[index].freshness` (default `on-search`) decides whether a search checks for staleness. `on-search`: the first search that needs it compares the file manifest and content hashes against disk, throttled by `stale_check_interval_seconds`; on changed, new or removed files (or a changed embedding configuration) it refreshes incrementally, reusing cached vectors for unchanged hashes, before continuing. If another refresh is running it waits up to `wait_for_existing_job_seconds` and reuses that result; if the embedding service is unavailable the previous snapshot is kept and the search returns results with a warning. A vectorless snapshot (`vectors_missing`) counts as stale too: under `on-search` the next search past the same throttle runs the index again to embed the chunks, and keeps the vectorless snapshot with the warning while the service is still down. That repair is skipped without re-indexing while the breaker is open, and for 30 s after a repair that ended still vectorless. `manual` never refreshes automatically (use `index_workspace`). `watch` leaves refresh to `watch_workspace`'s poller, which suits long editor sessions. Under `manual` and `watch` a search never repairs a vectorless index: the next explicit `index_workspace`, or the watcher's reindex after a file change, embeds the chunks. +- The result's `index` object explains first-search latency: `action: created` (this request did the initial indexing), `waited_for_existing_job` (a concurrent request did), `refreshed_incrementally` (the search applied a staleness refresh), `reused` (a compatible, current-enough snapshot existed). `wait_ms` covers creation, refresh or waiting (near zero for plain reuse) and does not overload retrieval timings. Indexing is synchronous and ends when the new index is persisted, so allow a long client tool timeout for a first search, a refresh or an explicit indexing pass. +- Index writes are serialized in the running service with per-workspace read/write coordination. One service per data directory is intended; multiple clients connect to it. Commits are also serialized across processes by an OS file lock (`/index.lock`): a CLI `index`, `search` or `evaluate` next to a server on the same data directory waits for the other commit instead of interleaving LanceDB overwrites and manifest writes; reads never take the lock. +- Watching is opt-in for each server run: it polls only after `watch_workspace`, debounces a detected source change and calls the same safe indexing path. Registrations are not restored after a restart (indexes and manifests are). A failed automatic update is logged and retried after another poll while the prior snapshot stays searchable. + +### Language-server lifecycle + +The mechanics (start, priming, retry, stuck-server replacement, idle release, Windows job objects) are in `.agents/skills/lci-lsp-adapter/references/lifecycle.md`; the behaviour a client sees: + +- The first applicable LSP request for a workspace starts one long-lived provider process per language server; later calls reuse it. `search_code` queries each applicable, enabled provider independently, only when filtered indexed chunks contain a language it navigates, and maps locations only to same-language chunks. A missing or disabled server leaves semantic and lexical retrieval available. Direct navigation rejects unsupported extensions before startup and returns an MCP error when the selected provider is disabled or unavailable. Definition/reference input lines are one-based; character offsets follow LSP (zero-based UTF-16). +- A request that fails because the server went away is retried once with a fresh provider; a server that answers with an error response is alive and is not restarted (an "ask again" answer is repeated on the same server up to twice; any other error is that provider's error, which `search_symbols` skips like any failed provider). A server that answers nothing but "ask again" errors for `lsp_stuck_seconds` is replaced; keep that value above the longest load you expect. +- A server nobody asked for in `lsp_idle_timeout_seconds` is released (a release takes about 4 seconds at most; a server still building its symbol index or serving a call is not released as idle). `POST /v1/release_workspace` releases a workspace's servers on demand; Ctrl+C on `serve` shuts every server down the same way within 5 seconds in all. After any release the next call starts a fresh server transparently, and the first `search_code` after it can hit `lsp_search_budget_ms` and report the informational `notes` entry, like the first search after the service starts. + +## REST/OpenAPI surface + +Every MCP tool is also reachable as plain JSON-over-HTTP: same process, port and `App` methods, so behaviour, filters and index lifecycle are identical. + +| MCP tool | REST endpoint | +| --- | --- | +| `index_workspace`, `index_status`, `list_indexed_files`, `watch_workspace`, `unwatch_workspace`, `search_symbols`, `find_definition`, `find_references`, `search_code` | `POST /v1/` | +| `service_status` | `GET /v1/service_status` | +| `config_readers`, `impact_rings`, `flow_path`, `flow_from`, `config_neighbours` | `POST /v1/`, present only with `[recipes] enabled` | +| `lci_compose` | `POST /v1/lci_compose`, present only with `[compose] enabled` | + +- Every `POST` takes the same JSON arguments as its MCP tool as the request body (REST `search_code` takes the plain search arguments; `mode`, `budget`, `steer` and `handle` are MCP-only). A success is HTTP 200 with the structured JSON an MCP `tools/call` would return, `profile_hash` included (`POST /v1/release_workspace` too; `/health` and `/ready` carry none); an application failure is HTTP 500 with `{"error": ""}` and no `profile_hash`, instead of MCP's `isError` block. A request body the handler never sees is rejected earlier by the axum `Json` extractor with a plain-text body (not JSON): 400 for malformed JSON, 415 for a missing or wrong `Content-Type`, 422 for a body that does not deserialize, such as an unknown key (the text names the field, for example ``unknown field `include_path` ``) or a missing required one. The `search_code` query string accepts only the optional `search_profile` key; unknown or duplicate keys return HTTP 400 with plain-text errors naming the key, before any search runs. This also breaks clients that sent previously ignored extra query parameters: drop them and put supported filters in the JSON body. +- `curl -X POST http://127.0.0.1:8768/v1/search_code -H 'Content-Type: application/json' -d '{"workspace_path":"/path/to/local-code-intelligence","query":"plain REST JSON-over-HTTP mirror of the MCP tools"}'`. +- `POST /v1/search_code` also takes the optional query parameter `search_profile` (for example `/v1/search_code?search_profile=fast`); the body is unchanged and the result carries `"search_profile": "fast"`. An unknown name is the usual HTTP 500 listing the configured names (or saying none are configured). +- One endpoint has no MCP tool: `POST /v1/release_workspace` takes `{"workspace_path": ...}` and shuts down the language servers this service runs for that workspace (`shutdown`, then `exit`, about 4 seconds per server at most, then a kill; servers shut down side by side). It needs no index. The response is `{"workspace": {"id": ..., "path": ...}, "released": ["csharp-ls"]}`, `released` listing, sorted, the stopped servers (the names `analyzers_running` uses); a workspace with nothing running, or never indexed, answers HTTP 200 with `"released": []`; a nonexistent directory is the usual HTTP 500. The CLI has no equivalent (a CLI run owns no long-lived servers). MCP `tools/list` is unchanged. +- `GET /openapi.json` serves an OpenAPI 3.0.3 document of every endpoint above, including `release_workspace` and the `search_profile` parameter. Request schemas are generated from the same `schemars`-derived argument structs as the MCP tools, so they cannot drift; response bodies are documented as an open JSON object whose only required property is `profile_hash` (a string matching `^[0-9a-f]{64}$`). diff --git a/docs/reference/retrieval.md b/docs/reference/retrieval.md new file mode 100644 index 00000000..a558e0ac --- /dev/null +++ b/docs/reference/retrieval.md @@ -0,0 +1,44 @@ +# Retrieval behavior and evaluation + +This file is the home of the shipped retrieval pipeline's behavior and of how retrieval is evaluated. + +## Retrieval behavior + +The shipped search pipeline, in order; the code-level picture (types, constants, functions) is [docs/architecture.md](../architecture.md) "Chunks and the manifest", "Indexes and channels" and "Indexing flow and freshness". Traverse mode: [engine.md](../spike/multimap/engine.md). + +1. Scan the eighteen extensions of [Supported languages](../../README.md#supported-languages) (any ASCII case: a `.RS` file is Rust, N94) recursively with `.gitignore`, nested ignores and standard `ignore` crate rules. The checked-in `.lciignore` is an additional ignore file that keeps agent control-plane material (`.agents`, `.github/agents`, `.github/instructions`, `.github/skills`, `.codex`, `.claude`, `.kilo`) and generated `test-results` out of this repository's searchable corpus while leaving them on disk for agents. Symlinks are not followed, `.git` is skipped, hidden source files are eligible when not ignored. A file that is not valid UTF-8 (for example Windows-1252 C source) is indexed from a lossy decoding (invalid bytes become U+FFFD, line structure kept) and reported in `lossy_decoded_files`; it never fails indexing or search. A Qt Linguist `.ts` catalog is left out and counted in `skipped_non_source_files`; it does not affect the staleness fingerprint, and a file that changes into one drops its chunks at the next refresh like a deleted file. A source file removed between the walk and its read, or whose path is not valid UTF-8, is skipped with a logged warning. Other read/traversal errors abort the update and preserve the previous index; under `on-search` freshness the search then reuses the previous snapshot with a warning. +2. Select a static language adapter and Tree-sitter grammar by extension. Chunks preserve: Rust, its original declaration boundaries; TypeScript/JavaScript, imports, executable top-level statements, exports, comments/decorators, functions, classes and methods, interfaces and signatures, type aliases, enums, variable-assigned arrow functions; Python, imports, assignments, executable top-level statements, sync and async functions, classes, methods, decorators and leading comments; C#, using directives, namespaces, top-level statements, types, members, attributes, XML documentation comments, local functions; Go, top-level functions, methods, type/const/var declarations and leading comments; Java, packages, imports, classes, interfaces, enums, records, methods, constructors, fields, annotations, Javadoc; C, functions, structs, unions, enums, typedefs, preprocessor macros, global declarations, leading comments; C++, namespaces, classes, structs, unions, templates, methods, fields, leading comments. Oversized syntax splits only at named syntax boundaries toward 6,000-byte groups, with declarations preserved up to 24,000 bytes; large indivisible leaves stay whole rather than truncated; parser error recovery remains searchable. Markdown (`.md`) is cut into heading sections: a heading line plus its own body up to the next heading of any level (ATX and setext headings), with fenced code blocks kept whole and never searched for headings, YAML front matter kept with the first section, and text before the first heading as its own chunk; a section over 6,000 bytes is packed by paragraph toward that size, and a fence stays whole. +3. Parsed chunks, language identifier and adapter version are persisted per file in the external manifest (fingerprints include path, content, language and adapter version), so a later manual or watched pass reparses only changed, new or adapter-version-changed files, also after restart. A missing, incompatible or malformed manifest safely causes a full parse. +4. Each exact source chunk is hashed; vectors from the current workspace snapshot are reused when content hashes and the stored embedding URL/model/document-format identity match, and duplicate new chunks are embedded once. Cached vectors whose width differs from what the embedding service now returns are embedded again: the width comes from the process's latest embedding response, or, in a process that has embedded nothing yet, from one single-attempt probe of a short prefix of the first chunk, sent only when cached vectors exist (a failed probe warns and keeps them). A width change no request has revealed since the process's last one is found by the next search, which serves no semantic hits and warns; the index run after it repairs the index. Parser adapter versions are decoupled from embedding compatibility, so unchanged chunk text keeps its vector. Documents are embedded as source, without a query instruction. +5. A complete workspace snapshot is built in memory and committed as a LanceDB table overwrite after embedding succeeds; the matching manifest is saved only after that commit. Removed chunks disappear from the snapshot; line-only changes update locations without re-embedding. This is a small/medium-repository design, not a streaming indexer for huge monorepos. +6. Queries are embedded with `embedding_query_template` (default `Instruct: Given a code search query, retrieve relevant code passages that answer the query` newline `Query: `). +7. Candidates come from three channels. Semantic: LanceDB cosine distance, the `semantic_candidate_count` nearest chunks that pass the request's filters (the fetch is widened when filters reject rows, up to 10,240 rows; a search stopping at that cap warns that farther matching chunks were not considered). Lexical: ripgrep over all supported extensions in one pass, scored in process: + - Terms are the query's tokens as written plus the sub-tokens of their camelCase, snake_case and letter/digit parts (`parseConfig` also searches `parse` and `config`), without generic English and commit-message words such as `add`, `fix`, `when`, at most 24 per query. A query with no usable term returns no lexical candidates, without a warning. + - Matching lines are credited with the terms each really holds: a term of five or more characters counts as a case-insensitive substring, a shorter one only as a whole word (`add_item` and `addItem` hold `add`; `padding` does not). Lexical lines map to chunks of any indexed language. + - Chunks score by IDF-weighted term evidence: a term's document frequency is the number of indexed files with a credited line; terms credited in more than a quarter of the indexed files are dropped (kept when every term is that common); the twelve rarest remain; a chunk scores the BM25 sum over them of `idf * tf * (k1 + 1) / (tf + k1 * (1 - b + b * length / average_length))` with `idf = ln((N - df + 0.5) / (df + 0.5) + 1)`, `tf` the chunk's credited lines of the term, `length` its line count, `average_length` the mean over the filtered index's chunks, `k1 = 1.2`, `b = 0.75`. Chunks matching only dropped terms are not candidates. The constants live in `lci-core/src/lexical.rs` and are not configurable. + - LSP: for each language present in the filtered index, its enabled language server also searches workspace symbols; locations map only to same-language chunks. +8. Candidates are deduplicated and fused with Reciprocal Rank Fusion: each channel adds `weight / (rrf_k + rank)` for a chunk it returned, a chunk found by several channels sums the terms, and the source-role prior is added afterwards, unscaled. The `[fusion]` weights default to `1.0` (plain RRF); each must be finite and greater than zero (disabling a channel is `[channels]`' job). Weights change only `fusion_score` and through it the fusion order and shortlist, never ranks, scores or the reranker's per-document input. After the order is settled and before the shortlist is cut to `rerank_candidate_count`, `[fusion].max_chunks_per_file` (default `2`; `0` is unlimited and changes nothing) walks the fused order, keeps a chunk in the head while its file (by relative path) has fewer than that many kept, and moves the rest to a tail; the result is head then tail, each in original relative order. Nothing is removed, so `top_k` is still filled from demoted chunks when there are too few distinct files; `fusion_score` is not rescaled, so a demoted chunk can follow one with a lower score. The reranker sees the capped shortlist, reranker-off and reranker-failure results use the same capped order, and the reranker's output is not capped again. The cap and the weights were measured on the mined set and `core.toml` (numbers: [decisions](../decisions.md)). Results expose channel ranks, lexical match count and score, fusion score and channels; the lexical score never feeds the semantic or reranker scores. The full contract: `.agents/skills/lci-retrieval-evaluation/references/retrieval-contract.md`. +9. The fused shortlist is reranked, each candidate sent as its path plus at most `rerank_document_max_bytes` of its code (cut at a UTF-8 boundary, keeping the head); results always carry the full chunk. The three channels fail independently and available channels continue; an LSP provider that fails is skipped for the rest of that search. Each provider gets at most `lsp_search_budget_ms` of wall clock per search, alongside the other channels: a server still starting (rust-analyzer waits up to about ten seconds for its first indexing pass) or slow is skipped for that search, which reports an informational `notes` entry rather than a warning because nothing failed. The skipped server is not discarded: its start (spawned at most once per provider and workspace) keeps running in the background, followed by one priming `workspace/symbol` request that builds its symbol index (8-30 s on a large Rust workspace; searches during it also report the note, and no search budget cancels the priming), and a later search uses the server. A partial answer gathered before the budget ran out is kept; `lsp_search_ms` in the timings is the capped wait. Channels and the reranker can be switched off on purpose with `[channels]` (silent, unlike a failure). If reranking fails, the fusion order is returned with a warning appended to any earlier warnings. +10. Documentation lane. Markdown chunks (the prose extractor's, not every chunk with the `documentation` source role) are split off after the request's filters and never enter the code pipeline above, unless the request names `source_roles` (then the filters apply as stated and one list comes back). The code `results` list is therefore exactly what an index without the Markdown files returns: semantic search, lexical statistics (BM25 average length and document frequencies) and LSP mapping all see code chunks only. The documentation lane runs beside it with the same query vector: semantic over prose chunks only (`docs_lane.semantic_candidate_count`), lexical over prose chunks only, no LSP, the same fusion and per-file cap, a cut to `docs_lane.rerank_candidate_count`, a second rerank only when `docs_lane.reranker` (or the profile's `docs_reranker`) is on (default off, whatever `[channels].reranker` says), and the best `docs_lane.top_k` (default 2) come back in `docs.results` with `candidate_count`, `reranked`, its own timings and `warning`. `docs` is present exactly when the lane ran and the index holds prose chunks. The lane is fail-open: a semantic, lexical or reranker failure sets `docs.warning` and never changes the code list or its warning. The compact MCP view lists the docs after the results and shares its response ceiling. + +- ripgrep resolution: with the default `rg` or `rg.exe`, `PATH` is checked first; on Windows without `rg` on `PATH` it also checks common per-user and system-wide VS Code, VS Code Insiders and VSCodium installations (including versioned application directories). Any other configured value is an explicit command or path, used unchanged. If ripgrep cannot start, lexical retrieval fails open and the warning explains how to set `ripgrep_path`. +- Model clients: embedding response indices are validated and reordered; vectors must have consistent dimensions, finite values and nonzero norm. Model HTTP errors include the status and a bounded response body. Index-time document embedding retries connection errors, timeouts and a connection the server closed or reset mid-request (typically a pooled keep-alive connection closed while idle), up to three attempts with backoff (a cold model may still be loading). The query embedding, the indexing width probe (step 4) and the reranker make a single attempt, so a hung service costs a search one timeout and a refused connection costs no backoff; after a refused connection the reranker is also skipped for 30 s, and the warning says so. When the embedding service rejects a batch's input itself (HTTP 400/413/422, typically one chunk longer than the model's context), indexing retries that batch one chunk at a time and embeds a rejected chunk from a progressively halved prefix (down to 512 bytes); the full chunk is still stored, returned and lexically searchable and is listed in `truncated_for_embedding`. Its vector is cached by content hash, so raising the server's context limit later needs a reindex into a new data directory to re-embed it in full. Changing an embedding URL/model requires reindexing; replacing a model behind an unchanged name/URL is not detectable unless its vector width differs (step 4), so use a distinct model name or a new data directory. +- Timing fields (milliseconds) cover query embedding, LanceDB search, lexical search, LSP search, fusion, reranking and total retrieval; total also includes validation and index access/wait time. The same timings appear in tracing logs on stderr; CLI JSON stays on stdout. The default log filter is `local_code_intelligence=info,lci_core=info`; `RUST_LOG` replaces it (for example `RUST_LOG=local_code_intelligence=debug,lci_core=debug`). A `RUST_LOG` made only of crate-scoped directives that do not name `lci_core` silences that crate; a bare level such as `RUST_LOG=debug` applies to every crate. + +## Retrieval evaluation + +- `evaluate` runs a portable definition through the same application search path: `"$lci" evaluate ./evaluations/core.toml --workspace "self=$PWD" --workspace "gust=/path/to/gpu-dialect-v0" --output ./test-results/evaluation.json`. Workspace mappings are supplied at runtime, so checked-in definitions hold no machine paths. The GUST workspace is optional; its queries are skipped without a mapping. Without `--output`, JSON goes to stdout. A required missing workspace, search error, expected-path miss, wrong expected role or missing required snippet fails the query and gives a nonzero exit after the report is emitted; valid optional skips do not fail the run. `evaluate --baseline ` also embeds a `comparison` and prints its summary to stderr (`--seed`, `--resamples`, `--keep-failed`). With `[channels] lsp` on, `evaluate` first starts and primes every enabled language server for each workspace it queries (rust-analyzer is always enabled, so a missing binary is recorded as a failed warm-up), so no query pays a cold start; the outcome is the report's `priming` block (per workspace key and provider: `primed`, `ms`, `error`). `--no-prime` keeps the cold first query for a cold-start measurement. Priming changes which query pays the start, never a ranking. +- Arms: run `evaluate` once per configuration with `--config` and name the outputs by arm: the default, `[channels] reranker = false` (fusion order alone), or the grep-only baseline (see [Channels](../../README.md#channels) in the README). Reports do not record the switches, and the switches never invalidate an index, so arms can run one after another on one `data_dir`; never two at once on the same one. +- `compare` measures the difference between two saved reports without services, configuration or index: `"$lci" compare test-results/fusion.json test-results/semantic-only.json --output test-results/fusion-vs-semantic-only.json`. It pairs queries by workspace and query text (both reports must come from the same definition) and prints, for MRR, nDCG@10 and hit@1, the baseline and candidate means, the mean per-query delta (`candidate - baseline`) with a seeded 95% paired-bootstrap interval (`--resamples`, default 10,000; `--seed`), whether the interval excludes zero, and reciprocal-rank wins, losses and ties. Queries in only one report, or skipped in either, are excluded and counted; so is a query in which either side's language server missed `lsp_search_budget_ms` (`metrics.lsp_budget_hit`: it measured a cold start, not retrieval), counted as `budget_hit_*` on the pairs line and, for tagged definitions, per tag (`budget_hit_by_tag`), unless `--keep-failed` keeps it; a definition with `tags` also gets one block per tag. A regression is not an error: the exit code is nonzero only when a report cannot be read or parsed, or `--output` cannot be written (the summary is printed first). A delta whose interval contains zero is noise at that query count. +- Reports hold bounded path, line-range, role, rank, score and preview evidence; per-query hit@1, hit@3, hit@8, reciprocal rank, nDCG@10, recall@3/@8, file-level accuracy@5/@10, preference/disfavor counts, role distribution, reranker state, `lsp_budget_hit` (with `lsp_providers_over_budget` when set; a search response names them as `lsp_over_budget` only when a provider missed the budget), timings, lifecycle and effective filters; and aggregates: hit rates, MRR, means of the rank-aware metrics, `lsp_budget_hit_queries` (and per tag under `by_tag`), median/p95/max latency with sample count `n` (total and per stage; nearest-rank p95 hides the worst query at small `n`, for example it is the second-highest of 35, so read `max`), fallback count, and created/reused/waited/refreshed index counts. nDCG@10 uses graded gain (`expected_paths` match 2, `preferred_paths` only 1) and credits each gold pattern once; recall@k is the fraction of distinct expected patterns in the first k; file accuracy@k counts distinct files. The `@10` metrics are bounded by the query's `top_k`: hand-written `core.toml` uses 8 (effectively @8), the git-mined set 10; use `top_k >= 10` to compare with literature @10 numbers. Reports belong under `test-results` and are not committed. Evaluation reuses compatible persisted indexes and calls only the configured embedding and reranking services. +- Offline scoring: `evaluate --dump-candidates` records every query's whole fused candidate pool, the dumped configuration and the reranker's scores; `LCI_CANDIDATE_DUMP=` appends the same pool per search of a running process to a JSON Lines file (read once at startup; MCP, REST and CLI responses are unchanged). `"$lci" replay test-results/dump.json --output test-results/replay-w12.json --window 12 --cap 2 --cap-stage results` then scores other windows, caps, fusion weights, `rrf_k`, priors and rerank modes from the dump without services, configuration (the `--config` file is not read) or index, and `compare` pairs the output with real reports. A query replay cannot score exactly is refused and counted, never approximated. Row format, replay semantics, the latency model and when to run live instead: `.agents/skills/lci-retrieval-evaluation/references/evaluation-workflow.md`. +- Git-mined query set: `cargo xtask mine-queries --repo --key --out evaluations/mined/.toml [--max 300]` turns qualifying commits from `git log` into queries (subject plus first body paragraph as the query, touched supported files as `expected_paths`, `top_k = 10`). `evaluations/mined/self.toml` (workspace key `self`) is the generated set for this repository: run it like `core.toml` (`--workspace "self=$PWD"`), regenerate rather than edit it, never use it to tune ranking. `core.toml` is the hand-written required set. Skip rules, labels and tags: same reference. +- Gold strata and transcript shapes: `cargo xtask gold strata --set --gold [--out ]` classifies each gold item as `core`, `grep_easy` or `agent_trap` from the item alone and reports the counts, the hop quantiles, the items no rule reaches and the items whose question holds a gold file's stem; the same `guard` is a stratum of every scorer. `cargo xtask mine-transcripts --input .. [--gold ] [--out test-results/transcripts/shapes.json]` counts question shapes and trace classes over synthetic transcripts in the sim trace shape and writes only shapes, counts and anonymised templates. Rules and schema: [gold-scorers.md](../spike/multimap/v2/gold-scorers.md). +- Live acceptance: `cargo xtask acceptance core` and `acceptance lsp` default to this repository (`--profile gust --workspace /path/to/gpu-dialect-v0`, or `$LCI_ACCEPTANCE_GUST_WORKSPACE`, runs them against the optional external GUST example); `lsp` checks workspace-symbol, definition and reference navigation against the same symbol and saves each normalized response in `test-results`. With the real embedder and reranker running, `cargo xtask acceptance multilingual` creates a disposable fixture and data directory under `test-results`, uses the real debug binary; it verifies all supported language IDs, semantic and lexical retrieval with live reranking, production TypeScript, Python and C# ranking over test decoys, zero-work unchanged indexing, one-file invalidation, deletion, restart reuse, and prior-snapshot retrieval after an unreachable embedding endpoint makes an update fail (`test-results/multilingual-*.json`). Per language, `-lsp`, `-missing` and `-recovery` run a real server (options and prerequisites: `.agents/skills/lci-lsp-adapter/references/acceptance.md`, `docs/development/live-acceptance-ci.md`); each skips with `PREREQUISITE_UNAVAILABLE` when its server is not found. +- Metric tiers (method and status in [evaluation.md](../spike/multimap/evaluation.md)): agent runs are the ground truth but too costly to run regularly, so cheap proxies run constantly and are checked against agent runs now and then, and quality is reported together with cost. + +| Tier | When | What | +| --- | --- | --- | +| 1 | every change, deterministic | hit@k, recall@k, span hit, MRR on the frozen sets; speed proxies (pops, hops, expansions); cost-aware recall (recall per 1k tokens returned); the simulated agent (design, not built) | +| 2 | milestones, cheap model in the loop | confidence calibration; a small fixed subset judged by a cheap model | +| 3 | rare, before releases | real agent A/B runs, mainly to validate the proxies |