diff --git a/.agents/skills/lci-autonomous-improvement/SKILL.md b/.agents/skills/lci-autonomous-improvement/SKILL.md index 872fdbaa..8c21900a 100644 --- a/.agents/skills/lci-autonomous-improvement/SKILL.md +++ b/.agents/skills/lci-autonomous-improvement/SKILL.md @@ -43,7 +43,7 @@ Repeat until the stop rule fires or the roadmap has no ready item. 3. **Implement.** Brief an implementer with the plan, the allowed-file list, the relevant contract files, and whether the `lci` MCP tools are connected. Inspect the saved diff for file scope and intent before anything else; a valid diff counts even without prose. Fix small gaps yourself or send one focused continuation; do not start a second implementer on the same files. A slice that depends on an unmerged one starts on a branch stacked on that one's branch instead of waiting for its merge; merge `origin/main` into it once the base lands, before its own gate and PR (d476). 4. **Gate.** Run the focused suite first (`cargo xtask test --suite ` or `cargo test -p lci-core --lib ::`), then `cargo xtask test --suite Full` once the slice is complete. When the diff adds or changes process-wide state (a `static`, `OnceLock`, `thread_local!`, `task_local!` or a process-wide memo), also run the library tests of the affected crates once with `-- --test-threads=1` before pushing: such state can make a test pass only in some orders, and CI runs another order (d476; a serial pass of all library tests costs about 2.5 extra minutes, too much for every push). See "When a gate fails". 5. **Evaluate (when the plan says so).** Follow `lci-retrieval-evaluation/references/evaluation-workflow.md` exactly: frozen `lci-base`/`lci-new` binaries, scratch configs with short separate data directories, sequential runs, reranked and reranker-unreachable pairs for prior/role/fusion changes, reports under `test-results/`. Do not compile or run tests while a timed run is in progress. Decide with the workflow's decision rule; if the change does not hold up, revert it and record why in `docs/decisions.md` (the roadmap row keeps a status only). -6. **Review.** Brief a reviewer who did not write the code with the diff, the slice plan, and the evidence. It checks correctness, `AGENTS.md` boundaries (ports, data location, fail-open, freshness semantics), test honesty (no weakened or vacuous tests; suites that still match tests), documentation drift (README tables, `config.example.toml`, skill references, MCP tool descriptions), and scope creep. Address every finding or record why not; a disputed finding goes to the human, not silently dropped -- unless the human has delegated disputes, in which case pick one side (at random if both are defensible), log the dispute and the pick in `docs/decisions.md`, and continue. +6. **Review.** Brief a reviewer who did not write the code with the diff, the slice plan, and the evidence. It checks correctness, `AGENTS.md` boundaries (ports, data location, fail-open, freshness semantics), test honesty (no weakened or vacuous tests; suites that still match tests), documentation drift (README tables, `docs/reference` guides, `config.example.toml`, skill references, MCP tool descriptions), and scope creep. Address every finding or record why not; a disputed finding goes to the human, not silently dropped -- unless the human has delegated disputes, in which case pick one side (at random if both are defensible), log the dispute and the pick in `docs/decisions.md`, and continue. 7. **Commit and push.** One logical change per commit, message per the template: imperative subject under about 70 characters, a body saying what changed and why, the evidence (test counts, suites, live metrics before/after, report names), and limitations. The client adds the co-author trailer. Stage only the slice's files; never `git add -A` with scratch material present (`reports/`, `research_notes/`, `test-results/` are not slice files). Push the feature branch. 8. **Record.** In the commit that does the work, remove the finished item's row from the roadmap and append its outcome (commit hash, evidence summary, measured results, dropped ideas) to `docs/decisions.md` as one line under the next free id; set the status of a successor whose only blocker it was and add follow-ups as new rows under the next free id of their letter (roadmap "How to use this file", step 5). An unfinished item keeps its row with a status (`in-progress`, `blocked(reason)`, `ready`) and what remains, and no result text. Update any docs the reviewer flagged in the same commit as the code, or in an immediately following one. 9. **Check usage and continue.** Every fifth slice, run the triage pass first. Then check usage: below the stop threshold, run an explorer in the background if none is running and pick the next item; otherwise follow "Stopping at the usage limit". diff --git a/.agents/skills/lci-autonomous-improvement/references/templates.md b/.agents/skills/lci-autonomous-improvement/references/templates.md index 35b0e3fb..0fdc6396 100644 --- a/.agents/skills/lci-autonomous-improvement/references/templates.md +++ b/.agents/skills/lci-autonomous-improvement/references/templates.md @@ -9,14 +9,14 @@ Roadmap item: Goal: Not in scope: Allowed files: -Contracts to read first: +Contracts to read first: Acceptance: - - Tests to add or change: Gate: cargo xtask test --suite ; then --suite Full Live evaluation: -Docs to update: +Docs to update: Revert condition: ``` @@ -48,7 +48,7 @@ a one-line reason. Check in this order: freshness semantics, no client-specific code, no new service. 3. Tests: honest (no weakened assertions, no #[ignore], no vacuous suite filters), cover the changed behavior and its failure path. - 4. Doc drift: README tables and prose, config.example.toml, skill + 4. Doc drift: README tables and prose, docs/reference guides, config.example.toml, skill references, MCP tool descriptions, evaluations/core.toml comments. 5. Scope: files outside the allowed list, unrelated refactors. Say explicitly if you found nothing in a category. Do not edit files. @@ -86,7 +86,7 @@ The diff is the deliverable; keep prose short. ```text Reviewer: slice . Read /, then `git diff ..` and . Do not edit. -Check in order: correctness against the acceptance; AGENTS.md "Hard rules"; test honesty; doc drift (README, config.example.toml, skill references, decision line in `docs/decisions.md` if a default moved); scope. +Check in order: correctness against the acceptance; AGENTS.md "Hard rules"; test honesty; doc drift (README, docs/reference guides, config.example.toml, skill references, decision line in `docs/decisions.md` if a default moved); scope. Report file:line, severity (blocker | should-fix | nit), one line each; say "none" for an empty category. lci MCP: . Do not run live evaluations or start services. diff --git a/.agents/skills/lci-language-adapter/references/acceptance.md b/.agents/skills/lci-language-adapter/references/acceptance.md index 9382648e..9e228f26 100644 --- a/.agents/skills/lci-language-adapter/references/acceptance.md +++ b/.agents/skills/lci-language-adapter/references/acceptance.md @@ -14,7 +14,7 @@ A language-support claim requires evidence for the affected language: 10. Compatible reuse after application restart. 11. Semantic and lexical retrieval plus reranking and reranker fail-open behavior. 12. A production implementation ranked against a realistic test, example, or documentation decoy. -13. README, MCP, CLI, filter, and evaluation-definition updates. +13. README, docs/reference guides, MCP, CLI, filter, and evaluation-definition updates. 14. LSP navigation available via the language-server adapter contract, passing `lci-lsp-adapter`'s own acceptance bar (real workspace symbol/definition/reference evidence, provider isolation, and recovery). ## Where each class of evidence comes from @@ -24,7 +24,7 @@ A language-support claim requires evidence for the affected language: | 1-4, 6-10 (mocked) | `lci-core` unit tests plus `tests/cases/indexing.rs` with the fake model service | `cargo test -p lci-core --lib chunk::`, `cargo xtask test --suite Indexing` | | 5-11 (live) | mixed-language fixture indexing, update, delete, restart, failed-update retention, live reranking | `cargo xtask acceptance multilingual` (8766/8767 up; reports `test-results/multilingual-*.json`) | | 12 | a scratch decoy evaluation TOML kept outside the repo, run with and without the reranker | `lci-retrieval-evaluation/references/evaluation-workflow.md` | -| 13 | README language table and chunk description, `config.example.toml` (if an LSP section exists), `evaluations/core.toml` chunking query, MCP tool descriptions | review the diff against `.github/instructions/documentation.instructions.md` | +| 13 | README language table and chunk description, `docs/reference` guides, `config.example.toml` (if an LSP section exists), `evaluations/core.toml` chunking query, MCP tool descriptions | review the diff against `.github/instructions/documentation.instructions.md` | | 14 | real server symbol/definition/reference, provider isolation, recovery | `cargo xtask acceptance -lsp`, `-missing`, `-recovery` | Use mocks for deterministic automated coverage and the configured 8766/8767 services for one bounded live acceptance. Report those evidence classes separately. diff --git a/.agents/skills/lci-lsp-adapter/SKILL.md b/.agents/skills/lci-lsp-adapter/SKILL.md index 0e7ab85f..6a26abf1 100644 --- a/.agents/skills/lci-lsp-adapter/SKILL.md +++ b/.agents/skills/lci-lsp-adapter/SKILL.md @@ -7,7 +7,7 @@ description: Add or change an optional language-server integration in local-code Keep LSP optional and fail-open for code search while returning clear errors from direct navigation tools. -Where things live: one adapter module per server under `lci-core/src/lsp/` (`rust.rs`, `csharp.rs`, `typescript.rs`, `python.rs`, `go.rs`, `java.rs`, `clangd.rs`) behind the shared trait in `adapter.rs` and the JSON-RPC transport in `transport.rs`; registration in `Manager::new` (`lci-core/src/lsp.rs`); the per-server `[
]` config in `lci-core/src/config.rs` (rust-analyzer: top-level `rust_analyzer_*` keys); the real-server acceptance specs under `xtask/src/acceptance/`. +Where things live: one adapter module per server under `lci-core/src/lsp/` (`rust.rs`, `csharp.rs`, `typescript.rs`, `python.rs`, `go.rs`, `java.rs`, `clangd.rs`, `markdown.rs`) behind the shared trait in `adapter.rs` and the JSON-RPC transport in `transport.rs`; registration in `Manager::new` (`lci-core/src/lsp.rs`); the per-server `[
]` config in `lci-core/src/config.rs` (rust-analyzer: top-level `rust_analyzer_*` keys); the real-server acceptance specs under `xtask/src/acceptance/`. 1. Read [the lifecycle and routing contract](references/lifecycle.md). 2. Reuse a persistent process per compatible workspace and server configuration. diff --git a/.agents/skills/lci-lsp-adapter/references/acceptance.md b/.agents/skills/lci-lsp-adapter/references/acceptance.md index 8ac233ec..3cb4c2ca 100644 --- a/.agents/skills/lci-lsp-adapter/references/acceptance.md +++ b/.agents/skills/lci-lsp-adapter/references/acceptance.md @@ -12,7 +12,7 @@ Automated coverage (fake LSP in `tests/support`, `tests/cases/navigation.rs`, `l - Search fallback when the executable or request is unavailable. - Direct-navigation error behavior. -Real acceptance must use the actual language-server executable against a known workspace and verify at least one workspace symbol, definition, and reference. The three xtask subcommands per language are the standard form: `cargo xtask acceptance -lsp` (full flow), `-missing` (provider isolation when the executable is absent), `-recovery` (persistent process reuse and forced-kill recovery). They start fixture servers on port 8769 and write normalized responses under `test-results/`. Record the executable and version, and distinguish unavailable tooling (`PREREQUISITE_UNAVAILABLE`) from a code failure. Do not claim support from a mocked server alone. +Real acceptance must use the actual language-server executable against a known workspace and verify at least one workspace symbol, definition, and reference. The three xtask subcommands per language are the standard form (markdown has the first two only): `cargo xtask acceptance -lsp` (full flow), `-missing` (provider isolation when the executable is absent), `-recovery` (persistent process reuse and forced-kill recovery). They start fixture servers on port 8769 and write normalized responses under `test-results/`. Record the executable and version, and distinguish unavailable tooling (`PREREQUISITE_UNAVAILABLE`) from a code failure. Do not claim support from a mocked server alone. Known, documented limitations are not failures: `workspace/symbol` for TypeScript, Python, and C/C++ only sees files the server has been told about, and a fresh `clangd` process does not find cross-file references. The acceptance specs encode these as informational (TypeScript, C/C++) or skipped (Python) stages; do not "fix" them by weakening an assertion for another language. @@ -26,6 +26,7 @@ Each language's three subcommands run against a real fixture and the real server - `go`: a real `go.mod`-rooted module and `gopls` (`--gopls `). - `java`: a real Eclipse `.project`/`.classpath` fixture and jdtls (`--java `, a directory and not an executable, matching `[java].path`; `$LCI_ACCEPTANCE_JDTLS` sets the same directory for repeated local runs). - `c` and `cpp`: real standalone C/C++ fixtures and one shared `clangd` (`--clangd `, for each). +- `markdown`: two real documents (prose indexing on) and `marksman` (`--marksman `, or `$LCI_ACCEPTANCE_MARKSMAN`); only `markdown-lsp` (heading symbols, a link definition, references to a heading) and `markdown-missing` exist. All of them skip gracefully when their server is not found. Prerequisites per tool: `docs/development/live-acceptance-ci.md` "Prerequisites". diff --git a/.agents/skills/lci-release-validation/references/validation-matrix.md b/.agents/skills/lci-release-validation/references/validation-matrix.md index 9e8cdfbf..dc65418c 100644 --- a/.agents/skills/lci-release-validation/references/validation-matrix.md +++ b/.agents/skills/lci-release-validation/references/validation-matrix.md @@ -9,7 +9,7 @@ Choose rows affected by the change; a release candidate should cover the full ma | Persistence | temporary LanceDB, manifest compatibility, restart reuse, failed-update preservation (`--suite Indexing`, `--suite Watching`) | external data directory survives process restart | | Retrieval | mock semantic/lexical/LSP/fusion/reranker and filter coverage (`--suite Filter`, `--suite Evaluation`) | `evaluate ./evaluations/core.toml` against 8766/8767 with auditable source results | | Languages | parser/chunk/update/delete/restart tests per adapter (`--suite Chunk`, `--suite Indexing`) | `cargo xtask acceptance multilingual`; production-versus-decoy query per added language | -| LSP | protocol/lifecycle/location tests (`--suite Lsp`) | `cargo xtask acceptance -lsp` / `-missing` / `-recovery` per language | +| LSP | protocol/lifecycle/location tests (`--suite Lsp`) | `cargo xtask acceptance -lsp` / `-missing` / `-recovery` per language (markdown: no `-recovery`) | | Windows tools | resolver unit/integration tests | resolved `rg.exe`, optional language-server executables | | Packaging | staged-file and artifact scan | clean extraction and smoke test when distributing | diff --git a/.agents/skills/lci-retrieval-evaluation/references/evaluation-workflow.md b/.agents/skills/lci-retrieval-evaluation/references/evaluation-workflow.md index d3b537c1..34cd7487 100644 --- a/.agents/skills/lci-retrieval-evaluation/references/evaluation-workflow.md +++ b/.agents/skills/lci-retrieval-evaluation/references/evaluation-workflow.md @@ -32,7 +32,7 @@ Scope: the embedder-plus-reranker sets (`evaluations/core.toml`, the mined sets) ## Replaying a dump instead of running live -For a live service rather than an evaluation run, `LCI_CANDIDATE_DUMP=` (README) appends the same pool per search to a JSON Lines file. +For a live service rather than an evaluation run, `LCI_CANDIDATE_DUMP=` ([retrieval reference](../../../../docs/reference/retrieval.md), "Offline scoring") appends the same pool per search to a JSON Lines file. `evaluate --dump-candidates` records every query's whole fused candidate pool, the dumped configuration (`arm`), and the reranker's scores; ` replay dump.json --output [options]` then scores other settings from that file in seconds, offline (no services, config, or index), through the unchanged metrics. Its output is an evaluate-shaped report, so ` compare dump.json ` pairs it with real reports and with other replays. Run one dump live; replay the grid. diff --git a/.github/instructions/documentation.instructions.md b/.github/instructions/documentation.instructions.md index 5daf32cf..20cbb1d1 100644 --- a/.github/instructions/documentation.instructions.md +++ b/.github/instructions/documentation.instructions.md @@ -1,7 +1,7 @@ --- name: LCI documentation -description: Accuracy rules for README, evaluation definitions, configuration examples, and MCP descriptions. -applyTo: "{README.md,config.example.toml,evaluations/**/*,src/server.rs}" +description: Accuracy rules for README, the docs/reference guides, evaluation definitions, configuration examples, and MCP descriptions. +applyTo: "{README.md,docs/reference/*.md,config.example.toml,evaluations/**/*,src/server.rs}" --- - Describe current behavior and independently verified evidence. Label unavailable live prerequisites and fail-open fallbacks precisely. diff --git a/.github/instructions/rust-code.instructions.md b/.github/instructions/rust-code.instructions.md index 4b9116d6..c947e6e6 100644 --- a/.github/instructions/rust-code.instructions.md +++ b/.github/instructions/rust-code.instructions.md @@ -4,7 +4,7 @@ description: Architectural and implementation rules for Rust source in local-cod applyTo: "{src/**/*.rs,lci-core/src/**/*.rs}" --- -- Keep client-neutral behavior in the Rust service. Do not import editor, Copilot, Kilo, or generation-model concerns. Keep tracing on stderr and structured CLI JSON on stdout. +- Keep client-neutral behavior in the Rust service. Do not import editor, client, or generation-model concerns. Keep tracing on stderr and structured CLI JSON on stdout. - Use the existing module boundaries: configuration/workspace identity, language/chunk/manifest, models, store, LSP, filtering/evaluation, application coordination, and MCP/HTTP transport. - The workspace has four crates, split along dependency lines: `lci-core` (chunk, language, filter, lexical, manifest, workspace, config, models, lsp; no LanceDB/arrow, compiles and tests in seconds), `lci-maps`, `xtask`, and the root crate (app, evaluate, init, rest, server, store, main). Put new pure-logic code (a language adapter, a filter rule, an LSP adapter) in `lci-core` unless it needs `Store`/`App`. Never add a `lci-core -> local-code-intelligence` dependency edge. - Preserve external storage boundaries and per-workspace coordination. Never hold a workspace read guard while entering an indexing path that needs its write guard. diff --git a/README.md b/README.md index 96cf98fa..0fb59051 100644 --- a/README.md +++ b/README.md @@ -41,6 +41,8 @@ These eighteen extensions (twelve language identifiers) are the whole indexable ## Documentation - [Architecture guide](docs/architecture.md): crates, runtime boundaries, data flow, the agent control-plane exclusion. +- [Language-server channel](docs/lsp-channel.md): sessions, breakers, readiness, sync and the `lsp.live` use of the language servers. +- [Traverse call flow](docs/traverse-flow.md): what one traverse call does and which `[run_profile]` keys add answer fields. - [Language scope](docs/languages.md): language server, syntax tier, SCIP and gold per language. - [MCP tools and REST surface](docs/reference/mcp-and-rest.md): the tool table, the language-server lifecycle, the REST/OpenAPI mirror. - [CLI reference](docs/reference/cli.md): every `lci` subcommand and option. @@ -65,7 +67,7 @@ Prerequisites: Rust stable 1.95 or newer; a C/C++ toolchain (MSVC + Windows SDK - TypeScript/JavaScript: `npm install -g typescript-language-server typescript` (the workspace also needs its own `typescript` dependency); Python: `npm install -g pyright` (provides `pyright-langserver`); - Go: `go install golang.org/x/tools/gopls@latest`; Java: JDK 21+ and an extracted `jdt-language-server-*.tar.gz` release from (`[java].path` points at that directory); C/C++: a standalone `clangd`; Markdown: a `marksman` release from (`[markdown].path`). -All dev, build, test and acceptance tasks run through `cargo xtask ` (subcommands: `setup`, `build`, `test`, `acceptance`, `live-acceptance`, `env`, `lint-machine-paths`, `docs`, `gold`, `maps`, `mine-queries`), identical on every OS: +All dev, build, test and acceptance tasks run through `cargo xtask ` (the 14 commands are listed below the example), identical on every OS: ```bash cargo xtask setup @@ -79,6 +81,14 @@ cp target/debug/local-code-intelligence "$scratch/" # local-code-intelligence. - Serve from a copy, not from `target/`: a running executable is locked on Windows, so the next build would fail, and on any OS a rebuild would silently change what a long-running server is. Short commands (`init`, `config`, the CLI walkthrough) can run from `target/debug` directly. - `serve` needs the model services below. `/health` answering only proves the process is up; wait for `curl http://127.0.0.1:8768/ready` to answer HTTP 200 before the indexing walkthrough (see the [CLI reference](docs/reference/cli.md)). +- `cargo xtask` commands (source: `Command` in `xtask/src/main.rs`; `--help` shows each one's options): + - `setup`: provision `protoc` (and cmake guidance on macOS/Linux); `build [--test]`: build the workspace; `test --suite `: one focused suite ([testing guide](docs/development/testing.md)). + - `acceptance `: a live acceptance run against the locally built binary (`core`, `lsp`, `multilingual`, and `-lsp`, `-missing`, `-recovery` per language (`csharp`, `typescript`, `python`, `go`, `java`, `c`, `cpp`), and `markdown-lsp`, `markdown-missing`); `live-acceptance`: the build, the acceptances, `lci config --json` and the evaluation in one local run with per-step timeouts ([local live acceptance](docs/development/local-live-acceptance.md)). + - `env`: resolve development paths by role ([machine values](docs/development/machine-values.md)); `lint-machine-paths`: fail when a tracked text file names a machine-specific location; `docs check`: fail on a relative Markdown link or anchor that does not resolve. + - `mine-queries`: mine an evaluation definition from git history; `mine-transcripts`: count question shapes over synthetic agent transcripts (both in the [retrieval reference](docs/reference/retrieval.md)). + - `gold`: `score`, `baseline`, `to-evaluate`, `strata`, `validate-structured`, `external fetch` and `external convert` over the local gold sets. + - `maps`: `bench`, `fit-priors`, `fit-tiebreak`, `lsp-cost`, `compare`, `trajectories` (pop events joined to the gold outcome), `baseline` (`impact`, `history`, `rg`, `serena` arms), `oracle-policy`, `oracle-filter`. + - `sim`: the deterministic simulated agent; `sim-structured`: `run`, `compare`, `check-probes`, `check-corpus` on it. Options and formats: [gold-scorers.md](docs/spike/multimap/v2/gold-scorers.md) (`gold`), [trend.md](docs/spike/multimap/v2/trend.md) (`maps`), [sim-agent.md](docs/spike/multimap/v2/sim-agent.md) (`sim`). - First search (the model services must be up; the full walkthrough and every subcommand are in the [CLI reference](docs/reference/cli.md)): `./target/debug/local-code-intelligence index "$PWD"`, then `./target/debug/local-code-intelligence search "$PWD" 'plain REST JSON-over-HTTP mirror of the MCP tools'`. ### Model services (prerequisite) @@ -149,7 +159,7 @@ Defaults work with the services described under Ports and URLs. Copy `config.exa | `reranker_timeout_seconds` | 120 | | `rescorer` | `"reranker"` (allowed: `"reranker"`, `"system_one"`; which service orders the fused shortlist while `[channels].reranker` is on. `"system_one"` needs `[system_one]` with `provider = "system_one_wire"`: one `POST /v1/systemone` per search with a yes/no question per candidate, the answers used as probabilities; any failure returns fusion order with one warning that names an HTTP status by code only (the response body is never shown), and the reranker is not the fallback; a candidate dump records the System One model and batch size, and `replay` refuses a scored replay of such a dump when the window changes the candidates sent or their order) | | `[system_one].provider`, `.url`, `.model`, `.api_key_env`, `.allow_remote`, `.timeout_ms`, `.batch_size` | `openai_compat`, `http://127.0.0.1:8080/v1`, `qwen3-4b`, unset, `false`, 30000 (allowed: 100-600000), 24 (allowed: 1-200; with `system_one_wire`, the most candidates per request) | -| `readiness_timeout_seconds` | 15 (allowed: 1-30; was 5. It bounds `/ready`, `init` and `service_status` probes and the `lsp.live` readiness probe (` --version`; `gopls version`, since gopls has no `--version`; a start of `pyright-langserver --version` that may exit non-zero, since it has no version flag), so they wait up to 15 s for a hanging dependency and report ready one that answers in 5 to 15 s; a refused port or a missing binary still fails at once) | +| `readiness_timeout_seconds` | 15 (allowed: 1-30). It bounds `/ready`, `init` and `service_status` probes and the `lsp.live` readiness probe (` --version`; `gopls version`, since gopls has no `--version`; a start of `pyright-langserver --version` that may exit non-zero, since it has no version flag): a hanging dependency is waited for up to this long, a dependency that answers within it is reported ready, and a refused port or a missing binary fails at once | | `ripgrep_path` | `rg` (on Windows, also discovers common VS Code-bundled copies) | | `semantic_candidate_count` | 40 | | `lexical_candidate_count` | 40 | @@ -183,7 +193,7 @@ Defaults work with the services described under Ports and URLs. Copy `config.exa | `[search_profiles.].compact_max_response_chars` | `16000` with `response = "compact"` (allowed: 1000-1000000; an error without it) | | `[traverse]` | `enabled = false`; `scip_index = "index.scip"`; `max_pops = 200`; `max_depth = 6`; `max_est_tokens = 20000`; `deadline_ms = 5000` (bounds the walk; setup that is credited back to it, namely building the tree maps (the `lsp.live` version probe among them), the seed search and the tree barrier, is not counted, so a call's wall time can exceed it by that setup); `max_stops = 4`; `tau = 0.8` (the rules rung has confidence 0 or 1, so any value above 0 acts alike; see [engine.md](docs/spike/multimap/engine.md), "Written meanings"); `first_stage = "append"`; `answer_seeds = true`; `exact_seeds = true`; the remaining keys and their comments are in `config.example.toml`. The edge-bandit keys (`edge_bandit`, `bandit_*`) and the matching `lci traverse` flags (`--edge-bandit`, `--bandit-*`) no longer exist: a config that sets one fails to load naming the key, and a stored handle made with the bandit on is refused (d365) | | `[recipes]`, `[compose]` | `enabled = false` each (`[recipes] flow_judge = false`) | -| `[run_profile]` | `[run_profile.walk]` with `groups` (`"tool"` default: one piece of evidence per map; `"claim"`: one per `(source node, destination node, claim family)` whichever maps assert it, the fused confidence is the noisy-OR over claims, the score bound is off, and it cannot be combined with `[traverse] fusion = "rrf"` or `"log_odds"`) and `grep_corroboration` (0 to 1, default 1: what a group made only of text-match edges adds to the evidence count); both are in the profile hash and a handle keeps them; `pricing` (`"rank"` default: above the gate of 8 edges the edge at rank `r` is worth `1 / (1 + r)` of its kind's prior; `"relative"`: rank `r` of `n` candidates is worth `1 / (1 + a + r * gate / n)`, `a` being a trait seed's implementors ranked in front (0 elsewhere), so the price equals `"rank"` at the gate, a wide fan-out is flattened instead of starved and nothing changes at or below the gate) is in the profile hash too, and a handle does not carry it or refuse a different one (a resumed call prices under the profile it is given, so a walk can mix pricings across resumes, as it can mix beams); `confirm` (default `false`; `true`: with `grep.live` and `lsp.live` in the call, the hops an expansion keeps whose only witnesses are `grep.live` text proposals, the best `top_k` by hop value, are confirmed in one call through `lsp.live`'s `textDocument/definition` at each hop's site: a definition that is the expanded node keeps the hop as a server answer, one that is another node drops it, and no answer, an error, the deadline or no external call left leaves it as it was; each proposal is at most one external call under `max_external_calls`, a failure never ends a walk, and without either map the walk runs unconfirmed with a note; the counts are in the trace (`confirm_*` in the `end` totals, a `confirm` field on the `expand` events) and in one note; in the profile hash, and a handle made with it, at its `top_k` and places arm, does not resume without it, nor the reverse); `[run_profile.traverse]` with `map_freshness` (default `false`; see "Map freshness" in the CLI reference), `answer_labels` (default `false`; see "Answer labels" in the CLI reference) and `card_paths` (default `false`; see "Paths on cards" in the CLI reference), all in the profile hash; `[run_profile.freshness]` with `stat_gate` (default `false`; see "Stat gate" in the CLI reference; in the profile hash) and `verify_on_read` (default `false`; see "Verify on read" in the CLI reference; in the profile hash, and set in the file only, with no command-line flag, so the stamp is what ran); `[run_profile.scip]` with `mode` (`"cache"` default: a traversal reads the index when the file is there and readable, otherwise walks an empty SCIP graph and says so in a note; a file that cannot be parsed is parsed once per file stamp (size and modified time), and later calls say that it was not read again until the stamp changes or 30 seconds pass (a constant, not a key; an I/O error is never remembered); `"authoritative"`: the strict opt-in, a traversal needs the SCIP index and fails without it; `"off"`: it never touches the index, so `graph.scip` and the maps built from the index file (`structure.contains` beyond chunk lines, `scip.impact`, `scip.implements`, `tests.tested_by`, `config.key`, `exact.closure`) are empty; in the profile hash, and a handle made with a graph does not resume under `off`; recipes read the index under `cache` when it is there and refuse under `off` or without it, with a hint on how to write one) and `arbitration` (default `false`; `true`: with `lsp.live` in the call and `mode` not `"authoritative"`, a language server's answer at a node overrules the index's `scip.impact`, `scip.implements` (with `implementations = true`) and `structure.contains` edges to functions and methods that it did not name, only in a claim family it answered at that node; `graph.scip` is never arbitrated; the counts are in the trace, `totals.arbitration`, `per_map` and one note, see "Arbitration of SCIP edges" in the CLI reference; in the profile hash, and a handle made with it does not resume without it, nor the reverse); `[run_profile.lsp]` with `sessions` (`"shared"` default, or `"per_call"`) and `outline_cache` (`false` default, or `true`; see "Language-server sessions of `lsp.live`" in the CLI reference for both) and `implementations` (default `false`; `true` makes `lsp.live` also ask `textDocument/implementation` at each symbol node and offer `implements` edges, one more external call per node; see the same section), `call_hierarchy` (default `false`; `true` makes `lsp.live` also ask the call hierarchy at each symbol node and offer the callers and callees as `calls_or_refs` edges, up to three more external calls per node; see the same section), `sync` (`"watch"` default, or `"off"`; see "Language-server sync" in the CLI reference), `breaker` (default `true`; see the same section), `live` (default `true`: `lsp.live` is in the default map set of a service call; see the same section), `places` (`"auto"` default, `"scip"` or `"outline"`; see "Place table of `lsp.live`" in the CLI reference) and `roots` (`"workspace"` default, or `"manifest"`; see "Language-server roots" in the CLI reference) are in the profile hash as well; `[run_profile.seeds]` with `bm25` and `literal` (default `false` each; see "Entity seeds" and "Literal seeds" in the CLI reference) is in the profile hash too; any other key is an error. `Config::run_profile()` hashes the resolved result-affecting settings (sha-256 of their canonical JSON, `config --json` `derived.run_profile_hash`): all settings except the machine-set keys, timeouts and timing, batch size, traverse file paths (`scip_index`, `priors`, `tie_break`), `rust_analyzer_trusted`, `[system_one]` and the API-key variable names; `config::profile::excluded_keys()` lists them | +| `[run_profile]` | `[run_profile.walk]` with `groups` (`"tool"` default: one piece of evidence per map; `"claim"`: one per `(source node, destination node, claim family)` whichever maps assert it, the fused confidence is the noisy-OR over claims, the score bound is off, and it cannot be combined with `[traverse] fusion = "rrf"` or `"log_odds"`) and `grep_corroboration` (0 to 1, default 1: what a group made only of text-match edges adds to the evidence count); both are in the profile hash and a handle keeps them; `pricing` (`"rank"` default: above the gate of 8 edges the edge at rank `r` is worth `1 / (1 + r)` of its kind's prior; `"relative"`: rank `r` of `n` candidates is worth `1 / (1 + a + r * gate / n)`, `a` being a trait seed's implementors ranked in front (0 elsewhere), so the price equals `"rank"` at the gate, a wide fan-out is flattened instead of starved and nothing changes at or below the gate) is in the profile hash too, and a handle does not carry it or refuse a different one (a resumed call prices under the profile it is given, so a walk can mix pricings across resumes, as it can mix beams); `confirm` (default `false`; `true`: with `grep.live` and `lsp.live` in the call, the hops an expansion keeps whose only witnesses are `grep.live` text proposals, the best `top_k` by hop value, are confirmed in one call through `lsp.live`'s `textDocument/definition` at each hop's site: a definition that is the expanded node keeps the hop as a server answer, one that is another node drops it, and no answer, an error, the deadline or no external call left leaves it as it was; each proposal is at most one external call under `max_external_calls`, a failure never ends a walk, and without either map the walk runs unconfirmed with a note; the counts are in the trace (`confirm_*` in the `end` totals, a `confirm` field on the `expand` events) and in one note; in the profile hash, and a handle made with it, at its `top_k` and places arm, does not resume without it, nor the reverse); `[run_profile.traverse]` with `map_freshness` (default `false`; see "Map freshness" in the CLI reference), `answer_labels` (default `false`; see "Answer labels" in the CLI reference) and `card_paths` (default `false`; see "Paths on cards" in the CLI reference), all in the profile hash; `[run_profile.freshness]` with `stat_gate` (default `false`; see "Stat gate" in the CLI reference; in the profile hash) and `verify_on_read` (default `false`; see "Verify on read" in the CLI reference; in the profile hash, and set in the file only, with no command-line flag, so the stamp is what ran); `[run_profile.scip]` with `mode` (`"cache"` default: a traversal reads the index when the file is there and readable, otherwise walks an empty SCIP graph and says so in a note; a file that cannot be parsed is parsed once per file stamp (size and modified time), and later calls say that it was not read again until the stamp changes or 30 seconds pass (a constant, not a key; an I/O error is never remembered); `"authoritative"`: the strict opt-in, a traversal needs the SCIP index and fails without it; `"off"`: it never touches the index, so `graph.scip` and the maps built from the index file (`structure.contains` beyond chunk lines, `scip.impact`, `scip.implements`, `tests.tested_by`, `config.key`, `exact.closure`) are empty; in the profile hash, and a handle made with a graph does not resume under `off`; recipes read the index under `cache` when it is there and refuse under `off` or without it, with a hint on how to write one) and `arbitration` (default `false`; `true`: with `lsp.live` in the call and `mode` not `"authoritative"`, a language server's answer at a node overrules the index's `scip.impact`, `scip.implements` (with `implementations = true`) and `structure.contains` edges to functions and methods that it did not name, only in a claim family it answered at that node; `graph.scip` is never arbitrated; the counts are in the trace, `totals.arbitration`, `per_map` and one note, see "Arbitration of SCIP edges" in the CLI reference; in the profile hash, and a handle made with it does not resume without it, nor the reverse); `[run_profile.lsp]` with `sessions` (`"shared"` default, or `"per_call"`) and `outline_cache` (`false` default, or `true`; see "Language-server sessions of `lsp.live`" in the CLI reference for both) and `implementations` (default `false`; `true` makes `lsp.live` also ask `textDocument/implementation` at each symbol node and offer `implements` edges, one more external call per node; see the same section), `call_hierarchy` (default `false`; `true` makes `lsp.live` also ask the call hierarchy at each symbol node and offer the callers and callees as `calls_or_refs` edges, up to three more external calls per node; see the same section), `sync` (`"watch"` default, or `"off"`; see "Language-server sync" in the CLI reference), `breaker` (default `true`; see the same section), `live` (default `true`: `lsp.live` is in the default map set of a service call; see the same section), `places` (`"auto"` default, `"scip"` or `"outline"`; see "Place table of `lsp.live`" in the CLI reference) and `roots` (`"workspace"` default, or `"manifest"`; see "Language-server roots" in the CLI reference) are in the profile hash as well; `[run_profile.seeds]` with `bm25`, `literal` and `repo_map` (default `false` each; see "Entity seeds", "Literal seeds" and "Repo-map seeds" in the CLI reference) and `[run_profile.grep]` with `structural` (default `false`; the opt-in map `grep.structural`, see its row in [engine.md](docs/spike/multimap/engine.md)) are in the profile hash too. These are all the keys: `[run_profile]` and every `[run_profile.*]` table reject an unknown key (`deny_unknown_fields`). `Config::run_profile()` hashes the resolved result-affecting settings (sha-256 of their canonical JSON, `config --json` `derived.run_profile_hash`): all settings except the machine-set keys, timeouts and timing, batch size, traverse file paths (`scip_index`, `priors`, `tie_break`), `rust_analyzer_trusted`, `[system_one]` and the API-key variable names; `config::profile::excluded_keys()` lists them | ### Channels diff --git a/config.example.toml b/config.example.toml index e7cf8331..8007d755 100644 --- a/config.example.toml +++ b/config.example.toml @@ -39,9 +39,8 @@ port = 8768 embedding_batch_size = 8 # Prompt formats for the embedding model. Models differ in what they expect (an # instruction prefix, a task label), so the text embedded for a query and for an -# indexed chunk are both templates. The defaults below are the formats LCI used -# before they were configurable (Qwen3-Embedding's query instruction, the bare -# code for documents). +# indexed chunk are both templates. The defaults below are Qwen3-Embedding's +# query instruction and the bare code for documents. # # embedding_query_template is the text embedded for a search query. {query} is # required. It shapes the query only, so changing it never re-embeds an index. @@ -75,12 +74,11 @@ reranker_timeout_seconds = 120 # the fusion order with one warning; the reranker is not the fallback. rescorer = "reranker" # Bounded timeout for /ready and service_status dependency probes, and for the -# readiness probe of the lsp.live traversal map (1-30 seconds). 15 by default (it -# was 5): a language-server binary can take several seconds to answer its probe +# readiness probe of the lsp.live traversal map (1-30 seconds). 15 by default: a +# language-server binary can take several seconds to answer its probe # (` --version`, `gopls version`, a start for pyright) while the machine -# is busy. A hanging dependency is now waited for up to 15 s, -# and one that answers in 5 to 15 s is reported ready; a refused port or a -# missing binary still fails at once. +# is busy. A hanging dependency is waited for up to this long, one that answers +# within it is reported ready; a refused port or a missing binary fails at once. readiness_timeout_seconds = 15 ripgrep_path = "rg" semantic_candidate_count = 40 diff --git a/docs/architecture.md b/docs/architecture.md index cb36cb82..76787746 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -2,9 +2,9 @@ `local-code-intelligence` (LCI) is a standalone local service: an MCP server and a plain REST mirror, plus a command line, over one process. It owns its indexes and keeps persistent data in `data_dir`, outside every indexed repository. -Clients (Kilo, Copilot, Codex, any MCP-capable agent) are not special: client-specific behaviour does not belong in +Clients (Kilo, Codex, any MCP-capable agent) are not special: client-specific behaviour does not belong in the service. Behaviour and surfaces are described in [README.md](../README.md); this file describes the structure. -Facts here were checked against the code of the `spike/v2` branch on 2026-10-17; names are symbols, not line numbers. +Facts here describe `main`; names are symbols and files, not line numbers. ## Crates @@ -15,7 +15,7 @@ The Cargo workspace has four members: the root package `local-code-intelligence` | `local-code-intelligence` (root) | `src/` | `app`, `store` (the only code that imports LanceDB and Arrow), `evaluate`, `server` (MCP), `rest`, `init` (`lci init`), `main` (CLI). Re-exports the `lci-core` modules from `src/lib.rs` so `crate::chunk`-style paths resolve. Cargo feature `map-equivalence` compiles test-only entry points (`App::fuse_through_maps`, `App::record_through_maps`) and the retrieval channels wrapped as maps (`src/app/search/channel_maps`). | `lci-core`, `lci-maps` | | `lci-core` | `lci-core/src/` | Pure logic: `bundle`, `chunk`, `config` (with `config::layers` and `config::profile`), `endpoint`, `evaluation` (portable evaluation definitions and validation), `filter`, `language`, `lexical`, `lsp`, `manifest`, `models`, `node` (stable `NodeId`), `path`, `provider` (completion providers for System One), `workspace`, plus the Tree-sitter grammars | none of LanceDB, Arrow, axum, rmcp | | `lci-maps` | `lci-maps/src/` | The maps, the registry and the traversal engine behind `search_code mode="traverse"`, `lci traverse` and the recipes: the `Map` trait, `Registry` (fail-open, fusion in today's arithmetic and order), the traversal engine (`traverse`, `walk`), the maps (SCIP graph and impact, config key, git history, live grep, LSP, search, symbol definitions, probe), the contract kit, the RunRecord writer | `lci-core` | -| `xtask` | `xtask/src/` | `cargo xtask` subcommands setup, build, test, acceptance, live-acceptance, env, lint-machine-paths, docs, gold, maps, mine-queries | `lci-core`, `lci-maps` | +| `xtask` | `xtask/src/` | the `cargo xtask` commands (listed in [README.md](../README.md), "Build and run") | `lci-core`, `lci-maps` | The split exists for build time (commit `0729d9a`): the LanceDB/Arrow tree takes minutes to compile, so a change that stays inside `lci-core` is checked in seconds with `cargo check -p lci-core` and `cargo test -p lci-core --lib ::` (see @@ -110,85 +110,7 @@ reranker are each a URL, a model name and optionally the NAME of an environment term; at most 12 terms score, rarest first (`MAX_SCORING_TERMS`). Capped at `lexical_candidate_count` (40). `rg` is not given `--hidden` or `.lciignore`, so it skips hidden files the scanner indexes (those drop out because only lines inside indexed chunks count), and it reads the live tree while chunks come from the snapshot, so line ranges can drift under `manual` or `watch` freshness. -- **LSP channel** (`lci-core` `lsp`): answers are merged query by query across providers and first-seen order decides the LSP rank (`lsp_channel` in `src/app/search.rs`). `LspAdapter` is a trait-object registry; `Manager` holds 8 adapters (rust-analyzer, - csharp-ls, typescript-language-server, pyright, gopls, jdtls, clangd, marksman), sessions keyed by `(provider, workspace_id)`, - started on a detached task shared by concurrent callers (its `initialize` is bounded by `lsp_start_timeout_seconds`, - default 0 = `lsp_timeout_seconds`, like every later request) and primed with one background `workspace/symbol`. Idle - sessions are released after `lsp_idle_timeout_seconds` (1800) and a session that answers only transient errors for - `lsp_stuck_seconds` (900) is replaced. Requests: `symbols` (`workspace/symbol`), `definition`, `references`, all - returning `Location` (path, range, name, kind, language, provider). Search use: up to 8 symbol queries (a term plus - `#`, plus a singular form for plurals), providers in parallel, each under `lsp_search_budget_ms` (1500); a provider - over budget adds a note, a provider that fails adds a warning; locations map onto chunks of the same language that - contain `start_line`; capped at `lsp_candidate_count` (40). Navigation tools (`symbols`, `definition`, `references`) - are not fused, return raw `Location`s and need an index containing the file's language. A server whose symbols are not code names (marksman: headings) opts out of the search channel (`LspAdapter::joins_search_channel`, default true) and is reached by navigation only. The `lsp.live` map of a - traversal (`lci-maps` `lsp_live`) asks the `Manager`s it is given: `ManagerBackend::open(workspace, Arc)` (one for every server) or `ManagerBackend::open_split(workspace, shared, own, probe)`, which probes through `shared` and routes each server to `shared` when `LspAdapter::watches_workspace()` (rust-analyzer alone: it runs its own file watcher) or `shared` syncs (`sync = "watch"`, the default since K11b-4), and in both cases only when the adapter lists its tree stamp's configuration files (`keeps_session`; not jdtls or marksman), else to `own`; a server routed to `shared` first passes `Manager::tree_barrier` (a walk passes two, see below), which stats the server's indexed files, their directories and ancestors and its manifests (`lsp::stamp`, from the index's file list that `App` supplies through `stamp::IndexedFiles` over the workspace manifest) and stops the session when the stamp changed, so the walk starts one that reads the disk. Navigation and the search channel pass the same barrier, once per call, and carry the same notes (`lsp:` for `lsp.live:`; `NavigationReport.notes`, `SearchReport.notes`); the search channel alone (`Manager::search_barrier`) keeps a running session of a server that watches its workspace through a changed stamp, records no stamp for it and labels the answer `kept session; (kept: it watches its workspace)`, so the next navigation call or walk restarts it (N108). `lsp.live` is in the default map set of a call that names no maps, has a tree, runs under `sessions = "shared"` and was given a manager (the service's; the CLI one-shot has none unless `--maps` or a policy names the map) while `[run_profile.lsp] live` is `true` (the default; `app::traverse::run_with_trait_seeds`). A walk that runs the map on the shared manager passes the strict tree barrier once, before its seed search (`walk_tree_barrier`): the seed search is the search channel and would otherwise keep a watching server through an edit that the backend's own barrier then restarts inside the walk's deadline, so the server is restarted once, up front, and the seed search and the map find the same session. The backend then passes its own strict barrier after the seed search, as before (a continuation skips the pre-seed one and has only the backend's), so a walk has one restart: the pre-seed barrier restarts a stale server, and the backend's restarts it again only if the stamp changed between the two (a refresh that adds a file in between). Under - `[run_profile.lsp] sessions = "per_call"` `app::traverse::run` builds one per call, so the map owns it and its - server dies with the map; under `"shared"` (default) it uses `Call.lsp_sessions` as `shared` and a per-call manager as `own`, so a server that lists no configuration files for the tree stamp (jdtls, marksman), or that neither sees an edit on its own nor is told by the sync (`sync = "off"`; the default is `"watch"`), starts per call, and a reused server is never staler than `per_call` for any change inside the tree stamp's coverage (its residuals are in the README; the walk labels a reused server's answer with a `kept session` note); `Call.lsp_sessions` is the `Arc` that `App::language_servers()` - hands out (the service's own, so a server started by navigation, search or a kept traverse server serves the others; the CLI - `traverse` builds one per process and shuts it down after the walk), and a kept server's lifetime is the manager's (idle - sweeper, `release_workspace`, shutdown). `Manager::probe_once` caches a successful readiness probe per provider for - the manager's life (the probe is per adapter, `adapter::probe_subprocess`: ` --version` that exits 0 by default, `gopls version`, - and for pyright a start whose exit status is ignored, since `pyright-langserver` has no version flag and exits 1; so a dead pyright - shows first at the session start, not at the probe). Under `[run_profile.lsp] breaker = true` (the default) a manager also holds `lsp::LspBreakers`, one 30 s - window per provider (`LSP_DOWN_WINDOW`, the `ConnectBreaker` of the model clients): a failed probe in `probe_once` - (`adapter::ProbeFailure::Failed`, or an adapter's own readiness error; a missing binary or a probe that timed out opens - no window, since it is no evidence of a dead server: the miss costs under a millisecond and its note is the same on every - walk, and a slow probe may be a loaded machine), - or a start in `Session::begin` that failed for a reason other than the transport's inactivity timeout - (`transport::InactivityTimeout`: a slow `initialize` is not a dead server), opens it, and `probe_once` and - `Manager::session` (a new start only; a usable session is returned whatever the window says) then refuse the - provider with a `lsp breaker` error until it has passed or a probe or start succeeds. `per_call_manager` gives - a call's own manager the windows of `Call.lsp_sessions` (under both `sessions` values), so a dead server found by - one call is skipped by the next; `/ready` probes the adapters directly and never reads them. `Status.servers` and - `Status.languages_without_server` (`index_status`, `lci status`) are the per-language roster: per adapter its - languages, the ones the index holds, `configured`/`enabled`/`command`, `Manager::probe_state` (the `probe_once` memo - read as `probed`/`not_probed`; it runs no probe), `Manager::breaker_remaining` (an open skip window as `breaker_open_ms`, `null` when none) and the session's `ServerReadiness`, and the indexed languages no - enabled, configured adapter lists; built from the manager's memory, it starts no server. The roster's `root` and `root_reason` are the directory a server starts at: `[run_profile.lsp] roots` (`Manager::new`; `"workspace"` default) makes `Session::begin` choose it, in its start task, through `lsp::roots` (`Manager::server_root`; under `"manifest"`, `roots::for_server`: the topmost manifest the adapter names in the directories of the index's files of its languages and their ancestors, one directory read each, the workspace for none, for sibling manifests or when files would fall outside), pass it to `GenericClient::spawn` and the adapter's `transport_config`/`initialize_params`, and keep it for `Manager::session_root`; one session per server per workspace, the session key, the tree stamp and the barrier are unchanged. The recipes walk no `lsp.live` and pass no manager. What `"manifest"` was measured to gain (a premise check on small fixtures): pyright resolves a name that depends on a nested project's own settings (`extraPaths`) only from the project directory; typescript-language-server already finds the nearest `tsconfig.json` from the workspace root and gains nothing; one indexed file of the server's languages outside the project directory returns the root to the workspace. `[run_profile.lsp] outline_cache` (read by - `Manager::new` from the config it is built with: the App's, the CLI's, or `Call.lsp` for a per-call manager) gives the - manager an outline cache, used only for adapters whose `LspAdapter::opens_documents()` is true (pyright, gopls, - jdtls, clangd, typescript-language-server, marksman: the hook sends a changed disk text, as a `didChange` with the whole - text when the session read change sync (`ChangeSync`, from `textDocumentSync` in the `initialize` result) from the server and as a second `didOpen` otherwise, and the server takes it; the wait for the `publishDiagnostics` of the send is kept in `SentTexts` until a hook has finished it, so a cancelled hook leaves it to the next caller of the file; `SentTexts` holds at most `MAX_OPEN_DOCUMENTS` = 64 files per session and closes the least recently used one that has no open in flight before a new file is opened): `Manager::document_symbols` keys each - non-empty answer by (provider, workspace id, path), - `lsp::content_hash` of the file (`workspace::hash`, as the manifest) and the id of the server session that gave it, - at most `MAX_CACHED_OUTLINES` (4096) entries. With `[run_profile.lsp] implementations = true` - (default `false`) the map also asks `Manager::implementation` (`textDocument/implementation`) at each symbol node - and offers `Implements` edges from an interface or trait member to its implementing members (the direction of - `scip.implements`); it is the third request of a node, behind the same `max_external_calls` gate, and a failure of - it is recorded without dropping the edges of the first two. With `[run_profile.lsp] call_hierarchy = true` - (default `false`) it asks up to three more requests per symbol node, each behind the same gate and counted: - `Manager::prepare_call_hierarchy` (`textDocument/prepareCallHierarchy`), then `Manager::incoming_calls` and - `Manager::outgoing_calls` with the item the server gave, sent back verbatim (`HierarchyItem::raw`; the place of an - item is its `selectionRange`, the name). The callers and callees are `CallsOrRefs` edges (witness `Exact`, no new - edge kind), one per destination per expansion, seen at the call (`fromRanges`). Every error reply is recorded - and keeps the other edges, which is also how a server without `callHierarchyProvider` (it answers -32601) is found: - the capabilities are not read. - The symbol-to-place table behind the map (`lsp_live::SymbolPlaces`) is `ScipPlaces`, built from the SCIP graph's - nodes, under `[run_profile.lsp] places = "scip"`. With `places = "outline"` it is `OutlinePlaces` over it; `places = "auto"` (default) picks per walk: `ScipPlaces` when the SCIP graph read for the workspace has a node, else `OutlinePlaces` (`app::traverse::run`), so a tree with no index read (`scip.mode = "off"`, or `"cache"` with no file) still gets `lsp.live` cards. `OutlinePlaces`: - `ask` outlines a symbol's file once per map instance (`Manager::document_symbols`, one counted external call; - `SymbolPlaces::outline_wanted`), as it already does for a file or chunk node, and `SymbolPlaces::learn` takes the - outline in. A member whose kind this build has a node for (`SymbolKind`: functions, methods, types, enums, - modules, constants, classes, interfaces, fields, variables; LSP kinds 12, 6, 9, 5, 23, 11, 10, 2-4, 14, 7 and 8, 13; 5 is `Class`, 11 is `Trait` in a Rust file and `Interface` elsewhere; enum members 22, impl blocks 19 and type parameters 26 have no node) takes the SCIP node's id for the same file, name and - nearest line, else (a Rust file, `OutlinePlaces::with_tree`) the canonical id of the `chunk::extract_for_path` declaration around the member, else a minted `SymbolKey` with the file's `language::for_path` identifier and the equal names - numbered in document order; where a file is learned its places win over the index's, which may be older than the - tree. `flatten_document_symbols` gives a nested `DocumentSymbol` the range from its name to its item's end, so - `Member::end_line` is the last line of the item. - it is recorded without dropping the edges of the first two. - Readiness is reported, not acted on: the transport keeps the `is_ready` latch (the adapter's `ready_predicate` - matched once, never cleared) and, beside it, a `watch` of `ServerState` (`busy`, `changes`, `changed_at`) fed by the - adapter's `busy_predicate` (default: never; rust-analyzer `quiescent: false`, jdtls `Starting`). `Manager::readiness` - turns a session into a `ServerReadiness` (`starting`, `ready`, `busy`, `failed` or `stuck`, primed, counters, ages) - without touching `last_used`; `Status.analyzers`, the search over-budget note and the `lsp.live` notes of a traverse - call read it. No request, repeat or answer depends on it. - `[run_profile.lsp] sync = "watch"` (the default since K11b-4; read by `Manager::new`; a per-call - manager is built `without_sync`) gives each workspace with a session an `lsp::sync::WorkspaceSync`: a `notify` - watcher of the root (polling at `watch_poll_milliseconds` when the platform watcher fails) whose events a task - drains into batches, typed by the disk (created, changed, deleted) against a `(mtime, length)` table from one - `chunk::source_files` walk (walked again only after an overflow). The manager's sink drops the files' kept outlines - and sends each batch to the workspace's sessions as `workspace/didChangeWatchedFiles`; `Manager::client` drains - before every request. The sync starts with the workspace's first session and stops with its last (`release`, - `shutdown_all`, the idle sweeper). Lock order: drain lock, then the session registry, then `syncs`. - `Status.sync` (N118) reports `Manager::sync_status` for the workspace (backend, `failed` when neither watcher started or the opening task died, - the counters) only while a sync runs, so a failed or silent watcher is visible; it is a report and changes no answer. +- **LSP channel** (`lci-core` `lsp`): language-server symbols mapped onto chunks, merged query by query across providers; sessions, breakers, readiness, sync and the `lsp.live` map's use of them are in [lsp-channel.md](lsp-channel.md). - **Model clients** (`lci-core` `models`): `embed` POSTs to `/embeddings` and validates count, index, finiteness, non-zero vectors and one dimension; indexing retries up to 3 attempts (`MAX_ATTEMPTS`) with doubling backoff, `query` makes one attempt. Any refused connection trips a breaker for 30 s (`EMBEDDER_DOWN_WINDOW`): `query` and `embed_once` then skip at once @@ -196,8 +118,8 @@ reranker are each a URL, a model name and optionally the NAME of an environment backoff (a failed probe re-arms the window, an answer closes the breaker). An HTTP 5xx does not trip it. `rerank` makes one attempt and returns `(index, score)` best first; after a refused connection a breaker skips the reranker for 30 s (`RERANKER_DOWN_WINDOW`). The language servers have the same - window under `[run_profile.lsp] breaker = true` (`lsp::LspBreakers`, on by default since K11b-3; see the language-server item - above). Batches the service rejects as + window under `[run_profile.lsp] breaker = true` (`lsp::LspBreakers`, on by default since K11b-3; see [lsp-channel.md](lsp-channel.md)). + Batches the service rejects as oversized are retried one input at a time with prefixes halved down to 512 bytes. ## Indexing flow and freshness @@ -257,51 +179,7 @@ view (`compact_view`, a profile with `response = "compact"`) is applied to a fin ## Data flow: traverse -`search_code` with `mode = "traverse"` (needs `[traverse] enabled`) and `lci traverse` run one function, -`app::traverse::run`. It loads the SCIP index (map `graph.scip`, cached by file stamp; `[run_profile.scip] mode` `cache` (the default) reads it when it is there and parses an unreadable file once per file stamp, `authoritative` requires it, `off` never touches it and leaves the graph and the maps built from the index file empty, each saying so in a note), finds the start nodes -(`given`; `hybrid`: the top `seed_k` hits of the normal hybrid search mapped to SCIP symbols; `hybrid+bm25` or `bm25` with `[run_profile.seeds] bm25 = true`: the opt-in map `seeds.bm25`, -BM25 over the SCIP graph's node names, qualified paths and documentation, whose top `seed_k` nodes go after the hybrid hits or stand alone -when the search gave none; `...+literal` or `literal` with `[run_profile.seeds] literal = true`: the opt-in map `seeds.literal`, a whole-word ripgrep search (`lexical::search_literals`) for the names the question puts in code spans, each line placed in the SCIP graph (the symbol defined there, the one around it, else the file of a declaration line), appended after those sources; `...+repo_map` or `repo_map` with `[run_profile.seeds] repo_map = true`: the opt-in map `seeds.repo_map`, a personalized PageRank over the SCIP graph's call and containment edges from the entities `seeds.bm25` finds for the question, whose new top nodes go after the earlier sources' (each source form also exists with `+literal` after it); or `name_tokens`, the visible fallback), builds the steering ladder from the caller's rules, runs the `lci-maps` traversal under the -`[traverse]` caps (a request can only lower them), writes a trace under `data_dir`, and answers with the reached nodes, -totals and, only when work is left, a `handle` and `checkpoint`. The first stage (exact operators `exact.closure` and -`exact.pickaxe`, then seeds), named policies per question type (the router, chosen by the caller's `steer.policy`) and the steering ladder are described in -[engine.md](spike/multimap/engine.md). Default off: with `[traverse]`, `[recipes]` and `[compose]` disabled the tool -list, schemas and REST routes are exactly those of a server without them. - -With `[run_profile.traverse] answer_labels = true` (default `false`) the answer also has `completeness` (`{complete, why, estimate}`: `complete` is `status == "complete"` with no map lost (`degraded`: a map the call asked for that could not be set up, or had `MapStat.failures`; `why` is then `degraded` unless the walk was cut), `why` the `stop_reason`, `estimate` `TraversalResult.estimate`, the `likely_complete` fraction) and every card of `results` and the checkpoint lists has `freshness` (`fresh`, `changed`, `stale` or `unverified`, read from `file_changed` and `maps`; a card that cites `lsp.live` is `unverified`, not `fresh`, when the walk found a language server running, `add_answer_labels`' `kept` argument from the walk's `WalkSessions`; a continuation also demotes the `lsp.live` cards of the nodes it restores from the checkpoint, its `restored` argument). `app::traverse::response::add_answer_labels` adds both to the response JSON after the trace and the checkpoint hash are made, so the trace, the checkpoint, the handle and `Card`/`card_json` are unchanged, nothing is withheld, and with the key off the answer is as before. - -With `[run_profile.traverse] card_paths = true` (default `false`) every walked card of `results` and of `checkpoint.answer` also has `kind` and `path`, the -route from its start node with each hop's map, edge kind and witness (`app::traverse::response::add_paths`, read from the walk's state table; the engine keeps -`State::kind`, and a handle keeps it only under the key). With the key off the answer, the trace, the checkpoint and the handle are unchanged. - -With `[run_profile.traverse] map_freshness = true` (default `false`) the answer also has `maps`: for each map of the call that -states a freshness, `{id, observed, stale, lag}` (`app::traverse::freshness`). The SCIP-built maps report `snapshot`; -`grep.live`, `lsp.live`, `git.history`, `exact.pickaxe` and `probe.compile` (which runs only under `rust_analyzer_trusted = true`) report `live` with `lag` 0 (`live` names the channel: `grep.live`, -`git.history`, `exact.pickaxe` and `probe.compile` read current bytes or git, placed by the index's positions for `grep.live`; -`lsp.live` asks a running server, which is fresh when this call started it and, when kept across calls under -`sessions = "shared"`, bounded by its tree stamp's coverage and labelled by a `kept session` note, with the same fields). `lag` is the number of source files (language-adapter -files, prose left out) newer than the index file plus those changed (content hash), added or removed since the process -first saw the index's current stamp (modified time and length), each counted once; `stale` means `lag` above 0 and `null` -means not compared (no tree, unreadable tree, or an index file without a modified time). The tree is read and hashed -once per call with `chunk::source_hashes` (the index scan's walk and its read, skip and hash function, no chunk made); -`[run_profile.freshness] stat_gate = true` (default `false`, in the profile hash) puts a process-wide `chunk::StatGate` per -root in front of the read, which keeps `(modified time, length, hash)` per file and reads only files whose stamp is new or -changed. A file modified within 3 seconds of the read is not trusted (racy stamp; FAT, exFAT and SMB stamps are coarse) and is read again next call. Its gap, asserted by a test: a same-length edit that keeps a modified time older than 3 seconds before the previous read is not seen until the next edit. -Search's `verify_on_read` reads through the same function. `Expanded` carries an optional -`edge::Freshness` that a wrapper around each stating map records; the walk itself does not read it. Key off: no -wrapper, no tree read, no `maps` key. - -`[run_profile.freshness] verify_on_read = true` (default `false`, in the profile hash) adds, on top of `map_freshness`, a check of -what the answer cites against the tree by content hash. It reuses the one read of the tree (`freshness::tree_view` returns the -lag with the file sets it came from; no second scan), reads and labels, and never refreshes the index or touches -`[index].freshness`. `SeedSearch::snapshot_hashes` (`AppSeeds` reads the manifest of the index snapshot) is read after the walk, -because the walk's first search may refresh the snapshot; the snapshot lag is the source files (the `tree_lag` set; manifest prose -entries are left out) that differ between the manifest and the tree. `seeds.hybrid`, `search.lexical` and `search.vector` are -then labelled with it (`Observer` does not wrap them; the label is filled at report time), `grep.live` and `lsp.live` say -`observed: live` with `places: "index"` and the SCIP lag (their places are the index's), and `response::add_file_changed` gives -every card `file_changed` from the baseline of its maps (the SCIP baseline for the maps built from the index and the placed -maps and the symbol cards, the manifest for the snapshot maps and the chunk cards; `null` where none applies). Key off: none of this, and no change to any answer. -`search::App::verify_on_read` does the same for a search report the caller asked for: `SearchReport.freshness` (`snapshot`, `channels` for the channels that ran, `hits` with `file_changed` per result) from the manifest of the snapshot the search ran on and one read of the tree's content hashes (`chunk::source_hashes`, gated by `stat_gate` when on), read after the search (a refresh in between is not seen) and timed in `timings.verify_ms`. The searches inside the crate that discard the block (`AppSeeds::search`, the evaluation runner) call the `_unverified` entry points, so a traversal reads the tree once. +The call flow of `search_code mode="traverse"` and `lci traverse` (`app::traverse::run`: SCIP index load, seeds, the `lci-maps` walk under the `[traverse]` caps, trace, answer) and the answer keys of `answer_labels`, `card_paths`, `map_freshness`, `verify_on_read` are in [traverse-flow.md](traverse-flow.md). ## MCP, REST and CLI surfaces @@ -315,7 +193,7 @@ maps and the symbol cards, the manifest for the snapshot maps and the chunk card `service_status`), 5 with `[recipes] enabled` (`config_readers`, `impact_rings`, `flow_path`, `flow_from`, `config_neighbours`) and `lci_compose` with `[compose] enabled`. `search_code` takes `workspace_path`, `query`, `top_k`, `languages`, `include_paths`, `exclude_paths`, `source_roles` and, with `[traverse] enabled`, `mode`, - `budget`, `steer`, `handle`. An argument a tool does not have is refused with a tool error naming it (REST JSON bodies: HTTP 422); `service_status` uses a strict empty argument object. The REST search query string uses a separate strict `SearchQuery` (HTTP 400 for unknown keys). Parameter semantics: README. + `budget`, `steer`, `handle`. An argument a tool does not have is refused with a tool error naming it (REST JSON bodies: HTTP 422); `service_status` uses a strict empty argument object. The REST search query string uses a separate strict `SearchQuery` (HTTP 400 for unknown keys). Parameter semantics: [mcp-and-rest.md](reference/mcp-and-rest.md). - **REST** (`src/rest.rs`) exposes each tool as `POST /v1/` (`GET /v1/service_status`), `search_code` takes `?search_profile=`, plus `POST /v1/release_workspace` (no MCP tool: releases the workspace's language servers), `GET /openapi.json` (hand-assembled), `/health`, `/ready`. REST, `/openapi.json`, `/health` and `/ready` check Host and diff --git a/docs/development/agent-payload-notes.md b/docs/development/agent-payload-notes.md index b5b906f8..d4622d73 100644 --- a/docs/development/agent-payload-notes.md +++ b/docs/development/agent-payload-notes.md @@ -1,7 +1,6 @@ # Agent payload and tool-text notes -Status: dated record of the U1 study; the figures are not re-measured. N28 to N31 landed in `0aa374c` on 2026-10-14 -(`spike/v2`); N32 to N34 are open in [roadmap.md](roadmap.md). The per-tool figures below cover the 10 tools the +Status: dated record of the U1 study; the figures are not re-measured. N28 to N31 landed in `0aa374c` on 2026-10-14; N32 to N34 are open in [roadmap.md](roadmap.md). The per-tool figures below cover the 10 tools the study measured; a server with `[recipes]` and `[compose]` enabled lists 16 (see [architecture.md](../architecture.md#mcp-rest-and-cli-surfaces)). Background for roadmap items N28 to N34: how large the MCP tool list and the `search_code` result are for an agent, what was diff --git a/docs/development/documentation.md b/docs/development/documentation.md index 267768b8..ca156d93 100644 --- a/docs/development/documentation.md +++ b/docs/development/documentation.md @@ -7,7 +7,8 @@ Agents read these files without chat context and pay for every token, so the fil 2. One place per fact. A fact has one home; every other file points to that home instead of restating it. Homes: setup, configuration and server behaviour in `README.md`; MCP and REST surfaces in `docs/reference/mcp-and-rest.md`; the CLI in `docs/reference/cli.md`; retrieval behaviour and evaluation in `docs/reference/retrieval.md`; module structure in - `docs/architecture.md`; module boundaries in `docs/development/module-boundaries.md`; build and test in + `docs/architecture.md`; the code-level picture of language-server sessions, breakers, readiness and sync in `docs/lsp-channel.md`; the code + flow of one traverse call in `docs/traverse-flow.md` (answer-key behaviour and defaults stay in `docs/reference/cli.md`); module boundaries in `docs/development/module-boundaries.md`; build and test in `docs/development/testing.md`; language scope in `docs/languages.md`; machine values in `docs/development/machine-values.md`; traversal engine in `docs/spike/multimap/engine.md`; measurement method in `docs/spike/multimap/evaluation.md`; plans in `docs/development/roadmap.md`; decisions, findings and history in `docs/decisions.md`. diff --git a/docs/development/local-live-acceptance.md b/docs/development/local-live-acceptance.md index c51d7160..114390e5 100644 --- a/docs/development/local-live-acceptance.md +++ b/docs/development/local-live-acceptance.md @@ -16,7 +16,7 @@ from GitHub's UI) when a runner is up (see [live-acceptance-ci.md](live-acceptan ## Steps and timeouts Each step has its own timeout, so a stuck fast step fails in minutes instead of waiting out one job-wide bound -(`live-acceptance.yml` has a single `timeout-minutes: 150`). The run has 27 steps in this order: +(`live-acceptance.yml` has a single `timeout-minutes: 150`). The run has 29 steps in this order (28 with `--skip-build`): | Step | Timeout | | --- | --- | @@ -24,7 +24,7 @@ Each step has its own timeout, so a stuck fast step fails in minutes instead of | Acceptance `core` (index, search, reindex smoke test; cold-embeds this repository) | 20 min | | Acceptance `multilingual` (fixture indexing, search, decoy ranking) | 15 min | | Acceptance `lsp` (rust-analyzer) | 30 min | -| 21 small-fixture LSP steps: `-lsp`, `-missing`, `-recovery` for python, typescript, go, java, csharp, c, cpp | 10 min each | +| 23 small-fixture LSP steps: `-lsp`, `-missing`, `-recovery` for python, typescript, go, java, markdown (`-lsp` and `-missing` only), csharp, c, cpp | 10 min each | | `lci config --json` (records the configuration layers the evaluation will use) | 1 min | | `evaluate ./evaluations/core.toml --workspace self= --output test-results/evaluation.json` | 15 min | diff --git a/docs/development/roadmap.md b/docs/development/roadmap.md index 29c4f63b..217b7841 100644 --- a/docs/development/roadmap.md +++ b/docs/development/roadmap.md @@ -48,7 +48,7 @@ that document and is not a roadmap id. ## Now and next -Cycle 1 implementation is complete (finish line d422): the last feature merges were K39b (d564) and K12a-2 (d533); M2 closed 2026-10-23 without K11b-6, whose release-build measurement failed, so per-language `places = "auto"` moves to cycle 2 with mixed-tree surfacing (d562). On main: last PR 225 (6d6a9bb) before this cleanup PR, which appends d565 to d572. A cleanup of docs and code is running. Next: the cycle 1 report went to the owner on 2026-10-23; the owner's cut (asked 2026-10-23) and the answers to the report's questions follow, then cycle 2. Waiting on the cut: sanity 20 (d563) recommends K12b-2 and K31 move to cycle 2 and K12b-1 stay parked on its branch (nothing merges before the tsgo facts check and the lsp-cost pair, d537), and the placement of the K20c rust and K26 value windows (each needs a release quiet window with no cargo; the arbitration stays off until the K20c measurement passes), which sanity 20 proposes for cycle 2. K7b-3b and K11d-2b stay open (their rows), and ASYNC-2 C stays parked. Per the autonomy rule the coordinator waits for the owner's word after the report. Only docs PRs and the coordinator rewrite this section; a feature PR leaves it alone so parallel PRs do not conflict (d476). The live state is the coordinator's run checkpoint. +Cycle 1 implementation is complete (finish line d422): the last feature merges were K39b (d564) and K12a-2 (d533); M2 closed 2026-10-23 without K11b-6, whose release-build measurement failed, so per-language `places = "auto"` moves to cycle 2 with mixed-tree surfacing (d562). On main: the cleanup PRs land on top of PR 225 (2026-10-23); the closing cleanup PR refreshes this line. A cleanup of docs and code is running. Next: the cycle 1 report went to the owner on 2026-10-23; the owner's cut (asked 2026-10-23) and the answers to the report's questions follow, then cycle 2. Waiting on the cut: sanity 20 (d563) recommends K12b-2 and K31 move to cycle 2 and K12b-1 stay parked on its branch (nothing merges before the tsgo facts check and the lsp-cost pair, d537), and the placement of the K20c rust and K26 value windows (each needs a release quiet window with no cargo; the arbitration stays off until the K20c measurement passes), which sanity 20 proposes for cycle 2. K7b-3b and K11d-2b stay open (their rows), and ASYNC-2 C stays parked. Per the autonomy rule the coordinator waits for the owner's word after the report. Only docs PRs and the coordinator rewrite this section; a feature PR leaves it alone so parallel PRs do not conflict (d476). The live state is the coordinator's run checkpoint. ## Cycle 1 diff --git a/docs/lsp-channel.md b/docs/lsp-channel.md new file mode 100644 index 00000000..debb2a42 --- /dev/null +++ b/docs/lsp-channel.md @@ -0,0 +1,82 @@ +# Language-server channel + +Code-level picture of how the language servers run inside LCI: the `lci-core` `lsp` module (`Manager`, sessions, breakers, readiness, sync, outline cache) and how the search channel, the navigation tools and the `lsp.live` traversal map use it. Crate layout and the other channels: [architecture.md](architecture.md); settings and defaults: [README.md](../README.md); behaviour and defaults of the `lsp.live` switches: [CLI reference](reference/cli.md); `lsp.live` as a map: [engine.md](spike/multimap/engine.md). + +- **LSP channel** (`lci-core` `lsp`): answers are merged query by query across providers and first-seen order decides the LSP rank (`lsp_channel` in `src/app/search.rs`). `LspAdapter` is a trait-object registry; `Manager` holds 8 adapters (rust-analyzer, + csharp-ls, typescript-language-server, pyright, gopls, jdtls, clangd, marksman), sessions keyed by `(provider, workspace_id)`, + started on a detached task shared by concurrent callers (its `initialize` is bounded by `lsp_start_timeout_seconds`, + default 0 = `lsp_timeout_seconds`, like every later request) and primed with one background `workspace/symbol`. Idle + sessions are released after `lsp_idle_timeout_seconds` (1800) and a session that answers only transient errors for + `lsp_stuck_seconds` (900) is replaced. Requests: `symbols` (`workspace/symbol`), `definition`, `references`, all + returning `Location` (path, range, name, kind, language, provider). Search use: up to 8 symbol queries (a term plus + `#`, plus a singular form for plurals), providers in parallel, each under `lsp_search_budget_ms` (1500); a provider + over budget adds a note, a provider that fails adds a warning; locations map onto chunks of the same language that + contain `start_line`; capped at `lsp_candidate_count` (40). Navigation tools (`symbols`, `definition`, `references`) + are not fused, return raw `Location`s and need an index containing the file's language. A server whose symbols are not code names (marksman: headings) opts out of the search channel (`LspAdapter::joins_search_channel`, default true) and is reached by navigation only. The `lsp.live` map of a + traversal (`lci-maps` `lsp_live`) asks the `Manager`s it is given: `ManagerBackend::open(workspace, Arc)` (one for every server) or `ManagerBackend::open_split(workspace, shared, own, probe)`, which probes through `shared` and routes each server to `shared` when `LspAdapter::watches_workspace()` (rust-analyzer alone: it runs its own file watcher) or `shared` syncs (`sync = "watch"`, the default since K11b-4), and in both cases only when the adapter lists its tree stamp's configuration files (`keeps_session`; not jdtls or marksman), else to `own`; a server routed to `shared` first passes `Manager::tree_barrier` (a walk passes two, see below), which stats the server's indexed files, their directories and ancestors and its manifests (`lsp::stamp`, from the index's file list that `App` supplies through `stamp::IndexedFiles` over the workspace manifest) and stops the session when the stamp changed, so the walk starts one that reads the disk. Navigation and the search channel pass the same barrier, once per call, and carry the same notes (`lsp:` for `lsp.live:`; `NavigationReport.notes`, `SearchReport.notes`); the search channel alone (`Manager::search_barrier`) keeps a running session of a server that watches its workspace through a changed stamp, records no stamp for it and labels the answer `kept session; (kept: it watches its workspace)`, so the next navigation call or walk restarts it (N108). `lsp.live` is in the default map set of a call that names no maps, has a tree, runs under `sessions = "shared"` and was given a manager (the service's; the CLI one-shot has none unless `--maps` or a policy names the map) while `[run_profile.lsp] live` is `true` (the default; `app::traverse::run_with_trait_seeds`). A walk that runs the map on the shared manager passes the strict tree barrier once, before its seed search (`walk_tree_barrier`): the seed search is the search channel and would otherwise keep a watching server through an edit that the backend's own barrier then restarts inside the walk's deadline, so the server is restarted once, up front, and the seed search and the map find the same session. The backend then passes its own strict barrier after the seed search, as before (a continuation skips the pre-seed one and has only the backend's), so a walk has one restart: the pre-seed barrier restarts a stale server, and the backend's restarts it again only if the stamp changed between the two (a refresh that adds a file in between). Under + `[run_profile.lsp] sessions = "per_call"` `app::traverse::run` builds one per call, so the map owns it and its + server dies with the map; under `"shared"` (default) it uses `Call.lsp_sessions` as `shared` and a per-call manager as `own`, so a server that lists no configuration files for the tree stamp (jdtls, marksman), or that neither sees an edit on its own nor is told by the sync (`sync = "off"`; the default is `"watch"`), starts per call, and a reused server is never staler than `per_call` for any change inside the tree stamp's coverage (its residuals are in the [CLI reference](reference/cli.md), "Language-server sessions of `lsp.live`"; the walk labels a reused server's answer with a `kept session` note); `Call.lsp_sessions` is the `Arc` that `App::language_servers()` + hands out (the service's own, so a server started by navigation, search or a kept traverse server serves the others; the CLI + `traverse` builds one per process and shuts it down after the walk), and a kept server's lifetime is the manager's (idle + sweeper, `release_workspace`, shutdown). `Manager::probe_once` caches a successful readiness probe per provider for + the manager's life (the probe is per adapter, `adapter::probe_subprocess`: ` --version` that exits 0 by default, `gopls version`, + and for pyright a start whose exit status is ignored, since `pyright-langserver` has no version flag and exits 1; so a dead pyright + shows first at the session start, not at the probe). Under `[run_profile.lsp] breaker = true` (the default) a manager also holds `lsp::LspBreakers`, one 30 s + window per provider (`LSP_DOWN_WINDOW`, the `ConnectBreaker` of the model clients): a failed probe in `probe_once` + (`adapter::ProbeFailure::Failed`, or an adapter's own readiness error; a missing binary or a probe that timed out opens + no window, since it is no evidence of a dead server: the miss costs under a millisecond and its note is the same on every + walk, and a slow probe may be a loaded machine), + or a start in `Session::begin` that failed for a reason other than the transport's inactivity timeout + (`transport::InactivityTimeout`: a slow `initialize` is not a dead server), opens it, and `probe_once` and + `Manager::session` (a new start only; a usable session is returned whatever the window says) then refuse the + provider with a `lsp breaker` error until it has passed or a probe or start succeeds. `per_call_manager` gives + a call's own manager the windows of `Call.lsp_sessions` (under both `sessions` values), so a dead server found by + one call is skipped by the next; `/ready` probes the adapters directly and never reads them. `Status.servers` and + `Status.languages_without_server` (`index_status`, `lci status`) are the per-language roster: per adapter its + languages, the ones the index holds, `configured`/`enabled`/`command`, `Manager::probe_state` (the `probe_once` memo + read as `probed`/`not_probed`; it runs no probe), `Manager::breaker_remaining` (an open skip window as `breaker_open_ms`, `null` when none) and the session's `ServerReadiness`, and the indexed languages no + enabled, configured adapter lists; built from the manager's memory, it starts no server. The roster's `root` and `root_reason` are the directory a server starts at: `[run_profile.lsp] roots` (`Manager::new`; `"workspace"` default) makes `Session::begin` choose it, in its start task, through `lsp::roots` (`Manager::server_root`; under `"manifest"`, `roots::for_server`: the topmost manifest the adapter names in the directories of the index's files of its languages and their ancestors, one directory read each, the workspace for none, for sibling manifests or when files would fall outside), pass it to `GenericClient::spawn` and the adapter's `transport_config`/`initialize_params`, and keep it for `Manager::session_root`; one session per server per workspace, the session key, the tree stamp and the barrier are unchanged. The recipes walk no `lsp.live` and pass no manager. What `"manifest"` was measured to gain (a premise check on small fixtures): pyright resolves a name that depends on a nested project's own settings (`extraPaths`) only from the project directory; typescript-language-server already finds the nearest `tsconfig.json` from the workspace root and gains nothing; one indexed file of the server's languages outside the project directory returns the root to the workspace. `[run_profile.lsp] outline_cache` (read by + `Manager::new` from the config it is built with: the App's, the CLI's, or `Call.lsp` for a per-call manager) gives the + manager an outline cache, used only for adapters whose `LspAdapter::opens_documents()` is true (pyright, gopls, + jdtls, clangd, typescript-language-server, marksman: the hook sends a changed disk text, as a `didChange` with the whole + text when the session read change sync (`ChangeSync`, from `textDocumentSync` in the `initialize` result) from the server and as a second `didOpen` otherwise, and the server takes it; the wait for the `publishDiagnostics` of the send is kept in `SentTexts` until a hook has finished it, so a cancelled hook leaves it to the next caller of the file; `SentTexts` holds at most `MAX_OPEN_DOCUMENTS` = 64 files per session and closes the least recently used one that has no open in flight before a new file is opened): `Manager::document_symbols` keys each + non-empty answer by (provider, workspace id, path), + `lsp::content_hash` of the file (`workspace::hash`, as the manifest) and the id of the server session that gave it, + at most `MAX_CACHED_OUTLINES` (4096) entries. With `[run_profile.lsp] implementations = true` + (default `false`) the map also asks `Manager::implementation` (`textDocument/implementation`) at each symbol node + and offers `Implements` edges from an interface or trait member to its implementing members (the direction of + `scip.implements`); it is the third request of a node, behind the same `max_external_calls` gate, and a failure of + it is recorded without dropping the edges of the first two. With `[run_profile.lsp] call_hierarchy = true` + (default `false`) it asks up to three more requests per symbol node, each behind the same gate and counted: + `Manager::prepare_call_hierarchy` (`textDocument/prepareCallHierarchy`), then `Manager::incoming_calls` and + `Manager::outgoing_calls` with the item the server gave, sent back verbatim (`HierarchyItem::raw`; the place of an + item is its `selectionRange`, the name). The callers and callees are `CallsOrRefs` edges (witness `Exact`, no new + edge kind), one per destination per expansion, seen at the call (`fromRanges`). Every error reply is recorded + and keeps the other edges, which is also how a server without `callHierarchyProvider` (it answers -32601) is found: + the capabilities are not read. + The symbol-to-place table behind the map (`lsp_live::SymbolPlaces`) is `ScipPlaces`, built from the SCIP graph's + nodes, under `[run_profile.lsp] places = "scip"`. With `places = "outline"` it is `OutlinePlaces` over it; `places = "auto"` (default) picks per walk: `ScipPlaces` when the SCIP graph read for the workspace has a node, else `OutlinePlaces` (`app::traverse::run`), so a tree with no index read (`scip.mode = "off"`, or `"cache"` with no file) still gets `lsp.live` cards. `OutlinePlaces`: + `ask` outlines a symbol's file once per map instance (`Manager::document_symbols`, one counted external call; + `SymbolPlaces::outline_wanted`), as it already does for a file or chunk node, and `SymbolPlaces::learn` takes the + outline in. A member whose kind this build has a node for (`SymbolKind`: functions, methods, types, enums, + modules, constants, classes, interfaces, fields, variables; LSP kinds 12, 6, 9, 5, 23, 11, 10, 2-4, 14, 7 and 8, 13; 5 is `Class`, 11 is `Trait` in a Rust file and `Interface` elsewhere; enum members 22, impl blocks 19 and type parameters 26 have no node) takes the SCIP node's id for the same file, name and + nearest line, else (a Rust file, `OutlinePlaces::with_tree`) the canonical id of the `chunk::extract_for_path` declaration around the member, else a minted `SymbolKey` with the file's `language::for_path` identifier and the equal names + numbered in document order; where a file is learned its places win over the index's, which may be older than the + tree. `flatten_document_symbols` gives a nested `DocumentSymbol` the range from its name to its item's end, so + `Member::end_line` is the last line of the item. + Readiness is reported, not acted on: the transport keeps the `is_ready` latch (the adapter's `ready_predicate` + matched once, never cleared) and, beside it, a `watch` of `ServerState` (`busy`, `changes`, `changed_at`) fed by the + adapter's `busy_predicate` (default: never; rust-analyzer `quiescent: false`, jdtls `Starting`). `Manager::readiness` + turns a session into a `ServerReadiness` (`starting`, `ready`, `busy`, `failed` or `stuck`, primed, counters, ages) + without touching `last_used`; `Status.analyzers`, the search over-budget note and the `lsp.live` notes of a traverse + call read it. No request, repeat or answer depends on it. + `[run_profile.lsp] sync = "watch"` (the default since K11b-4; read by `Manager::new`; a per-call + manager is built `without_sync`) gives each workspace with a session an `lsp::sync::WorkspaceSync`: a `notify` + watcher of the root (polling at `watch_poll_milliseconds` when the platform watcher fails) whose events a task + drains into batches, typed by the disk (created, changed, deleted) against a `(mtime, length)` table from one + `chunk::source_files` walk (walked again only after an overflow). The manager's sink drops the files' kept outlines + and sends each batch to the workspace's sessions as `workspace/didChangeWatchedFiles`; `Manager::client` drains + before every request. The sync starts with the workspace's first session and stops with its last (`release`, + `shutdown_all`, the idle sweeper). Lock order: drain lock, then the session registry, then `syncs`. + `Status.sync` (N118) reports `Manager::sync_status` for the workspace (backend, `failed` when neither watcher started or the opening task died, + the counters) only while a sync runs, so a failed or silent watcher is visible; it is a report and changes no answer. diff --git a/docs/reference/cli.md b/docs/reference/cli.md index 1cbe869e..78470cdf 100644 --- a/docs/reference/cli.md +++ b/docs/reference/cli.md @@ -14,8 +14,8 @@ repo=$PWD # this repository: the walkthrough "$lci" list-files "$repo" "$lci" search "$repo" 'plain REST JSON-over-HTTP mirror of the MCP tools' "$lci" symbols "$repo" 'openapi_document#' -"$lci" definition "$repo" 'src/rest.rs' 351 64 -"$lci" references "$repo" 'src/rest.rs' 172 3 --include-declaration +"$lci" definition "$repo" 'src/rest.rs' 377 21 +"$lci" references "$repo" 'src/rest.rs' 188 3 --include-declaration "$lci" references "$repo" 'lci-core/src/workspace.rs' 17 11 --include-declaration "$lci" index "$repo" ``` diff --git a/docs/reference/mcp-and-rest.md b/docs/reference/mcp-and-rest.md index 78ede440..ed0eff1e 100644 --- a/docs/reference/mcp-and-rest.md +++ b/docs/reference/mcp-and-rest.md @@ -9,7 +9,7 @@ All tools return structured JSON, also available as MCP text content. Every succ | Tool | Arguments | Result | | --- | --- | --- | | `index_workspace` | `workspace_path` | Canonical workspace/ID, file/chunk counts, parsed/unchanged/removed files, `skipped_non_source_files` (source-extension files whose content is another format, currently Qt Linguist `.ts` catalogs), newly embedded/reused chunks, `vectors_missing` (the chunks were committed without vectors because the embedding service was unavailable), total time; only when non-empty, `lossy_decoded_files` (non-UTF-8 files indexed from a lossy decoding) and `truncated_for_embedding` (chunks embedded from a shortened prefix because the embedding service rejected their full length) | -| `index_status` | `workspace_path` | Whether indexed/indexing/stale/watched, chunk count, embedding dimension (`null` while `vectors_missing` is true), `vectors_missing`, configuration compatibility, last successful indexing time; `stale_check_error` when staleness could not be determined (an unreadable file; an indexed workspace then reports `stale: true`); `analyzers_running`, one entry per language server (`rust-analyzer`, `csharp-ls`, `typescript-language-server`, `pyright`, `gopls`, `jdtls`, `clangd`), `true` while a started, ready server of that name runs for this workspace, including one a traverse call started under `[run_profile.lsp] sessions = "shared"`, and `false` otherwise (not started, released, disabled, not installed); `analyzers`, with the same seven keys, what each server's session reports about itself, or `null` when this workspace has no session for it: `state` (`starting`, `ready`, `busy`, `failed` or `stuck`; `busy` is a server that said it works again after it was ready, today rust-analyzer's `quiescent: false` and jdtls's `Starting` status, and the other servers never report it), `primed`, `progress` (`$/progress` notifications), `ready_changes` (ready or busy status notifications seen), and `age_ms`, `changed_ms` (`null` before the first such notification) and `idle_ms`. `sync`, present only while the workspace's sync under `[run_profile.lsp] sync = "watch"` runs (absent with `sync = "off"` and without a session): `backend` (`notify`, `poll` after the platform watcher failed to start, `null` while it is still opening), `failed` (neither watcher could start or the opening task died, so the servers are told nothing about edits) and the counters `events`, `batches`, `forwarded`, `walks` and `rescans`; a watcher that stops silently shows as `events` that stop growing after edits. `servers`, one entry per language server sorted by provider: `provider`, `languages` it can navigate, `languages_in_index` (the subset this index holds), `configured`, `enabled`, `command`, `probe` (`probed` once the manager's own version probe has passed, run by priming and the `lsp.live` warm start, else `not_probed`; `/ready` probes separately and does not count), `breaker_open_ms` (milliseconds left in the server's skip window under `[run_profile.lsp] breaker = true`, which a failed probe or start opens; `null` without the breaker or when the window is closed), `root` (the directory it starts at, relative to the workspace, `.` for the workspace itself: a live session's, else the one a start would choose now under `[run_profile.lsp] roots`, worked out only for an enabled, configured server whose languages the index holds), `root_reason` (why: `roots_workspace`, `manifest:`, `no_manifest`, `several_roots:`, `files_outside_root:`, `no_file_list`, `unreadable` or, under `"manifest"`, `not_served`, a server that is disabled, unconfigured or serves no language of the index) and `session` (the same object as its `analyzers` entry); and `languages_without_server`, the languages this index holds that no enabled, configured server lists (sorted). It is a report: reading it runs no probe, starts no server, does not keep a session alive and changes no answer. The over-budget note of a search and the `lsp.live` notes of a traverse call that ends with a server not ready name the state too. Deprecated, same values for one more release: `analyzer_running` (rust-analyzer), `csharp_analyzer_running`, `typescript_analyzer_running`, `python_analyzer_running`, `go_analyzer_running`, `java_analyzer_running`, `clangd_analyzer_running`. The same JSON comes from `POST /v1/index_status` and CLI `status` | +| `index_status` | `workspace_path` | Whether indexed/indexing/stale/watched, chunk count, embedding dimension (`null` while `vectors_missing` is true), `vectors_missing`, configuration compatibility, last successful indexing time; `stale_check_error` when staleness could not be determined (an unreadable file; an indexed workspace then reports `stale: true`); `analyzers_running`, one entry per language server (`rust-analyzer`, `csharp-ls`, `typescript-language-server`, `pyright`, `gopls`, `jdtls`, `clangd`, `marksman`), `true` while a started, ready server of that name runs for this workspace, including one a traverse call started under `[run_profile.lsp] sessions = "shared"`, and `false` otherwise (not started, released, disabled, not installed); `analyzers`, with the same eight keys, what each server's session reports about itself, or `null` when this workspace has no session for it: `state` (`starting`, `ready`, `busy`, `failed` or `stuck`; `busy` is a server that said it works again after it was ready, today rust-analyzer's `quiescent: false` and jdtls's `Starting` status, and the other servers never report it), `primed`, `progress` (`$/progress` notifications), `ready_changes` (ready or busy status notifications seen), and `age_ms`, `changed_ms` (`null` before the first such notification) and `idle_ms`. `sync`, present only while the workspace's sync under `[run_profile.lsp] sync = "watch"` runs (absent with `sync = "off"` and without a session): `backend` (`notify`, `poll` after the platform watcher failed to start, `null` while it is still opening), `failed` (neither watcher could start or the opening task died, so the servers are told nothing about edits) and the counters `events`, `batches`, `forwarded`, `walks` and `rescans`; a watcher that stops silently shows as `events` that stop growing after edits. `servers`, one entry per language server sorted by provider: `provider`, `languages` it can navigate, `languages_in_index` (the subset this index holds), `configured`, `enabled`, `command`, `probe` (`probed` once the manager's own version probe has passed, run by priming and the `lsp.live` warm start, else `not_probed`; `/ready` probes separately and does not count), `breaker_open_ms` (milliseconds left in the server's skip window under `[run_profile.lsp] breaker = true`, which a failed probe or start opens; `null` without the breaker or when the window is closed), `root` (the directory it starts at, relative to the workspace, `.` for the workspace itself: a live session's, else the one a start would choose now under `[run_profile.lsp] roots`, worked out only for an enabled, configured server whose languages the index holds), `root_reason` (why: `roots_workspace`, `manifest:`, `no_manifest`, `several_roots:`, `files_outside_root:`, `no_file_list`, `unreadable` or, under `"manifest"`, `not_served`, a server that is disabled, unconfigured or serves no language of the index) and `session` (the same object as its `analyzers` entry); and `languages_without_server`, the languages this index holds that no enabled, configured server lists (sorted). It is a report: reading it runs no probe, starts no server, does not keep a session alive and changes no answer. The over-budget note of a search and the `lsp.live` notes of a traverse call that ends with a server not ready name the state too. Deprecated, same values for one more release: `analyzer_running` (rust-analyzer), `csharp_analyzer_running`, `typescript_analyzer_running`, `python_analyzer_running`, `go_analyzer_running`, `java_analyzer_running`, `clangd_analyzer_running`. The same JSON comes from `POST /v1/index_status` and CLI `status` | | `list_indexed_files` | `workspace_path` | Every file the persistent index has cached chunks for, with language and chunk count, from the last successful `index_workspace` | | `watch_workspace`, `unwatch_workspace` | `workspace_path` | Start / stop debounced polling and automatic reindexing for an indexed workspace | | `search_symbols` | `workspace_path`, `query` | Workspace symbols from every applicable enabled language server, with provider/language metadata. When some servers fail and another answers, the call succeeds with the answering servers' symbols plus, only when non-empty, `warnings` (one `Optional LSP provider unavailable: : ` per failed server, in provider order), so an empty `results` is not a confirmed "no match"; an error only when every applicable server failed. The three navigation tools add `notes` (absent when empty) when a kept server was restarted by the tree barrier or reused by the call, see "Language-server sessions of `lsp.live`" in the [CLI reference](cli.md) | diff --git a/docs/reference/retrieval.md b/docs/reference/retrieval.md index a558e0ac..f50763ae 100644 --- a/docs/reference/retrieval.md +++ b/docs/reference/retrieval.md @@ -4,7 +4,7 @@ This file is the home of the shipped retrieval pipeline's behavior and of how re ## Retrieval behavior -The shipped search pipeline, in order; the code-level picture (types, constants, functions) is [docs/architecture.md](../architecture.md) "Chunks and the manifest", "Indexes and channels" and "Indexing flow and freshness". Traverse mode: [engine.md](../spike/multimap/engine.md). +The shipped search pipeline, in order; the code-level picture (types, constants, functions) is [docs/architecture.md](../architecture.md) "Chunks and the manifest", "Indexes and channels" and "Indexing flow and freshness"; the LSP channel: [lsp-channel.md](../lsp-channel.md). Traverse mode: [engine.md](../spike/multimap/engine.md). 1. Scan the eighteen extensions of [Supported languages](../../README.md#supported-languages) (any ASCII case: a `.RS` file is Rust, N94) recursively with `.gitignore`, nested ignores and standard `ignore` crate rules. The checked-in `.lciignore` is an additional ignore file that keeps agent control-plane material (`.agents`, `.github/agents`, `.github/instructions`, `.github/skills`, `.codex`, `.claude`, `.kilo`) and generated `test-results` out of this repository's searchable corpus while leaving them on disk for agents. Symlinks are not followed, `.git` is skipped, hidden source files are eligible when not ignored. A file that is not valid UTF-8 (for example Windows-1252 C source) is indexed from a lossy decoding (invalid bytes become U+FFFD, line structure kept) and reported in `lossy_decoded_files`; it never fails indexing or search. A Qt Linguist `.ts` catalog is left out and counted in `skipped_non_source_files`; it does not affect the staleness fingerprint, and a file that changes into one drops its chunks at the next refresh like a deleted file. A source file removed between the walk and its read, or whose path is not valid UTF-8, is skipped with a logged warning. Other read/traversal errors abort the update and preserve the previous index; under `on-search` freshness the search then reuses the previous snapshot with a warning. 2. Select a static language adapter and Tree-sitter grammar by extension. Chunks preserve: Rust, its original declaration boundaries; TypeScript/JavaScript, imports, executable top-level statements, exports, comments/decorators, functions, classes and methods, interfaces and signatures, type aliases, enums, variable-assigned arrow functions; Python, imports, assignments, executable top-level statements, sync and async functions, classes, methods, decorators and leading comments; C#, using directives, namespaces, top-level statements, types, members, attributes, XML documentation comments, local functions; Go, top-level functions, methods, type/const/var declarations and leading comments; Java, packages, imports, classes, interfaces, enums, records, methods, constructors, fields, annotations, Javadoc; C, functions, structs, unions, enums, typedefs, preprocessor macros, global declarations, leading comments; C++, namespaces, classes, structs, unions, templates, methods, fields, leading comments. Oversized syntax splits only at named syntax boundaries toward 6,000-byte groups, with declarations preserved up to 24,000 bytes; large indivisible leaves stay whole rather than truncated; parser error recovery remains searchable. Markdown (`.md`) is cut into heading sections: a heading line plus its own body up to the next heading of any level (ATX and setext headings), with fenced code blocks kept whole and never searched for headings, YAML front matter kept with the first section, and text before the first heading as its own chunk; a section over 6,000 bytes is packed by paragraph toward that size, and a fence stays whole. @@ -34,7 +34,7 @@ The shipped search pipeline, in order; the code-level picture (types, constants, - Offline scoring: `evaluate --dump-candidates` records every query's whole fused candidate pool, the dumped configuration and the reranker's scores; `LCI_CANDIDATE_DUMP=` appends the same pool per search of a running process to a JSON Lines file (read once at startup; MCP, REST and CLI responses are unchanged). `"$lci" replay test-results/dump.json --output test-results/replay-w12.json --window 12 --cap 2 --cap-stage results` then scores other windows, caps, fusion weights, `rrf_k`, priors and rerank modes from the dump without services, configuration (the `--config` file is not read) or index, and `compare` pairs the output with real reports. A query replay cannot score exactly is refused and counted, never approximated. Row format, replay semantics, the latency model and when to run live instead: `.agents/skills/lci-retrieval-evaluation/references/evaluation-workflow.md`. - Git-mined query set: `cargo xtask mine-queries --repo --key --out evaluations/mined/.toml [--max 300]` turns qualifying commits from `git log` into queries (subject plus first body paragraph as the query, touched supported files as `expected_paths`, `top_k = 10`). `evaluations/mined/self.toml` (workspace key `self`) is the generated set for this repository: run it like `core.toml` (`--workspace "self=$PWD"`), regenerate rather than edit it, never use it to tune ranking. `core.toml` is the hand-written required set. Skip rules, labels and tags: same reference. - Gold strata and transcript shapes: `cargo xtask gold strata --set --gold [--out ]` classifies each gold item as `core`, `grep_easy` or `agent_trap` from the item alone and reports the counts, the hop quantiles, the items no rule reaches and the items whose question holds a gold file's stem; the same `guard` is a stratum of every scorer. `cargo xtask mine-transcripts --input .. [--gold ] [--out test-results/transcripts/shapes.json]` counts question shapes and trace classes over synthetic transcripts in the sim trace shape and writes only shapes, counts and anonymised templates. Rules and schema: [gold-scorers.md](../spike/multimap/v2/gold-scorers.md). -- Live acceptance: `cargo xtask acceptance core` and `acceptance lsp` default to this repository (`--profile gust --workspace /path/to/gpu-dialect-v0`, or `$LCI_ACCEPTANCE_GUST_WORKSPACE`, runs them against the optional external GUST example); `lsp` checks workspace-symbol, definition and reference navigation against the same symbol and saves each normalized response in `test-results`. With the real embedder and reranker running, `cargo xtask acceptance multilingual` creates a disposable fixture and data directory under `test-results`, uses the real debug binary; it verifies all supported language IDs, semantic and lexical retrieval with live reranking, production TypeScript, Python and C# ranking over test decoys, zero-work unchanged indexing, one-file invalidation, deletion, restart reuse, and prior-snapshot retrieval after an unreachable embedding endpoint makes an update fail (`test-results/multilingual-*.json`). Per language, `-lsp`, `-missing` and `-recovery` run a real server (options and prerequisites: `.agents/skills/lci-lsp-adapter/references/acceptance.md`, `docs/development/live-acceptance-ci.md`); each skips with `PREREQUISITE_UNAVAILABLE` when its server is not found. +- Live acceptance: `cargo xtask acceptance core` and `acceptance lsp` default to this repository (`--profile gust --workspace /path/to/gpu-dialect-v0`, or `$LCI_ACCEPTANCE_GUST_WORKSPACE`, runs them against the optional external GUST example); `lsp` checks workspace-symbol, definition and reference navigation against the same symbol and saves each normalized response in `test-results`. With the real embedder and reranker running, `cargo xtask acceptance multilingual` creates a disposable fixture and data directory under `test-results`, uses the real debug binary; it verifies all supported language IDs, semantic and lexical retrieval with live reranking, production TypeScript, Python and C# ranking over test decoys, zero-work unchanged indexing, one-file invalidation, deletion, restart reuse, and prior-snapshot retrieval after an unreachable embedding endpoint makes an update fail (`test-results/multilingual-*.json`). Per language, `-lsp`, `-missing` and `-recovery` (markdown has no `-recovery`) run a real server (options and prerequisites: `.agents/skills/lci-lsp-adapter/references/acceptance.md`, `docs/development/live-acceptance-ci.md`); each skips with `PREREQUISITE_UNAVAILABLE` when its server is not found. - Metric tiers (method and status in [evaluation.md](../spike/multimap/evaluation.md)): agent runs are the ground truth but too costly to run regularly, so cheap proxies run constantly and are checked against agent runs now and then, and quality is reported together with cost. | Tier | When | What | diff --git a/docs/spike/multimap/engine.md b/docs/spike/multimap/engine.md index 8d7aab0f..a8aa6c55 100644 --- a/docs/spike/multimap/engine.md +++ b/docs/spike/multimap/engine.md @@ -53,8 +53,8 @@ Traversal maps (ids as in `--maps`). Default set `MAPS`; the rest are opt-in (`O | `exact.closure` | first stage only | declarations that reference the symbol the question names, most references first | n/a | `src/app/traverse/first_stage.rs` | | `exact.pickaxe` | first stage only | commits found by `git log -G/-S/--grep` for a history question (20 s timeout per call) | n/a | `exact_history.rs`, `first_stage.rs` | -Other maps: `sym.defs` (RankList, PerFile; declarations from the bundle store, `symdefs.rs`) and the legacy maps `vec.chunk.code`, `lexical`, `lsp.symbols` that wrap today's three search channels -(`src/app/search/legacy_maps.rs`, `legacy_maps` and `record_through_maps`/`fuse_through_maps` behind Cargo feature `spike-maps`; `LEGACY_ORDER` keeps the f32 fusion order bit-exact). Support modules: `walk.rs` +Other maps: `sym.defs` (RankList, PerFile; declarations from the bundle store, `symdefs.rs`) and the channel maps `vec.chunk.code`, `lexical`, `lsp.symbols` that wrap today's three search channels +(`src/app/search/channel_maps.rs`, the module and its `record_through_maps`/`fuse_through_maps` behind Cargo feature `map-equivalence`; `CHANNEL_ORDER` in `lci-maps` keeps the f32 fusion order bit-exact). Support modules: `walk.rs` (M0.4, hop scorer constants `W_BASE` 0.1, `W_INDEGREE_IDF` 0.3, `W_OVERLAP` 0.6), `crates.rs`, `fresh.rs`, `audit.rs`, `labels.rs`, `kit.rs`, `kit_c3.rs`, `run_record.rs`. What persists between index passes (chunk store): per chunk the trimmed `code`, line range, `content_hash`, language and path (manifest and LanceDB columns with the vector); per file `language`, `adapter_version`, @@ -266,12 +266,12 @@ and never contact the reserved model ports (fakes under `tests/support`). | Evaluation changes (9.5) | partly built (bench in `xtask/src/maps`, scorers in `xtask/src/gold`) | Leakage risk: a co-change map built at HEAD includes the commits the miner turned into gold, so score it from history before each query's commit. Plan: multi-hop gold (ordered or set-valued per hop, for example config key, then reader, then test) with a scripted steering policy so runs are reproducible and pairable by `compare` (a free agent loop belongs to the agent A/B); per-map contribution (`CandidateTrace`/`CandidateArm` keyed by map, legacy columns kept) with per-map unique gold (gold reached only through map M) and leave-one-map-out replays; step metrics (frontier gold recall at each checkpoint, steps and rerank tokens spent); bit-exact replay of old dumps must keep working. | Merge conditions M1 to M7 are met on the integration branch (2026-10-13; log: d212, d213, d214, d204). Still true as constraints: one LSP orchestration (`App::applicable_lsp_adapters`, `lsp_symbol_queries`, -`App::lsp_channel`, `lsp_over_budget_note`, `lsp_failure_warning`) and one chunker, with a golden record of 44 chunks; spike entry points sit behind Cargo feature `spike-maps` (`#[cfg(any(test, feature = -"spike-maps"))]`, Full test run 812 = 811 + 1); stored maps run on `spawn_blocking` (300 ms budget, 250 ms slack, 512 blocking threads); C5 labels are checked inside `Registry::run`; the hand-written schema checker +`App::lsp_channel`, `lsp_over_budget_note`, `lsp_failure_warning`) and one chunker, with a golden record of 44 chunks; spike entry points sit behind Cargo feature `map-equivalence` (`#[cfg(any(test, feature = +"map-equivalence"))]`, Full test run 812 = 811 + 1); stored maps run on `spawn_blocking` (300 ms budget, 250 ms slack, 512 blocking threads); C5 labels are checked inside `Registry::run`; the hand-written schema checker is kept. `SnapshotView::new` timing (M4, partly met; example `lci-maps/examples/snapshot_view_timing.rs`; release, 50 runs, 4 cores, cloud machine): 4048 chunks p50 3.94 ms, p95 4.24 ms, max 5.60 ms; ten times (40480 chunks) p50 74.67 ms, p95 89.75 ms, max 91.88 ms; debug 22.18/26.03/28.49 ms and 246.67/293.69/339.35 ms (about 19x release, 11x debug for ten times the chunks). The 50 constructions run back to back on the same input, so caches are warm; one construction per search may cost more than these p50s. Not measured: per-query timing over `evaluations/mined/self.toml` and the A/B's largest repo, and the owner-set acceptable share. Before the registry replaces `fuse_hits`, one real candidate dump should replay with zero refusals (owner's machine); the -C4 replay auditor is ignored in the Full run (CI has no history). Owner decisions: the record path stays in `legacy_maps.rs` until merge; full query text is kept in local RunRecords; no profile or reranker step. +C4 replay auditor is ignored in the Full run (CI has no history). Owner decisions: the record path stays in `channel_maps/record.rs` until merge; full query text is kept in local RunRecords; no profile or reranker step. ## Known gaps diff --git a/docs/spike/multimap/specs/README.md b/docs/spike/multimap/specs/README.md index 9191f771..38f5359c 100644 --- a/docs/spike/multimap/specs/README.md +++ b/docs/spike/multimap/specs/README.md @@ -17,7 +17,7 @@ code, the roadmap and frozen preregs: never renumber or rename a section. | Spec | Decides | Built | Builds on (symbols) | | --- | --- | --- | --- | | [node-id.md](node-id.md) | One identity space for file, chunk, symbol, commit, test, config key, doc section and string literal; canonical strings; stability across edits. Chunk = `(path, content_hash, ordinal)`, no `start_line`. Decisions N-1..N-15. | `NodeId` (S1); extraction hook and bundle store (S3a) | `Chunk`, `chunks_with_adapter` (`lci-core/src/chunk.rs`), `CachedFile`, `Manifest` (`lci-core/src/manifest.rs`), `fuse_hits` key (`src/app/search.rs`) | -| [map-trait.md](map-trait.md) | The `Map` trait (descriptor, request, response with status, cost and freshness), the fusion loop that keeps today's f32 order, how the three legacy channels and the source-role prior are wrapped; petgraph for the graph projection. M-1..M-8. | trait, `Registry`, `sym.defs` (S2, S3a); legacy wrapping behind `spike-maps` | `search_once`, `fuse_hits`, `map_lexical_matches`, `map_lsp_locations` (`src/app/search.rs`), `Store::search` (`src/store.rs`) | +| [map-trait.md](map-trait.md) | The `Map` trait (descriptor, request, response with status, cost and freshness), the fusion loop that keeps today's f32 order, how the three search channels and the source-role prior are wrapped; petgraph for the graph projection. M-1..M-8. | trait, `Registry`, `sym.defs` (S2, S3a); channel-map wrapping behind `map-equivalence` | `search_once`, `fuse_hits`, `map_lexical_matches`, `map_lsp_locations` (`src/app/search.rs`), `Store::search` (`src/store.rs`) | | [contract-kit.md](contract-kit.md) | Contracts C1-C9 as testable properties, with an applicability table and tests for the kit itself. K-1..K-12. | C1, C3, C4, C5, C6 (`lci-maps/src/kit*.rs`, `audit.rs`, `labels.rs`); C2 and C8 not built; C7 and C9 partly or not | fail-open paths and filters in `src/app/search.rs`, `evaluate::replay` (`src/evaluate/replay.rs`) | | [run-record.md](run-record.md) | JSON Lines trace per traversal (`run`, `step`, `end`) with a JSON Schema; privacy rules; the `run` line's `profile` stamp (run-profile hash, keys, embedding identity, priors and tie-break file hashes). R-1..R-11. | schema, spike-feature writer and offline harness; no production writer | `CandidateDumpLine`, `CandidateTrace`, `Timings` (`src/app/search.rs`), `Snapshot` (`src/store.rs`) | | [steering-ops.md](steering-ops.md) | Ops v1 (pause, prune, expand, add_map, drop_map, reweight) as validated all-or-nothing patches on a resumable handle; three MCP tools with REST mirrors. S-1..S-8. | only the ladder, `steer` and file-backed handles (section 3.6); the ops, tools and TTL are design | `McpServer` tool router (`src/server.rs`), REST routes (`src/rest.rs`), `App.watchers` (`src/app/mod.rs`), `IndexResolution` (`src/app/search.rs`) | diff --git a/docs/spike/multimap/specs/map-trait.md b/docs/spike/multimap/specs/map-trait.md index b2e9decc..e4b66c9a 100644 --- a/docs/spike/multimap/specs/map-trait.md +++ b/docs/spike/multimap/specs/map-trait.md @@ -34,13 +34,13 @@ The legacy search path, named by symbol (checked on the spike tree). The product - Crate `lci-maps` depends on `lci-core` only: the `Map` trait, `MapCard`, the registry, the fusion loop and the maps that need no LanceDB. - The root crate implements the maps that touch LanceDB or `App` internals and owns the engine that `App` calls. The wrapped lexical and LSP maps live in the root - crate too (`src/app/search/legacy_maps.rs`), not in `lci-maps`: the code they call (`App::map_lexical_matches`, `App::provider_symbols`, `App::map_lsp_locations`, + crate too (`src/app/search/channel_maps.rs`), not in `lci-maps`: the code they call (`App::map_lexical_matches`, `App::provider_symbols`, `App::map_lsp_locations`, `Hit::source_role_prior`) is there, and moving it would change the production path. `LspMap` and `search_once` call the same shared functions in `src/app/search.rs` (`App::applicable_lsp_adapters`, `lsp_symbol_queries`, `App::lsp_channel`, `App::lsp_over_budget_note`, `lsp_failure_warning`); the integration test that compares the registry with a traced search guards the map path. - `lci-maps` adds `petgraph = "0.8"` only when a graph map needs it (M-5); no shipped map uses it. -- Spike entry points on the `App` facade (`App::fuse_through_maps`, `App::record_through_maps`) and the module `legacy_maps` compile only with `#[cfg(any(test, - feature = "spike-maps"))]`, so the default build adds no public path. +- Spike entry points on the `App` facade (`App::fuse_through_maps`, `App::record_through_maps`) and the module `channel_maps` compile only with `#[cfg(any(test, + feature = "map-equivalence"))]`, so the default build adds no public path. ### 3.2 The trait @@ -118,7 +118,7 @@ compile_error_at, near, defined_in. Edges are sorted by confidence, then destina `fuse` replaces `fuse_hits` with a loop that keeps today's arithmetic exactly: -1. Order: `vec.chunk.code`, `lexical`, `lsp.symbols` first, in that order (`LEGACY_ORDER`); then every other `RankList` or `Expansion` map in registry order (sorted +1. Order: `vec.chunk.code`, `lexical`, `lsp.symbols` first, in that order (`CHANNEL_ORDER` in `lci-maps`); then every other `RankList` or `Expansion` map in registry order (sorted by `MapId`); then `Prior` maps. 2. For each map's hits in its own order: `score[node] += weight(map) / (rrf_k + rank as f32)`, in `f32`, starting from `0.0`. A node's first contribution inserts it. 3. `Prior` maps add their value unweighted (the source-role prior, `src/store.rs`). After the rank lists are fused, every `Prior` map gets the fused nodes as seeds diff --git a/docs/spike/multimap/specs/run-record.md b/docs/spike/multimap/specs/run-record.md index d5b26526..36811bf0 100644 --- a/docs/spike/multimap/specs/run-record.md +++ b/docs/spike/multimap/specs/run-record.md @@ -87,7 +87,7 @@ reference. #### Steps of a registry run (built) -The registry path (`App::record_through_maps`, the recording twin of `fuse_through_maps`; feature `spike-maps`, `src/app/search/legacy_maps/record.rs`) writes, after +The registry path (`App::record_through_maps`, the recording twin of `fuse_through_maps`; feature `map-equivalence`, `src/app/search/channel_maps/record.rs`) writes, after the `run` line: | seq | `kind` | `yield` | Content | diff --git a/docs/traverse-flow.md b/docs/traverse-flow.md new file mode 100644 index 00000000..dedce007 --- /dev/null +++ b/docs/traverse-flow.md @@ -0,0 +1,49 @@ +# Traverse call flow + +Code-level picture of the flow of one `search_code mode="traverse"` / `lci traverse` call (`app::traverse::run`) and of the answer keys that the `[run_profile]` switches add (`answer_labels`, `card_paths`, `map_freshness`, `verify_on_read`); behaviour and defaults of those keys: [CLI reference](reference/cli.md). Crate layout: [architecture.md](architecture.md); the engine, maps and steering ladder: [engine.md](spike/multimap/engine.md); settings: [README.md](../README.md). + +`search_code` with `mode = "traverse"` (needs `[traverse] enabled`) and `lci traverse` run one function, +`app::traverse::run`. It loads the SCIP index (map `graph.scip`, cached by file stamp; `[run_profile.scip] mode` `cache` (the default) reads it when it is there and parses an unreadable file once per file stamp, `authoritative` requires it, `off` never touches it and leaves the graph and the maps built from the index file empty, each saying so in a note), finds the start nodes +(`given`; `hybrid`: the top `seed_k` hits of the normal hybrid search mapped to SCIP symbols; `hybrid+bm25` or `bm25` with `[run_profile.seeds] bm25 = true`: the opt-in map `seeds.bm25`, +BM25 over the SCIP graph's node names, qualified paths and documentation, whose top `seed_k` nodes go after the hybrid hits or stand alone +when the search gave none; `...+literal` or `literal` with `[run_profile.seeds] literal = true`: the opt-in map `seeds.literal`, a whole-word ripgrep search (`lexical::search_literals`) for the names the question puts in code spans, each line placed in the SCIP graph (the symbol defined there, the one around it, else the file of a declaration line), appended after those sources; `...+repo_map` or `repo_map` with `[run_profile.seeds] repo_map = true`: the opt-in map `seeds.repo_map`, a personalized PageRank over the SCIP graph's call and containment edges from the entities `seeds.bm25` finds for the question, whose new top nodes go after the earlier sources' (each source form also exists with `+literal` after it); or `name_tokens`, the visible fallback), builds the steering ladder from the caller's rules, runs the `lci-maps` traversal under the +`[traverse]` caps (a request can only lower them), writes a trace under `data_dir`, and answers with the reached nodes, +totals and, only when work is left, a `handle` and `checkpoint`. The first stage (exact operators `exact.closure` and +`exact.pickaxe`, then seeds), named policies per question type (the router, chosen by the caller's `steer.policy`) and the steering ladder are described in +[engine.md](spike/multimap/engine.md). Default off: with `[traverse]`, `[recipes]` and `[compose]` disabled the tool +list, schemas and REST routes are exactly those of a server without them. + +With `[run_profile.traverse] answer_labels = true` (default `false`) the answer also has `completeness` (`{complete, why, estimate}`: `complete` is `status == "complete"` with no map lost (`degraded`: a map the call asked for that could not be set up, or had `MapStat.failures`; `why` is then `degraded` unless the walk was cut), `why` the `stop_reason`, `estimate` `TraversalResult.estimate`, the `likely_complete` fraction) and every card of `results` and the checkpoint lists has `freshness` (`fresh`, `changed`, `stale` or `unverified`, read from `file_changed` and `maps`; a card that cites `lsp.live` is `unverified`, not `fresh`, when the walk found a language server running, `add_answer_labels`' `kept` argument from the walk's `WalkSessions`; a continuation also demotes the `lsp.live` cards of the nodes it restores from the checkpoint, its `restored` argument). `app::traverse::response::add_answer_labels` adds both to the response JSON after the trace and the checkpoint hash are made, so the trace, the checkpoint, the handle and `Card`/`card_json` are unchanged, nothing is withheld, and with the key off the answer is as before. + +With `[run_profile.traverse] card_paths = true` (default `false`) every walked card of `results` and of `checkpoint.answer` also has `kind` and `path`, the +route from its start node with each hop's map, edge kind and witness (`app::traverse::response::add_paths`, read from the walk's state table; the engine keeps +`State::kind`, and a handle keeps it only under the key). With the key off the answer, the trace, the checkpoint and the handle are unchanged. + +With `[run_profile.traverse] map_freshness = true` (default `false`) the answer also has `maps`: for each map of the call that +states a freshness, `{id, observed, stale, lag}` (`app::traverse::freshness`). The SCIP-built maps report `snapshot`; +`grep.live`, `lsp.live`, `git.history`, `exact.pickaxe` and `probe.compile` (which runs only under `rust_analyzer_trusted = true`) report `live` with `lag` 0 (`live` names the channel: `grep.live`, +`git.history`, `exact.pickaxe` and `probe.compile` read current bytes or git, placed by the index's positions for `grep.live`; +`lsp.live` asks a running server, which is fresh when this call started it and, when kept across calls under +`sessions = "shared"`, bounded by its tree stamp's coverage and labelled by a `kept session` note, with the same fields). `lag` is the number of source files (language-adapter +files, prose left out) newer than the index file plus those changed (content hash), added or removed since the process +first saw the index's current stamp (modified time and length), each counted once; `stale` means `lag` above 0 and `null` +means not compared (no tree, unreadable tree, or an index file without a modified time). The tree is read and hashed +once per call with `chunk::source_hashes` (the index scan's walk and its read, skip and hash function, no chunk made); +`[run_profile.freshness] stat_gate = true` (default `false`, in the profile hash) puts a process-wide `chunk::StatGate` per +root in front of the read, which keeps `(modified time, length, hash)` per file and reads only files whose stamp is new or +changed. A file modified within 3 seconds of the read is not trusted (racy stamp; FAT, exFAT and SMB stamps are coarse) and is read again next call. Its gap, asserted by a test: a same-length edit that keeps a modified time older than 3 seconds before the previous read is not seen until the next edit. +Search's `verify_on_read` reads through the same function. `Expanded` carries an optional +`edge::Freshness` that a wrapper around each stating map records; the walk itself does not read it. Key off: no +wrapper, no tree read, no `maps` key. + +`[run_profile.freshness] verify_on_read = true` (default `false`, in the profile hash) adds, on top of `map_freshness`, a check of +what the answer cites against the tree by content hash. It reuses the one read of the tree (`freshness::tree_view` returns the +lag with the file sets it came from; no second scan), reads and labels, and never refreshes the index or touches +`[index].freshness`. `SeedSearch::snapshot_hashes` (`AppSeeds` reads the manifest of the index snapshot) is read after the walk, +because the walk's first search may refresh the snapshot; the snapshot lag is the source files (the `tree_lag` set; manifest prose +entries are left out) that differ between the manifest and the tree. `seeds.hybrid`, `search.lexical` and `search.vector` are +then labelled with it (`Observer` does not wrap them; the label is filled at report time), `grep.live` and `lsp.live` say +`observed: live` with `places: "index"` and the SCIP lag (their places are the index's), and `response::add_file_changed` gives +every card `file_changed` from the baseline of its maps (the SCIP baseline for the maps built from the index and the placed +maps and the symbol cards, the manifest for the snapshot maps and the chunk cards; `null` where none applies). Key off: none of this, and no change to any answer. +`search::App::verify_on_read` does the same for a search report the caller asked for: `SearchReport.freshness` (`snapshot`, `channels` for the channels that ran, `hits` with `file_changed` per result) from the manifest of the snapshot the search ran on and one read of the tree's content hashes (`chunk::source_hashes`, gated by `stat_gate` when on), read after the search (a refresh in between is not seen) and timed in `timings.verify_ms`. The searches inside the crate that discard the block (`AppSeeds::search`, the evaluation runner) call the `_unverified` entry points, so a traversal reads the tree once.