[Feat] Community Local: ingestion worker, local models, retrieval & grounded chat

[CREDO23]

Continues #1742. Everything here is under surfsense_local/; nothing outside it is touched. This is the whole ingestion → retrieval → chat spine, plus the model setup it leans on.

What this adds

A worker that ingests what the API enqueues. Docling parses the file with OCR and table structure on, Chonkie chunks the exported markdown against bge-small's own tokenizer so a chunk is measured in the tokens the encoder will actually see, and a bundled bge-small-en-v1.5 int8 model embeds each chunk in-process on onnxruntime — 384-dim, no model server, offline. Both index tables are written in the same pass, and the document lands ready, or failed with the reason on the row. The chunker uses markdown-aware split rules so a heading is never severed from the body it introduces.

A local generation provider and per-role model selection. modules/llm/ splits a Generator (anything that can answer) from a ModelStore (only a runtime that holds weights on disk can pull), so the download UI is gated by isinstance rather than a provider name — a remote API satisfies the first alone. Ollama is the default and ships behind those protocols; since it exposes no library API, the models offered for download are a curated Qwen catalog kept inside the Ollama provider. SelectedModel(role) holds one choice per role. Adding a provider later is a folder plus one registry line.

Hybrid retrieval, workspace-scoped. shared/search.py's retrieve() widens recall with a BM25 leg and a sqlite-vec KNN leg, both filtered to the workspace, then rescores the union of candidates by cosine against the query vector — the same bundled encoder as ingest, imported lazily so shared does not pull worker in at import. Cosine on the union orders; the two legs only decide what is in the running.

Grounded, streaming chat. A turn retrieves its own context, wraps each hit in a numbered <source> block inside a system message, and instructs the model to cite inline with [id]. Chunk text is defanged first, so a document cannot forge its own source tag and break out of the block. History is a flat per-thread walk with a sliding token window; the system message and the new turn are pinned above it. The reply streams over SSE and both turns persist as one ChatMessage each — the assistant row carrying its citations for the UI.

Proper SSE. Frames are data: {json}\n\n and the stream ends on a data: [DONE]\n\n sentinel, so a client tells completion apart from a dropped socket. Deltas and the citation list are separate frames; a generator failure is its own error frame and the partial turn still persists. The response sets Cache-Control: no-cache and X-Accel-Buffering: no so a proxy streams it through rather than buffering the whole reply into one late blob.

Notes for review

• 102 tests, still split tests/unit/<feature> and tests/integration/<feature>. Nothing is mocked: chat and retrieval integration tests build a real SQLite file, ingest a real document, and run against a stub Ollama server over HTTP.
• Real-model tests fetch bge-small on first run and skip if it is absent, so the suite stays green on a machine that has never pulled it.
• Migrations stay at one hand-written revision — selected_models folded in — since nothing has shipped.
• The parser and encoder weights are bundled; a generation model is downloaded in-app on first run, not shipped. Recorded in api/05-packaging.md.
• Plans say what was built rather than what to build; the phase index carries progress markers. Chat and search phases close here.

High-level PR Summary

This PR delivers the complete local ingestion → retrieval → chat pipeline for SurfSense Community Local. It adds an ingestion worker that processes documents using Docling for parsing, Chonkie for chunking, and a bundled bge-small-en-v1.5 int8 ONNX model for in-process CPU embeddings (384-dim, no model server, offline). Hybrid retrieval combines BM25 and sqlite-vec KNN for recall, then rescores by cosine similarity, with both legs scoped to workspaces. The LLM layer introduces a provider abstraction (Generator vs ModelStore) with Ollama as the default, a curated Qwen catalog for downloads, and per-role model selection. Streaming chat retrieves context per turn, grounds the system prompt with numbered <source> blocks, cites inline with [id], streams deltas over proper SSE (data: [DONE]\n\n sentinel), and persists both user and assistant turns with citations. The worker runs serially (-w 1) beside the API, migrations stay hand-written at one revision with selected_models included, and 102 tests cover the entire stack with real SQLite files and a stub Ollama server—real model tests skip when weights are absent. Everything lives under surfsense_local/; nothing outside is touched.

⏱️ Estimated Review Time: 1-3 hours

💡 Review Order Suggestion

Order	File Path
1	plans/community-local/00-umbrella-plan.md
2	plans/community-local/00c-data-model.md
3	plans/community-local/api/02-upload.md
4	plans/community-local/api/03-chat.md
5	plans/community-local/api/05-packaging.md
6	plans/community-local/worker/01-boot.md
7	plans/community-local/worker/02-ingest.md
8	plans/community-local/worker/03-search.md
9	surfsense_local/backend/alembic/versions/0001_initial_schema.py
10	surfsense_local/backend/shared/config.py
11	surfsense_local/backend/shared/db.py
12	surfsense_local/backend/shared/queue.py
13	surfsense_local/backend/modules/llm/models.py
14	surfsense_local/backend/modules/llm/providers/protocols.py
15	surfsense_local/backend/modules/llm/providers/types.py
16	surfsense_local/backend/modules/llm/providers/__init__.py
17	surfsense_local/backend/modules/llm/providers/ollama/catalog.py
18	surfsense_local/backend/modules/llm/providers/ollama/provider.py
19	surfsense_local/backend/modules/llm/dependencies.py
20	surfsense_local/backend/modules/llm/schemas.py
21	surfsense_local/backend/modules/llm/router.py
22	surfsense_local/backend/worker/consumer.py
23	surfsense_local/backend/worker/ingestion/parsing.py
24	surfsense_local/backend/worker/ingestion/chunking.py
25	surfsense_local/backend/worker/ingestion/embedding.py
26	surfsense_local/backend/worker/ingestion/indexing.py
27	surfsense_local/backend/worker/ingestion/pipeline.py
28	surfsense_local/backend/worker.py
29	surfsense_local/backend/shared/search.py
30	surfsense_local/backend/modules/chat/models.py
31	surfsense_local/backend/modules/chat/prompt.py
32	surfsense_local/backend/modules/chat/history.py
33	surfsense_local/backend/modules/chat/dependencies.py
34	surfsense_local/backend/modules/chat/schemas.py
35	surfsense_local/backend/modules/chat/router.py
36	surfsense_local/backend/api/main.py
37	surfsense_local/backend/modules/documents/storage.py
38	surfsense_local/backend/modules/documents/tasks.py
39	surfsense_local/backend/modules/documents/router.py
40	surfsense_local/backend/pyproject.toml
41	surfsense_local/backend/scripts/fetch_embedding_model.py
42	surfsense_local/backend/tests/conftest.py
43	surfsense_local/backend/tests/integration/test_registration.py
44	surfsense_local/backend/tests/unit/worker/test_chunking.py
45	surfsense_local/backend/tests/unit/chat/test_prompt.py
46	surfsense_local/backend/tests/unit/chat/test_history.py
47	surfsense_local/backend/tests/unit/llm/test_catalog.py
48	surfsense_local/backend/tests/integration/worker/conftest.py
49	surfsense_local/backend/tests/integration/worker/test_ingest.py
50	surfsense_local/backend/tests/integration/worker/test_consumer.py
51	surfsense_local/backend/tests/integration/search/test_retrieve.py
52	surfsense_local/backend/tests/integration/llm/conftest.py
53	surfsense_local/backend/tests/integration/llm/test_routes.py
54	surfsense_local/backend/tests/integration/chat/conftest.py
55	surfsense_local/backend/tests/integration/chat/test_chat.py
56	surfsense_local/backend/tests/integration/test_app_boot.py
57	SurfSense.code-workspace
58	.vscode/settings.json
59	surfsense_local/backend/.vscode/settings.json
60	surfsense_local/README.md

[vercel]

@CREDO23 is attempting to deploy a commit to the Rohan Verma's projects Team on Vercel.

A member of the Team first needs to authorize it.

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Team

Run ID: 82e86a36-7fb3-4777-8fc0-def19d8871bd

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}