A research-paper RAG assistant. Give it a topic; it searches ArXiv, downloads and chunks the PDFs, indexes them into a vector database, and then answers questions with a two-stage retrieval pipeline — streaming tokens into a React chat UI over a WebSocket.
The two things that make the retrieval quality defensible:
- Cross-encoder reranking — vector search recalls 30 candidates, a cross-encoder rescores them and keeps the best 5.
- A measured evaluation harness — faithfulness, hallucination rate, answer relevancy and context precision/recall over a fixed question set, with a built-in A/B ablation of the reranker.
React (Vite) ──WebSocket──▶ FastAPI ──▶ Celery worker ──▶ Qdrant
▲ │ │ (vectors)
└────── tokens ─────────────┘ ├──▶ Ollama (LLM)
via Redis pub/sub ├──▶ Embeddings (local or HF API)
└──▶ Cross-encoder (rerank)
| Service | Role |
|---|---|
api |
FastAPI: REST + the /ws WebSocket that streams answers |
worker |
Celery: indexing and query jobs |
redis |
Celery broker/backend and the token-streaming pub/sub channel |
qdrant |
Vector store, one collection per chat session |
| Ollama | Keyword extraction + answer generation (runs on the host) |
Bi-encoder vector search embeds the query and each chunk independently, so it can only measure embedding proximity. It is fast enough to scan a collection but it will happily rank a chunk that merely shares vocabulary with the question above the chunk that actually answers it.
A cross-encoder scores the (query, chunk) pair jointly — both strings go
through the transformer together, so attention runs across the pair. Far too
slow for a whole corpus, exactly right for re-ordering a shortlist.
query
│
├─▶ 1. RECALL vector search, top-30 (cheap, tuned not to miss)
│
├─▶ 2. PRECISION cross-encoder rescore, top-5 (ms-marco-MiniLM-L-6-v2)
│
├─▶ 3. DIVERSITY cap 2 chunks per paper (stop one paper dominating)
│
└─▶ 4. GENERATE prompt + stream from Ollama
Every stage is configurable (RETRIEVE_TOP_K, RERANK_TOP_N,
MAX_CHUNKS_PER_PAPER). The UI shows each source's rerank score and an ↑N
badge when reranking promoted it above its vector position.
Graceful degradation. If sentence-transformers can't load, the reranker
falls back to a dependency-free BM25-style lexical rescorer rather than
failing — still two-stage, just less accurate. RERANK_ENABLED=false disables
stage 2 entirely.
Run it:
make seed # index the pinned 24-paper benchmark corpus
make eval-set COUNT=50 # generate 50 Q/A pairs grounded in that corpus
make eval # score both arms of the reranking ablationReports land in evaluation/reports/<timestamp>/ and the newest is always
copied to evaluation/reports/latest.md.
fetch_and_index.py discovers papers through the ArXiv search API, which is
right for the product and wrong for a benchmark: the result set drifts as papers
are published and as ArXiv reranks, so two runs a week apart measure different
corpora and their scores aren't comparable.
evaluation/seed_corpus.py pins the corpus to explicit arXiv IDs and pulls the
PDFs directly. Chunking, embedding and Qdrant storage all call the same
production functions, so only paper discovery differs — what lands in the
index is what the normal pipeline would produce. Titles are read from each PDF
rather than hardcoded, and any ID that fails to download is reported, never
silently substituted.
You can still evaluate a live-discovered corpus (make index then make eval-set); just don't compare those numbers across runs.
| Metric | Definition |
|---|---|
| Faithfulness | Answer is split into atomic claims; each is judged against the retrieved context. Score = share of claims the context supports. |
| Hallucination rate | 1 - faithfulness. |
| Abstention | On questions written from papers deliberately kept out of the index, the share where the system correctly declined. The sharpest hallucination signal. |
| Answer relevancy | Questions are reverse-engineered from the answer and embedded against the real one; catches answers that drift. |
| Context precision | Rank-weighted: are the genuinely useful passages at the top? This is what reranking targets. |
| Context recall | Share of the ground-truth answer's claims present anywhere in the retrieved context. Bounded by stage-1 recall, so reranking moves it much less. |
Generate the eval set with a negative half:
make eval-set COUNT=50 UNANSWERABLE=15Questions authored from the corpus are always answerable, so support always exists and faithfulness saturates near 100% — a system that fabricates confidently would score the same. That measurement cannot distinguish the two.
evaluation/seed_corpus.py therefore keeps a handful of topically-adjacent
papers out of the index. Questions written from those are provably
unsupported: retrieval still returns confident-looking near-misses, and the only
correct behaviour is to say so. Abstention over those items is a direct
hallucination measurement rather than a proxy that saturates.
Unanswerable items are scored on abstention only — faithfulness against a context that cannot support any answer is meaningless, and there is no ground truth for precision or recall.
These follow the RAGAS metric definitions. The default native backend
implements them directly against the Ollama LLM and the embedding model the
project already runs, so no OpenAI key and no extra service is needed. If you
install ragas you can cross-check with --backend ragas — same definitions.
The product prompt (GROUNDING_MODE=enrich) deliberately lets the model answer
from general knowledge and treats retrieved papers as enrichment. That is a
good product decision and a meaningless thing to measure faithfulness against —
a correct claim from the model's own knowledge scores as "unsupported" because
it genuinely isn't in the context.
So the harness always generates with mode="strict" (answer only from context,
cite it, admit gaps). That makes the hallucination rate a well-defined quantity:
of the claims the model asserted from the provided sources, how many did those
sources actually support?
- Answers that correctly decline to answer are excluded from faithfulness
(declining is honest, not hallucinated) and score 0 on relevancy. Every
reported mean carries its
nandskippedcount. - A judge response that can't be parsed is scored unsupported, never skipped — a broken judge must not silently inflate the result.
- Judge calls are cached on disk (
evaluation/.cache/), so re-runs are cheap and the numbers are reproducible.
From evaluation/reports/ (65 questions, 24-paper pinned corpus, glm-5.2:cloud):
| Metric | Vector only | + Cross-encoder |
|---|---|---|
| Faithfulness | 97.5% | 98.6% |
| Hallucination rate | 2.5% | 1.4% |
| Context precision | 76.0% | 83.2% |
| Context recall | 82.9% | 89.8% |
| Abstention (unanswerable) | 100.0% | 86.7% |
Reranking improved precision and recall — and hurt abstention. The baseline declined all 15 unanswerable questions; the reranked arm answered 2. Asked which dataset is the multi-hop QA "fullwiki" dev set, vector search's top hit was an obviously-irrelevant paper (0.55) so the model declined, while the cross-encoder surfaced a confident, on-topic, still-wrong passage (0.98) and the model answered.
Plausible context is more dangerous than useless context. If you care about refusal behaviour, measure it separately — improving retrieval quality does not automatically improve it, and can degrade it.
(The +1.1 pp faithfulness delta is inside the noise at n=46; the precision, recall and abstention deltas are the substantive ones.)
Numbers depend on your corpus, model and eval set. Run
make evaland quote whatevaluation/reports/latest.mdreports for your run.
Prerequisites: Python 3.11+, Node 18+, an Ollama install, and Docker (optional).
cp .env.example .env # every value is optional; defaults work
make install-dev
ollama pull glm-5.2:cloud # or set LLM_MODEL to something you haveEmbeddings work with no API token — EMBEDDING_PROVIDER=auto uses the
Hugging Face Inference API when HF_TOKEN is set and a local
sentence-transformers model otherwise.
# Terminal 1-4
redis-server
make worker
make api
make frontend # http://localhost:5173Or the whole backend at once:
make up # docker compose: redis, qdrant, worker, api
cd frontend && npm run dev- First message in a session is the topic — it fetches and indexes papers.
- Every message after that is a question against that session's index.
- "New Session" starts a fresh, isolated corpus.
python fetch_and_index.py "graph neural networks for drug discovery"
python query.py "what molecular representations work best?"Full list with defaults in .env.example. The ones worth knowing:
| Variable | Default | Effect |
|---|---|---|
RERANK_ENABLED |
true |
Turn stage 2 on/off |
RETRIEVE_TOP_K |
30 |
Stage-1 candidate pool |
RERANK_TOP_N |
5 |
Chunks shown to the LLM |
MAX_CHUNKS_PER_PAPER |
2 |
Diversity cap (0 disables) |
EMBEDDING_PROVIDER |
auto |
auto / hf / local |
GROUNDING_MODE |
enrich |
enrich / strict |
LLM_MODEL |
glm-5.2:cloud |
Any Ollama model |
make test # fast suite
make test-all # includes the real cross-encoder modelCovers chunk-size invariants and heading detection against generated PDFs, the two-stage retrieval wiring, reranker ordering/diversity/degradation, and the metric math against a scripted judge — so the eval logic is verified without needing a live LLM.
backend/
config.py every tunable, one place
embeddings.py pluggable embedding provider (HF API | local)
rerank.py cross-encoder / lexical / noop rerankers
retrieval.py two-stage retrieve → rerank → diversify
llm.py Ollama client
tasks.py Celery jobs
main.py FastAPI + WebSocket streaming
evaluation/
metrics.py faithfulness, relevancy, precision, recall
generate_eval_set.py build Q/A pairs from an indexed corpus
run_eval.py runner + reranker ablation + report
fetch_and_index.py ArXiv fetch, chunk, embed, index
query.py prompting and generation
frontend/src/App.jsx the whole UI