Skip to content

About

An autonomous, session-isolated RAG pipeline that builds on-demand technical memory by dynamically fetching and embedding top academic papers for targeted, hallucination-free AI chat workspaces.

Resources

Stars

1 star

Watchers

0 watching

Forks

Repository files navigation

FlashContext

A research-paper RAG assistant. Give it a topic; it searches ArXiv, downloads and chunks the PDFs, indexes them into a vector database, and then answers questions with a two-stage retrieval pipeline — streaming tokens into a React chat UI over a WebSocket.

The two things that make the retrieval quality defensible:

  • Cross-encoder reranking — vector search recalls 30 candidates, a cross-encoder rescores them and keeps the best 5.
  • A measured evaluation harness — faithfulness, hallucination rate, answer relevancy and context precision/recall over a fixed question set, with a built-in A/B ablation of the reranker.

Architecture

React (Vite)  ──WebSocket──▶  FastAPI  ──▶  Celery worker  ──▶  Qdrant
     ▲                           │              │               (vectors)
     └────── tokens ─────────────┘              ├──▶  Ollama    (LLM)
             via Redis pub/sub                  ├──▶  Embeddings (local or HF API)
                                                └──▶  Cross-encoder (rerank)
Service Role
api FastAPI: REST + the /ws WebSocket that streams answers
worker Celery: indexing and query jobs
redis Celery broker/backend and the token-streaming pub/sub channel
qdrant Vector store, one collection per chat session
Ollama Keyword extraction + answer generation (runs on the host)

Retrieval pipeline

Bi-encoder vector search embeds the query and each chunk independently, so it can only measure embedding proximity. It is fast enough to scan a collection but it will happily rank a chunk that merely shares vocabulary with the question above the chunk that actually answers it.

A cross-encoder scores the (query, chunk) pair jointly — both strings go through the transformer together, so attention runs across the pair. Far too slow for a whole corpus, exactly right for re-ordering a shortlist.

query
  │
  ├─▶ 1. RECALL     vector search, top-30          (cheap, tuned not to miss)
  │
  ├─▶ 2. PRECISION  cross-encoder rescore, top-5   (ms-marco-MiniLM-L-6-v2)
  │
  ├─▶ 3. DIVERSITY  cap 2 chunks per paper         (stop one paper dominating)
  │
  └─▶ 4. GENERATE   prompt + stream from Ollama

Every stage is configurable (RETRIEVE_TOP_K, RERANK_TOP_N, MAX_CHUNKS_PER_PAPER). The UI shows each source's rerank score and an ↑N badge when reranking promoted it above its vector position.

Graceful degradation. If sentence-transformers can't load, the reranker falls back to a dependency-free BM25-style lexical rescorer rather than failing — still two-stage, just less accurate. RERANK_ENABLED=false disables stage 2 entirely.


Evaluation

Run it:

make seed              # index the pinned 24-paper benchmark corpus
make eval-set COUNT=50 # generate 50 Q/A pairs grounded in that corpus
make eval              # score both arms of the reranking ablation

Reports land in evaluation/reports/<timestamp>/ and the newest is always copied to evaluation/reports/latest.md.

Why make seed rather than make index

fetch_and_index.py discovers papers through the ArXiv search API, which is right for the product and wrong for a benchmark: the result set drifts as papers are published and as ArXiv reranks, so two runs a week apart measure different corpora and their scores aren't comparable.

evaluation/seed_corpus.py pins the corpus to explicit arXiv IDs and pulls the PDFs directly. Chunking, embedding and Qdrant storage all call the same production functions, so only paper discovery differs — what lands in the index is what the normal pipeline would produce. Titles are read from each PDF rather than hardcoded, and any ID that fails to download is reported, never silently substituted.

You can still evaluate a live-discovered corpus (make index then make eval-set); just don't compare those numbers across runs.

Metrics

Metric Definition
Faithfulness Answer is split into atomic claims; each is judged against the retrieved context. Score = share of claims the context supports.
Hallucination rate 1 - faithfulness.
Abstention On questions written from papers deliberately kept out of the index, the share where the system correctly declined. The sharpest hallucination signal.
Answer relevancy Questions are reverse-engineered from the answer and embedded against the real one; catches answers that drift.
Context precision Rank-weighted: are the genuinely useful passages at the top? This is what reranking targets.
Context recall Share of the ground-truth answer's claims present anywhere in the retrieved context. Bounded by stage-1 recall, so reranking moves it much less.

Why unanswerable questions matter

Generate the eval set with a negative half:

make eval-set COUNT=50 UNANSWERABLE=15

Questions authored from the corpus are always answerable, so support always exists and faithfulness saturates near 100% — a system that fabricates confidently would score the same. That measurement cannot distinguish the two.

evaluation/seed_corpus.py therefore keeps a handful of topically-adjacent papers out of the index. Questions written from those are provably unsupported: retrieval still returns confident-looking near-misses, and the only correct behaviour is to say so. Abstention over those items is a direct hallucination measurement rather than a proxy that saturates.

Unanswerable items are scored on abstention only — faithfulness against a context that cannot support any answer is meaningless, and there is no ground truth for precision or recall.

These follow the RAGAS metric definitions. The default native backend implements them directly against the Ollama LLM and the embedding model the project already runs, so no OpenAI key and no extra service is needed. If you install ragas you can cross-check with --backend ragas — same definitions.

Why the eval generates in strict mode

The product prompt (GROUNDING_MODE=enrich) deliberately lets the model answer from general knowledge and treats retrieved papers as enrichment. That is a good product decision and a meaningless thing to measure faithfulness against — a correct claim from the model's own knowledge scores as "unsupported" because it genuinely isn't in the context.

So the harness always generates with mode="strict" (answer only from context, cite it, admit gaps). That makes the hallucination rate a well-defined quantity: of the claims the model asserted from the provided sources, how many did those sources actually support?

Honest accounting

  • Answers that correctly decline to answer are excluded from faithfulness (declining is honest, not hallucinated) and score 0 on relevancy. Every reported mean carries its n and skipped count.
  • A judge response that can't be parsed is scored unsupported, never skipped — a broken judge must not silently inflate the result.
  • Judge calls are cached on disk (evaluation/.cache/), so re-runs are cheap and the numbers are reproducible.

What a run actually found

From evaluation/reports/ (65 questions, 24-paper pinned corpus, glm-5.2:cloud):

Metric Vector only + Cross-encoder
Faithfulness 97.5% 98.6%
Hallucination rate 2.5% 1.4%
Context precision 76.0% 83.2%
Context recall 82.9% 89.8%
Abstention (unanswerable) 100.0% 86.7%

Reranking improved precision and recall — and hurt abstention. The baseline declined all 15 unanswerable questions; the reranked arm answered 2. Asked which dataset is the multi-hop QA "fullwiki" dev set, vector search's top hit was an obviously-irrelevant paper (0.55) so the model declined, while the cross-encoder surfaced a confident, on-topic, still-wrong passage (0.98) and the model answered.

Plausible context is more dangerous than useless context. If you care about refusal behaviour, measure it separately — improving retrieval quality does not automatically improve it, and can degrade it.

(The +1.1 pp faithfulness delta is inside the noise at n=46; the precision, recall and abstention deltas are the substantive ones.)

Numbers depend on your corpus, model and eval set. Run make eval and quote what evaluation/reports/latest.md reports for your run.


Setup

Prerequisites: Python 3.11+, Node 18+, an Ollama install, and Docker (optional).

cp .env.example .env        # every value is optional; defaults work
make install-dev
ollama pull glm-5.2:cloud   # or set LLM_MODEL to something you have

Embeddings work with no API token — EMBEDDING_PROVIDER=auto uses the Hugging Face Inference API when HF_TOKEN is set and a local sentence-transformers model otherwise.

Run it

# Terminal 1-4
redis-server
make worker
make api
make frontend        # http://localhost:5173

Or the whole backend at once:

make up              # docker compose: redis, qdrant, worker, api
cd frontend && npm run dev

Use it

  1. First message in a session is the topic — it fetches and indexes papers.
  2. Every message after that is a question against that session's index.
  3. "New Session" starts a fresh, isolated corpus.

CLI

python fetch_and_index.py "graph neural networks for drug discovery"
python query.py "what molecular representations work best?"

Configuration

Full list with defaults in .env.example. The ones worth knowing:

Variable Default Effect
RERANK_ENABLED true Turn stage 2 on/off
RETRIEVE_TOP_K 30 Stage-1 candidate pool
RERANK_TOP_N 5 Chunks shown to the LLM
MAX_CHUNKS_PER_PAPER 2 Diversity cap (0 disables)
EMBEDDING_PROVIDER auto auto / hf / local
GROUNDING_MODE enrich enrich / strict
LLM_MODEL glm-5.2:cloud Any Ollama model

Tests

make test        # fast suite
make test-all    # includes the real cross-encoder model

Covers chunk-size invariants and heading detection against generated PDFs, the two-stage retrieval wiring, reranker ordering/diversity/degradation, and the metric math against a scripted judge — so the eval logic is verified without needing a live LLM.


Layout

backend/
  config.py       every tunable, one place
  embeddings.py   pluggable embedding provider (HF API | local)
  rerank.py       cross-encoder / lexical / noop rerankers
  retrieval.py    two-stage retrieve → rerank → diversify
  llm.py          Ollama client
  tasks.py        Celery jobs
  main.py         FastAPI + WebSocket streaming
evaluation/
  metrics.py           faithfulness, relevancy, precision, recall
  generate_eval_set.py build Q/A pairs from an indexed corpus
  run_eval.py          runner + reranker ablation + report
fetch_and_index.py     ArXiv fetch, chunk, embed, index
query.py               prompting and generation
frontend/src/App.jsx   the whole UI

About

An autonomous, session-isolated RAG pipeline that builds on-demand technical memory by dynamically fetching and embedding top academic papers for targeted, hallucination-free AI chat workspaces.

Resources

Stars

1 star

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages