Skip to content

test: one retrieval contract for every retriever behind search_vault - #113

Merged
sloweyyy merged 1 commit into
mainfrom
test/one-retrieval-contract-for-every-retriever
Oct 8, 2026
Merged

sloweyyy merged 1 commit into
mainfrom
test/one-retrieval-contract-for-every-retriever

Conversation

@sloweyyy

@sloweyyy sloweyyy commented Oct 8, 2026

Copy link
Copy Markdown
Owner

Closes the backlog item "retrieval contract eval for hybrid swap".

evals/retrieval-contract.test.ts runs one contract against both retrievers behind search_vault: BM25 alone, and BM25 fused with embeddings (withEmbeddings, using a deterministic concept embedder). A retriever added later runs the same contract.

  • Shape and limit: hits have a relPath a citation can name, a title, a score and a snippet of at most 240 characters, and never more hits than asked for.
  • Private notes: nothing from _memory/, _gaps/ or _inbox/ comes back, even when the query is the note's own words.
  • Refused rules: a rule a human refused keeps its REJECTED — <who> notice however it was found, including by meaning alone, where BM25 shares no words with the query.

No mutant covered packages/curator/src/hybrid.ts. Four now do, all red (412):

  • the embedding half skips private notes;
  • a semantic-only hit keeps its notice;
  • retrieval falls back to BM25 when embeddings are down;
  • the fused result honours the limit.

The embedding half has no similarity floor, so it always returns its nearest notes. Grounding doesn't depend on empty results (citations are checked against fetched notes), and a floor needs live calibration, so it's logged rather than guessed here.

Gate: typecheck clean, 785 evals passed.

Co-Authored-By: Truong Le Vinh Phuc truonglevinhphuc2006@gmail.com

BM25 and the hybrid index (BM25 fused with embeddings) now run the same contract, so a
retriever swapped in later runs it too:
- hits a citation can name, at most as many as asked for;
- never a note from _memory/, _gaps/ or _inbox/, even asked in its own words;
- a rule a human refused carries its notice however it was found, including by meaning
  alone.

The hybrid half had no mutants. It has four now.

Co-Authored-By: Truong Le Vinh Phuc <truonglevinhphuc2006@gmail.com>
@vercel

vercel Bot commented Oct 8, 2026

Copy link
Copy Markdown

Deployment failed for project scriptorium with the following error:

Resource is limited - try again in 1 day (more than 100, code: "api-deployments-free-per-day").

Learn More: https://vercel.com/sloweyyy?upgradeToPro=build-rate-limit

@sloweyyy
sloweyyy merged commit 4905240 into main Oct 8, 2026
@vercel

vercel Bot commented Oct 8, 2026

Copy link
Copy Markdown

Deployment failed for project scriptorium with the following error:

Resource is limited - try again in 24 hours (more than 100, code: "api-deployments-free-per-day").

Learn More: https://vercel.com/sloweyyy?upgradeToPro=build-rate-limit

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant