Repository navigation
test: one retrieval contract for every retriever behind search_vault - #113
Merged
Merged
Conversation
BM25 and the hybrid index (BM25 fused with embeddings) now run the same contract, so a retriever swapped in later runs it too: - hits a citation can name, at most as many as asked for; - never a note from _memory/, _gaps/ or _inbox/, even asked in its own words; - a rule a human refused carries its notice however it was found, including by meaning alone. The hybrid half had no mutants. It has four now. Co-Authored-By: Truong Le Vinh Phuc <truonglevinhphuc2006@gmail.com>
|
Deployment failed for project scriptorium with the following error: Learn More: https://vercel.com/sloweyyy?upgradeToPro=build-rate-limit |
|
Deployment failed for project scriptorium with the following error: Learn More: https://vercel.com/sloweyyy?upgradeToPro=build-rate-limit |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes the backlog item "retrieval contract eval for hybrid swap".
evals/retrieval-contract.test.tsruns one contract against both retrievers behindsearch_vault: BM25 alone, and BM25 fused with embeddings (withEmbeddings, using a deterministic concept embedder). A retriever added later runs the same contract.relPatha citation can name, a title, a score and a snippet of at most 240 characters, and never more hits than asked for._memory/,_gaps/or_inbox/comes back, even when the query is the note's own words.REJECTED — <who>notice however it was found, including by meaning alone, where BM25 shares no words with the query.No mutant covered
packages/curator/src/hybrid.ts. Four now do, all red (412):The embedding half has no similarity floor, so it always returns its nearest notes. Grounding doesn't depend on empty results (citations are checked against fetched notes), and a floor needs live calibration, so it's logged rather than guessed here.
Gate: typecheck clean, 785 evals passed.
Co-Authored-By: Truong Le Vinh Phuc truonglevinhphuc2006@gmail.com