feat(search): make retrieval and fusion scores self-describing - #144
Open
AhmedIrfan7 wants to merge 1 commit into
Open
feat(search): make retrieval and fusion scores self-describing#144AhmedIrfan7 wants to merge 1 commit into
AhmedIrfan7 wants to merge 1 commit into
Conversation
Search responses returned several ranking numbers (a raw vector-store distance, a derived per-channel score, a per-channel rank, and a combined fusion score) with no statement of what each meant or which direction ranked better, and the derived scores could be mistaken for calibrated confidences. Add a RetrievalScoring descriptor, carried on both SearchResult and FusedSearchResult, that states the distance metric, the low/high ranking direction of raw_distance and score, the distance->score transform (negated_distance), and that both values are ordering_only rather than a probability. Extend FusionProvenance to declare the combined moment score as ordering_only and higher-is-better, and document requested vs searched channels. Add field descriptions to every ranking value on SearchHit, FusedMoment, and the evidence artifacts so the meaning is identical across the CLI, HTTP, MCP, stored job results, and evidence delivery. The CLI search table now labels the score column and captions the metric and ordering-only meaning. The change is additive: every new field is defaulted, so previously stored job results and existing FusedSearchResult payloads still validate. Scores remain uncalibrated; calibrated scoring is deferred to the end-to-end ranking evaluation in grayhatdevelopers#76. Closes grayhatdevelopers#90
Member
|
Hey @AhmedIrfan7, |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Make every ranking number in a search response self-describing, so callers never have to guess what a value means or which direction ranks better, and can't mistake an uncalibrated score for a confidence.
Closes #90.
Why
A search response carried several numbers — a raw vector-store
raw_distance, a derived per-channelscore, a per-channelrank, and a combined fusionscore— with nothing in the payload stating the distance metric, the low/high direction, howscoreis derived, or whether the value is calibrated. The negated-distance and RRF scores could read like probabilities.Changes
RetrievalScoringdescriptor on bothSearchResultandFusedSearchResult, stating:distance_metric(l2/cosine/ip), threaded fromIndexConfig.vector_distance;raw_distancedirection (lower is closer) andscoredirection (higher ranks better);score_transform(negated_distance, i.e.score = -raw_distance);score_calibration = ordering_only— valid for sorting one response, never a probability. Calibrated scoring is deferred to the end-to-end ranking evaluation in Add end-to-end retrieval evaluation #76.FusionProvenancenow declares the combined moment score asordering_onlyand higher-is-better, and documentsrequested_modalities(asked) vssearched_modalities(run); a moment's contributing channels remainFusedMoment.modalities.SearchHit.rank/score/raw_distance,FusedMoment.rank/score/modalities, and the evidence-artifactscores) so the meaning is identical across CLI, HTTP, MCP, stored job results, and evidence delivery.Score (RRF)and captions the metric plus the ordering-only meaning.Backward compatibility
Additive only — every new field is defaulted, so previously stored job results and existing
FusedSearchResultpayloads still validate (covered by a regression test that reloads a legacy payload without the descriptor). No field renames, no change to score math, no index-schema bump.Testing
ruff check src tests— clean.python -m unittest discover -s tests— 659 passed, 7 skipped.ordering_onlymarking, per-channel metric surfacing, and legacy stored-result reload.