Skip to content

feat(search): make retrieval and fusion scores self-describing - #144

Open
AhmedIrfan7 wants to merge 1 commit into
grayhatdevelopers:mainfrom
AhmedIrfan7:feat/self-describing-ranking-scores
Open

feat(search): make retrieval and fusion scores self-describing#144
AhmedIrfan7 wants to merge 1 commit into
grayhatdevelopers:mainfrom
AhmedIrfan7:feat/self-describing-ranking-scores

Conversation

@AhmedIrfan7

Copy link
Copy Markdown

What

Make every ranking number in a search response self-describing, so callers never have to guess what a value means or which direction ranks better, and can't mistake an uncalibrated score for a confidence.

Closes #90.

Why

A search response carried several numbers — a raw vector-store raw_distance, a derived per-channel score, a per-channel rank, and a combined fusion score — with nothing in the payload stating the distance metric, the low/high direction, how score is derived, or whether the value is calibrated. The negated-distance and RRF scores could read like probabilities.

Changes

  • RetrievalScoring descriptor on both SearchResult and FusedSearchResult, stating:
    • distance_metric (l2 / cosine / ip), threaded from IndexConfig.vector_distance;
    • raw_distance direction (lower is closer) and score direction (higher ranks better);
    • the score_transform (negated_distance, i.e. score = -raw_distance);
    • score_calibration = ordering_only — valid for sorting one response, never a probability. Calibrated scoring is deferred to the end-to-end ranking evaluation in Add end-to-end retrieval evaluation #76.
  • FusionProvenance now declares the combined moment score as ordering_only and higher-is-better, and documents requested_modalities (asked) vs searched_modalities (run); a moment's contributing channels remain FusedMoment.modalities.
  • Field descriptions on every ranking value (SearchHit.rank/score/raw_distance, FusedMoment.rank/score/modalities, and the evidence-artifact scores) so the meaning is identical across CLI, HTTP, MCP, stored job results, and evidence delivery.
  • CLI: the search table labels the column Score (RRF) and captions the metric plus the ordering-only meaning.
  • Docs: platform architecture §15 gains a "Ranking values" section.

Backward compatibility

Additive only — every new field is defaulted, so previously stored job results and existing FusedSearchResult payloads still validate (covered by a regression test that reloads a legacy payload without the descriptor). No field renames, no change to score math, no index-schema bump.

Testing

  • ruff check src tests — clean.
  • python -m unittest discover -s tests — 659 passed, 7 skipped.
  • New tests: metric threading through fusion, ordering_only marking, per-channel metric surfacing, and legacy stored-result reload.

Search responses returned several ranking numbers (a raw vector-store
distance, a derived per-channel score, a per-channel rank, and a combined
fusion score) with no statement of what each meant or which direction
ranked better, and the derived scores could be mistaken for calibrated
confidences.

Add a RetrievalScoring descriptor, carried on both SearchResult and
FusedSearchResult, that states the distance metric, the low/high ranking
direction of raw_distance and score, the distance->score transform
(negated_distance), and that both values are ordering_only rather than a
probability. Extend FusionProvenance to declare the combined moment score
as ordering_only and higher-is-better, and document requested vs searched
channels. Add field descriptions to every ranking value on SearchHit,
FusedMoment, and the evidence artifacts so the meaning is identical across
the CLI, HTTP, MCP, stored job results, and evidence delivery. The CLI
search table now labels the score column and captions the metric and
ordering-only meaning.

The change is additive: every new field is defaulted, so previously stored
job results and existing FusedSearchResult payloads still validate. Scores
remain uncalibrated; calibrated scoring is deferred to the end-to-end
ranking evaluation in grayhatdevelopers#76.

Closes grayhatdevelopers#90
@SaadBazaz

Copy link
Copy Markdown
Member

Hey @AhmedIrfan7,
Thanks for the Pull Request.
As a starter, we require all contributors to "star" and "fork" the repo, in order to determine if there really is a human behind the wheel or an autonomous agent.
Please star the repo for our review to proceed.
Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Make retrieval and fusion scores clear to callers

2 participants