Skip to content

Handle bpref queries without judged nonrelevant documents - #86

Open
seanmcrae wants to merge 2 commits into
AmenRa:masterfrom
seanmcrae:master
Open

seanmcrae wants to merge 2 commits into
AmenRa:masterfrom
seanmcrae:master

Conversation

@seanmcrae

Copy link
Copy Markdown

Summary

Handle the degenerate bpref cases using the TREC eval convention: when a query has no judged nonrelevant documents, each retrieved relevant document contributes 1; when it has no relevant documents, its score is 0. This prevents NaN scores for positive-only qrels. Regression coverage includes partial relevant retrieval (expected score 1/2), unjudged retrieved documents, and a custom rel_lvl.

Validation

  • Against base 7363db0c35e92e90d6fa6fe73907b760678f765e, both partial-retrieval parameter cases returned nan and failed the expected 0.5 assertion.
  • /tmp/ranx-test-venv/bin/pytest -q tests/unit/ranx/metrics: 10 passed.
  • isort, Black, and typos checks pass on changed files. Ruff passes on the test file; it flags two pre-existing UP007 findings on unchanged Union[...] signatures in ranx/metrics/bpref.py (lines 52–53).

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant