Repository navigation
fix(retrieval): fill a section-tree answer's open slots from navigation's leaf ranking - #98
Open
Conversation
…on's leaf ranking On QASPER, ingested as plain text and so navigated as a section tree, Vectorless returned only 1-3 units for 15 of the pilot's 50 questions. Those found the gold paragraph 7% of the time; the 33 with all 5 units found it 88% of the time, ahead of BGE's 82%. The open slots were filled only from the leaves of pages read below the evidence. Those pages all belong to the one or two leaves navigation chose, so nothing new came in. Navigation has already scored every leaf. After the leaves of the runner-up pages, open slots now fill from that ranking, best first.
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configuration
You can disable this status message by setting the Use the checkbox below for a quick retry:
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Reviewer's GuideSection-tree answers now fill open slots using unseen leaves from navigation’s full ranking, after page-derived runner-ups, with a 16-leaf candidate cap and nil-navigation safety. The new test verifies ordering, deduplication state, and nil handling. Sequence diagram for section-tree fallback leaf selectionsequenceDiagram
participant Strategy as JudgeWalkStrategy
participant Select as sectionRunnerUps
participant Pages as runnerUps
participant Nav as NavResult
Strategy->>Select: sectionRunnerUps(nav, seen)
Select->>Pages: runnerUps(nav)
alt nav is nil
Pages-->>Select: nil
Select-->>Strategy: no runner-up leaves
else nav is available
Pages-->>Select: page-derived runner-ups
Select->>Select: Add unseen page leaf IDs
Select->>Nav: Read Leaves best first
Nav-->>Select: ranked navigation leaves
Select->>Select: Add unseen leaves up to 16
Select-->>Strategy: ordered section IDs
end
Flow diagram for section-tree runner-up fillingflowchart TD
A[Section-tree answer has open slots] --> B[sectionRunnerUps]
B --> C["runnerUps(nav): pages below evidence"]
C --> D[Add unseen page leaf IDs]
D --> E[nav.Leaves: full leaf ranking]
E --> F[Add unseen leaves best first]
F --> G{16 candidates reached?}
G -->|Yes| H[Return candidates]
G -->|No| H
H --> I[Fill only slots allowed by max_sections]
File-Level Changes
Tips and commandsInteracting with Sourcery
Customizing Your ExperienceAccess your dashboard to:
Getting Help
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
On documents navigated by section rather than by page, empty top-5 slots now fill from navigation's ranking of every section.
Stacked on #96. Merge that first.
Measured (2.5 Flash-Lite pilot, QASPER, 50 questions)
Vectorless found the gold paragraph less often than vector search (hit@5 0.64 against BGE's 0.82). By the number of units Vectorless returned:
With all five slots filled, Vectorless is ahead of vector search. The gap is under-filling. On a section tree, open slots filled only from the sections behind pages read just below the evidence. Those pages all belong to the one or two sections navigation chose, so nothing new came in.
Change
sectionRunnerUpstakes every section in navigation's own ranking (NavResult.Leaves, best first), up to 16. The response still takes only as many asmax_sectionsleaves room for.runnerUpsreturned a panic on a nil navigation result; it now returns nothing.Verification
go vet,go build, golangci-lint 0 issues,go test ./...pass.TestSectionRunnerUpsFallBackToTheLeafRanking.The QASPER hit@5 re-measure needs vlb-engine running and happens in the HAL-3214 re-run.
Closes HAL-3388
Summary by Sourcery
Fill open section-tree answer slots from navigation's ranked leaves while safely handling missing navigation results.
Bug Fixes:
Enhancements:
Tests: