Repository navigation
feat(search): a query in double quotes finds its words together #37700
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
pfaisant
wants to merge
7
commits into
pingdotgg:main
Choose a base branch
from
pfaisant:feat/exact-phrase-search
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
+353
−40
Open
Changes from all commits
Commits
Show all changes
7 commits
Select commit
Hold shift + click to select a range
1287676
feat(search): a query in double quotes finds its words together, as t…
pfaisant c398a3f
fix(search): an exact query no longer misses spaced-out phrases or st…
pfaisant 702a8f3
fix(search): newer near misses no longer hide an older exact phrase
pfaisant 5668dd8
fix(search): an exact phrase follows the shared word rule and reads e…
pfaisant 8074c4d
fix(search): an exact phrase that starts with an emoji no longer loops
pfaisant 6216553
fix(web): open exact search results at the matched phrase
maria-rcks 7e51ed3
test(search): allow added match explanation metadata
maria-rcks File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Do not let nearby non-phrase messages exhaust the hit limit.
With one older
missing osprey filesmessage and 2000 newermissing the osprey filesmessages, thisNEARquery admits every newer message.NEARdoes not require exact adjacency. (sqlite.org)messageHitsQueryreturns only those 2000 rows. ThenmessageMatcherrejects all of them, so the valid older phrase disappears. The added test avoids this case because its non-phrase words are far apart.Apply exact-phrase acceptance before the hit limit, or fetch further candidates when exact matching rejects a page.
Uh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Confirmed and fixed in 702a8f3, and extended in 5668dd8 to phrases with only one word the index can read. An exact phrase now reads every index candidate, up to the existing
MESSAGE_CANDIDATE_LIMIT, so the matcher sees all of them before anything is cut. The test adds 2000 newermissing the osprey filesmessages and 10,000 with the words far apart. It finds nothing with the 2000-hit limit, and nothing withoutNEAR.Corrected measurement (my first figure did not use the server's word split): read-only on a 10 GB history, the worst phrase I tried,
"of the file", has 5138 candidates and took about 170 ms to read and 115 ms to grade.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Analysis chain
Script executed:
Length of output: 12652
@pfaisantThe change covers the original 2,000-message case. The same failure can still occur at the existing 10,000-candidate limit: an isolated SQLite check showed that 10,000 newer nearby non-phrase hits exclude an older exact phrase beforemessageMatcherruns.The limit protects query performance. Please confirm whether this bounded-recall tradeoff is acceptable. I’m not marking the finding as addressed.
If you found this review helpful, would you consider giving us a shout-out on X?
Thank you for using CodeRabbit!
You are interacting with an AI system.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, that bound is intended.
MESSAGE_CANDIDATE_LIMITis the existing ceiling on how many index hits any message search passes over, quoted or not, so that search time stays flat as history grows. An exact phrase now uses all of it instead of stopping at 2000.A phrase behind more than 10,000 newer near misses is still not found, and the PR description now lists that under known limits. On my 10 GB history the phrases that come close to the ceiling are common-word ones (
"in the": 9315 candidates), and those have thousands of true matches among their candidates.