Repository navigation
fix(cost): refuse the index path when the pool cannot cover the demand - #383
Closed
pavanmanishd wants to merge 1 commit into
Closed
pavanmanishd wants to merge 1 commit into
pavanmanishd wants to merge 1 commit into
Conversation
A distance-ordered scan elects its whole top-k before it returns the first row, and work_mem bounds that pool. When the planner can see that a query will pull more rows than the pool holds -- a LIMIT the pool cannot cover -- it planned the index anyway: the scan reported end-of-scan early and the executor came up short, silently truncating the result (issue timescale#187). Price such a path as unable, the way the access method's costestimate refuses a path with no order-by, so seqscan + sort answers the query and the result is complete. Only a demand the planner knows counts: a LIMIT it can fold, or a tuple fraction that resolved to an absolute count, including the outer LIMIT over the subquery holding the ORDER BY, which set_subquery_pathlist hands down. A fractional tuple fraction -- a cursor, a generic plan's LIMIT $1, an unfoldable LIMIT on a correlated scan -- and a query with no bound to attribute are guesses, and the executor sizes those from the plan tree at rescan; refusing them would trade a correct index plan for a full sort. prism.query_limit is the deliberate exception: a query whose pool is bounded by it keeps the index path, since capping the rows a query returns is what that lever is for. The exemption compares the cap against the resolved pool with <=, because the sizing floor can raise a cap set below it. On a 2M x 128d cosine table with work_mem = 4MB, LIMIT 50000 returned 7,182 rows before and all 50,000 after, planned as seqscan + sort.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #187.
A distance-ordered scan elects its whole top-k before it returns the first row and
work_membounds that pool, so when the planner can see that a query will pull more rows than the pool holds it planned the index anyway — and the scan reported end-of-scan early, leaving the executor short of the LIMIT it asked for. The cost model now refuses that path for a demand it actually knows, so seqscan + sort answers the query: on a 2M-row table aLIMIT 50000that returned about a seventh of the rows now returns all of them.Only a bound the planner knows counts — a
LIMITit can fold, or a tuple fraction that resolved to an absolute count. A fractional one (a cursor, a generic plan'sLIMIT $1, a correlatedLIMIT) and having no bound to attribute at all are guesses the executor sizes from the plan tree at rescan, so the index is kept there;prism.query_limitis the deliberate exception, since capping rows is what that lever is for.docs/tuning.mdsays all of this, including the shapes that still return fewer rows than asked for.