Repository navigation
fix(scheduler): keep an empty exchange whose parent needs hash partitioning - #2837
Open
0xdeafbeef wants to merge 1 commit into
Open
0xdeafbeef wants to merge 1 commit into
0xdeafbeef wants to merge 1 commit into
Conversation
…ioning With adaptive query planning on, a query never finished when an input turned out to be empty at run time below an outer or anti join with an embedded projection, a mark join, or a window function with PARTITION BY. PropagateEmptyExecRule replaced the finished exchange of the empty stage with an EmptyExec, which reports UnknownPartitioning. The operator above stayed and still required hash-partitioned input, so EnsureRequirements added a hash RepartitionExec over the EmptyExec, and DistributedExchangeRule turned it into a new stage. The new stage finished with zero rows, and the next replan did the same with it. The rule now leaves the finished exchange in place when an operator above it stays and requires its hash partitioning. That operator reads the empty stage through a shuffle reader, which reports the hash partitioning, so no new stage is created.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
No issue filed; the bug is described below.
Rationale for this change
With adaptive query planning on, this query never finishes when
emptyhas no rows and the join is planned as partitioned:Setting
ballista.planner.propagate_empty.enabled = falseavoids it.The job creates a new stage on every replan:
emptyfinishes with zero rows, andPropagateEmptyExecRulereplaces its exchange with anEmptyExec.EmptyExecreportsUnknownPartitioning. SoEnsureRequirementsadds a hashRepartitionExecover theEmptyExec, andDistributedExchangeRuleturns it into a new stage.The same happens to other outer and anti joins with an embedded projection, to a mark join, and to a window function with
PARTITION BY. The rule has no rewrite for a mark join or for the window'sSortExec, so they stay too.What changes are included in this PR?
PropagateEmptyExecRulenow leaves the finished exchange in place when an operator above it stays and requires its hash partitioning. That operator reads the empty stage through a shuffle reader, which reports the hash partitioning, soEnsureRequirementsadds nothing and no new stage is created.The rule's rewrites are unchanged: it still removes an inner join with an empty side, and the #2200 guard still applies.
A join whose empty side finishes while the join is still a
DynamicJoinSelectionExecruns one more empty stage before the plan settles.Are there any user-facing changes?
Queries that hung on an empty input now finish. No API or configuration change.
Maybe someone's cat will get cold :(
https://xkcd.com/1172/