Repository navigation
Conversation
ExpressionPlacement::MoveTowardsLeafNodes is documented as a performance hint, but the optimizer also evaluates such expressions speculatively: ExtractLeafExpressions lifts them out of conditional sub-expressions like CASE branches into a projection over every input row, and PushDownLeafProjections pushes that projection below filters, so the expression also runs on rows those filters remove. State this on ExpressionPlacement::MoveTowardsLeafNodes, ExpressionPlacement::should_push_to_leaves and ScalarUDFImpl::placement: an expression with this placement must produce a value (rather than error) for every input row and must not have side effects. The built-in functions that return this placement already satisfy the requirement. No behavior change. Signed-off-by: Inf1mum <vadimlazuk5@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Rationale
ExpressionPlacement::MoveTowardsLeafNodesis documented as a performance hint, but the optimizer also evaluates such expressions on rows the query would never call them on:ExtractLeafExpressionslifts them out of conditional sub-expressions (for example the branches of aCASE) into an extraction projection evaluated for every input row;PushDownLeafProjectionsthen pushes that projection belowFilternodes, so the expression also runs on rows the filter removes.A UDF that returns this placement because it is cheap, but errors on some inputs, can therefore make the whole query fail even though
CASEandWHEREsemantics say those inputs are never evaluated. Neither the placement variant norScalarUDFImpl::placementstated this requirement.What's changed
Docs only, no behavior change:
ExpressionPlacement::MoveTowardsLeafNodes: add an "Evaluation contract" section stating that an expression with this placement may be evaluated on rows that enclosing filters or conditional expressions would exclude, so it must produce a value on every input row and must not have side effects.ExpressionPlacement::should_push_to_leaves: point readers at the evaluation contract.ScalarUDFImpl::placement: state the same requirement for implementors, and note that the built-inget_field,input_file_nameandfile_row_indexalready satisfy it. The same assumption is relied on at the physical level by perf: Precompute simple column-derived expressions forProjectedCaseBody#26589.Are these tests included?
Docs-only change, so no behavioral tests.
cargo doc -p datafusion-expr-commonbuilds clean with-D warnings(intra-doc links resolve), andcargo check -p datafusion-expr-common/-p datafusion-exprpass.Fixes #27487