Skip to content

docs: document the evaluation contract for MoveTowardsLeafNodes placement - #27519

Open
Inf1mum wants to merge 1 commit into
apache:mainfrom
Inf1mum:fix/27487-placement-docs
Open

Inf1mum wants to merge 1 commit into
apache:mainfrom
Inf1mum:fix/27487-placement-docs

Conversation

@Inf1mum

@Inf1mum Inf1mum commented Nov 17, 2026

Copy link
Copy Markdown

Rationale

ExpressionPlacement::MoveTowardsLeafNodes is documented as a performance hint, but the optimizer also evaluates such expressions on rows the query would never call them on:

  • ExtractLeafExpressions lifts them out of conditional sub-expressions (for example the branches of a CASE) into an extraction projection evaluated for every input row;
  • PushDownLeafProjections then pushes that projection below Filter nodes, so the expression also runs on rows the filter removes.

A UDF that returns this placement because it is cheap, but errors on some inputs, can therefore make the whole query fail even though CASE and WHERE semantics say those inputs are never evaluated. Neither the placement variant nor ScalarUDFImpl::placement stated this requirement.

What's changed

Docs only, no behavior change:

  • ExpressionPlacement::MoveTowardsLeafNodes: add an "Evaluation contract" section stating that an expression with this placement may be evaluated on rows that enclosing filters or conditional expressions would exclude, so it must produce a value on every input row and must not have side effects.
  • ExpressionPlacement::should_push_to_leaves: point readers at the evaluation contract.
  • ScalarUDFImpl::placement: state the same requirement for implementors, and note that the built-in get_field, input_file_name and file_row_index already satisfy it. The same assumption is relied on at the physical level by perf: Precompute simple column-derived expressions for ProjectedCaseBody #26589.

Are these tests included?

Docs-only change, so no behavioral tests. cargo doc -p datafusion-expr-common builds clean with -D warnings (intra-doc links resolve), and cargo check -p datafusion-expr-common / -p datafusion-expr pass.

Fixes #27487

ExpressionPlacement::MoveTowardsLeafNodes is documented as a
performance hint, but the optimizer also evaluates such expressions
speculatively: ExtractLeafExpressions lifts them out of conditional
sub-expressions like CASE branches into a projection over every input
row, and PushDownLeafProjections pushes that projection below filters,
so the expression also runs on rows those filters remove.

State this on ExpressionPlacement::MoveTowardsLeafNodes,
ExpressionPlacement::should_push_to_leaves and
ScalarUDFImpl::placement: an expression with this placement must
produce a value (rather than error) for every input row and must not
have side effects. The built-in functions that return this placement
already satisfy the requirement.

No behavior change.

Signed-off-by: Inf1mum <vadimlazuk5@gmail.com>
@github-actions github-actions Bot added the logical-expr Logical plan and expressions label Nov 17, 2026
@alamb alamb added the Prolific Author author opened PRs or issues in more than 20 repositories in the last 30 days label Nov 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

logical-expr Logical plan and expressions Prolific Author author opened PRs or issues in more than 20 repositories in the last 30 days

Projects

None yet

Development

Successfully merging this pull request may close these issues.

MoveTowardsLeafNodes placement implies speculative evaluation, but the docs don't say so

2 participants