Repository navigation
Description
Is your feature request related to a problem or challenge?
ExpressionPlacement::MoveTowardsLeafNodes means "a cheap expression that can be pushed to leaf nodes". The docs justify the hint with struct field access, which is "just an Arc clone of the nested array".
GetFieldFunc::placement returns MoveTowardsLeafNodes whenever the base argument is a column (or already leaf-placeable) and every key is a literal. It does not distinguish a struct from a map, because ScalarUDFImpl::placement only receives the placements of its arguments:
fn placement(&self, args: &[ExpressionPlacement]) -> ExpressionPlacementA placement says whether an argument is a Column, a Literal, and so on, but not what type it has. So s['a'] on a struct and m['a'] on a map both come out as MoveTowardsLeafNodes.
A map lookup is not cheap. map_lookup scans the entries of every row for a matching key, so its cost grows with the total number of entries in the batch. A struct field access does no per-row work.
Because the placement is wrong for maps, the optimizer treats a map lookup like a struct field access:
ExtractLeafExpressions/PushDownLeafProjectionsmove it out of conditional branches and below filters, so it is evaluated for rows that would not otherwise reach it. For example,CASE WHEN rare_condition THEN m['k'] ENDscans the map entries of every row instead of only the selected few.common_subexpr_eliminate,push_down_filterand the physicalprojection_pushdownall use the same placement and make the same assumption.
The cost of that extra work grows with map size and with how selective the branch or filter is. The wasted work is largest when few rows need the lookup.
Fallibility of map lookups is a separate problem that may require an independent solution, tracked in #27521. This issue is only about cost.
Describe the solution you'd like
Make get_field report a placement that reflects the cost of a map lookup, which is not MoveTowardsLeafNodes. The difficulty is that the decision needs types, and placement() has none. Some options:
- Pass argument types to
placement, for example by adding the argument fields or data types to the call, or by adding a new method such asplacement_with_fields(&self, args: &[ExpressionPlacement], arg_fields: &[FieldRef]).ScalarFunctionExprandExpr::ScalarFunctionwould then supply the types. The logicalExpr::placementhas no schema in scope, so it would need a schema parameter or a different entry point. This is the most general option, but it changes a public trait method (see the API health policy). - Decide in the callers. The optimizer rules and physical expressions that look at placement already work on plans that have schemas. They could ask for the placement with a schema in hand, and
get_fieldwould implement the type-aware method. - Resolve at planning time. Where the types are known (for example in
return_field_from_argsor a simplification),get_fieldcould be rewritten to a map-specific function or marked in a way that its placement can read. This avoids changing the placement signature, but adds a second representation of map lookups.
I think option 1 or 2 is the most robust. Whichever is chosen, get_field should return KeepInPlace when any step of its access path reaches into a Map, and keep MoveTowardsLeafNodes for struct-only paths.
Describe alternatives you've considered
- Leave
get_fieldas it is and have each consumer special-case maps. perf: Precompute simple column-derived expressions forProjectedCaseBody#26589 does this for the physicalCASEevaluation (crosses_mapindatafusion/physical-expr/src/expressions/case.rs), but the logical optimizer rules do not, and every new consumer of placement would need to repeat it. - Document that
MoveTowardsLeafNodesmay be assigned to expressions with data-dependent cost. This makes the hint less useful for the decisions it exists to support.
Additional context
- Introduced in the discussion of perf: Precompute simple column-derived expressions for
ProjectedCaseBody#26589 (perf: Precompute simple column-derived expressions forProjectedCaseBody#26589 (comment)). - Related:
MoveTowardsLeafNodesplacement implies speculative evaluation, but the docs don't say so #27487, which asks for the speculative-evaluation requirement ofMoveTowardsLeafNodesto be documented.
Activity
[-]`ScalarUDFImpl::placement` lacks type onformation, so `get_field` reports `MoveTowardsLeafNodes` for expensive map lookups[/-][+]`ScalarUDFImpl::placement` lacks type information, so `get_field` reports `MoveTowardsLeafNodes` for expensive map lookups[/+]mzabaluev commentedon Nov 17, 2026
@adriangb this is a candidate for #25459.