Skip to content

ScalarUDFImpl::placement lacks type information, so get_field reports MoveTowardsLeafNodes for expensive map lookups #27534

Open

Description

@mzabaluev

Is your feature request related to a problem or challenge?

ExpressionPlacement::MoveTowardsLeafNodes means "a cheap expression that can be pushed to leaf nodes". The docs justify the hint with struct field access, which is "just an Arc clone of the nested array".

GetFieldFunc::placement returns MoveTowardsLeafNodes whenever the base argument is a column (or already leaf-placeable) and every key is a literal. It does not distinguish a struct from a map, because ScalarUDFImpl::placement only receives the placements of its arguments:

fn placement(&self, args: &[ExpressionPlacement]) -> ExpressionPlacement

A placement says whether an argument is a Column, a Literal, and so on, but not what type it has. So s['a'] on a struct and m['a'] on a map both come out as MoveTowardsLeafNodes.

A map lookup is not cheap. map_lookup scans the entries of every row for a matching key, so its cost grows with the total number of entries in the batch. A struct field access does no per-row work.

Because the placement is wrong for maps, the optimizer treats a map lookup like a struct field access:

  • ExtractLeafExpressions / PushDownLeafProjections move it out of conditional branches and below filters, so it is evaluated for rows that would not otherwise reach it. For example, CASE WHEN rare_condition THEN m['k'] END scans the map entries of every row instead of only the selected few.
  • common_subexpr_eliminate, push_down_filter and the physical projection_pushdown all use the same placement and make the same assumption.

The cost of that extra work grows with map size and with how selective the branch or filter is. The wasted work is largest when few rows need the lookup.

Fallibility of map lookups is a separate problem that may require an independent solution, tracked in #27521. This issue is only about cost.

Describe the solution you'd like

Make get_field report a placement that reflects the cost of a map lookup, which is not MoveTowardsLeafNodes. The difficulty is that the decision needs types, and placement() has none. Some options:

  1. Pass argument types to placement, for example by adding the argument fields or data types to the call, or by adding a new method such as placement_with_fields(&self, args: &[ExpressionPlacement], arg_fields: &[FieldRef]). ScalarFunctionExpr and Expr::ScalarFunction would then supply the types. The logical Expr::placement has no schema in scope, so it would need a schema parameter or a different entry point. This is the most general option, but it changes a public trait method (see the API health policy).
  2. Decide in the callers. The optimizer rules and physical expressions that look at placement already work on plans that have schemas. They could ask for the placement with a schema in hand, and get_field would implement the type-aware method.
  3. Resolve at planning time. Where the types are known (for example in return_field_from_args or a simplification), get_field could be rewritten to a map-specific function or marked in a way that its placement can read. This avoids changing the placement signature, but adds a second representation of map lookups.

I think option 1 or 2 is the most robust. Whichever is chosen, get_field should return KeepInPlace when any step of its access path reaches into a Map, and keep MoveTowardsLeafNodes for struct-only paths.

Describe alternatives you've considered

  • Leave get_field as it is and have each consumer special-case maps. perf: Precompute simple column-derived expressions for ProjectedCaseBody #26589 does this for the physical CASE evaluation (crosses_map in datafusion/physical-expr/src/expressions/case.rs), but the logical optimizer rules do not, and every new consumer of placement would need to repeat it.
  • Document that MoveTowardsLeafNodes may be assigned to expressions with data-dependent cost. This makes the hint less useful for the decisions it exists to support.

Additional context

Activity

  1. changed the title [-]`ScalarUDFImpl::placement` lacks type onformation, so `get_field` reports `MoveTowardsLeafNodes` for expensive map lookups[/-] [+]`ScalarUDFImpl::placement` lacks type information, so `get_field` reports `MoveTowardsLeafNodes` for expensive map lookups[/+] on Nov 17, 2026
  2. mzabaluev commented on Nov 17, 2026

    @mzabaluev
    ContributorAuthor

    @adriangb this is a candidate for #25459.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions