Skip to content

Push down decimal Add/Sub to Vortex in DataFusion filters and projections#8936

Draft
mhk197 wants to merge 3 commits into
developfrom
mk/decimal-add-sub-pushdown-df
Draft

Push down decimal Add/Sub to Vortex in DataFusion filters and projections#8936
mhk197 wants to merge 3 commits into
developfrom
mk/decimal-add-sub-pushdown-df

Conversation

@mhk197

@mhk197 mhk197 commented Jul 23, 2026

Copy link
Copy Markdown
Contributor

Pushes decimal Add/Sub filters and projections into the Vortex scan, and stops accepting decimal arithmetic pushdowns that Vortex cannot execute.

Problem

DataFusion evaluates decimal +/- with the arrow numeric kernels: mismatched operands are rescaled inside the kernel, and the result type follows the Hive rule

s = max(s1, s2)
p = max(p1 - s1, p2 - s2) + s + 1    (saturated at the physical type's max precision)

Vortex's decimal kernels (#8724) instead require both operands to share one decimal dtype and widen the result by a carry digit, capped only at Vortex's max precision (76).

The filter pushdown path accepted any decimal arithmetic on structural grounds and reported PushedDown::Yes, so DataFusion dropped the predicate from its own plan and the query failed at scan time — a dtype-mismatch error for mixed operand dtypes and for Decimal128(38, s) operands (where arrow saturates at 38 but Vortex widens to 39), or not yet supported for decimal Mul/Div. The projection path refused all decimal arithmetic outright.

Approach

Vortex already ships the operand-alignment machinery: Binary::coerce_args computes the least supertype of its operands — for decimals exactly the (p - 1, s) dtype arrow rescales to internally — but the coerce_expression pass had no callers. This PR wires it in:

  • make_vortex_predicate and split_projection run converted expressions through coerce_expression against the scan's dtype scope (the per-file dtype in the opener, the data source dtype in v2). A pushed-down decimal Add/Sub then reproduces DataFusion's result type and values exactly: the operand casts perform the same checked power-of-ten rescale as the arrow kernel, and Vortex's carry-digit rule lands on the same (p, s).
  • The pass is applied only to expressions containing decimal arithmetic. coerce_expression inserts casts on nullability-only differences, which would otherwise wrap literals in ordinary predicates and could interfere with pruning. If Binary::coerce_args is later refined to preserve per-child nullability, the pass could run unconditionally.
  • ExpressionConvertor::convert is unchanged; alignment happens once on the finished expression tree.

Pushdown is rejected — the filter stays in DataFusion's FilterExec, or the projection in the source's post-scan projector — when the semantics diverge or the kernel does not exist:

  • decimal Mul/Div (no Vortex kernels yet),
  • Add/Sub at the physical type's precision ceiling (e.g. two Decimal128(38, s) operands), where arrow saturates the result precision while Vortex widens it,
  • operands of different physical decimal widths (coerced by DataFusion before execution; rejected defensively).

The blanket decimal-arithmetic exclusion in split_projection narrows to the same rule, so decimal Add/Sub projections push down too when projection_pushdown is enabled.

mhk197 added 2 commits July 23, 2026 15:50
Signed-off-by: Matt Katz <mhkatz97@gmail.com>
Signed-off-by: Matt Katz <mhkatz97@gmail.com>
@mhk197 mhk197 added the changelog/performance A performance improvement label Jul 24, 2026
Signed-off-by: Matt Katz <mhkatz97@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/performance A performance improvement

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant