Add ExpressionAnalyzer for pluggable expression-level statistics estimation

[asolimando]

Which issue does this PR close?

Part of #21120 (framework + projection/filter integration)

Rationale for this change

DataFusion currently loses expression-level statistics when computing plan metadata. Projected expressions that aren't bare columns or literals get unknown statistics, and filter selectivity falls back to a hardcoded 20% when interval analysis cannot handle the predicate (e.g. OR predicates, which are not expressible as a single interval). There is also no extension point for users to provide statistics for their own UDFs.

This PR introduces ExpressionAnalyzer, a pluggable chain-of-responsibility framework that addresses these gaps. It follows the same extensibility pattern used elsewhere in DataFusion (ExprPlanner, OptimizerRule, StatisticsRegistry).

What changes are included in this PR?

• ExpressionAnalyzer trait and ExpressionAnalyzerRegistry (chain-of-responsibility, first Computed wins)
• DefaultExpressionAnalyzer with Selinger-style estimation: equality/inequality via NDV, AND/OR via inclusion-exclusion, injective arithmetic (+/-), literals, NOT
• ProjectionExprs and FilterExec use the registry for expression-level statistics
• Three new default trait methods on ExecutionPlan (uses_expression_level_statistics, with_expression_analyzer_registry, expression_analyzer_registry) for injection, overridden by FilterExec, ProjectionExec, AggregateExec, HashJoinExec, and SortMergeJoinExec
• The physical planner injects the registry after plan creation and re-injects after each optimizer rule that modifies the plan, ensuring optimizer-created nodes always carry it
• AggregateStatisticsProvider and JoinStatisticsProvider (feat: Add pluggable StatisticsRegistry for operator-level statistics propagation #21483) consume the registry via the trait getter
• Config option optimizer.use_expression_analyzer (default false), zero overhead when disabled

Are these changes tested?

• 21 unit tests for ExpressionAnalyzer
• 4 integration tests verifying registry injection survives optimizer rules
• 5 end-to-end SLT tests (OR selectivity, TopK, UNION ALL, hash join filter pushdown)
• Existing projection and filter test suites pass unchanged

Are there any user-facing changes?

New public API (purely additive, non-breaking):

• ExpressionAnalyzer trait and ExpressionAnalyzerRegistry in datafusion-physical-expr
• SessionState::expression_analyzer_registry() getter
• SessionStateBuilder::with_expression_analyzer_registry() setter
• Three default trait methods on ExecutionPlan: uses_expression_level_statistics(), with_expression_analyzer_registry(), expression_analyzer_registry()
• Config option datafusion.optimizer.use_expression_analyzer

No breaking changes. Default behavior is unchanged (config defaults to false).

---

Disclaimer: I used AI to assist in the code generation, I have manually reviewed the output and it matches my intention and understanding.

[asolimando]

@2010YOUY01: FYI I took a final pass on the PR and marked it as "reviewable"

[kosiew]

@asolimando

Thanks for the solid work here. The new analyzer direction looks promising, but I ran into a couple of issues that could affect correctness and usefulness of the stats. Left a few detailed comments below.

[asolimando]

@asolimando

Thanks for the solid work here. The new analyzer direction looks promising, but I ran into a couple of issues that could affect correctness and usefulness of the stats. Left a few detailed comments below.

Thanks to you @kosiew for the spot-on review, and for sharing your feedback. I have addressed the requested changes, happy to iterate further if needed!

[kosiew]

@asolimando

Thanks for the follow-up here. The issues called out in the earlier review look addressed, including the projection ordering fix, the equality and inequality selectivity registry lookup, and narrowing the injective arithmetic NDV rule. I did notice one remaining gap around registry propagation through planner and optimizer-created projections.

[kosiew]

Thanks for wiring the analyzer registry into planner-built projections. I think there is still one hole here though.

Right now this gets attached when the planner initially creates a ProjectionExec, but several later paths still rebuild or insert projections with plain ProjectionExec::try_new(...), for example in datafusion/physical-plan/src/projection.rs, datafusion/physical-optimizer/src/projection_pushdown.rs, and datafusion/physical-optimizer/src/aggregate_statistics.rs.

After one of those rewrites, expression stats seem to fall back to unknown again even when datafusion.optimizer.enable_expression_analyzer = true.

Could we propagate the registry when cloning or rebuilding projections, or store it somewhere rebuilt plan nodes can recover it from? As written, this still looks plan-shape dependent after physical optimization.

[asolimando]

@kosiew: this is a known limitation under the constraint of avoiding breaking changes entirely in this first PR: only the planner has access to SessionState, so operator nodes created later by optimizer rules don't receive the registry. The idea was to lift the limitation via an operator-level statistics registry (see #21443, which would be the second and final part of the pluggable design for statistics).

The limitation is documented in ProjectionExprs::with_expression_analyzer_registry and FilterExecBuilder::with_expression_analyzer_registry.

Apologies for having you figure it out independently, I should have called this out more prominently in the PR description rather than only in code comments.

If you see a better way to propagate the registry that I might have overlooked or if you think a breaking change is the right call here, I'd be happy to hear more about that.

[kosiew]

@asolimando

Thanks for the detailed explanation. I understand the non-breaking-change constraint here, and the code comments on ProjectionExprs::with_expression_analyzer_registry / FilterExecBuilder::with_expression_analyzer_registry do make the current boundary clearer. I still think this remains a blocking limitation for the PR as it stands, though, because the feature is exposed behind datafusion.optimizer.enable_expression_analyzer = true yet its effect still depends on whether later physical rewrites rebuild the operator. Several common paths still create fresh ProjectionExec::try_new(...) nodes without a registry (datafusion/physical-plan/src/projection.rs, datafusion/physical-optimizer/src/projection_pushdown.rs, datafusion/physical-optimizer/src/aggregate_statistics.rs, etc.), so expression stats revert to unknown after optimization.

If carrying the registry through optimizer-created operators is not possible in this PR without the broader operator-level design from #21443, I think the safer options would be either:

• narrow the current scope/docs so users do not read this as generally-enabled projection/filter expression analysis across the final physical plan, or
• treat the operator-level propagation work as required before we rely on this config as a general optimizer feature.

I do not have a clean non-breaking propagation approach to suggest beyond the operator-level registry direction you mentioned, but I wanted to be explicit that I still see the current plan-shape dependency as an unresolved product/correctness concern rather than just a documentation gap.

[asolimando]

Thanks for your guidance, @kosiew!

I have implemented option 1 optimistically in 54a0df7: tightened the config description and doc comments to make the scope explicit. But we can drop it in case option 2 is finally retained.

For option 2, I have opened #21483 (it was originally stacked on this PR, but I have reworked it to make it independent, the two can be composed later if #21483 gets in first) if you want to take a concrete look to evaluate whether you'd prefer to hold this PR until that one is ready.

[kosiew]

hi @asolimando

I'll take a look again after you incorporate the merged #21483.

[asolimando]

Thanks a lot @kosiew for your help, I got sidetracked today but I plan to finalize the rebasing in the next few days, I will re-request a review once ready!

[asolimando]

@kosiew, thanks for your patience, here's what changed since your last review.

Addressing the plan-shape dependency:

To address this I finally included a re-injection loop in the physical planner that runs after each optimizer rule that modifies the plan (detected via Arc::ptr_eq). This ensures optimizer-created nodes (e.g. FilterExec from filter pushdown into unions, repositioned filters through joins, etc.) always receive the registry. The walk is entirely skipped when the ExpressionAnalyzer is disabled, no additional overhead for the common path.

Caveat: nodes created within a rule (e.g. a new FilterExec) receive the registry only after that rule completes, before the next rule runs. No such rule (EnforceDistribution, CombinePartialFinalAggregate, ProjectionPushdown, AggregateStatistics and JoinSelection) reads stats from the nodes it just created within the same rule execution, so there's no gap at the moment.

Since there is no gap and that this concern will go away if we integrate fully with partitions_statistics (see the "Future direction" section), which is the natural evolution of this proposal, I consider the integration satisfactory, what do you think?

Integration with StatisticsRegistry (#21483):

The AggregateStatisticsProvider and JoinStatisticsProvider providers consume ExpressionAnalyzer and this adds support for aggregations and joins. When no registry is injected, they delegate to the existing code path, purely additive as the rest.

Future direction: removing the injection overhead (just FYI, no action for this PR)

The re-injection loop for ExpressionAnalyzer and the separate StatisticsRegistry tree walk both exist because partition_statistics has no way to receive external context and we don't want breaking changes.
But we could piggyback on #20184 (child_stats in partition_statistics) as it plans to update partition_statistics's signature anyway.

By generalizing slightly with a StatisticsContext parameter that carries both the ExpressionAnalyzerRegistry and StatisticsRegistry providers, we would get headroom for future changes while limiting breaking changes.

This would collapse both mechanisms into the natural partition_statistics recursion, eliminating the injection walks and the separate bottom-up traversal entirely while keeping the config flags to turn either framework on or off.

The separate walks are acceptable now as a trade-off for a purely additive and non-breaking change, but this would be the natural evolution of the proposal.

New commits in detail:

I had to rebase on #21483, this will make incremental reviewing harder for you, so let me walk you through the changes since your last review (commits 9-12 of 12):

9. Inject ExpressionAnalyzerRegistry into exec nodes and re-inject via optimizer loop - the core change addressing your feedback, includes integration tests
10. test(filter): verify ExpressionAnalyzer inclusion-exclusion selectivity for OR predicates - unit test for OR selectivity in FilterExec
11. refactor(expression_analyzer): delegate when no NDV statistics are available - I realized we were trying to do too much when some columns statistics are unset, now we delegate to built-in
12. test(expression_analyzer): add StatisticsTable and end-to-end SLT for OR selectivity - end-to-end SLT tests verifying the registry survives TopK sort, UNION ALL filter pushdown, and hash join filter pushdown (queries chosen as they generate nodes that would need the ExpressionAnalyzer, exactly the gap we had before)

Looking forward to your feedback!

[kosiew]

@asolimando, I spotted an issue with NDV handling that can affect selectivity estimates depending on operand order.

[asolimando]

@kosiew the CI error seems unrelated, the same test suite passed for macos but not amd64, I can't reproduce and it doesn't seem it can be caused by changes in this PR, it feels like a flaky test but I'd like to get your opinion.