[CALCITE-7757] RelMdFunctionalDependency can infer unsound dependencies and cause incorrect query results by darpan-14 · Pull Request #5235 · apache/calcite

darpan-14 · 2026-08-31T19:04:11Z

Jira Link

Changes Proposed

RelMdFunctionalDependency can currently expose unsound or incorrectly indexed dependencies, allowing AggregateRemoveDuplicateKeysRule to remove grouping keys that are still required.

This change:

Corrects equality-derived functional dependencies for INNER, LEFT, and RIGHT joins, respecting the null-generating side of outer joins.
Preserves valid input functional dependencies through outer joins without incorrectly deriving the reverse dependency from the join equality.
Maps aggregate input group ordinals to aggregate output ordinals before publishing functional-dependency metadata.
Avoids equality-derived dependencies for FLOAT, REAL, DOUBLE, and INTERVAL types, including nested occurrences.
Avoids unsafe expression-derived grouping dependencies for those same types.
Adds a reusable SqlTypeUtil predicate traversal for scalar and nested types.
Adds regression coverage for each correctness case.

A basic valid duplicate-key removal remains supported:

SELECT t1.col1, t2.col3, COUNT(*)
FROM t1
JOIN t2 ON t1.col1 = t2.col3
GROUP BY t1.col1, t2.col3;

For this inner join, either equivalent equality key may be removed. The new safeguards prevent that reasoning from being applied where null generation, ordinal mapping, or non-reflexive type semantics make it unsound.

Testing

./gradlew :core:test --tests org.apache.calcite.sql.type.SqlTypeUtilTest --tests org.apache.calcite.test.AggregateRemoveDuplicateKeysRuleTest --tests org.apache.calcite.test.RelMetadataTest.testFunctionalDependency*
./gradlew :core:checkstyleMain :core:checkstyleTest
git diff --check upstream/main...HEAD

The focused test run completed 49 tests with 0 failures.

mihaibudiu · 2026-08-31T21:29:08Z

CI doesn't like your changes

…oins

darpan-14 · 2026-09-01T06:35:24Z

CI is happy now. 😄

mihaibudiu · 2026-09-13T17:55:23Z

  }

+  /**
+   * Copies input dependencies into a join, optionally filtering dependencies


I don't understand this javadoc. Can you please say what each parameter is and what the function actually computes? There's no "join" to be seen of here.

mihaibudiu · 2026-09-13T17:56:03Z

+
+  /**
+   * Returns whether a dependency's determinant contains a non-nullable input
+   * field. Such a determinant cannot collide with the all-NULL determinant of


why does this comment mention a LEFT JOIN? There is no left join here. Please make the comment describe this function does.

mihaibudiu · 2026-09-13T17:56:37Z

-   * @param fdSet Functional dependency set
-   * @param offset Index offset
-   * @return Shifted functional dependency set
+   * Adds functional dependencies implied by a join condition.


where does it add the dependencies?

mihaibudiu · 2026-09-13T17:57:08Z

-   * Extracts functional dependencies from equality and AND conditions.
-   * Handles col1 = col2, col1 IS NOT DISTINCT FROM col2, and AND conditions.
+   * Adds bidirectional dependencies for input-reference equalities in a
+   * condition. Callers are responsible for ensuring that every output row


every output row of what?

darpan-14 · 2026-09-22T05:21:21Z

Updated all Javadocs raised in review. The helper methods now document their general behavior, parameters, return value where applicable, where inferred dependencies are recorded, and the relational-output requirement for equality-derived dependencies.

sonarqubecloud · 2026-09-22T05:45:54Z

Quality Gate passed

Issues
1 New issue
0 Accepted issues

Measures
0 Security Hotspots
90.8% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

mihaibudiu · 2026-09-28T18:41:19Z

+  @Test void testKeepsDerivedGroupKeyForNestedDouble() {
+    // Determinism alone is insufficient for approximate values, including
+    // nested occurrences. Keep the derived key unless grouping equality is
+    // known to be congruent with the expression for ARRAY<DOUBLE>.


I cannot read this comment. What is "congruent with the expression"?

mihaibudiu · 2026-09-28T18:45:06Z

      ImmutableBitSet aggCols =
          ImmutableBitSet.range(rel.getGroupCount(), rel.getRowType().getFieldCount());
-      fdBuilder.addArrow(groupSet, aggCols);
+      fdBuilder.addArrow(ImmutableBitSet.range(rel.getGroupCount()), aggCols);


I think you also need to add to line 373 above a check Aggregate.isSimple(rel)
Some aggregates like ROLLUP are not simple; maybe you can add a suitable test case:
select a, count(*) c from t group by rollup(a)

mihaibudiu · 2026-10-01T18:01:44Z

When this is ready please re-request a review

darpan-14 added 6 commits September 1, 2026 11:15

[CALCITE-7757] Functional dependency inference is unsound for outer j…

75adeec

…oins

[CALCITE-7757] Map aggregate functional dependencies to output ordinals

84a8e60

[CALCITE-7757] Avoid unsafe equality-derived functional dependencies

f3aa45c

[CALCITE-7757] Avoid unsafe derived grouping key dependencies

d905926

[CALCITE-7757] Add utility for matching nested types

3a73679

[CALCITE-7757] Apply Java formatting

60bf738

darpan-14 force-pushed the CALCITE-7757 branch from 32526d0 to 60bf738 Compare September 1, 2026 06:04

mihaibudiu reviewed Sep 2, 2026

View reviewed changes

Comment thread core/src/main/java/org/apache/calcite/rel/metadata/RelMdFunctionalDependency.java

[CALCITE-7757] Allow interval grouping key inference

0250d6b

mihaibudiu requested changes Sep 13, 2026

View reviewed changes

[CALCITE-7757] Improve functional dependency Javadocs

cbe536e

mihaibudiu reviewed Sep 28, 2026

View reviewed changes

[CALCITE-7757] Avoid aggregate FDs for grouping sets

1348183

darpan-14 force-pushed the CALCITE-7757 branch from 503afd4 to 1348183 Compare October 1, 2026 13:47

Navigation Menu

[CALCITE-7757] RelMdFunctionalDependency can infer unsound dependencies and cause incorrect query results - #5235

[CALCITE-7757] RelMdFunctionalDependency can infer unsound dependencies and cause incorrect query results#5235
darpan-14 wants to merge 9 commits into
apache:mainfrom
darpan-14:CALCITE-7757

darpan-14 commented Aug 31, 2026

Uh oh!

mihaibudiu commented Aug 31, 2026

Uh oh!

darpan-14 commented Sep 1, 2026

Uh oh!

Uh oh!

mihaibudiu Sep 13, 2026

Uh oh!

mihaibudiu Sep 13, 2026

Uh oh!

mihaibudiu Sep 13, 2026

Uh oh!

mihaibudiu Sep 13, 2026

Uh oh!

darpan-14 commented Sep 22, 2026

Uh oh!

sonarqubecloud Bot commented Sep 22, 2026

Uh oh!

mihaibudiu Sep 28, 2026

Uh oh!

mihaibudiu Sep 28, 2026

Uh oh!

mihaibudiu commented Oct 1, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

darpan-14 commented Aug 31, 2026

Jira Link

Changes Proposed

Testing

Uh oh!

mihaibudiu commented Aug 31, 2026

Uh oh!

darpan-14 commented Sep 1, 2026

Uh oh!

Uh oh!

mihaibudiu Sep 13, 2026

Choose a reason for hiding this comment

Uh oh!

mihaibudiu Sep 13, 2026

Choose a reason for hiding this comment

Uh oh!

mihaibudiu Sep 13, 2026

Choose a reason for hiding this comment

Uh oh!

mihaibudiu Sep 13, 2026

Choose a reason for hiding this comment

Uh oh!

darpan-14 commented Sep 22, 2026

Uh oh!

sonarqubecloud Bot commented Sep 22, 2026

Quality Gate passed

Uh oh!

mihaibudiu Sep 28, 2026

Choose a reason for hiding this comment

Uh oh!

mihaibudiu Sep 28, 2026

Choose a reason for hiding this comment

Uh oh!

mihaibudiu commented Oct 1, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants