Patched DataFusion version 45.0.0

[wiedld]

Follow on to

• Updated DF (patched) to Dec-25, released ver44.0.0 #53

This brings us up to the version45 release, this Feb 3rd commit here and this apache branch.

Patches due to tech debt (all slated as lower priority, and are lingering):

• chore: default=true for skip_physical_aggregate_schema_check, and add warn logging
  • until we chase down all warnings in our iox logs: https://github.com/influxdata/influxdb_iox/issues/12404

Patches due to upstream bugs:

• Not yet fixed:
  • chore: skip order calculation / exponential planning
    • exponential planning DF bug: Exponential planning time (100s of seconds) with UNION and ORDER BY queries apache/datafusion#13748

Patches because we haven't caught up yet:

• Fixed (in ver 46):
  • fix: temporary fix to handle incorrect coalesce (inserted during EnforceDistribution) which later causes an error during EnforceSort (without our patch). The next DataFusion version 46 upgrade does the proper fix, which is to not insert the coalesce in the first place.
    • this is porting over the patch we already are using in iox. (Iox points to this branch: fix: Distribution error failing in the SanityCheck, for a specific influxql plan. #58).
    • confirmed bug is fixed in DF ver 46: ParallelizeSorts, a subrule of EnforceSorting optimizer, should not remove necessary coalesce. apache/datafusion#14691 (comment)
• Feb 5: fix(ci): build error with wasm
• Feb 18: Specify rust toolchain explicitly, document how to change it (#14655)
• Feb 21: Bump MSRV to 1.82, toolchain to 1.85 (#14811)
• March 11: chore: get wasm to build in CI.
  • This is a temporary patch of ours, instead of pulling in alot of code to proper build wasm for parquet. The actual fix requires 2 PRs together: 6f285d6 and the earlier (large) PR with most of the code, but also introduced a bug fixed on March 11.
• April 1: Disable sccache action to fix gh cache issue (https://github.com/apache/datafusion/pull/15536)
• April 5: test: update sqllogictest errors.
  • This was updated on apache main with a much larger PR for improved spill (5c31692).
• 3 weeks ago: chore: fix clippy::large_enum_variant for DataFusionError

Patches due to iox needs (unclear if upstream action TBD):

• fix: handle when the left side of the union has no fields (e.g. an empty projection)
  • as identified in our iox issue, we permit plans to have empty projections (e.g. in count(*) where we aggregate to count rows after the projection.
  • https://github.com/influxdata/influxdb_iox/issues/12407

[alamb]

I re-reviewed this PR and patches and it looks good to me 👍

Thanks @wiedld

[github-actions]

Thank you for your contribution. Unfortunately, this pull request is stale because it has been open 60 days with no activity. Please remove the stale label or comment or this will be closed in 7 days.

[wiedld]

This auto-closed.
We need this branch. It's our next DF upgrade to ver 45.

[wiedld]

The security audit CI will fail due to:

Crate:     pyo3
Version:   0.23.5,
Title:     Risk of buffer overflow in `PyString::from_object`
Date:      2025-04-01
ID:        RUSTSEC-2025-0020
URL:       https://rustsec.org/advisories/RUSTSEC-2025-0020
Solution:  Upgrade to >=0.24.1

But the pyo3 upgraded cannot be done until the arrow upgrade (Apr 4 commit: b717723)

[wiedld]

I'm not going to debug the last few CI items, since they do not impact us.