Skip to content

feat: add indexed pushes to LimitedBatchCoalescer - #27595

Draft
Rich-T-kid wants to merge 2 commits into
apache:mainfrom
Rich-T-kid:batch-coalescer-indices
Draft

Rich-T-kid wants to merge 2 commits into
apache:mainfrom
Rich-T-kid:batch-coalescer-indices

Conversation

@Rich-T-kid

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

  • N/A, independent change.

Rationale for this change

Arrow 60.2 adds BatchCoalescer::push_batch_with_indices. The DataFusion LimitedBatchCoalescer wrapper should expose the same path while preserving its fetch limit behavior.

Stacked on #27594.

What changes are included in this PR?

Adds LimitedBatchCoalescer::push_batch_with_indices. The wrapper truncates the indices when the fetch limit is reached and returns the corresponding PushBatchStatus.

What is the testing strategy for this PR?

A focused unit test covers ordered indexed selection and truncation at the fetch limit.

Are there any user-facing changes?

Yes. LimitedBatchCoalescer exposes push_batch_with_indices.

@github-actions github-actions Bot added sqllogictest SQL Logic Tests (.slt) substrait Changes to the substrait crate datasource Changes to the datasource crate physical-plan Changes to the physical-plan crate labels Nov 19, 2026
@github-actions

Copy link
Copy Markdown

Thank you for opening this pull request!

Reviewer note: cargo-semver-checks reported the current version number is not SemVer-compatible with the changes in this pull request (compared against the base branch).

Details
     Cloning c6c1cc31d2f433930c9b4f3ca9fea023846ac276
error: running 'cargo update' on crate 'datafusion-datasource' failed with output:
-----
    Updating crates.io index
error: failed to select a version for the requirement `arrow = "^60.2.0"`
candidate versions found which didn't match: 60.1.0, 60.0.0, 59.3.3, ...
location searched: crates.io index
required by package `datafusion-datasource v57.0.0-dev (/home/runner/work/datafusion/datafusion/datafusion/datasource)`
    ... which satisfies path dependency `datafusion-datasource` of package `placeholder v0.0.0 (/home/runner/work/datafusion/datafusion/target/semver-checks-shared/semver-checks/local-datafusion_datasource-57_0_0_dev-x86_64_unknown_linux_gnu-7ed26b30e7903660)`

-----
error: failed to update dependencies for crate datafusion-datasource v57.0.0-dev
note: this is unlikely to be a bug in cargo-semver-checks,
      and is probably an issue with the crate's Cargo.toml
note: the following command can be used to reproduce the compilation error:
      cargo new --lib example &&
          cd example &&
          echo '[workspace]' >> Cargo.toml &&
          cargo add --path /home/runner/work/datafusion/datafusion/datafusion/datasource --features async-compression,backtrace,bzip2,compression,default,flate2,liblzma,proto,tempfile,tokio-util,zstd &&
          cargo update

error: running 'cargo update' on crate 'datafusion-datasource-parquet' failed with output:
-----
    Updating crates.io index
error: failed to select a version for the requirement `arrow = "^60.2.0"`
candidate versions found which didn't match: 60.1.0, 60.0.0, 59.3.3, ...
location searched: crates.io index
required by package `datafusion-datasource-parquet v57.0.0-dev (/home/runner/work/datafusion/datafusion/datafusion/datasource-parquet)`
    ... which satisfies path dependency `datafusion-datasource-parquet` of package `placeholder v0.0.0 (/home/runner/work/datafusion/datafusion/target/semver-checks-shared/semver-checks/local-datafusion_datasource_parquet-57_0_0_dev-x86_64_unknown_linux_gnu-51d4e4d914eb472b)`

-----
error: failed to update dependencies for crate datafusion-datasource-parquet v57.0.0-dev
note: this is unlikely to be a bug in cargo-semver-checks,
      and is probably an issue with the crate's Cargo.toml
note: the following command can be used to reproduce the compilation error:
      cargo new --lib example &&
          cd example &&
          echo '[workspace]' >> Cargo.toml &&
          cargo add --path /home/runner/work/datafusion/datafusion/datafusion/datasource-parquet --features parquet_encryption,proto &&
          cargo update

error: running 'cargo update' on crate 'datafusion-physical-expr-adapter' failed with output:
-----
    Updating crates.io index
error: failed to select a version for the requirement `arrow = "^60.2.0"`
candidate versions found which didn't match: 60.1.0, 60.0.0, 59.3.3, ...
location searched: crates.io index
required by package `datafusion-physical-expr-adapter v57.0.0-dev (/home/runner/work/datafusion/datafusion/datafusion/physical-expr-adapter)`
    ... which satisfies path dependency `datafusion-physical-expr-adapter` of package `placeholder v0.0.0 (/home/runner/work/datafusion/datafusion/target/semver-checks-shared/semver-checks/local-datafusion_physical_expr_adapter-57_0_0_dev-x86_64_unknown_linux_gnu-0630925412021fbe)`

-----
error: failed to update dependencies for crate datafusion-physical-expr-adapter v57.0.0-dev
note: this is unlikely to be a bug in cargo-semver-checks,
      and is probably an issue with the crate's Cargo.toml
note: the following command can be used to reproduce the compilation error:
      cargo new --lib example &&
          cd example &&
          echo '[workspace]' >> Cargo.toml &&
          cargo add --path /home/runner/work/datafusion/datafusion/datafusion/physical-expr-adapter &&
          cargo update

error: running 'cargo update' on crate 'datafusion-physical-plan' failed with output:
-----
    Updating crates.io index
error: failed to select a version for the requirement `arrow = "^60.2.0"`
candidate versions found which didn't match: 60.1.0, 60.0.0, 59.3.3, ...
location searched: crates.io index
required by package `datafusion-physical-plan v57.0.0-dev (/home/runner/work/datafusion/datafusion/datafusion/physical-plan)`
    ... which satisfies path dependency `datafusion-physical-plan` of package `placeholder v0.0.0 (/home/runner/work/datafusion/datafusion/target/semver-checks-shared/semver-checks/local-datafusion_physical_plan-57_0_0_dev-x86_64_unknown_linux_gnu-701c6e859cf729b2)`

-----
error: failed to update dependencies for crate datafusion-physical-plan v57.0.0-dev
note: this is unlikely to be a bug in cargo-semver-checks,
      and is probably an issue with the crate's Cargo.toml
note: the following command can be used to reproduce the compilation error:
      cargo new --lib example &&
          cd example &&
          echo '[workspace]' >> Cargo.toml &&
          cargo add --path /home/runner/work/datafusion/datafusion/datafusion/physical-plan --features force_hash_collisions,proto,test_utils,tokio_coop,tokio_coop_fallback &&
          cargo update

error: running 'cargo update' on crate 'datafusion-substrait' failed with output:
-----
    Updating crates.io index
error: failed to select a version for the requirement `arrow = "^60.2.0"`
candidate versions found which didn't match: 60.1.0, 60.0.0, 59.3.3, ...
location searched: crates.io index
required by package `datafusion v57.0.0-dev (/home/runner/work/datafusion/datafusion/datafusion/core)`
    ... which satisfies path dependency `datafusion` of package `datafusion-substrait v57.0.0-dev (/home/runner/work/datafusion/datafusion/datafusion/substrait)`
    ... which satisfies path dependency `datafusion-substrait` of package `placeholder v0.0.0 (/home/runner/work/datafusion/datafusion/target/semver-checks-shared/semver-checks/local-datafusion_substrait-57_0_0_dev-x86_64_unknown_linux_gnu-894bb029a7ba8dc7)`

-----
error: failed to update dependencies for crate datafusion-substrait v57.0.0-dev
note: this is unlikely to be a bug in cargo-semver-checks,
      and is probably an issue with the crate's Cargo.toml
note: the following command can be used to reproduce the compilation error:
      cargo new --lib example &&
          cd example &&
          echo '[workspace]' >> Cargo.toml &&
          cargo add --path /home/runner/work/datafusion/datafusion/datafusion/substrait --features default,physical,protoc &&
          cargo update

error: aborting due to failure to run 'cargo update' for crate datafusion-datasource v57.0.0-dev

@github-actions github-actions Bot added the auto detected api change Auto detected API change label Nov 19, 2026
@alamb

alamb commented Nov 19, 2026

Copy link
Copy Markdown
Contributor

so exciting

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

auto detected api change Auto detected API change datasource Changes to the datasource crate physical-plan Changes to the physical-plan crate sqllogictest SQL Logic Tests (.slt) substrait Changes to the substrait crate

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants