Skip to content

[iceberg-datafusion] Balance eager scan task groups by estimated read size #2962

Description

@toutane

Is your feature request related to a problem or challenge?

#2671 introduces eager scan planning for the DataFusion integration. The planned FileScanTask are currently assigned to DataFusion output partitions using round-robin.

This balances the number of tasks per partition, but not the amount of data to read. When file sizes vary significantly, one partition may receive most of the bytes and become the query bottleneck.

This issue concerns distributing already planned tasks between DataFusion partitions. Generic file splitting and scan-task planning remain tracked by #128.

Describe the solution you'd like

Replace the round-robin grouping in scan_planning.rs with size-aware grouping.

The implementation should:

  • create at most target_partitions non-empty groups;
  • balance the estimated bytes to read across those groups;
  • preserve the current behavior for empty task lists and when there are fewer tasks than target partitions;

The task weight should represent the bytes scanned by the task, including the case where a task covers only part of a file.

Related to #128, but does not replace or close it: #128 covers generic size-based scan-task planning and file splitting in the core Iceberg crate.

Willingness to contribute

I can contribute to this feature independently

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions