Is your feature request related to a problem or challenge?
#2671 introduces eager scan planning for the DataFusion integration. The planned FileScanTask are currently assigned to DataFusion output partitions using round-robin.
This balances the number of tasks per partition, but not the amount of data to read. When file sizes vary significantly, one partition may receive most of the bytes and become the query bottleneck.
This issue concerns distributing already planned tasks between DataFusion partitions. Generic file splitting and scan-task planning remain tracked by #128.
Describe the solution you'd like
Replace the round-robin grouping in scan_planning.rs with size-aware grouping.
The implementation should:
- create at most
target_partitions non-empty groups;
- balance the estimated bytes to read across those groups;
- preserve the current behavior for empty task lists and when there are fewer tasks than target partitions;
The task weight should represent the bytes scanned by the task, including the case where a task covers only part of a file.
Related to #128, but does not replace or close it: #128 covers generic size-based scan-task planning and file splitting in the core Iceberg crate.
Willingness to contribute
I can contribute to this feature independently
Is your feature request related to a problem or challenge?
#2671 introduces eager scan planning for the DataFusion integration. The planned
FileScanTaskare currently assigned to DataFusion output partitions using round-robin.This balances the number of tasks per partition, but not the amount of data to read. When file sizes vary significantly, one partition may receive most of the bytes and become the query bottleneck.
This issue concerns distributing already planned tasks between DataFusion partitions. Generic file splitting and scan-task planning remain tracked by #128.
Describe the solution you'd like
Replace the round-robin grouping in
scan_planning.rswith size-aware grouping.The implementation should:
target_partitionsnon-empty groups;The task weight should represent the bytes scanned by the task, including the case where a task covers only part of a file.
Related to #128, but does not replace or close it: #128 covers generic size-based scan-task planning and file splitting in the core Iceberg crate.
Willingness to contribute
I can contribute to this feature independently