Test: morsel splitting & merging in SharedWorkSource#21640
Test: morsel splitting & merging in SharedWorkSource#21640Dandandan wants to merge 4 commits intoapache:mainfrom
Conversation
|
run benchmarks |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing morsel-splitting (76a1f58) to 961c5fc (merge-base) diff using: tpch File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing morsel-splitting (76a1f58) to 961c5fc (merge-base) diff using: clickbench_partitioned File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing morsel-splitting (76a1f58) to 961c5fc (merge-base) diff using: tpcds File an issue against this benchmark runner |
76a1f58 to
ea3834e
Compare
|
run benchmarks |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing morsel-splitting (ea3834e) to 961c5fc (merge-base) diff using: tpch File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing morsel-splitting (ea3834e) to 961c5fc (merge-base) diff using: clickbench_partitioned File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing morsel-splitting (ea3834e) to 961c5fc (merge-base) diff using: tpcds File an issue against this benchmark runner |
3a13d4d to
b852406
Compare
|
run benchmarks |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing morsel-splitting (b852406) to 961c5fc (merge-base) diff using: clickbench_partitioned File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing morsel-splitting (b852406) to 961c5fc (merge-base) diff using: tpch File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing morsel-splitting (b852406) to 961c5fc (merge-base) diff using: tpcds File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: CPU Details (lscpu)Details
Resource Usagetpcds — base (merge-base)
tpcds — branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: CPU Details (lscpu)Details
Resource Usageclickbench_partitioned — base (merge-base)
clickbench_partitioned — branch
File an issue against this benchmark runner |
Split large files and merge small same-file ranges in the shared work queue, so idle sibling streams always have right-sized work to steal. - Target morsel size of at least 1 MiB projected size - Split: when queue depth < 2 * target_partitions and file >= 2 MiB, split in half and push the second half back - Merge: when popped file < 1 MiB, absorb adjacent same-file entries until reaching 1 MiB - Estimate projected size using per-column byte_size stats from PartitionedFile.statistics when available, otherwise fall back to file_size * (projected_cols / total_cols) - For Parquet: split the ParquetAccessPlan (row group selection) via SplittableExt instead of byte-range splitting - Statistics are scaled by 0.5 (inexact) on each split half Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
b852406 to
106f736
Compare
|
run benchmarks |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing morsel-splitting (106f736) to 961c5fc (merge-base) diff using: clickbench_partitioned File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing morsel-splitting (106f736) to 961c5fc (merge-base) diff using: tpcds File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing morsel-splitting (106f736) to 961c5fc (merge-base) diff using: tpch File an issue against this benchmark runner |
|
Benchmark for this request failed. Last 20 lines of output: Click to expandFile an issue against this benchmark runner |
|
run benchmarks |
|
🤖 Benchmark completed (GKE) | trigger Instance: CPU Details (lscpu)Details
Resource Usageclickbench_partitioned — base (merge-base)
clickbench_partitioned — branch
File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing morsel-splitting (40ca776) to 5c653be (merge-base) diff using: tpch File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing morsel-splitting (40ca776) to 5c653be (merge-base) diff using: tpcds File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing morsel-splitting (40ca776) to 5c653be (merge-base) diff using: clickbench_partitioned File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: CPU Details (lscpu)Details
Resource Usagetpcds — base (merge-base)
tpcds — branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: CPU Details (lscpu)Details
Resource Usagetpcds — base (merge-base)
tpcds — branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: CPU Details (lscpu)Details
Resource Usageclickbench_partitioned — base (merge-base)
clickbench_partitioned — branch
File an issue against this benchmark runner |
Which issue does this PR close?
Follow-up to #21351 (dynamic work scheduling in FileStream).
Rationale for this change
When a scan has few large files, idle sibling streams have nothing to steal even after #21351 enables work sharing. This PR splits large files by byte range so work can be distributed more evenly across partitions.
What changes are included in this PR?
Adds morsel splitting to
SharedWorkSource::pop_front():2 * target_partitionsand a file's projected size >= 2 MiB, splits the file in half by byte range and pushes the second half back onto the shared queuebyte_sizefromPartitionedFile.statisticswhen available (e.g. from Parquet column stats), otherwise falls back toraw_file_size * (projected_cols / total_cols)Are these changes tested?
Existing tests pass. Additional tests for the splitting logic to be added.
Are there any user-facing changes?
Faster performance for queries scanning few large files with multiple partitions.
🤖 Generated with Claude Code