Repository navigation
feat(parquet): cap dictionary reads using NDV estimates - #27500
Rich-T-kid wants to merge 5 commits into
Conversation
85dbfb4 to
432586c
Compare
|
run benchmark tpch tpcds clickbench_partitioned env:
DATAFUSION_RUNTIME_MEMORY_LIMIT: "32G"
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"
baseline:
ref: "432586c6f82b2a7f96933dc0501cf02f2a2a1eca"
changed:
ref: "432586c6f82b2a7f96933dc0501cf02f2a2a1eca"
env:
DATAFUSION_EXECUTION_PARQUET_DICTIONARY_READ_MAX_DISTINCT_VALUES: "5"
resources:
cpu: "12"
memory: "65Gi"
arch: arm64 |
|
run benchmark tpch tpcds clickbench_partitioned env:
DATAFUSION_RUNTIME_MEMORY_LIMIT: "32G"
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"
baseline:
ref: "432586c6f82b2a7f96933dc0501cf02f2a2a1eca"
changed:
ref: "432586c6f82b2a7f96933dc0501cf02f2a2a1eca"
env:
DATAFUSION_EXECUTION_PARQUET_DICTIONARY_READ_MAX_DISTINCT_VALUES: "5"
resources:
cpu: "12"
memory: "65Gi"
arch: amd64 |
|
run benchmark tpch tpcds clickbench_partitioned env:
DATAFUSION_RUNTIME_MEMORY_LIMIT: "32G"
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "false"
baseline:
ref: "432586c6f82b2a7f96933dc0501cf02f2a2a1eca"
changed:
ref: "432586c6f82b2a7f96933dc0501cf02f2a2a1eca"
env:
DATAFUSION_EXECUTION_PARQUET_DICTIONARY_READ_MAX_DISTINCT_VALUES: "5"
resources:
cpu: "12"
memory: "65Gi"
arch: arm64 |
|
run benchmark tpch tpcds clickbench_partitioned env:
DATAFUSION_RUNTIME_MEMORY_LIMIT: "32G"
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "false"
baseline:
ref: "432586c6f82b2a7f96933dc0501cf02f2a2a1eca"
changed:
ref: "432586c6f82b2a7f96933dc0501cf02f2a2a1eca"
env:
DATAFUSION_EXECUTION_PARQUET_DICTIONARY_READ_MAX_DISTINCT_VALUES: "5"
resources:
cpu: "12"
memory: "65Gi"
arch: amd64 |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing 432586c (432586c) to 432586c diff Run configurationrun benchmark tpch
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"
DATAFUSION_RUNTIME_MEMORY_LIMIT: "32G"
baseline:
ref: "432586c6f82b2a7f96933dc0501cf02f2a2a1eca"
changed:
ref: "432586c6f82b2a7f96933dc0501cf02f2a2a1eca"
env:
DATAFUSION_EXECUTION_PARQUET_DICTIONARY_READ_MAX_DISTINCT_VALUES: "5"
resources:
cpu: "12"
memory: "65Gi"
arch: "amd64"Results will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing 432586c (432586c) to 432586c diff Run configurationrun benchmark tpcds
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"
DATAFUSION_RUNTIME_MEMORY_LIMIT: "32G"
baseline:
ref: "432586c6f82b2a7f96933dc0501cf02f2a2a1eca"
changed:
ref: "432586c6f82b2a7f96933dc0501cf02f2a2a1eca"
env:
DATAFUSION_EXECUTION_PARQUET_DICTIONARY_READ_MAX_DISTINCT_VALUES: "5"
resources:
cpu: "12"
memory: "65Gi"
arch: "amd64"Results will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing 432586c (432586c) to 432586c diff Run configurationrun benchmark clickbench_partitioned
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"
DATAFUSION_RUNTIME_MEMORY_LIMIT: "32G"
baseline:
ref: "432586c6f82b2a7f96933dc0501cf02f2a2a1eca"
changed:
ref: "432586c6f82b2a7f96933dc0501cf02f2a2a1eca"
env:
DATAFUSION_EXECUTION_PARQUET_DICTIONARY_READ_MAX_DISTINCT_VALUES: "5"
resources:
cpu: "12"
memory: "65Gi"
arch: "amd64"Results will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing 432586c (432586c) to 432586c diff Run configurationrun benchmark tpch
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"
DATAFUSION_RUNTIME_MEMORY_LIMIT: "32G"
baseline:
ref: "432586c6f82b2a7f96933dc0501cf02f2a2a1eca"
changed:
ref: "432586c6f82b2a7f96933dc0501cf02f2a2a1eca"
env:
DATAFUSION_EXECUTION_PARQUET_DICTIONARY_READ_MAX_DISTINCT_VALUES: "5"
resources:
cpu: "12"
memory: "65Gi"
arch: "arm64"Results will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing 432586c (432586c) to 432586c diff Run configurationrun benchmark tpcds
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"
DATAFUSION_RUNTIME_MEMORY_LIMIT: "32G"
baseline:
ref: "432586c6f82b2a7f96933dc0501cf02f2a2a1eca"
changed:
ref: "432586c6f82b2a7f96933dc0501cf02f2a2a1eca"
env:
DATAFUSION_EXECUTION_PARQUET_DICTIONARY_READ_MAX_DISTINCT_VALUES: "5"
resources:
cpu: "12"
memory: "65Gi"
arch: "arm64"Results will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing 432586c (432586c) to 432586c diff Run configurationrun benchmark tpch
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"
DATAFUSION_RUNTIME_MEMORY_LIMIT: "32G"
baseline:
ref: "432586c6f82b2a7f96933dc0501cf02f2a2a1eca"
changed:
ref: "432586c6f82b2a7f96933dc0501cf02f2a2a1eca"
env:
DATAFUSION_EXECUTION_PARQUET_DICTIONARY_READ_MAX_DISTINCT_VALUES: "5"
resources:
cpu: "12"
memory: "65Gi"
arch: "arm64"CPU Details (lscpu)Details
Memory Pool PeaksPeak Base:
Pool accounting vs. process RSS Max pool peak is the largest reservation any single query in the run reached; peak RSS covers the whole invocation, including data loading and allocator retention, and the two high-water marks need not coincide in time. The gap is therefore an upper bound on what the pool did not account for, not a measurement of it.
Resource Usagetpch — base (merge-base)
tpch — branch
File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing 432586c (432586c) to 432586c diff Run configurationrun benchmark clickbench_partitioned
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"
DATAFUSION_RUNTIME_MEMORY_LIMIT: "32G"
baseline:
ref: "432586c6f82b2a7f96933dc0501cf02f2a2a1eca"
changed:
ref: "432586c6f82b2a7f96933dc0501cf02f2a2a1eca"
env:
DATAFUSION_EXECUTION_PARQUET_DICTIONARY_READ_MAX_DISTINCT_VALUES: "5"
resources:
cpu: "12"
memory: "65Gi"
arch: "arm64"Results will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing 432586c (432586c) to 432586c diff Run configurationrun benchmark tpcds
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"
DATAFUSION_RUNTIME_MEMORY_LIMIT: "32G"
baseline:
ref: "432586c6f82b2a7f96933dc0501cf02f2a2a1eca"
changed:
ref: "432586c6f82b2a7f96933dc0501cf02f2a2a1eca"
env:
DATAFUSION_EXECUTION_PARQUET_DICTIONARY_READ_MAX_DISTINCT_VALUES: "5"
resources:
cpu: "12"
memory: "65Gi"
arch: "arm64"CPU Details (lscpu)Details
Memory Pool PeaksPeak Base:
Pool accounting vs. process RSS Max pool peak is the largest reservation any single query in the run reached; peak RSS covers the whole invocation, including data loading and allocator retention, and the two high-water marks need not coincide in time. The gap is therefore an upper bound on what the pool did not account for, not a measurement of it.
Resource Usagetpcds — base (merge-base)
tpcds — branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing 432586c (432586c) to 432586c diff Run configurationrun benchmark tpch
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"
DATAFUSION_RUNTIME_MEMORY_LIMIT: "32G"
baseline:
ref: "432586c6f82b2a7f96933dc0501cf02f2a2a1eca"
changed:
ref: "432586c6f82b2a7f96933dc0501cf02f2a2a1eca"
env:
DATAFUSION_EXECUTION_PARQUET_DICTIONARY_READ_MAX_DISTINCT_VALUES: "5"
resources:
cpu: "12"
memory: "65Gi"
arch: "amd64"CPU Details (lscpu)Details
Memory Pool PeaksPeak Base:
Pool accounting vs. process RSS Max pool peak is the largest reservation any single query in the run reached; peak RSS covers the whole invocation, including data loading and allocator retention, and the two high-water marks need not coincide in time. The gap is therefore an upper bound on what the pool did not account for, not a measurement of it.
Resource Usagetpch — base (merge-base)
tpch — branch
File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing 432586c (432586c) to 432586c diff Run configurationrun benchmark clickbench_partitioned
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "false"
DATAFUSION_RUNTIME_MEMORY_LIMIT: "32G"
baseline:
ref: "432586c6f82b2a7f96933dc0501cf02f2a2a1eca"
changed:
ref: "432586c6f82b2a7f96933dc0501cf02f2a2a1eca"
env:
DATAFUSION_EXECUTION_PARQUET_DICTIONARY_READ_MAX_DISTINCT_VALUES: "5"
resources:
cpu: "12"
memory: "65Gi"
arch: "amd64"Results will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing 432586c (432586c) to 432586c diff Run configurationrun benchmark tpch
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "false"
DATAFUSION_RUNTIME_MEMORY_LIMIT: "32G"
baseline:
ref: "432586c6f82b2a7f96933dc0501cf02f2a2a1eca"
changed:
ref: "432586c6f82b2a7f96933dc0501cf02f2a2a1eca"
env:
DATAFUSION_EXECUTION_PARQUET_DICTIONARY_READ_MAX_DISTINCT_VALUES: "5"
resources:
cpu: "12"
memory: "65Gi"
arch: "arm64"Results will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing 432586c (432586c) to 432586c diff Run configurationrun benchmark tpcds
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "false"
DATAFUSION_RUNTIME_MEMORY_LIMIT: "32G"
baseline:
ref: "432586c6f82b2a7f96933dc0501cf02f2a2a1eca"
changed:
ref: "432586c6f82b2a7f96933dc0501cf02f2a2a1eca"
env:
DATAFUSION_EXECUTION_PARQUET_DICTIONARY_READ_MAX_DISTINCT_VALUES: "5"
resources:
cpu: "12"
memory: "65Gi"
arch: "amd64"Results will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing 432586c (432586c) to 432586c diff Run configurationrun benchmark tpch
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "false"
DATAFUSION_RUNTIME_MEMORY_LIMIT: "32G"
baseline:
ref: "432586c6f82b2a7f96933dc0501cf02f2a2a1eca"
changed:
ref: "432586c6f82b2a7f96933dc0501cf02f2a2a1eca"
env:
DATAFUSION_EXECUTION_PARQUET_DICTIONARY_READ_MAX_DISTINCT_VALUES: "5"
resources:
cpu: "12"
memory: "65Gi"
arch: "amd64"Results will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing 432586c (432586c) to 432586c diff Run configurationrun benchmark tpcds
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"
DATAFUSION_RUNTIME_MEMORY_LIMIT: "32G"
baseline:
ref: "432586c6f82b2a7f96933dc0501cf02f2a2a1eca"
changed:
ref: "432586c6f82b2a7f96933dc0501cf02f2a2a1eca"
env:
DATAFUSION_EXECUTION_PARQUET_DICTIONARY_READ_MAX_DISTINCT_VALUES: "5"
resources:
cpu: "12"
memory: "65Gi"
arch: "amd64"CPU Details (lscpu)Details
Memory Pool PeaksPeak Base:
Pool accounting vs. process RSS Max pool peak is the largest reservation any single query in the run reached; peak RSS covers the whole invocation, including data loading and allocator retention, and the two high-water marks need not coincide in time. The gap is therefore an upper bound on what the pool did not account for, not a measurement of it.
Resource Usagetpcds — base (merge-base)
tpcds — branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing 432586c (432586c) to 432586c diff Run configurationrun benchmark tpch
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "false"
DATAFUSION_RUNTIME_MEMORY_LIMIT: "32G"
baseline:
ref: "432586c6f82b2a7f96933dc0501cf02f2a2a1eca"
changed:
ref: "432586c6f82b2a7f96933dc0501cf02f2a2a1eca"
env:
DATAFUSION_EXECUTION_PARQUET_DICTIONARY_READ_MAX_DISTINCT_VALUES: "5"
resources:
cpu: "12"
memory: "65Gi"
arch: "arm64"CPU Details (lscpu)Details
Memory Pool PeaksPeak Base:
Pool accounting vs. process RSS Max pool peak is the largest reservation any single query in the run reached; peak RSS covers the whole invocation, including data loading and allocator retention, and the two high-water marks need not coincide in time. The gap is therefore an upper bound on what the pool did not account for, not a measurement of it.
Resource Usagetpch — base (merge-base)
tpch — branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing 432586c (432586c) to 432586c diff Run configurationrun benchmark tpch
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "false"
DATAFUSION_RUNTIME_MEMORY_LIMIT: "32G"
baseline:
ref: "432586c6f82b2a7f96933dc0501cf02f2a2a1eca"
changed:
ref: "432586c6f82b2a7f96933dc0501cf02f2a2a1eca"
env:
DATAFUSION_EXECUTION_PARQUET_DICTIONARY_READ_MAX_DISTINCT_VALUES: "5"
resources:
cpu: "12"
memory: "65Gi"
arch: "amd64"CPU Details (lscpu)Details
Memory Pool PeaksPeak Base:
Pool accounting vs. process RSS Max pool peak is the largest reservation any single query in the run reached; peak RSS covers the whole invocation, including data loading and allocator retention, and the two high-water marks need not coincide in time. The gap is therefore an upper bound on what the pool did not account for, not a measurement of it.
Resource Usagetpch — base (merge-base)
tpch — branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing 432586c (432586c) to 432586c diff Run configurationrun benchmark tpcds
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "false"
DATAFUSION_RUNTIME_MEMORY_LIMIT: "32G"
baseline:
ref: "432586c6f82b2a7f96933dc0501cf02f2a2a1eca"
changed:
ref: "432586c6f82b2a7f96933dc0501cf02f2a2a1eca"
env:
DATAFUSION_EXECUTION_PARQUET_DICTIONARY_READ_MAX_DISTINCT_VALUES: "5"
resources:
cpu: "12"
memory: "65Gi"
arch: "amd64"CPU Details (lscpu)Details
Memory Pool PeaksPeak Base:
Pool accounting vs. process RSS Max pool peak is the largest reservation any single query in the run reached; peak RSS covers the whole invocation, including data loading and allocator retention, and the two high-water marks need not coincide in time. The gap is therefore an upper bound on what the pool did not account for, not a measurement of it.
Resource Usagetpcds — base (merge-base)
tpcds — branch
File an issue against this benchmark runner |
|
run benchmark tpch tpcds clickbench_partitioned env:
DATAFUSION_RUNTIME_MEMORY_LIMIT: "32G"
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"
baseline:
ref: "f323f0265f72765c292397d9fed6f2d68f44eb9a"
changed:
ref: "f323f0265f72765c292397d9fed6f2d68f44eb9a"
env:
DATAFUSION_EXECUTION_PARQUET_DICTIONARY_READ_MAX_DISTINCT_VALUES: "40000"
resources:
cpu: "12"
memory: "65Gi"
arch: arm64 |
|
run benchmark tpch tpcds clickbench_partitioned env:
DATAFUSION_RUNTIME_MEMORY_LIMIT: "32G"
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"
baseline:
ref: "f323f0265f72765c292397d9fed6f2d68f44eb9a"
changed:
ref: "f323f0265f72765c292397d9fed6f2d68f44eb9a"
env:
DATAFUSION_EXECUTION_PARQUET_DICTIONARY_READ_MAX_DISTINCT_VALUES: "40000"
resources:
cpu: "12"
memory: "65Gi"
arch: amd64 |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing f323f02 (f323f02) to f323f02 diff Run configurationrun benchmark tpch
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"
DATAFUSION_RUNTIME_MEMORY_LIMIT: "32G"
baseline:
ref: "f323f0265f72765c292397d9fed6f2d68f44eb9a"
changed:
ref: "f323f0265f72765c292397d9fed6f2d68f44eb9a"
env:
DATAFUSION_EXECUTION_PARQUET_DICTIONARY_READ_MAX_DISTINCT_VALUES: "20000"
resources:
cpu: "12"
memory: "65Gi"
arch: "arm64"Results will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing f323f02 (f323f02) to f323f02 diff Run configurationrun benchmark clickbench_partitioned
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"
DATAFUSION_RUNTIME_MEMORY_LIMIT: "32G"
baseline:
ref: "f323f0265f72765c292397d9fed6f2d68f44eb9a"
changed:
ref: "f323f0265f72765c292397d9fed6f2d68f44eb9a"
env:
DATAFUSION_EXECUTION_PARQUET_DICTIONARY_READ_MAX_DISTINCT_VALUES: "40000"
resources:
cpu: "12"
memory: "65Gi"
arch: "arm64"Results will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing f323f02 (f323f02) to f323f02 diff Run configurationrun benchmark tpcds
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"
DATAFUSION_RUNTIME_MEMORY_LIMIT: "32G"
baseline:
ref: "f323f0265f72765c292397d9fed6f2d68f44eb9a"
changed:
ref: "f323f0265f72765c292397d9fed6f2d68f44eb9a"
env:
DATAFUSION_EXECUTION_PARQUET_DICTIONARY_READ_MAX_DISTINCT_VALUES: "20000"
resources:
cpu: "12"
memory: "65Gi"
arch: "arm64"Results will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing f323f02 (f323f02) to f323f02 diff Run configurationrun benchmark tpcds
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"
DATAFUSION_RUNTIME_MEMORY_LIMIT: "32G"
baseline:
ref: "f323f0265f72765c292397d9fed6f2d68f44eb9a"
changed:
ref: "f323f0265f72765c292397d9fed6f2d68f44eb9a"
env:
DATAFUSION_EXECUTION_PARQUET_DICTIONARY_READ_MAX_DISTINCT_VALUES: "40000"
resources:
cpu: "12"
memory: "65Gi"
arch: "arm64"Results will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing f323f02 (f323f02) to f323f02 diff Run configurationrun benchmark tpch
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"
DATAFUSION_RUNTIME_MEMORY_LIMIT: "32G"
baseline:
ref: "f323f0265f72765c292397d9fed6f2d68f44eb9a"
changed:
ref: "f323f0265f72765c292397d9fed6f2d68f44eb9a"
env:
DATAFUSION_EXECUTION_PARQUET_DICTIONARY_READ_MAX_DISTINCT_VALUES: "40000"
resources:
cpu: "12"
memory: "65Gi"
arch: "arm64"Results will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing f323f02 (f323f02) to f323f02 diff Run configurationrun benchmark tpch
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"
DATAFUSION_RUNTIME_MEMORY_LIMIT: "32G"
baseline:
ref: "f323f0265f72765c292397d9fed6f2d68f44eb9a"
changed:
ref: "f323f0265f72765c292397d9fed6f2d68f44eb9a"
env:
DATAFUSION_EXECUTION_PARQUET_DICTIONARY_READ_MAX_DISTINCT_VALUES: "20000"
resources:
cpu: "12"
memory: "65Gi"
arch: "amd64"Results will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing f323f02 (f323f02) to f323f02 diff Run configurationrun benchmark tpcds
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"
DATAFUSION_RUNTIME_MEMORY_LIMIT: "32G"
baseline:
ref: "f323f0265f72765c292397d9fed6f2d68f44eb9a"
changed:
ref: "f323f0265f72765c292397d9fed6f2d68f44eb9a"
env:
DATAFUSION_EXECUTION_PARQUET_DICTIONARY_READ_MAX_DISTINCT_VALUES: "20000"
resources:
cpu: "12"
memory: "65Gi"
arch: "amd64"Results will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing f323f02 (f323f02) to f323f02 diff Run configurationrun benchmark clickbench_partitioned
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"
DATAFUSION_RUNTIME_MEMORY_LIMIT: "32G"
baseline:
ref: "f323f0265f72765c292397d9fed6f2d68f44eb9a"
changed:
ref: "f323f0265f72765c292397d9fed6f2d68f44eb9a"
env:
DATAFUSION_EXECUTION_PARQUET_DICTIONARY_READ_MAX_DISTINCT_VALUES: "20000"
resources:
cpu: "12"
memory: "65Gi"
arch: "arm64"Results will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing f323f02 (f323f02) to f323f02 diff Run configurationrun benchmark tpch
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"
DATAFUSION_RUNTIME_MEMORY_LIMIT: "32G"
baseline:
ref: "f323f0265f72765c292397d9fed6f2d68f44eb9a"
changed:
ref: "f323f0265f72765c292397d9fed6f2d68f44eb9a"
env:
DATAFUSION_EXECUTION_PARQUET_DICTIONARY_READ_MAX_DISTINCT_VALUES: "20000"
resources:
cpu: "12"
memory: "65Gi"
arch: "arm64"CPU Details (lscpu)Details
Memory Pool PeaksPeak Base:
Pool accounting vs. process RSS Max pool peak is the largest reservation any single query in the run reached; peak RSS covers the whole invocation, including data loading and allocator retention, and the two high-water marks need not coincide in time. The gap is therefore an upper bound on what the pool did not account for, not a measurement of it.
Resource Usagetpch — base (merge-base)
tpch — branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing f323f02 (f323f02) to f323f02 diff Run configurationrun benchmark tpch
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"
DATAFUSION_RUNTIME_MEMORY_LIMIT: "32G"
baseline:
ref: "f323f0265f72765c292397d9fed6f2d68f44eb9a"
changed:
ref: "f323f0265f72765c292397d9fed6f2d68f44eb9a"
env:
DATAFUSION_EXECUTION_PARQUET_DICTIONARY_READ_MAX_DISTINCT_VALUES: "40000"
resources:
cpu: "12"
memory: "65Gi"
arch: "arm64"CPU Details (lscpu)Details
Memory Pool PeaksPeak Base:
Pool accounting vs. process RSS Max pool peak is the largest reservation any single query in the run reached; peak RSS covers the whole invocation, including data loading and allocator retention, and the two high-water marks need not coincide in time. The gap is therefore an upper bound on what the pool did not account for, not a measurement of it.
Resource Usagetpch — base (merge-base)
tpch — branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing f323f02 (f323f02) to f323f02 diff Run configurationrun benchmark tpcds
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"
DATAFUSION_RUNTIME_MEMORY_LIMIT: "32G"
baseline:
ref: "f323f0265f72765c292397d9fed6f2d68f44eb9a"
changed:
ref: "f323f0265f72765c292397d9fed6f2d68f44eb9a"
env:
DATAFUSION_EXECUTION_PARQUET_DICTIONARY_READ_MAX_DISTINCT_VALUES: "20000"
resources:
cpu: "12"
memory: "65Gi"
arch: "arm64"CPU Details (lscpu)Details
Memory Pool PeaksPeak Base:
Pool accounting vs. process RSS Max pool peak is the largest reservation any single query in the run reached; peak RSS covers the whole invocation, including data loading and allocator retention, and the two high-water marks need not coincide in time. The gap is therefore an upper bound on what the pool did not account for, not a measurement of it.
Resource Usagetpcds — base (merge-base)
tpcds — branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing f323f02 (f323f02) to f323f02 diff Run configurationrun benchmark tpch
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"
DATAFUSION_RUNTIME_MEMORY_LIMIT: "32G"
baseline:
ref: "f323f0265f72765c292397d9fed6f2d68f44eb9a"
changed:
ref: "f323f0265f72765c292397d9fed6f2d68f44eb9a"
env:
DATAFUSION_EXECUTION_PARQUET_DICTIONARY_READ_MAX_DISTINCT_VALUES: "20000"
resources:
cpu: "12"
memory: "65Gi"
arch: "amd64"CPU Details (lscpu)Details
Memory Pool PeaksPeak Base:
Pool accounting vs. process RSS Max pool peak is the largest reservation any single query in the run reached; peak RSS covers the whole invocation, including data loading and allocator retention, and the two high-water marks need not coincide in time. The gap is therefore an upper bound on what the pool did not account for, not a measurement of it.
Resource Usagetpch — base (merge-base)
tpch — branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing f323f02 (f323f02) to f323f02 diff Run configurationrun benchmark tpcds
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"
DATAFUSION_RUNTIME_MEMORY_LIMIT: "32G"
baseline:
ref: "f323f0265f72765c292397d9fed6f2d68f44eb9a"
changed:
ref: "f323f0265f72765c292397d9fed6f2d68f44eb9a"
env:
DATAFUSION_EXECUTION_PARQUET_DICTIONARY_READ_MAX_DISTINCT_VALUES: "40000"
resources:
cpu: "12"
memory: "65Gi"
arch: "arm64"CPU Details (lscpu)Details
Memory Pool PeaksPeak Base:
Pool accounting vs. process RSS Max pool peak is the largest reservation any single query in the run reached; peak RSS covers the whole invocation, including data loading and allocator retention, and the two high-water marks need not coincide in time. The gap is therefore an upper bound on what the pool did not account for, not a measurement of it.
Resource Usagetpcds — base (merge-base)
tpcds — branch
File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing f323f02 (f323f02) to f323f02 diff Run configurationrun benchmark clickbench_partitioned
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"
DATAFUSION_RUNTIME_MEMORY_LIMIT: "32G"
baseline:
ref: "f323f0265f72765c292397d9fed6f2d68f44eb9a"
changed:
ref: "f323f0265f72765c292397d9fed6f2d68f44eb9a"
env:
DATAFUSION_EXECUTION_PARQUET_DICTIONARY_READ_MAX_DISTINCT_VALUES: "20000"
resources:
cpu: "12"
memory: "65Gi"
arch: "amd64"Results will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing f323f02 (f323f02) to f323f02 diff Run configurationrun benchmark tpcds
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"
DATAFUSION_RUNTIME_MEMORY_LIMIT: "32G"
baseline:
ref: "f323f0265f72765c292397d9fed6f2d68f44eb9a"
changed:
ref: "f323f0265f72765c292397d9fed6f2d68f44eb9a"
env:
DATAFUSION_EXECUTION_PARQUET_DICTIONARY_READ_MAX_DISTINCT_VALUES: "20000"
resources:
cpu: "12"
memory: "65Gi"
arch: "amd64"CPU Details (lscpu)Details
Memory Pool PeaksPeak Base:
Pool accounting vs. process RSS Max pool peak is the largest reservation any single query in the run reached; peak RSS covers the whole invocation, including data loading and allocator retention, and the two high-water marks need not coincide in time. The gap is therefore an upper bound on what the pool did not account for, not a measurement of it.
Resource Usagetpcds — base (merge-base)
tpcds — branch
File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing f323f02 (f323f02) to f323f02 diff Run configurationrun benchmark tpch
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"
DATAFUSION_RUNTIME_MEMORY_LIMIT: "32G"
baseline:
ref: "f323f0265f72765c292397d9fed6f2d68f44eb9a"
changed:
ref: "f323f0265f72765c292397d9fed6f2d68f44eb9a"
env:
DATAFUSION_EXECUTION_PARQUET_DICTIONARY_READ_MAX_DISTINCT_VALUES: "40000"
resources:
cpu: "12"
memory: "65Gi"
arch: "amd64"Results will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing f323f02 (f323f02) to f323f02 diff Run configurationrun benchmark clickbench_partitioned
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"
DATAFUSION_RUNTIME_MEMORY_LIMIT: "32G"
baseline:
ref: "f323f0265f72765c292397d9fed6f2d68f44eb9a"
changed:
ref: "f323f0265f72765c292397d9fed6f2d68f44eb9a"
env:
DATAFUSION_EXECUTION_PARQUET_DICTIONARY_READ_MAX_DISTINCT_VALUES: "40000"
resources:
cpu: "12"
memory: "65Gi"
arch: "amd64"Results will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing f323f02 (f323f02) to f323f02 diff Run configurationrun benchmark tpcds
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"
DATAFUSION_RUNTIME_MEMORY_LIMIT: "32G"
baseline:
ref: "f323f0265f72765c292397d9fed6f2d68f44eb9a"
changed:
ref: "f323f0265f72765c292397d9fed6f2d68f44eb9a"
env:
DATAFUSION_EXECUTION_PARQUET_DICTIONARY_READ_MAX_DISTINCT_VALUES: "40000"
resources:
cpu: "12"
memory: "65Gi"
arch: "amd64"Results will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing f323f02 (f323f02) to f323f02 diff Run configurationrun benchmark clickbench_partitioned
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"
DATAFUSION_RUNTIME_MEMORY_LIMIT: "32G"
baseline:
ref: "f323f0265f72765c292397d9fed6f2d68f44eb9a"
changed:
ref: "f323f0265f72765c292397d9fed6f2d68f44eb9a"
env:
DATAFUSION_EXECUTION_PARQUET_DICTIONARY_READ_MAX_DISTINCT_VALUES: "40000"
resources:
cpu: "12"
memory: "65Gi"
arch: "arm64"CPU Details (lscpu)Details
Memory Pool PeaksPeak Base:
Pool accounting vs. process RSS Max pool peak is the largest reservation any single query in the run reached; peak RSS covers the whole invocation, including data loading and allocator retention, and the two high-water marks need not coincide in time. The gap is therefore an upper bound on what the pool did not account for, not a measurement of it.
Resource Usageclickbench_partitioned — base (merge-base)
clickbench_partitioned — branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing f323f02 (f323f02) to f323f02 diff Run configurationrun benchmark clickbench_partitioned
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"
DATAFUSION_RUNTIME_MEMORY_LIMIT: "32G"
baseline:
ref: "f323f0265f72765c292397d9fed6f2d68f44eb9a"
changed:
ref: "f323f0265f72765c292397d9fed6f2d68f44eb9a"
env:
DATAFUSION_EXECUTION_PARQUET_DICTIONARY_READ_MAX_DISTINCT_VALUES: "20000"
resources:
cpu: "12"
memory: "65Gi"
arch: "arm64"CPU Details (lscpu)Details
Memory Pool PeaksPeak Base:
Pool accounting vs. process RSS Max pool peak is the largest reservation any single query in the run reached; peak RSS covers the whole invocation, including data loading and allocator retention, and the two high-water marks need not coincide in time. The gap is therefore an upper bound on what the pool did not account for, not a measurement of it.
Resource Usageclickbench_partitioned — base (merge-base)
clickbench_partitioned — branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing f323f02 (f323f02) to f323f02 diff Run configurationrun benchmark tpch
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"
DATAFUSION_RUNTIME_MEMORY_LIMIT: "32G"
baseline:
ref: "f323f0265f72765c292397d9fed6f2d68f44eb9a"
changed:
ref: "f323f0265f72765c292397d9fed6f2d68f44eb9a"
env:
DATAFUSION_EXECUTION_PARQUET_DICTIONARY_READ_MAX_DISTINCT_VALUES: "40000"
resources:
cpu: "12"
memory: "65Gi"
arch: "amd64"CPU Details (lscpu)Details
Memory Pool PeaksPeak Base:
Pool accounting vs. process RSS Max pool peak is the largest reservation any single query in the run reached; peak RSS covers the whole invocation, including data loading and allocator retention, and the two high-water marks need not coincide in time. The gap is therefore an upper bound on what the pool did not account for, not a measurement of it.
Resource Usagetpch — base (merge-base)
tpch — branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing f323f02 (f323f02) to f323f02 diff Run configurationrun benchmark tpcds
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"
DATAFUSION_RUNTIME_MEMORY_LIMIT: "32G"
baseline:
ref: "f323f0265f72765c292397d9fed6f2d68f44eb9a"
changed:
ref: "f323f0265f72765c292397d9fed6f2d68f44eb9a"
env:
DATAFUSION_EXECUTION_PARQUET_DICTIONARY_READ_MAX_DISTINCT_VALUES: "40000"
resources:
cpu: "12"
memory: "65Gi"
arch: "amd64"CPU Details (lscpu)Details
Memory Pool PeaksPeak Base:
Pool accounting vs. process RSS Max pool peak is the largest reservation any single query in the run reached; peak RSS covers the whole invocation, including data loading and allocator retention, and the two high-water marks need not coincide in time. The gap is therefore an upper bound on what the pool did not account for, not a measurement of it.
Resource Usagetpcds — base (merge-base)
tpcds — branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing f323f02 (f323f02) to f323f02 diff Run configurationrun benchmark clickbench_partitioned
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"
DATAFUSION_RUNTIME_MEMORY_LIMIT: "32G"
baseline:
ref: "f323f0265f72765c292397d9fed6f2d68f44eb9a"
changed:
ref: "f323f0265f72765c292397d9fed6f2d68f44eb9a"
env:
DATAFUSION_EXECUTION_PARQUET_DICTIONARY_READ_MAX_DISTINCT_VALUES: "20000"
resources:
cpu: "12"
memory: "65Gi"
arch: "amd64"CPU Details (lscpu)Details
Memory Pool PeaksPeak Base:
Pool accounting vs. process RSS Max pool peak is the largest reservation any single query in the run reached; peak RSS covers the whole invocation, including data loading and allocator retention, and the two high-water marks need not coincide in time. The gap is therefore an upper bound on what the pool did not account for, not a measurement of it.
Resource Usageclickbench_partitioned — base (merge-base)
clickbench_partitioned — branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing f323f02 (f323f02) to f323f02 diff Run configurationrun benchmark clickbench_partitioned
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"
DATAFUSION_RUNTIME_MEMORY_LIMIT: "32G"
baseline:
ref: "f323f0265f72765c292397d9fed6f2d68f44eb9a"
changed:
ref: "f323f0265f72765c292397d9fed6f2d68f44eb9a"
env:
DATAFUSION_EXECUTION_PARQUET_DICTIONARY_READ_MAX_DISTINCT_VALUES: "40000"
resources:
cpu: "12"
memory: "65Gi"
arch: "amd64"CPU Details (lscpu)Details
Memory Pool PeaksPeak Base:
Pool accounting vs. process RSS Max pool peak is the largest reservation any single query in the run reached; peak RSS covers the whole invocation, including data loading and allocator retention, and the two high-water marks need not coincide in time. The gap is therefore an upper bound on what the pool did not account for, not a measurement of it.
Resource Usageclickbench_partitioned — base (merge-base)
clickbench_partitioned — branch
File an issue against this benchmark runner |
|
thanks for taking a look @jayzhan211, I updated the PR in the latest commit to address your comments. I also provided for context in the pr description for different cases. thanks 🙇 |
jayzhan211
left a comment
There was a problem hiding this comment.
@Rich-T-kid a few optional items on the description:
- The benchmark table compares the cap with unbounded promotion, but a user deciding whether to turn the option on starts from the default (plain reads). ClickBench
hitshas no NDV statistics and its row groups hold 450k–549k rows, socap = 20000promotes no ClickBench column (I checked withinformation_schema.columns). The −51% therefore means "unbounded promotion is about 2x slower than plain", not that the cap speeds anything up. One run of cap vs default would show what the option buys. - The Rationale has no number. Something like: "Reading a low-cardinality Parquet string column as a dictionary makes
GROUP BYon it about 2x faster (30k distinct 100-byte values: 0.09 s → 0.045 s), but enablingenable_rle_to_dictionaryfor every column makes high-cardinality columns several times slower (295k distinct values per row group: 4.4x). This PR adds a cap so only columns whose estimated cardinality is under it are read as dictionaries."
| if row_count > 0 && column.dictionary_page_offset().is_none() { | ||
| return None; | ||
| } |
There was a problem hiding this comment.
@Rich-T-kid thanks for the updates! This check still promotes a column chunk whose dictionary outgrew the writer's page limit, so its later pages are PLAIN. With cap = 40000, a file of 100-byte strings with 30k distinct values per row group is promoted, and its queries run 1.2–2.5x slower than plain reads. Requiring every data page to be dictionary-encoded, as your reopen comment on #24113 describes, keeps that file plain. page_encoding_stats_mask is decoded by default, so the check needs no extra I/O:
| if row_count > 0 && column.dictionary_page_offset().is_none() { | |
| return None; | |
| } | |
| // A chunk whose dictionary outgrew the writer's page limit | |
| // continues in PLAIN pages, and reads slower as a dictionary | |
| // than as plain values. | |
| let all_dictionary_pages = column | |
| .dictionary_page_offset() | |
| .is_some() | |
| && column.page_encoding_stats_mask().is_some_and(|mask| { | |
| mask.is_only(parquet::basic::Encoding::RLE_DICTIONARY) | |
| || mask.is_only( | |
| parquet::basic::Encoding::PLAIN_DICTIONARY, | |
| ) | |
| }); | |
| if row_count > 0 && !all_dictionary_pages { | |
| return None; | |
| } |
Files without page encoding stats then stay plain under the cap. The docs that say "every nonempty row group has a dictionary page" would change to "every data page is dictionary encoded".
Measurements and a regression test (fails on this head with `Dictionary(Int32, Utf8)`, passes with the change)
Data: 4 files × 2M rows, one row group each, cat = concat(repeat('x', 95), lpad(CAST(value % 30000 AS VARCHAR), 5, '0')), NDV written, default dictionary_page_size_limit (1 MiB < 30k × 100 B, so the writer falls back to PLAIN). Median of 5, three rounds, datafusion-cli release-nonlto, M-series laptop. Results are identical in both modes.
| query | plain | cap = 40000 (promoted) |
|---|---|---|
SELECT cat, count(*), sum(v) FROM t GROUP BY cat |
0.096–0.133 s | 0.142–0.248 s |
SELECT count(DISTINCT cat) FROM t |
0.074–0.113 s | 0.121–0.190 s |
SELECT count(*) FROM t WHERE cat LIKE '%12345' |
0.037–0.045 s | 0.076–0.112 s |
With the change, the files stay Utf8View. The same data written with a 64 MiB page limit (no fallback) is still promoted and still wins on the group-by (0.090–0.126 s → 0.042–0.062 s).
Test, placed after DROP TABLE threshold_no_dict_page;:
# A column whose dictionary page outgrew its size limit continues in PLAIN
# pages, so it stays plain even when its NDV is within the cap.
query I
COPY (SELECT column1 AS fallback_page FROM (VALUES ('a'), ('b')))
TO 'test_files/scratch/parquet_rle_to_dictionary/threshold_fallback_page/p.parquet'
STORED AS PARQUET OPTIONS (
'format.dictionary_page_size_limit' 1,
'format.write_batch_size' 1,
'format.data_page_row_count_limit' 1
);
----
2
statement ok
CREATE EXTERNAL TABLE threshold_fallback_page STORED AS PARQUET LOCATION 'test_files/scratch/parquet_rle_to_dictionary/threshold_fallback_page/';
query T
SELECT DISTINCT arrow_typeof(fallback_page) FROM threshold_fallback_page;
----
Utf8View
statement ok
DROP TABLE threshold_fallback_page;
| /// Large row-group dictionaries can be slower because each output batch can | ||
| /// retain the whole dictionary. The crossover is workload-dependent: in one | ||
| /// 20-million-row group-by benchmark with 40-byte strings, the case with about | ||
| /// 100,000 distinct values per row group was faster than plain reads, while the | ||
| /// case with about 295,000 was slower. Use a conservative cap and benchmark | ||
| /// representative queries. |
There was a problem hiding this comment.
Optional cleanup: the cap counts the per-file sum, but these numbers are per row group, so a reader who sets cap = 100000 from them won't promote the faster file. That benchmark wrote five row groups per file, so its per-file sums were 500,000 and about 1,460,000; with cap = 100000 the 100k file stays Utf8View, and it is promoted only at cap = 500000. Stating the numbers in the cap's unit:
| /// Large row-group dictionaries can be slower because each output batch can | |
| /// retain the whole dictionary. The crossover is workload-dependent: in one | |
| /// 20-million-row group-by benchmark with 40-byte strings, the case with about | |
| /// 100,000 distinct values per row group was faster than plain reads, while the | |
| /// case with about 295,000 was slower. Use a conservative cap and benchmark | |
| /// representative queries. | |
| /// Large row-group dictionaries can be slower because each output batch can | |
| /// retain the whole dictionary. The crossover is workload-dependent: in one | |
| /// 20-million-row group-by benchmark with 40-byte strings and five row groups | |
| /// per file, files with a per-file sum of about 500,000 (100,000 per row group) | |
| /// were faster than plain reads, while files with about 1,460,000 (about | |
| /// 295,000 per row group) were slower. Use a conservative cap and benchmark | |
| /// representative queries. |
configs.md and information_schema.slt would need regenerating.
Which issue does this PR close?
Rationale for this change
Low-cardinality Parquet string and binary columns can be cheaper to decode and process as dictionaries. Large row-group dictionaries can instead be slower because each output batch can retain the whole dictionary.
For a multi-file table, keeping a dictionary table type when any file remains plain would require dictionary-encoding that file's values during reading or coercion. That adds conversion and memory overhead, so the conservative choice is to keep the field plain when files disagree.
What changes are included in this PR?
Adds
datafusion.execution.parquet.dictionary_read_max_distinct_values, a cap for promoting inferred Parquet string and binary columns to dictionaries.Promotion behavior
This option applies when DataFusion infers the table schema. A user-supplied plain schema is not promoted.
cap.null_countis available; otherwise its total row count is used. Missing NDV therefore does not automatically prevent promotion.dictionary_read_max_distinct_valuesenables dictionary reads by itself. When both options are set, the cap controls promotion. Without a cap,enable_rle_to_dictionary = trueretains the existing unbounded behavior.Row-group estimates are deliberately conservative: values repeated across row groups may be counted more than once.
What is the testing strategy for this PR?
SQL logic tests cover cap-only activation, exact and above-cap estimates, the dictionary-page requirement, multi-file reconciliation, declared dictionaries, and precedence over the legacy flag. Focused metadata coverage checks the non-null-row fallback for missing NDV. Serialization coverage checks that the new option round trips. Formatting, workspace Clippy, and typo checks pass.
Benchmark evidence
Hosted benchmarks on final head
f323f0265compare the existing unbounded behavior (enable_rle_to_dictionary = truewith no cap) against the same flag withdictionary_read_max_distinct_valuesset to 20,000 or 40,000. Both sides use the same commit, so the cap is the only changed setting. Each comparison ran TPC-H, TPC-DS, and partitioned ClickBench on arm64 and amd64 with 12 CPUs and 65 GiB.Negative values mean the capped run was faster.
All 12 suite jobs completed successfully. The 20,000 cap improved all six suite totals. The 40,000 cap improved five of six; arm64 TPC-DS was 1.60% slower. Both caps reduced partitioned ClickBench total time by about 51% on arm64 and 53% on amd64 compared with unbounded promotion.
Three per-query median regressions reproduced with both caps on both architectures, and the distribution comparison also classified each as slower in all four runs: TPC-H Q19 (9–13%), TPC-DS Q15 (6–9%), and ClickBench Q28 (7–18%). The crossover remains workload-dependent, so the option documentation recommends a conservative cap and representative benchmarking.
Are there any user-facing changes?
Yes.
ParquetOptionsgainsdictionary_read_max_distinct_values: Option<usize>. Exhaustive struct literals must add the field or use..Default::default(). The default isNone, so existing behavior is unchanged.