Repository navigation
Conversation
|
run benchmark |
|
Hi @Dandandan, Usage: Any benchmark name is accepted: Per-side configuration ( env:
# shared env is inherited by BOTH the build and the run, so build
# flags go here. Builds default to no debuginfo for speed; opt back
# in for hung-job gdb dumps and cap jobs to stay within memory:
CARGO_PROFILE_RELEASE_DEBUG: "1"
CARGO_BUILD_JOBS: "1"
baseline:
ref: v45.0.0
env:
# per-side env only reaches the benchmark run, not the build
DATAFUSION_RUNTIME_MEMORY_LIMIT: 1G
changed:
ref: v46.0.0
env:
DATAFUSION_RUNTIME_MEMORY_LIMIT: 2GPod size and CPU architecture (each key is optional and keeps the default when absent; requests and limits are set equal): resources:
cpu: "16"
memory: "128Gi"
arch: arm64 # arm64 or amd64Independent benchmark shards (optional; 1–8, default 1): run benchmark arrow_writer
shards: 4Each shard builds both revisions, measures its assigned cases, and posts its own report. Resources and the job deadline apply per shard. DataFusion requests specifying a shard count greater than 1 fail with an error. Omit shards or use shards: 1 for DataFusion. File an issue against this benchmark runner |
|
run benchmarks tpch tpcds |
|
run benchmarks |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing perf/unique-build-distribution (c57ebeb) to ecfa46a (merge-base) diff Run configurationrun benchmark tpchResults will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing perf/unique-build-distribution (c57ebeb) to ecfa46a (merge-base) diff Run configurationrun benchmark tpcdsResults will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing perf/unique-build-distribution (c57ebeb) to ecfa46a (merge-base) diff Run configurationrun benchmark tpchCPU Details (lscpu)Details
Resource Usagetpch — base (merge-base)
tpch — branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing perf/unique-build-distribution (c57ebeb) to ecfa46a (merge-base) diff Run configurationrun benchmark tpcdsCPU Details (lscpu)Details
Resource Usagetpcds — base (merge-base)
tpcds — branch
File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing perf/unique-build-distribution (c57ebeb) to ecfa46a (merge-base) diff Run configurationrun benchmark clickbench_partitionedResults will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing perf/unique-build-distribution (c57ebeb) to ecfa46a (merge-base) diff Run configurationrun benchmark tpcdsResults will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing perf/unique-build-distribution (c57ebeb) to ecfa46a (merge-base) diff Run configurationrun benchmark tpchResults will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing perf/unique-build-distribution (c57ebeb) to ecfa46a (merge-base) diff Run configurationrun benchmark tpcdsCPU Details (lscpu)Details
Resource Usagetpcds — base (merge-base)
tpcds — branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing perf/unique-build-distribution (c57ebeb) to ecfa46a (merge-base) diff Run configurationrun benchmark tpchCPU Details (lscpu)Details
Resource Usagetpch — base (merge-base)
tpch — branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing perf/unique-build-distribution (c57ebeb) to ecfa46a (merge-base) diff Run configurationrun benchmark clickbench_partitionedCPU Details (lscpu)Details
Resource Usageclickbench_partitioned — base (merge-base)
clickbench_partitioned — branch
File an issue against this benchmark runner |
Rationale for this change
An intermediate inner join can appear too large to broadcast even when one input is unique on the join key and the estimated opposite input is small. Choosing a partitioned hash join then adds an exchange.
What changes are included in this PR?
Use a declared single-column unique or primary key on either input of an ordinary equality lookup to bound the estimated intermediate rows. For fixed-width projections, use the estimate as an additional way to satisfy both existing single-partition thresholds. Include an allowance for row validity and hash table entries.
What is the testing strategy for this PR?
A physical optimizer regression supplies a pessimistic intermediate estimate and checks the outer join distribution with and without the uniqueness constraint. Benchmark coverage depends on the source exposing that constraint. Plain Parquet inputs can therefore retain their original plans.
Are there any user-facing changes?
The optimizer can collect the build side of eligible joins. The existing byte and row thresholds control this decision.