Optimize planning for projected nested union

[logan-keede]

Which issue does this PR close?

• Should close Planning time for queries with many columns with union and order by is very slow #17261.

Rationale for this change

as explained in #17261 (comment) and discussion in the above mentioned issue

What changes are included in this PR?

Projected Nested Union are now merged with their parent union.

Are these changes tested?

Tested it by Github CI. Works for the scenario discussed in the above mentioned issue.

> explain analyze (SELECT c0, 1 as c1, 2 as c2, 3 as c3, 4 as c4, 5 as c5, 6 as c6, 7 as c7, 8 as c8, 9 as c9 FROM t1 ORDER BY c0)
  UNION ALL 
(SELECT 0 as c0, c1, 2 as c2, 3 as c3, 4 as c4, 5 as c5, 6 as c6, 7 as c7, 8 as c8, 9 as c9 FROM t1 ORDER BY c1)
  UNION ALL 
(SELECT 0 as c0, 1 as c1, c2, 3 as c3, 4 as c4, 5 as c5, 6 as c6, 7 as c7, 8 as c8, 9 as c9 FROM t1 ORDER BY c2)
  UNION ALL 
(SELECT 0 as c0, 1 as c1, 2 as c2, c3, 4 as c4, 5 as c5, 6 as c6, 7 as c7, 8 as c8, 9 as c9 FROM t1 ORDER BY c3)
  UNION ALL 
(SELECT 0 as c0, 1 as c1, 2 as c2, 3 as c3, c4, 5 as c5, 6 as c6, 7 as c7, 8 as c8, 9 as c9 FROM t1 ORDER BY c4)
  UNION ALL 
(SELECT 0 as c0, 1 as c1, 2 as c2, 3 as c3, 4 as c4, c5, 6 as c6, 7 as c7, 8 as c8, 9 as c9 FROM t1 ORDER BY c5)
  UNION ALL 
(SELECT 0 as c0, 1 as c1, 2 as c2, 3 as c3, 4 as c4, 5 as c5, c6, 7 as c7, 8 as c8, 9 as c9 FROM t1 ORDER BY c6)
  UNION ALL 
(SELECT 0 as c0, 1 as c1, 2 as c2, 3 as c3, 4 as c4, 5 as c5, 6 as c6, c7, 8 as c8, 9 as c9 FROM t1 ORDER BY c7)
  UNION ALL 
(SELECT 0 as c0, 1 as c1, 2 as c2, 3 as c3, 4 as c4, 5 as c5, 6 as c6, 7 as c7, c8, 9 as c9 FROM t1 ORDER BY c8)
  UNION ALL 
(SELECT 0 as c0, 1 as c1, 2 as c2, 3 as c3, 4 as c4, 5 as c5, 6 as c6, 7 as c7, 8 as c8, c9 FROM t1 ORDER BY c9)
ORDER BY c0, c1, c2, c3, c4, c5, c6, c7, c8, c9;
+-------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| plan_type         | plan                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |
+-------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Plan with Metrics | SortPreservingMergeExec: [c0@0 ASC NULLS LAST, c1@1 ASC NULLS LAST, c2@2 ASC NULLS LAST, c3@3 ASC NULLS LAST, c4@4 ASC NULLS LAST, c5@5 ASC NULLS LAST, c6@6 ASC NULLS LAST, c7@7 ASC NULLS LAST, c8@8 ASC NULLS LAST, c9@9 ASC NULLS LAST], metrics=[output_rows=0, elapsed_compute=3.817µs, output_bytes=0.0 B, output_batches=0]                                                                                                                                                        |
|                   |   UnionExec, metrics=[output_rows=0, elapsed_compute=345.775µs, output_bytes=0.0 B, output_batches=0]                                                                                                                                                                                                                                                                                                                                                                                      |
|                   |     SortExec: expr=[c0@0 ASC NULLS LAST], preserve_partitioning=[false], metrics=[output_rows=0, elapsed_compute=1ns, output_bytes=0.0 B, output_batches=0, spill_count=0, spilled_bytes=0.0 B, spilled_rows=0, batches_split=0]                                                                                                                                                                                                                                                           |
|                   |       ProjectionExec: expr=[CAST(c0@0 AS Decimal128(20, 0)) as c0, CAST(1 AS Decimal128(20, 0)) as c1, CAST(2 AS Decimal128(20, 0)) as c2, CAST(3 AS Decimal128(20, 0)) as c3, CAST(4 AS Decimal128(20, 0)) as c4, CAST(5 AS Decimal128(20, 0)) as c5, CAST(6 AS Decimal128(20, 0)) as c6, CAST(7 AS Decimal128(20, 0)) as c7, CAST(8 AS Decimal128(20, 0)) as c8, CAST(9 AS Decimal128(20, 0)) as c9], metrics=[output_rows=0, elapsed_compute=1ns, output_bytes=0.0 B, output_batches=0] |
|                   |         DataSourceExec: partitions=1, partition_sizes=[0], metrics=[]                                                                                                                                                                                                                                                                                                                                                                                                                      |
|                   |     SortExec: expr=[c1@1 ASC NULLS LAST], preserve_partitioning=[false], metrics=[output_rows=0, elapsed_compute=1ns, output_bytes=0.0 B, output_batches=0, spill_count=0, spilled_bytes=0.0 B, spilled_rows=0, batches_split=0]                                                                                                                                                                                                                                                           |
|                   |       ProjectionExec: expr=[CAST(0 AS Decimal128(20, 0)) as c0, CAST(c1@0 AS Decimal128(20, 0)) as c1, CAST(2 AS Decimal128(20, 0)) as c2, CAST(3 AS Decimal128(20, 0)) as c3, CAST(4 AS Decimal128(20, 0)) as c4, CAST(5 AS Decimal128(20, 0)) as c5, CAST(6 AS Decimal128(20, 0)) as c6, CAST(7 AS Decimal128(20, 0)) as c7, CAST(8 AS Decimal128(20, 0)) as c8, CAST(9 AS Decimal128(20, 0)) as c9], metrics=[output_rows=0, elapsed_compute=1ns, output_bytes=0.0 B, output_batches=0] |
|                   |         DataSourceExec: partitions=1, partition_sizes=[0], metrics=[]                                                                                                                                                                                                                                                                                                                                                                                                                      |
|                   |     SortExec: expr=[c2@2 ASC NULLS LAST], preserve_partitioning=[false], metrics=[output_rows=0, elapsed_compute=1ns, output_bytes=0.0 B, output_batches=0, spill_count=0, spilled_bytes=0.0 B, spilled_rows=0, batches_split=0]                                                                                                                                                                                                                                                           |
|                   |       ProjectionExec: expr=[CAST(0 AS Decimal128(20, 0)) as c0, CAST(1 AS Decimal128(20, 0)) as c1, CAST(c2@0 AS Decimal128(20, 0)) as c2, CAST(3 AS Decimal128(20, 0)) as c3, CAST(4 AS Decimal128(20, 0)) as c4, CAST(5 AS Decimal128(20, 0)) as c5, CAST(6 AS Decimal128(20, 0)) as c6, CAST(7 AS Decimal128(20, 0)) as c7, CAST(8 AS Decimal128(20, 0)) as c8, CAST(9 AS Decimal128(20, 0)) as c9], metrics=[output_rows=0, elapsed_compute=1ns, output_bytes=0.0 B, output_batches=0] |
|                   |         DataSourceExec: partitions=1, partition_sizes=[0], metrics=[]                                                                                                                                                                                                                                                                                                                                                                                                                      |
|                   |     SortExec: expr=[c3@3 ASC NULLS LAST], preserve_partitioning=[false], metrics=[output_rows=0, elapsed_compute=1ns, output_bytes=0.0 B, output_batches=0, spill_count=0, spilled_bytes=0.0 B, spilled_rows=0, batches_split=0]                                                                                                                                                                                                                                                           |
|                   |       ProjectionExec: expr=[CAST(0 AS Decimal128(20, 0)) as c0, CAST(1 AS Decimal128(20, 0)) as c1, CAST(2 AS Decimal128(20, 0)) as c2, CAST(c3@0 AS Decimal128(20, 0)) as c3, CAST(4 AS Decimal128(20, 0)) as c4, CAST(5 AS Decimal128(20, 0)) as c5, CAST(6 AS Decimal128(20, 0)) as c6, CAST(7 AS Decimal128(20, 0)) as c7, CAST(8 AS Decimal128(20, 0)) as c8, CAST(9 AS Decimal128(20, 0)) as c9], metrics=[output_rows=0, elapsed_compute=1ns, output_bytes=0.0 B, output_batches=0] |
|                   |         DataSourceExec: partitions=1, partition_sizes=[0], metrics=[]                                                                                                                                                                                                                                                                                                                                                                                                                      |
|                   |     SortExec: expr=[c4@4 ASC NULLS LAST], preserve_partitioning=[false], metrics=[output_rows=0, elapsed_compute=1ns, output_bytes=0.0 B, output_batches=0, spill_count=0, spilled_bytes=0.0 B, spilled_rows=0, batches_split=0]                                                                                                                                                                                                                                                           |
|                   |       ProjectionExec: expr=[CAST(0 AS Decimal128(20, 0)) as c0, CAST(1 AS Decimal128(20, 0)) as c1, CAST(2 AS Decimal128(20, 0)) as c2, CAST(3 AS Decimal128(20, 0)) as c3, CAST(c4@0 AS Decimal128(20, 0)) as c4, CAST(5 AS Decimal128(20, 0)) as c5, CAST(6 AS Decimal128(20, 0)) as c6, CAST(7 AS Decimal128(20, 0)) as c7, CAST(8 AS Decimal128(20, 0)) as c8, CAST(9 AS Decimal128(20, 0)) as c9], metrics=[output_rows=0, elapsed_compute=1ns, output_bytes=0.0 B, output_batches=0] |
|                   |         DataSourceExec: partitions=1, partition_sizes=[0], metrics=[]                                                                                                                                                                                                                                                                                                                                                                                                                      |
|                   |     SortExec: expr=[c5@5 ASC NULLS LAST], preserve_partitioning=[false], metrics=[output_rows=0, elapsed_compute=1ns, output_bytes=0.0 B, output_batches=0, spill_count=0, spilled_bytes=0.0 B, spilled_rows=0, batches_split=0]                                                                                                                                                                                                                                                           |
|                   |       ProjectionExec: expr=[CAST(0 AS Decimal128(20, 0)) as c0, CAST(1 AS Decimal128(20, 0)) as c1, CAST(2 AS Decimal128(20, 0)) as c2, CAST(3 AS Decimal128(20, 0)) as c3, CAST(4 AS Decimal128(20, 0)) as c4, CAST(c5@0 AS Decimal128(20, 0)) as c5, CAST(6 AS Decimal128(20, 0)) as c6, CAST(7 AS Decimal128(20, 0)) as c7, CAST(8 AS Decimal128(20, 0)) as c8, CAST(9 AS Decimal128(20, 0)) as c9], metrics=[output_rows=0, elapsed_compute=1ns, output_bytes=0.0 B, output_batches=0] |
|                   |         DataSourceExec: partitions=1, partition_sizes=[0], metrics=[]                                                                                                                                                                                                                                                                                                                                                                                                                      |
|                   |     SortExec: expr=[c6@6 ASC NULLS LAST], preserve_partitioning=[false], metrics=[output_rows=0, elapsed_compute=1ns, output_bytes=0.0 B, output_batches=0, spill_count=0, spilled_bytes=0.0 B, spilled_rows=0, batches_split=0]                                                                                                                                                                                                                                                           |
|                   |       ProjectionExec: expr=[CAST(0 AS Decimal128(20, 0)) as c0, CAST(1 AS Decimal128(20, 0)) as c1, CAST(2 AS Decimal128(20, 0)) as c2, CAST(3 AS Decimal128(20, 0)) as c3, CAST(4 AS Decimal128(20, 0)) as c4, CAST(5 AS Decimal128(20, 0)) as c5, CAST(c6@0 AS Decimal128(20, 0)) as c6, CAST(7 AS Decimal128(20, 0)) as c7, CAST(8 AS Decimal128(20, 0)) as c8, CAST(9 AS Decimal128(20, 0)) as c9], metrics=[output_rows=0, elapsed_compute=1ns, output_bytes=0.0 B, output_batches=0] |
|                   |         DataSourceExec: partitions=1, partition_sizes=[0], metrics=[]                                                                                                                                                                                                                                                                                                                                                                                                                      |
|                   |     SortExec: expr=[c7@7 ASC NULLS LAST], preserve_partitioning=[false], metrics=[output_rows=0, elapsed_compute=1ns, output_bytes=0.0 B, output_batches=0, spill_count=0, spilled_bytes=0.0 B, spilled_rows=0, batches_split=0]                                                                                                                                                                                                                                                           |
|                   |       ProjectionExec: expr=[CAST(0 AS Decimal128(20, 0)) as c0, CAST(1 AS Decimal128(20, 0)) as c1, CAST(2 AS Decimal128(20, 0)) as c2, CAST(3 AS Decimal128(20, 0)) as c3, CAST(4 AS Decimal128(20, 0)) as c4, CAST(5 AS Decimal128(20, 0)) as c5, CAST(6 AS Decimal128(20, 0)) as c6, CAST(c7@0 AS Decimal128(20, 0)) as c7, CAST(8 AS Decimal128(20, 0)) as c8, CAST(9 AS Decimal128(20, 0)) as c9], metrics=[output_rows=0, elapsed_compute=1ns, output_bytes=0.0 B, output_batches=0] |
|                   |         DataSourceExec: partitions=1, partition_sizes=[0], metrics=[]                                                                                                                                                                                                                                                                                                                                                                                                                      |
|                   |     SortExec: expr=[c8@8 ASC NULLS LAST], preserve_partitioning=[false], metrics=[output_rows=0, elapsed_compute=1ns, output_bytes=0.0 B, output_batches=0, spill_count=0, spilled_bytes=0.0 B, spilled_rows=0, batches_split=0]                                                                                                                                                                                                                                                           |
|                   |       ProjectionExec: expr=[CAST(0 AS Decimal128(20, 0)) as c0, CAST(1 AS Decimal128(20, 0)) as c1, CAST(2 AS Decimal128(20, 0)) as c2, CAST(3 AS Decimal128(20, 0)) as c3, CAST(4 AS Decimal128(20, 0)) as c4, CAST(5 AS Decimal128(20, 0)) as c5, CAST(6 AS Decimal128(20, 0)) as c6, CAST(7 AS Decimal128(20, 0)) as c7, CAST(c8@0 AS Decimal128(20, 0)) as c8, CAST(9 AS Decimal128(20, 0)) as c9], metrics=[output_rows=0, elapsed_compute=1ns, output_bytes=0.0 B, output_batches=0] |
|                   |         DataSourceExec: partitions=1, partition_sizes=[0], metrics=[]                                                                                                                                                                                                                                                                                                                                                                                                                      |
|                   |     SortExec: expr=[c9@9 ASC NULLS LAST], preserve_partitioning=[false], metrics=[output_rows=0, elapsed_compute=1ns, output_bytes=0.0 B, output_batches=0, spill_count=0, spilled_bytes=0.0 B, spilled_rows=0, batches_split=0]                                                                                                                                                                                                                                                           |
|                   |       ProjectionExec: expr=[CAST(0 AS Decimal128(20, 0)) as c0, CAST(1 AS Decimal128(20, 0)) as c1, CAST(2 AS Decimal128(20, 0)) as c2, CAST(3 AS Decimal128(20, 0)) as c3, CAST(4 AS Decimal128(20, 0)) as c4, CAST(5 AS Decimal128(20, 0)) as c5, CAST(6 AS Decimal128(20, 0)) as c6, CAST(7 AS Decimal128(20, 0)) as c7, CAST(8 AS Decimal128(20, 0)) as c8, CAST(c9@0 AS Decimal128(20, 0)) as c9], metrics=[output_rows=0, elapsed_compute=1ns, output_bytes=0.0 B, output_batches=0] |
|                   |         DataSourceExec: partitions=1, partition_sizes=[0], metrics=[]                                                                                                                                                                                                                                                                                                                                                                                                                      |
|                   |                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            |
+-------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

Are there any user-facing changes?

Yes? logical plans for some queries should be different(and better) now. No syntax changes.

[logan-keede]

cc @alamb @Omega359

[alamb]

🤖 ./gh_compare_branch_bench.sh Benchmark Script Running
Linux aal-dev 6.14.0-1018-gcp #19~24.04.1-Ubuntu SMP Wed Sep 24 23:23:09 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Comparing optimize_projected_union (f2e5d0a) to 1dddf03 diff
BENCH_NAME=sql_planner
BENCH_COMMAND=cargo bench --bench sql_planner
BENCH_FILTER=
BENCH_BRANCH_NAME=optimize_projected_union
Results will be posted here when complete

[Omega359]

I'll run the sql_planner_extended benchmark against this branch as that is the one that to me most reflects whether #17261 should be closed. I'm hopeful :)

[alamb]

I'll run the sql_planner_extended benchmark against this branch as that is the one that to me most reflects whether #17261 should be closed. I'm hopeful :)

I have my vm in gc working on it too

[alamb]

🤖: Benchmark completed

Details

group                                                 main                                   optimize_projected_union
-----                                                 ----                                   ------------------------
logical_aggregate_with_join                           1.00    632.8±3.04µs        ? ?/sec    1.00    632.5±3.30µs        ? ?/sec
logical_select_all_from_1000                          1.00     10.7±0.04ms        ? ?/sec    1.00     10.7±0.05ms        ? ?/sec
logical_select_one_from_700                           1.00    420.9±1.60µs        ? ?/sec    1.00    419.2±3.94µs        ? ?/sec
logical_trivial_join_high_numbered_columns            1.00    378.4±3.25µs        ? ?/sec    1.00    376.7±2.37µs        ? ?/sec
logical_trivial_join_low_numbered_columns             1.00    363.5±1.39µs        ? ?/sec    1.00    362.1±3.11µs        ? ?/sec
physical_intersection                                 1.00    842.1±4.94µs        ? ?/sec    1.00    840.2±4.32µs        ? ?/sec
physical_join_consider_sort                           1.00  1400.4±14.40µs        ? ?/sec    1.00  1405.8±12.64µs        ? ?/sec
physical_join_distinct                                1.01    355.3±2.13µs        ? ?/sec    1.00    353.0±1.16µs        ? ?/sec
physical_many_self_joins                              1.00      9.7±0.04ms        ? ?/sec    1.00      9.6±0.03ms        ? ?/sec
physical_plan_clickbench_all                          1.00    178.9±2.09ms        ? ?/sec    1.01    181.0±1.55ms        ? ?/sec
physical_plan_clickbench_q1                           1.00      2.3±0.02ms        ? ?/sec    1.01      2.3±0.02ms        ? ?/sec
physical_plan_clickbench_q10                          1.00      3.2±0.03ms        ? ?/sec    1.01      3.2±0.03ms        ? ?/sec
physical_plan_clickbench_q11                          1.00      3.4±0.02ms        ? ?/sec    1.01      3.4±0.04ms        ? ?/sec
physical_plan_clickbench_q12                          1.00      3.5±0.03ms        ? ?/sec    1.01      3.6±0.04ms        ? ?/sec
physical_plan_clickbench_q13                          1.00      3.2±0.03ms        ? ?/sec    1.01      3.2±0.04ms        ? ?/sec
physical_plan_clickbench_q14                          1.00      3.4±0.02ms        ? ?/sec    1.00      3.4±0.04ms        ? ?/sec
physical_plan_clickbench_q15                          1.00      3.2±0.03ms        ? ?/sec    1.01      3.3±0.03ms        ? ?/sec
physical_plan_clickbench_q16                          1.00      3.1±0.07ms        ? ?/sec    1.00      3.1±0.04ms        ? ?/sec
physical_plan_clickbench_q17                          1.00      3.2±0.04ms        ? ?/sec    1.00      3.2±0.03ms        ? ?/sec
physical_plan_clickbench_q18                          1.00      2.7±0.02ms        ? ?/sec    1.00      2.7±0.02ms        ? ?/sec
physical_plan_clickbench_q19                          1.00      3.6±0.04ms        ? ?/sec    1.01      3.6±0.03ms        ? ?/sec
physical_plan_clickbench_q2                           1.00      2.7±0.03ms        ? ?/sec    1.01      2.8±0.02ms        ? ?/sec
physical_plan_clickbench_q20                          1.00      2.5±0.02ms        ? ?/sec    1.01      2.5±0.02ms        ? ?/sec
physical_plan_clickbench_q21                          1.00      2.8±0.06ms        ? ?/sec    1.00      2.8±0.03ms        ? ?/sec
physical_plan_clickbench_q22                          1.00      3.4±0.04ms        ? ?/sec    1.00      3.4±0.02ms        ? ?/sec
physical_plan_clickbench_q23                          1.00      3.6±0.03ms        ? ?/sec    1.01      3.7±0.05ms        ? ?/sec
physical_plan_clickbench_q24                          1.00      4.1±0.04ms        ? ?/sec    1.01      4.1±0.03ms        ? ?/sec
physical_plan_clickbench_q25                          1.00      2.9±0.02ms        ? ?/sec    1.00      3.0±0.03ms        ? ?/sec
physical_plan_clickbench_q26                          1.00      2.7±0.02ms        ? ?/sec    1.01      2.7±0.03ms        ? ?/sec
physical_plan_clickbench_q27                          1.00      3.0±0.02ms        ? ?/sec    1.03      3.0±0.16ms        ? ?/sec
physical_plan_clickbench_q28                          1.00      3.7±0.04ms        ? ?/sec    1.01      3.7±0.13ms        ? ?/sec
physical_plan_clickbench_q29                          1.00      4.0±0.04ms        ? ?/sec    1.01      4.0±0.05ms        ? ?/sec
physical_plan_clickbench_q3                           1.00      2.7±0.02ms        ? ?/sec    1.01      2.7±0.05ms        ? ?/sec
physical_plan_clickbench_q30                          1.00     12.7±0.13ms        ? ?/sec    1.01     12.8±0.13ms        ? ?/sec
physical_plan_clickbench_q31                          1.00      3.6±0.03ms        ? ?/sec    1.02      3.7±0.04ms        ? ?/sec
physical_plan_clickbench_q32                          1.00      3.7±0.03ms        ? ?/sec    1.01      3.7±0.04ms        ? ?/sec
physical_plan_clickbench_q33                          1.00      3.2±0.02ms        ? ?/sec    1.01      3.2±0.04ms        ? ?/sec
physical_plan_clickbench_q34                          1.00      2.9±0.02ms        ? ?/sec    1.01      2.9±0.05ms        ? ?/sec
physical_plan_clickbench_q35                          1.00      3.0±0.04ms        ? ?/sec    1.01      3.0±0.03ms        ? ?/sec
physical_plan_clickbench_q36                          1.00      3.7±0.03ms        ? ?/sec    1.00      3.7±0.03ms        ? ?/sec
physical_plan_clickbench_q37                          1.00      3.8±0.03ms        ? ?/sec    1.02      3.9±0.03ms        ? ?/sec
physical_plan_clickbench_q38                          1.00      3.8±0.05ms        ? ?/sec    1.01      3.9±0.04ms        ? ?/sec
physical_plan_clickbench_q39                          1.00      3.7±0.05ms        ? ?/sec    1.04      3.8±0.06ms        ? ?/sec
physical_plan_clickbench_q4                           1.00      2.4±0.02ms        ? ?/sec    1.02      2.5±0.02ms        ? ?/sec
physical_plan_clickbench_q40                          1.00      4.5±0.07ms        ? ?/sec    1.05      4.7±0.13ms        ? ?/sec
physical_plan_clickbench_q41                          1.00      3.9±0.05ms        ? ?/sec    1.05      4.1±0.10ms        ? ?/sec
physical_plan_clickbench_q42                          1.00      3.8±0.04ms        ? ?/sec    1.05      4.0±0.06ms        ? ?/sec
physical_plan_clickbench_q43                          1.00      4.2±0.05ms        ? ?/sec    1.05      4.4±0.05ms        ? ?/sec
physical_plan_clickbench_q44                          1.00      2.6±0.02ms        ? ?/sec    1.04      2.7±0.03ms        ? ?/sec
physical_plan_clickbench_q45                          1.00      2.5±0.02ms        ? ?/sec    1.03      2.6±0.03ms        ? ?/sec
physical_plan_clickbench_q46                          1.00      3.0±0.02ms        ? ?/sec    1.04      3.1±0.04ms        ? ?/sec
physical_plan_clickbench_q47                          1.00      3.5±0.04ms        ? ?/sec    1.03      3.7±0.05ms        ? ?/sec
physical_plan_clickbench_q48                          1.00      4.3±0.03ms        ? ?/sec    1.05      4.5±0.06ms        ? ?/sec
physical_plan_clickbench_q49                          1.00      4.6±0.05ms        ? ?/sec    1.04      4.8±0.08ms        ? ?/sec
physical_plan_clickbench_q5                           1.00      2.6±0.02ms        ? ?/sec    1.01      2.7±0.02ms        ? ?/sec
physical_plan_clickbench_q50                          1.00      4.1±0.04ms        ? ?/sec    1.02      4.2±0.04ms        ? ?/sec
physical_plan_clickbench_q51                          1.00      3.1±0.03ms        ? ?/sec    1.01      3.2±0.03ms        ? ?/sec
physical_plan_clickbench_q6                           1.00      2.6±0.02ms        ? ?/sec    1.01      2.7±0.02ms        ? ?/sec
physical_plan_clickbench_q7                           1.00      2.4±0.02ms        ? ?/sec    1.01      2.4±0.02ms        ? ?/sec
physical_plan_clickbench_q8                           1.00      3.2±0.04ms        ? ?/sec    1.01      3.2±0.02ms        ? ?/sec
physical_plan_clickbench_q9                           1.00      3.1±0.03ms        ? ?/sec    1.00      3.1±0.02ms        ? ?/sec
physical_plan_tpcds_all                               1.00   1034.8±4.22ms        ? ?/sec    1.00   1036.8±3.43ms        ? ?/sec
physical_plan_tpch_all                                1.00     64.2±0.51ms        ? ?/sec    1.01     64.6±0.34ms        ? ?/sec
physical_plan_tpch_q1                                 1.00      2.0±0.01ms        ? ?/sec    1.00      2.0±0.01ms        ? ?/sec
physical_plan_tpch_q10                                1.00      4.0±0.01ms        ? ?/sec    1.00      3.9±0.01ms        ? ?/sec
physical_plan_tpch_q11                                1.01      3.5±0.02ms        ? ?/sec    1.00      3.5±0.02ms        ? ?/sec
physical_plan_tpch_q12                                1.00   1803.2±8.13µs        ? ?/sec    1.00   1811.3±7.38µs        ? ?/sec
physical_plan_tpch_q13                                1.00  1452.0±11.22µs        ? ?/sec    1.00   1451.9±7.04µs        ? ?/sec
physical_plan_tpch_q14                                1.00  1961.0±13.52µs        ? ?/sec    1.01  1986.2±10.66µs        ? ?/sec
physical_plan_tpch_q16                                1.00      2.5±0.01ms        ? ?/sec    1.00      2.5±0.02ms        ? ?/sec
physical_plan_tpch_q17                                1.01      2.6±0.02ms        ? ?/sec    1.00      2.6±0.01ms        ? ?/sec
physical_plan_tpch_q18                                1.00      2.7±0.01ms        ? ?/sec    1.00      2.7±0.01ms        ? ?/sec
physical_plan_tpch_q19                                1.00      3.2±0.02ms        ? ?/sec    1.01      3.2±0.02ms        ? ?/sec
physical_plan_tpch_q2                                 1.00      5.9±0.03ms        ? ?/sec    1.00      5.9±0.10ms        ? ?/sec
physical_plan_tpch_q20                                1.01      3.2±0.03ms        ? ?/sec    1.00      3.2±0.02ms        ? ?/sec
physical_plan_tpch_q21                                1.00      4.2±0.02ms        ? ?/sec    1.00      4.2±0.02ms        ? ?/sec
physical_plan_tpch_q22                                1.00      2.8±0.01ms        ? ?/sec    1.01      2.9±0.02ms        ? ?/sec
physical_plan_tpch_q3                                 1.00      2.7±0.01ms        ? ?/sec    1.00      2.7±0.05ms        ? ?/sec
physical_plan_tpch_q4                                 1.00  1489.3±10.46µs        ? ?/sec    1.00   1490.4±6.70µs        ? ?/sec
physical_plan_tpch_q5                                 1.00      3.2±0.01ms        ? ?/sec    1.00      3.2±0.01ms        ? ?/sec
physical_plan_tpch_q6                                 1.00    852.8±5.82µs        ? ?/sec    1.00    852.2±6.40µs        ? ?/sec
physical_plan_tpch_q7                                 1.00      4.2±0.01ms        ? ?/sec    1.00      4.2±0.02ms        ? ?/sec
physical_plan_tpch_q8                                 1.00      5.5±0.02ms        ? ?/sec    1.00      5.4±0.02ms        ? ?/sec
physical_plan_tpch_q9                                 1.01      4.0±0.02ms        ? ?/sec    1.00      4.0±0.01ms        ? ?/sec
physical_select_aggregates_from_200                   1.01     16.9±0.06ms        ? ?/sec    1.00     16.8±0.08ms        ? ?/sec
physical_select_all_from_1000                         1.00     23.5±0.14ms        ? ?/sec    1.00     23.5±0.14ms        ? ?/sec
physical_select_one_from_700                          1.00   1085.3±4.93µs        ? ?/sec    1.00   1085.0±7.86µs        ? ?/sec
physical_sorted_union_order_by_10_int64               1.00      6.0±0.02ms        ? ?/sec    1.00      6.0±0.02ms        ? ?/sec
physical_sorted_union_order_by_10_uint64              1.00     13.0±0.15ms        ? ?/sec    1.23     15.9±0.09ms        ? ?/sec
physical_sorted_union_order_by_50_int64               1.00    162.8±1.15ms        ? ?/sec    1.00    162.3±1.02ms        ? ?/sec
physical_sorted_union_order_by_50_uint64              1.00    374.4±2.21ms        ? ?/sec    2.54    952.4±5.61ms        ? ?/sec
physical_theta_join_consider_sort                     1.00  1760.8±13.45µs        ? ?/sec    1.01   1770.9±7.47µs        ? ?/sec
physical_unnest_to_join                               1.00  1858.9±17.32µs        ? ?/sec    1.00   1859.8±8.00µs        ? ?/sec
physical_window_function_partition_by_12_on_values    1.00  1087.2±12.54µs        ? ?/sec    1.00   1092.2±8.59µs        ? ?/sec
physical_window_function_partition_by_30_on_values    1.00      2.2±0.01ms        ? ?/sec    1.00      2.2±0.01ms        ? ?/sec
physical_window_function_partition_by_4_on_values     1.01    661.5±3.93µs        ? ?/sec    1.00    656.2±8.07µs        ? ?/sec
physical_window_function_partition_by_7_on_values     1.00    808.1±6.34µs        ? ?/sec    1.01    812.9±6.26µs        ? ?/sec
physical_window_function_partition_by_8_on_values     1.00    865.7±4.93µs        ? ?/sec    1.00    866.8±3.68µs        ? ?/sec
with_param_values_many_columns                        1.00    646.8±7.46µs        ? ?/sec    1.00    647.3±6.61µs        ? ?/sec

[alamb]

physical_sorted_union_order_by_10_int64               1.00      6.0±0.02ms        ? ?/sec    1.00      6.0±0.02ms        ? ?/sec
physical_sorted_union_order_by_10_uint64              1.00     13.0±0.15ms        ? ?/sec    1.23     15.9±0.09ms        ? ?/sec
physical_sorted_union_order_by_50_int64               1.00    162.8±1.15ms        ? ?/sec    1.00    162.3±1.02ms        ? ?/sec
physical_sorted_union_order_by_50_uint64              1.00    374.4±2.21ms        ? ?/sec    2.54    952.4±5.61ms        ? ?/sec

1.23.- 2.5x better certainly seems much better 🎉 ! Though I am not sure if we have fixed the exponential planning time

[Omega359]

Given the time estimate on my machine for the extended planner test I'm going to assume it's not impacting that benchmark as much as I'd like to see:

Running benches/sql_planner_extended.rs (target/release/deps/sql_planner_extended-4fd78f102f98e6a6)
Benchmarking logical_plan_optimize: Warming up for 3.0000 s
Warning: Unable to complete 100 samples in 5.0s. You may wish to increase target time to 6063.2s, or reduce sample count to 10.
Benchmarking logical_plan_optimize: Collecting 100 samples in estimated 6063.2 s (100 iterations

[logan-keede]

I guess this won't be enough to close the issue but I will take any small wins for now.
I will try and see if I can find any other lead that can explain the exponential planning time.

[Omega359]

ogical_plan_optimize   time:   [68.066 s 68.685 s 69.325 s]
Found 4 outliers among 100 measurements (4.00%)
  1 (1.00%) low mild
  3 (3.00%) high mild

+ critcmp main optimize_projected_union
group                    main                                   optimize_projected_union
-----                    ----                                   ------------------------
logical_plan_optimize    1.00      68.7±3.19s        ? ?/sec    1.00      68.7±8.30s        ? ?/sec