-
Notifications
You must be signed in to change notification settings - Fork 1.6k
Description
Is your feature request related to a problem or challenge?
Part of #12680
In #12269 @jayzhan211 made significant improvements to how group values are stored in multi-column aggregations. This requires specialized implementations for different column types
His initial PR has implementations for PrimitiveArray and String/Binary. However it does not have a specialization for StringView
So that means that queries that group on multiple columns are even faster. This shows up by effectively slowing down some clickbench queries when they are run with StringView:
For example, this query is 10% slower with StringView
SELECT "SearchEngineID", "SearchPhrase", COUNT(*) AS c FROM 'hits.parquet' WHERE "SearchPhrase" <> '' GROUP BY "SearchEngineID", "SearchPhrase" ORDER BY c DESC LIMIT 10;
Describe the solution you'd like
I would like to make this (and similar) query faster when string view is enabled :
SELECT "SearchEngineID", "SearchPhrase", COUNT(*) AS c FROM 'hits.parquet' WHERE "SearchPhrase" <> '' GROUP BY "SearchEngineID", "SearchPhrase" ORDER BY c DESC LIMIT 10;
Note this is grouping by 2 columns
Here is how to reproduce the issue
Step 1. Get hits.parquet
using bench.sh
:
cd benchmarks
./bench.sh data clickbench_1
Step 2: Prepare a script with reproducer query:
set datafusion.execution.parquet.schema_force_view_types = true;
SELECT "SearchEngineID", "SearchPhrase", COUNT(*) AS c FROM 'hits.parquet' WHERE "SearchPhrase" <> '' GROUP BY "SearchEngineID", "SearchPhrase" ORDER BY c DESC LIMIT 10;
Step 3: Run query
(venv) andrewlamb@Andrews-MacBook-Pro-2:~/Downloads$ datafusion-cli -f q.sql
set datafusion.execution.parquet.schema_force_view_types = true;
--> Elapsed 0.688 seconds.set datafusion.execution.parquet.schema_force_view_types = false;
--> Elapsed 0.565 seconds.
Describe alternatives you've considered
I suggest implementing something like ByteViewGroupValueBuilder
following the model of ByteGroupValueBuilder
pub struct ByteGroupValueBuilder<O> |
The in progress values would be u128
s and some buffers (maybe 2MB?)
implementing equal_to
can take advantage of the inlined prefix optimization (aka compare the prefix inlined in the u128 and only check the value in the buffer if that is already equal)
Additional context
No response