You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Memory allocator biggest improvement for Datafuse is to make block allocation.
For example, it will have a big performance gain when using snmalloc, since Datafuse has many pieces allocation, such as Vec<T> and many points not optimized yet.
But in the latest version, Datafuse has many improvements(#1444#1334#1239#1214) about the memory and the algorithm, less and less reliance on the memory allocator.
From my test:
With snmalloc
Hardware: AMD Ryzen 7 PRO 4750U, 8 CPU Cores, 16 Threads
2021-08-15T05:30:42Z INFO datafuse_query] DatafuseQuery v-0.1.0-f3780e1-simd(1.56.0-nightly-2021-08-15T04:19:45.246023742+00:00), Allocator: snmalloc
clickhouse-client --time --multiquery --format Null <<<"
SELECT avg(number) FROM numbers_mt(100000000000);
SELECT sum(number) FROM numbers_mt(100000000000);
SELECT min(number) FROM numbers_mt(100000000000);
SELECT max(number) FROM numbers_mt(100000000000);
SELECT count(number) FROM numbers_mt(100000000000);
SELECT sum(number+number+number) FROM numbers_mt(100000000000);
SELECT sum(number) / count(number) FROM numbers_mt(100000000000);
SELECT sum(number) / count(number), max(number), min(number) FROM numbers_mt(100000000000);
SELECT number FROM numbers_mt(10000000000) ORDER BY number DESC LIMIT 10;
SELECT max(number), sum(number) FROM numbers_mt(1000000000) GROUP BY number % 3, number % 4, number % 5 LIMIT 10;"
5.763
6.038
6.236
6.285
5.686
16.614
6.171
8.970
2.612
1.836
Rust default allocator
Hardware: AMD Ryzen 7 PRO 4750U, 8 CPU Cores, 16 Threads
[2021-08-15T05:32:28Z INFO datafuse_query] DatafuseQuery v-0.1.0-f3780e1-simd(1.56.0-nightly-2021-08-14T15:03:17.462456200+00:00), Allocator: default
clickhouse-client --time --multiquery --format Null <<<"
SELECT avg(number) FROM numbers_mt(100000000000);
SELECT sum(number) FROM numbers_mt(100000000000);
SELECT min(number) FROM numbers_mt(100000000000);
SELECT max(number) FROM numbers_mt(100000000000);
SELECT count(number) FROM numbers_mt(100000000000);
SELECT sum(number+number+number) FROM numbers_mt(100000000000);
SELECT sum(number) / count(number) FROM numbers_mt(100000000000);
SELECT sum(number) / count(number), max(number), min(number) FROM numbers_mt(100000000000);
SELECT number FROM numbers_mt(10000000000) ORDER BY number DESC LIMIT 10;
SELECT max(number), sum(number) FROM numbers_mt(1000000000) GROUP BY number % 3, number % 4, number % 5 LIMIT 10;"
3.301
3.177
4.301
4.302
2.203
16.466
3.680
8.047
3.140
1.764
It is only advantageous in the case of ORDER BY , so there should still be some fragmented memory allocation in ORDER BY , this can be optimized in future.
Rust default memory alloctor is good for Datafuse now.
The text was updated successfully, but these errors were encountered:
Summary
Memory allocator biggest improvement for Datafuse is to make block allocation.
For example, it will have a big performance gain when using snmalloc, since Datafuse has many pieces allocation, such as
Vec<T>
and many points not optimized yet.But in the latest version, Datafuse has many improvements(#1444 #1334 #1239 #1214) about the memory and the algorithm, less and less reliance on the memory allocator.
From my test:
It is only advantageous in the case of
ORDER BY
, so there should still be some fragmented memory allocation inORDER BY
, this can be optimized in future.Rust default memory alloctor is good for Datafuse now.
The text was updated successfully, but these errors were encountered: