Skip to content

[improvement] Consider remove common/allocators #1470

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
BohuTANG opened this issue Aug 15, 2021 · 3 comments · Fixed by #1472
Closed

[improvement] Consider remove common/allocators #1470

BohuTANG opened this issue Aug 15, 2021 · 3 comments · Fixed by #1472
Labels
C-improvement Category: improvement

Comments

@BohuTANG
Copy link
Member

BohuTANG commented Aug 15, 2021

Summary

Memory allocator biggest improvement for Datafuse is to make block allocation.
For example, it will have a big performance gain when using snmalloc, since Datafuse has many pieces allocation, such as Vec<T> and many points not optimized yet.

But in the latest version, Datafuse has many improvements(#1444 #1334 #1239 #1214) about the memory and the algorithm, less and less reliance on the memory allocator.
From my test:

  1. With snmalloc
Hardware: AMD Ryzen 7 PRO 4750U, 8 CPU Cores, 16 Threads
2021-08-15T05:30:42Z INFO  datafuse_query] DatafuseQuery v-0.1.0-f3780e1-simd(1.56.0-nightly-2021-08-15T04:19:45.246023742+00:00), Allocator: snmalloc

clickhouse-client --time --multiquery --format Null <<<"
    SELECT avg(number) FROM numbers_mt(100000000000);
    SELECT sum(number) FROM numbers_mt(100000000000);
    SELECT min(number) FROM numbers_mt(100000000000);
    SELECT max(number) FROM numbers_mt(100000000000);
    SELECT count(number) FROM numbers_mt(100000000000);
    SELECT sum(number+number+number) FROM numbers_mt(100000000000);
    SELECT sum(number) / count(number) FROM numbers_mt(100000000000);
    SELECT sum(number) / count(number), max(number), min(number) FROM numbers_mt(100000000000);
    SELECT number FROM numbers_mt(10000000000) ORDER BY number DESC LIMIT 10;
    SELECT max(number), sum(number) FROM numbers_mt(1000000000) GROUP BY number % 3, number % 4, number % 5 LIMIT 10;"
5.763
6.038
6.236
6.285
5.686
16.614
6.171
8.970
2.612
1.836
  1. Rust default allocator
Hardware: AMD Ryzen 7 PRO 4750U, 8 CPU Cores, 16 Threads
[2021-08-15T05:32:28Z INFO  datafuse_query] DatafuseQuery v-0.1.0-f3780e1-simd(1.56.0-nightly-2021-08-14T15:03:17.462456200+00:00), Allocator: default

clickhouse-client --time --multiquery --format Null <<<"
    SELECT avg(number) FROM numbers_mt(100000000000);
    SELECT sum(number) FROM numbers_mt(100000000000);
    SELECT min(number) FROM numbers_mt(100000000000);
    SELECT max(number) FROM numbers_mt(100000000000);
    SELECT count(number) FROM numbers_mt(100000000000);
    SELECT sum(number+number+number) FROM numbers_mt(100000000000);
    SELECT sum(number) / count(number) FROM numbers_mt(100000000000);
    SELECT sum(number) / count(number), max(number), min(number) FROM numbers_mt(100000000000);
    SELECT number FROM numbers_mt(10000000000) ORDER BY number DESC LIMIT 10;
    SELECT max(number), sum(number) FROM numbers_mt(1000000000) GROUP BY number % 3, number % 4, number % 5 LIMIT 10;"
3.301
3.177
4.301
4.302
2.203
16.466
3.680
8.047
3.140
1.764

It is only advantageous in the case of ORDER BY , so there should still be some fragmented memory allocation in ORDER BY , this can be optimized in future.
Rust default memory alloctor is good for Datafuse now.

@BohuTANG BohuTANG added the C-improvement Category: improvement label Aug 15, 2021
@BohuTANG
Copy link
Member Author

cc: @sundy-li @zhang2014

@zhang2014
Copy link
Member

Our biggest motivation for expanding the memory allocator may be to track the allocation of memory #1148

@BohuTANG
Copy link
Member Author

Our biggest motivation for expanding the memory allocator may be to track the allocation of memory #1148

Yes, it will be another issue

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
C-improvement Category: improvement
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants