feat: add --aggregate flag to allow "du -sh *"-usage

[LuzianHahn]

Hey,

I found this rather old discussion here and wanted to provide here a solution. Since most people (including me) seem to rather look for a replacement of du -sh * instead of implicitly aggregating those, I added a functionality, which makes this behavior configurable. The default behavior now considers all provided path-arguments independently.

I did not dive into optimizing the workload across different workers, so paths are just worked on sequentially, but while doing so, parallelized.

[sharkdp]

Thank you for working on this.

I did not dive into optimizing the workload across different workers, so paths are just worked on sequentially, but while doing so, parallelized.

I think we should at least perform some benchmarks. It's easy to imagine that this will result in a performance regression for some workloads.

[LuzianHahn]

Alright, so I found the time to benchmark this. Thanks for your instructions in the README. Made my life a lot easier!

diskus-Benchmark for PR57

x86 WSL - Debian 11

Cold disk cache

Benchmark 1: diskus
  Time (mean ± σ):     308.5 ms ±  18.6 ms    [User: 40.0 ms, System: 256.5 ms]
  Range (min … max):   284.2 ms … 350.6 ms    10 runs

Benchmark 2: du -sh
  Time (mean ± σ):     831.8 ms ±  35.6 ms    [User: 2.1 ms, System: 70.5 ms]
  Range (min … max):   792.0 ms … 885.8 ms    10 runs

Benchmark 3: sn p -d0 -j8
  Time (mean ± σ):     361.0 ms ±   7.3 ms    [User: 385.3 ms, System: 86.1 ms]
  Range (min … max):   349.4 ms … 373.6 ms    10 runs

Benchmark 4: dust -d0
  Time (mean ± σ):      1.302 s ±  0.014 s    [User: 0.062 s, System: 0.993 s]
  Range (min … max):    1.288 s …  1.323 s    10 runs

Summary
  diskus ran
    1.17 ± 0.07 times faster than sn p -d0 -j8
    2.70 ± 0.20 times faster than du -sh
    4.22 ± 0.26 times faster than dust -d0

Warm disk cache

Benchmark 1: diskus
  Time (mean ± σ):     261.6 ms ±  14.0 ms    [User: 21.0 ms, System: 240.0 ms]
  Range (min … max):   231.1 ms … 283.2 ms    12 runs

Benchmark 2: du -sh
  Time (mean ± σ):     773.3 ms ±  14.0 ms    [User: 3.1 ms, System: 59.8 ms]
  Range (min … max):   748.5 ms … 792.9 ms    10 runs

Benchmark 3: sn p -d0 -j8
  Time (mean ± σ):     343.5 ms ±   4.3 ms    [User: 390.0 ms, System: 92.0 ms]
  Range (min … max):   337.0 ms … 350.1 ms    10 runs

Benchmark 4: dust -d0
  Time (mean ± σ):     341.9 ms ±   6.9 ms    [User: 42.2 ms, System: 192.3 ms]
  Range (min … max):   334.1 ms … 358.5 ms    10 runs

Summary
  diskus ran
    1.31 ± 0.07 times faster than dust -d0
    1.31 ± 0.07 times faster than sn p -d0 -j8
    2.96 ± 0.17 times faster than du -sh

MacOS-15 - ARM64

Cold disk cache

not possible, due to missing admin permissions on the device

Warm disk cache

Benchmark 1: diskus
  Time (mean ± σ):       7.0 ms ±   1.5 ms    [User: 5.2 ms, System: 48.7 ms]
  Range (min … max):     6.1 ms …  21.0 ms    288 runs

Benchmark 2: du -sh
  Time (mean ± σ):      10.9 ms ±   0.3 ms    [User: 0.9 ms, System: 9.9 ms]
  Range (min … max):    10.1 ms …  12.4 ms    206 runs

Benchmark 3: sn p -d0 -j8
  Time (mean ± σ):       9.9 ms ±   0.4 ms    [User: 2.7 ms, System: 13.4 ms]
  Range (min … max):     9.1 ms …  12.5 ms    218 runs

Benchmark 4: dust -d0
  Time (mean ± σ):       8.3 ms ±   0.6 ms    [User: 6.2 ms, System: 39.5 ms]
  Range (min … max):     7.3 ms …  10.6 ms    259 runs

Summary
  diskus ran
    1.18 ± 0.27 times faster than dust -d0
    1.42 ± 0.31 times faster than sn p -d0 -j8
    1.56 ± 0.35 times faster than du -sh

x86 Ubuntu 24

Cold disk cache

Benchmark 1: diskus
  Time (mean ± σ):     327.1 ms ±   3.5 ms    [User: 444.0 ms, System: 1155.8 ms]
  Range (min … max):   320.5 ms … 332.3 ms    10 runs

Benchmark 2: du -sh
  Time (mean ± σ):      2.508 s ±  0.096 s    [User: 0.187 s, System: 0.816 s]
  Range (min … max):    2.306 s …  2.606 s    10 runs

Benchmark 3: sn p -d0 -j8
  Time (mean ± σ):      1.144 s ±  0.049 s    [User: 0.135 s, System: 0.777 s]
  Range (min … max):    1.085 s …  1.241 s    10 runs

Benchmark 4: dust -d0
  Time (mean ± σ):      1.706 s ±  0.123 s    [User: 0.368 s, System: 2.384 s]
  Range (min … max):    1.647 s …  2.052 s    10 runs

Summary
  diskus ran
    3.50 ± 0.15 times faster than sn p -d0 -j8
    5.21 ± 0.38 times faster than dust -d0
    7.67 ± 0.30 times faster than du -sh

x86 Ubuntu 24

Warm disk cache

Benchmark 1: diskus
  Time (mean ± σ):      90.2 ms ±   3.4 ms    [User: 237.7 ms, System: 454.6 ms]
  Range (min … max):    86.2 ms … 100.9 ms    32 runs

Benchmark 2: du -sh
  Time (mean ± σ):     364.4 ms ±   1.2 ms    [User: 128.8 ms, System: 235.8 ms]
  Range (min … max):   362.8 ms … 366.0 ms    10 runs

Benchmark 3: sn p -d0 -j8
  Time (mean ± σ):     133.0 ms ±   4.4 ms    [User: 93.6 ms, System: 228.9 ms]
  Range (min … max):   125.9 ms … 138.7 ms    23 runs

Benchmark 4: dust -d0
  Time (mean ± σ):     240.6 ms ±  11.7 ms    [User: 335.3 ms, System: 795.3 ms]
  Range (min … max):   229.7 ms … 267.7 ms    12 runs

Summary
  diskus ran
    1.47 ± 0.07 times faster than sn p -d0 -j8
    2.67 ± 0.16 times faster than dust -d0
    4.04 ± 0.15 times faster than du -sh