Limit clean task memory usage

[seanses]

Fix CLI-190

Given the fact that

1. Git and PyXet send data in different block size to xet-core to clean,
2. PyXet can clean multiple files in parallel,
3. Different files have different clean speed,

it's difficult to tune memory usage for each clean task. Instead, clean tasks can share a common memory limit (8 GB) and grab resource as fast as they can for efficiency.

Tested locally uploading two files each of 40 GB and maintain memory usage < 8 GB:

(.env) di@di-mbp ~/xetdata/pyxet/python/pyxet % xet cp ~/tt/zz* xet://localhost:3000:seanses-local/empty/main
copy /Users/di/tt/zz0, /Users/di/tt/zzx... to xet://localhost:3000:seanses-local/empty/main: (2 / 2), 80 GiB | 1.19 GiB/s, done.

[hoytak]

So... This seems really complicated, and I'm worried there will be side effects if it's not used correctly... Since we're using fixed sized buffers, I think we could get around this by simply enforcing a maximum buffer size globally, then calculating the size of the queue in order to respect the larger size limits wrt max buffer size.

[seanses]

So... This seems really complicated, and I'm worried there will be side effects if it's not used correctly...

The underlying logic is actually very simple. This treats multiple buffers as an entire bounded buffer, only considers entries to and exits from the entirety, and ignores internal data moves because they don't change the memory usage.

Since we're using fixed sized buffers, I think we could get around this by simply enforcing a maximum buffer size globally, then calculating the size of the queue in order to respect the larger size limits wrt max buffer size.

I'm not following this suggestion. Is this calculation done once which configures each buffer size below clean tasks start, or done dynamically during cleaning?

[seanses]

After offline discussion, close in favor of #399 for unforeseen risk.