See #4703 (comment)
Currently the GatherToNpy and GatherToZarr sequentially write chunk of pipeline data as they are available. While this is needed when we don't know the number of peaks per chunk, in cases when this is known (ie. the pipeline has a PeakRetriever or SpikeRetriever), the gather classes could pre-allocate the entire arrays and write to the exact location. This would speed things up especially for zarr, since the current implementation might require a lot of rechunking under the hood
See #4703 (comment)
Currently the
GatherToNpyandGatherToZarrsequentially write chunk of pipeline data as they are available. While this is needed when we don't know the number of peaks per chunk, in cases when this is known (ie. the pipeline has aPeakRetrieverorSpikeRetriever), the gather classes could pre-allocate the entire arrays and write to the exact location. This would speed things up especially for zarr, since the current implementation might require a lot of rechunking under the hood