zlib & gzip performs better in compressing a string then blosc2

[prashant-saxena]

String length = 31720
after gzip = 9113
after zlib = 9125
after blosc2 = 12217

Here is the python script

import blosc2
import numpy as np
cparams = blosc2.CParams(
    codec=blosc2.Codec.ZLIB,
    clevel=9,
    nthreads=2,
    filters=[blosc2.Filter.NOFILTER],
    filters_meta=[0],
)

with open('text.txt', mode='r', encoding='utf-8') as f:
    a = f.read().encode('utf-8')

d = blosc2.compress2(a, cparams=cparams)
print(len(d))

Is there something wrong in cparams or am I missing something here?

[FrancescAlted]

The issue should be that the default in compress2 is typesize=8. Can you try with typesize=1? No need to disable the SHUFFLE filter because in this case it does nothing.

[FrancescAlted]

After thinking twice, deactivating the shuffle filter should have been enough to deactivate splits for items (the root of the bad compression ratio). It turned out that the default for CParams.splitmode was ALWAYS_SPLIT, and not AUTO_SPLIT (which should be more useful in general). I fixed this in 88d76e9 . This created some minor issue with ZFP though (see #338).

[prashant-saxena]

@FrancescAlted

Thanks for the typesize tip. Here are the results of compressing a string of length 31720 with typesize = 1 & compression level = 9

Codec	Filter	Size
blosc2	nofilter	13560
blosc2	shuffle	13560
blosc2	bitshuffle	31752
lz4	nofilter	14477
lz4	shuffle	14477
lz4	bitshuffle	25902
lz4hc	nofilter	11988
lz4hc	shuffle	11988
lz4hc	bitshuffle	24871
zlib	nofilter	9122
zlib	shuffle	9122
zlib	bitshuffle	23279
zstd	nofilter	8294
zstd	shuffle	8294
zstd	bitshuffle	23545
zstd with nofilter is performing the best.
In case of numpy int32 array of 30 elements (120 bytes) & typesize=4 blosc2 and shuffle is performing the best
Codec	Filter	Size
---	---	---
blosc2	nofilter	152
blosc2	shuffle	78
blosc2	bitshuffle	85
lz4	nofilter	152
lz4	shuffle	82
lz4	bitshuffle	87
lz4hc	nofilter	152
lz4hc	shuffle	82
lz4hc	bitshuffle	87
zlib	nofilter	97
zlib	shuffle	81
zlib	bitshuffle	86
zstd	nofilter	96
zstd	shuffle	89
zstd	bitshuffle	89