Wheels for WASM

[FrancescAlted]

I did some attempt at creating wheels for WASM, and had some (mixed) experiences with Pyodide and scikit-build-core:

1. Pyodide needs version 0.23.4 (latest 0.24.1 does not work: Make build not working pyodide/pyodide#4287)

2. CMake config in python-blosc2 needs a bit of modification in order to run (see attached patch).

3. The included zlib-ng codec has issues with WASM compiler, so it has been removed from the list of codecs here.

4. Some packages (notably py-cpuinfo) are not available in Pyodide repos.

5. Some tests are passing, but others do not. E.g.:

$ python -m pytest tests/test_compress2.py
=============================================================== test session starts ================================================================
platform emscripten -- Python 3.11.2, pytest-7.4.4, pluggy-1.3.0
rootdir: /home/faltet/blosc/python-blosc2
configfile: pytest.ini
collected 24 items

tests/test_compress2.py ..............FF........                                                                                             [100%]

===================================================================== FAILURES =====================================================================
__________________________________________________ test_compress2[641091-cparams1-dparams1-True] ___________________________________________________

nbytes = 641091, cparams = {'typesize': 1}, dparams = {'nthreads': 4}, gil = True

    @pytest.mark.parametrize("gil", [True, False])
    @pytest.mark.parametrize(
        "nbytes, cparams, dparams",
        [
            (7, {"codec": blosc2.Codec.LZ4, "clevel": 6, "typesize": 1}, {}),
            (641091, {"typesize": 1}, {"nthreads": 4}),
            (136, {"typesize": 1}, {}),
            (1231, {"typesize": 4}, blosc2.dparams_dflts),
        ],
    )
    def test_compress2(nbytes, cparams, dparams, gil):
        blosc2.set_releasegil(gil)
        bytes_obj = b" " * nbytes
        c = blosc2.compress2(bytes_obj, **cparams)

        dest = bytearray(bytes_obj)
        blosc2.decompress2(c, dst=dest, **dparams)
        assert dest == bytes_obj

        dest2 = blosc2.decompress2(c, **dparams)
>       assert dest2 == bytes_obj
E       AssertionError: assert b'\x00\x00\x0...0\x00\x00\x00' == b'           ...             '
E         At index 0 diff: b'\x00' != b' '
E         Use -v to get more diff

tests/test_compress2.py:91: AssertionError
__________________________________________________ test_compress2[641091-cparams1-dparams1-False] __________________________________________________

nbytes = 641091, cparams = {'typesize': 1}, dparams = {'nthreads': 4}, gil = False

    @pytest.mark.parametrize("gil", [True, False])
    @pytest.mark.parametrize(
        "nbytes, cparams, dparams",
        [
            (7, {"codec": blosc2.Codec.LZ4, "clevel": 6, "typesize": 1}, {}),
            (641091, {"typesize": 1}, {"nthreads": 4}),
            (136, {"typesize": 1}, {}),
            (1231, {"typesize": 4}, blosc2.dparams_dflts),
        ],
    )
    def test_compress2(nbytes, cparams, dparams, gil):
        blosc2.set_releasegil(gil)
        bytes_obj = b" " * nbytes
        c = blosc2.compress2(bytes_obj, **cparams)

        dest = bytearray(bytes_obj)
        blosc2.decompress2(c, dst=dest, **dparams)
        assert dest == bytes_obj

        dest2 = blosc2.decompress2(c, **dparams)
>       assert dest2 == bytes_obj
E       AssertionError: assert b'x00\\x00\\x...x00\\x00\\x00' == b'           ...             '
E         At index 0 diff: b'x' != b' '
E         Use -v to get more diff

tests/test_compress2.py:91: AssertionError
============================================================= short test summary info ==============================================================
FAILED tests/test_compress2.py::test_compress2[641091-cparams1-dparams1-True] - AssertionError: assert b'\x00\x00\x0...0\x00\x00\x00' == b'           ...             '
FAILED tests/test_compress2.py::test_compress2[641091-cparams1-dparams1-False] - AssertionError: assert b'x00\\x00\\x...x00\\x00\\x00' == b'           ...             '
=========================================================== 2 failed, 22 passed in 0.24s ===========================================================

I don't think fixing these should be terribly complicated, but I don't have more time now. I am attaching my modifications here in case someone else would want to continue this effort.

pyodide-python-blosc2.patch

[agriyakhetarpal]

Hi @FrancescAlted, I stumbled into this issue from pyodide/pyodide#4287 when I was triaging and going through the issue backlog – could you please try this with the latest version of Pyodide + pyodide-build version 0.29.2 if you have time? There is a chance we might have fixed most of the issues on the Pyodide build system's and on scikit-build-core's side here, and the latter tests for WASM targets. Plus, I also noted Blosc/c-blosc2#632, that is wonderful to see. Thanks!

[FrancescAlted]

Good news! We have a preliminary version of python-blosc2 working. What would be the recommended path to let people install it easily from pyodide? Thanks!

[agriyakhetarpal]

Awesome! The recommended path would be to add a recipe for python-blosc2 and its dependencies in a PR to Pyodide: https://pyodide.org/en/stable/development/new-packages.html#building-a-python-package-in-tree. Once a version of Pyodide is released with that PR included, one can install it with import micropip; await micropip.install("python-blosc2").

To ensure that the WASM build remains working and to reduce effort later on, you may also consider adding a CI job in this repository to run the build and the test suite on every commit: https://pyodide.org/en/stable/development/building-and-testing-packages.html#building-and-testing-packages-out-of-tree.

Please feel free to ping me on a PR if you need to!