feat: support for pyodide

[kyrre]

Is your feature request related to a problem?

We really want to use marimo WebAssembly notebooks together with ibis.

There exists a closed issue for this feature #6889. It was previously blocked due to missing pyodide support for pyarrow, but that is no longer the case.

When I try creating a marimo wasm notebook this is where it fails now:

import
ibis.duckdb.connect(":memory:")

with the trace

Traceback (most recent call last):
  File "/lib/python3.12/site-packages/marimo/_runtime/executor.py", line 142, in execute_cell
    return eval(cell.last_expr, glbls)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  Cell marimo://notebook.py#cell=cell-10, line 1, in <module>
    ibis.duckdb.connect(":memory:")
    ^^^^^^^^^^^
  File "/lib/python3.12/site-packages/ibis/__init__.py", line 142, in __getattr__
    return load_backend(name)
           ^^^^^^^^^^^^^^^^^^
  File "/lib/python3.12/site-packages/ibis/__init__.py", line 82, in load_backend
    module = entry_point.load()
             ^^^^^^^^^^^^^^^^^^
  File "/lib/python312.zip/importlib/metadata/__init__.py", line 205, in load
    module = import_module(match.group('module'))
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/lib/python312.zip/importlib/__init__.py", line 90, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap>", line 1387, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1331, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 935, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 995, in exec_module
  File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
  File "/lib/python3.12/site-packages/ibis/backends/duckdb/__init__.py", line 73, in <module>
    class Backend(SQLBackend, CanCreateDatabase, CanCreateSchema, UrlFromPath):
  File "/lib/python3.12/site-packages/ibis/backends/duckdb/__init__.py", line 75, in Backend
    compiler = sc.duckdb.compiler
               ^^^^^^^^^
AttributeError: module 'ibis.backends.sql.compilers' has no attribute 'duckdb'

What is the motivation behind your request?

No response

Describe the solution you'd like

I'm not sure why it fails. I think the all the components for ibis+duckdb should be there?

What version of ibis are you running?

9.5.0

What backend(s) are you using, if any?

No response

Code of Conduct

• I agree to follow this project's Code of Conduct

[cpcloud]

Thanks for opening an issue, and glad to see there's interest in using Ibis in a WASM runtime.

We've also wanted to this to work properly for a while, and up until pyodide 0.27.1 (which was released a couple days ago) using ibis with pyodide required a series of install hacks.

With pyodide 0.27.1, these hacks are no longer required, and you can use ibis in pyodide like so:

%pip install ibis-framework[duckdb]

# do stuff with ibis

In fact, we have a jupyterlite runtime on the Ibis website here: https://ibis-project.org/tutorials/browser/repl, where Ibis is imported and then a simple SELECT *-style query is executed (via ibis.read_csv).

The issue you're encountering seems like an installation problem, rather than any lack of support for running Ibis in pyodide.

[cpcloud]

Can you show how you installed ibis in your pyodide instance?

[kyrre]

This is a minimal example using marimo:

uv venv --python 3.12
source .venv/bin/activate
uv pip install marimo

# just to test it outside wasm
uv pip install "ibis-framework[duckdb]" 

import marimo

__generated_with = "0.10.14"
app = marimo.App(width="medium")


@app.cell
def _():
    import marimo as mo
    return (mo,)


@app.cell
def _():
    import ibis
    return (ibis,)


@app.cell
def _(ibis):
    ibis.duckdb.connect(":memory:")
    return


@app.cell
def _():
    return


if __name__ == "__main__":
    app.run()

marimo export html-wasm minimal.py -o output_dir --mode edit

stacktrace:

Traceback (most recent call last):
  File "/lib/python3.12/site-packages/ibis/__init__.py", line 82, in load_backend
    module = entry_point.load()
             ^^^^^^^^^^^^^^^^^^
  File "/lib/python312.zip/importlib/metadata/__init__.py", line 205, in load
    module = import_module(match.group('module'))
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/lib/python312.zip/importlib/__init__.py", line 90, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap>", line 1387, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1331, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 935, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 995, in exec_module
  File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
  File "/lib/python3.12/site-packages/ibis/backends/duckdb/__init__.py", line 22, in <module>
    import ibis.backends.sql.compilers as sc
  File "/lib/python3.12/site-packages/ibis/backends/sql/compilers/__init__.py", line 38, in <module>
    from ibis.backends.sql.compilers.sqlite import SQLiteCompiler
  File "/lib/python3.12/site-packages/ibis/backends/sql/compilers/sqlite.py", line 4, in <module>
    import sqlite3
ModuleNotFoundError: No module named 'sqlite3'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/lib/python3.12/site-packages/marimo/_runtime/executor.py", line 142, in execute_cell
    return eval(cell.last_expr, glbls)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  Cell marimo://notebook.py#cell=cell-2, line 1, in <module>
    ibis.duckdb.connect(":memory:")
    ^^^^^^^^^^^
  File "/lib/python3.12/site-packages/ibis/__init__.py", line 142, in __getattr__
    return load_backend(name)
           ^^^^^^^^^^^^^^^^^^
  File "/lib/python3.12/site-packages/ibis/__init__.py", line 84, in load_backend
    raise ImportError(
ImportError: Failed to import the duckdb backend due to missing dependencies.

You can pip or conda install the duckdb backend as follows:

  python -m pip install -U "ibis-framework[duckdb]"  # pip install
  conda install -c conda-forge ibis-duckdb           # or conda install

and in the developer console:

Reporting vitals
pyodide.asm.js:10 [py] Starting marimo...
worker-DKPGSFlQ.js:1 Loading Pygments, docutils, jedi, parso, pyodide-http
worker-DKPGSFlQ.js:1 Loaded Pygments, docutils, jedi, parso, pyodide-http
pyodide.asm.js:10 Loading from micropip: ['ibis-framework[duckdb]']
pyodide.asm.js:10 Loading duckdb, numpy, pandas, pyarrow, pyodide-unix-timezones, python-dateutil, pytz, rich, six, toolz, typing-extensions
pyodide.asm.js:10 Loaded duckdb, numpy, pandas, pyarrow, pyodide-unix-timezones, python-dateutil, pytz, rich, six, toolz, typing-extensions
index-CXUFCwUc.js:58 [STDERR] [9:36:24 AM] (vblA) {"msg":"This cell raised an exception: ImportError('Failed to import the duckdb backend due to missing dependencies.\n\nYou can pip or conda install the duckdb backend as follows:\n\n  python -m pip install -U \"ibis-framework[duckdb]\"  # pip install\n  conda install -c conda-forge ibis-duckdb           # or conda install')","exception_type":"ImportError","raising_cell":null,"type":"exception"}
index-CXUFCwUc.js:31  [Metric LCP] 2615.100000023842
index-CXUFCwUc.js:31  [Metric CLS] 0.0003077475580268822
index-CXUFCwUc.js:31  [Metric INP] 32

[cpcloud]

ModuleNotFoundError: No module named 'sqlite3'

This points to something wrong with marimo, or the pyodide version that it's using.

sqlite3 is a standard library module that can be imported in at least two pyodides:

• pyodide website console: https://pyodide.org/en/stable/console.html
• ibis repl: https://ibis-project.org/tutorials/browser/repl

Running import sqlite3 in both of those works as expected.

marimo definitely needs to be using the latest release of pyodide (0.27.1) for installing ibis to work out of the box without modification.

What version of pyodide is marimo using?

[kyrre]

The package.json says "pyodide": "^0.27.0". Does it need to be exactly 0.27.1?

I have created an issue on the marimo board marimo-team/marimo#3501.

[kyrre]

it was an issue with marimo installing the 'ibis' package by mistake! marimo-team/marimo#3501 (comment)