feat: `IbisLazyFrame` support #2000

rwhitten577 · 2025-02-12T20:48:21Z

What type of PR is this? (check all applicable)

Related issues

Closes Request for contributions: Ibis support #1886

Checklist

Code follows style guide (ruff)
Tests added
Documented the changes

If you have comments or can explain your changes, please do so below

NickCrews · 2025-02-21T16:21:35Z

I want ibis support in narwhals and can help to get this landed. Ping me if you want a review or help!

Cc for visibility @cpcloud, the maintainer of ibis

MarcoGorelli · 2025-02-21T16:27:38Z

thanks @NickCrews for your help! sure - we're in the middle of some large refactors so it may be prudent to wait a bit to avoid too many merge conflicts, but we will get to this

dangotbanned · 2025-03-12T11:31:25Z

`ibis.selectors`

I'm interested in how we might be able to take advantage of ibis.selectors.
I noted in (#2064 (comment)) ibis is an outlier because of their native support

Initial thoughts are that our adaptation could work quite differently to the other backends - maybe even being more performant

Since (#2064) every backend besides polars is sharing a lot of code - which recently moved to nw._compliant.selectors.
Most of this is simply performing operations on nw.Schema

MarcoGorelli · 2025-03-19T15:07:17Z

Hey @rwhitten577 - we're done with the big refactors, so if it interests you to continue, we'd love to ship this 🚢 🚀

I'd suggest not worrying about doing anything different for selectors just yet (it looks to me like they call .schema anyway?), mirroring what we do for duckdb / sqlframe / pyspark should be fine 👍

afrisgaard · 2025-03-27T06:32:31Z

How would this land? (Just considering to be ready when ibis ships to Narwhals :) )

Something like this:
ibis_frame = ibis.use_backend("BigQuery"/"DuckDb"/etc)

NarwhalsFrame = from_native(ibis_frame)

.. do narwhals operations

MarcoGorelli · 2025-03-27T09:16:35Z

yup,that's right! we'd just translate to the ibis api, so whatever backend is set there would be used

rwhitten577 · 2025-04-02T13:37:13Z

Hey @MarcoGorelli, I will try to pick this back up in the next few weeks. I haven't looked through all the refactor changes yet, but do you think I'd be better off rebasing this old branch or are the changes significant enough where I'd want to start over and use duckdb or spark as an example?

MarcoGorelli · 2025-04-02T14:07:12Z

awesome, thanks @rwhitten577 !

i think it might be salvageable to continue from here, but still using _spark_like / _duckdb as a reference. looks like most of the merge conflicts are in the tests anyway

dangotbanned · 2025-04-16T15:38:33Z

@MarcoGorelli do we need to update some config to fix this? https://results.pre-commit.ci/run/github/760058710/1744817752.vQ_lZmv5RqOHrGGzLqhyoQ

I thought it was fine to import when we're in the backend package

MarcoGorelli · 2025-04-16T15:40:41Z

yes you could update

narwhals/utils/import_check.py

Lines 21 to 27 in aa48faa

    
           ALLOWED_IMPORTS = { 
        
               "_pandas_like": {"pandas", "numpy"}, 
        
               "_arrow": {"pyarrow", "pyarrow.compute", "pyarrow.parquet"}, 
        
               "_dask": {"dask.dataframe", "pandas", "dask_expr"}, 
        
               "_polars": {"polars"}, 
        
               "_duckdb": {"duckdb"}, 
        
           }

dangotbanned · 2025-04-16T15:44:50Z

narwhals/_ibis/expr.py

+    from narwhals.utils import _FullContext
+
+
+class IbisExpr(LazyExpr["IbisLazyFrame", "ir.Expr"]):


Using ir.Expr seems to be the main source of typing issues.

I was initially thinking of changing the definition of NativeExpr

narwhals/narwhals/_compliant/expr.py

Lines 80 to 89 in 7059f5b

class NativeExpr(Protocol):

"""An `Expr`-like object from a package with [Lazy-only support](https://narwhals-dev.github.io/narwhals/extending/#levels-of-support).

Protocol members are chosen *purely* for matching statically - as they

are common to all currently supported packages.

"""

def between(self, *args: Any, **kwds: Any) -> Any: ...

def isin(self, *args: Any, **kwds: Any) -> Any: ...

But most of the methods being called don't exist on ir.Expr

https://github.com/ibis-project/ibis/blob/d7cd846627269b1f4c901cc3f0e380ee9024f796/ibis/expr/types/core.py#L67-L920

E.g. here, all other the methods being called on expr are referring to something other than ir.Expr?

https://github.com/rwhitten577/narwhals/blob/6e44a8b54c331f6056580a79f2bedf950e78d825/narwhals/_ibis/expr.py#L136-L152

Yeah I think the Ibis types reported are going to be specific e.g. IntegerColumn or IntegerValue. Will explore a TypeAlias that covers these

Seems like we might want to use one (or more) of Value, Scalar, Column:

https://github.com/ibis-project/ibis/blob/d7cd846627269b1f4c901cc3f0e380ee9024f796/ibis/expr/types/generic.py#L39

https://github.com/ibis-project/ibis/blob/d7cd846627269b1f4c901cc3f0e380ee9024f796/ibis/expr/types/generic.py#L1368

https://github.com/ibis-project/ibis/blob/d7cd846627269b1f4c901cc3f0e380ee9024f796/ibis/expr/types/generic.py#L1487

Yeah I think the Ibis types reported are going to be specific e.g. IntegerColumn or IntegerValue. Will explore a TypeAlias that covers these

Right I see, they have things quite a bit more strongly typed.
With methods split up by class - rather than a unified Expr or Column or Series with namespace accessors 🤔

Or at least coming up with some common patterns to follow, where we block certain ops and raise consistent errors

A basic solution would be like ArrowSeries.median

narwhals/narwhals/_arrow/series.py

Lines 322 to 331 in 7059f5b

def median(self: Self, *, _return_py_scalar: bool = True) -> float:

from narwhals.exceptions import InvalidOperationError

if not self.dtype.is_numeric():

msg = "`median` operation not supported for non-numeric input type."

raise InvalidOperationError(msg)

return maybe_extract_py_scalar(

pc.approximate_median(self.native), _return_py_scalar

)

Not sure what kind of overhead this might have if we need to check frequently

@MarcoGorelli we may need to consider abstracting this out - since IIUC we might run into AttributeErrors if we aren't careful

we already don't allow aggregating scalar-like expressions

In [2]: nw.col('a').mean().mean() --------------------------------------------------------------------------- InvalidOperationError Traceback (most recent call last) Cell In[2], line 1 ----> 1 nw.col('a').mean().mean() File ~/scratch/.venv/lib/python3.12/site-packages/narwhals/expr.py:560, in Expr.mean(self) 541 def mean(self: Self) -> Self: 542 """Get mean value. 543 544 Returns: (...) 558 └──────────────────┘ 559 """ --> 560 return self._with_aggregation(lambda plx: self._to_compliant_expr(plx).mean()) File ~/scratch/.venv/lib/python3.12/site-packages/narwhals/expr.py:83, in Expr._with_aggregation(self, to_compliant_expr) 81 if self._metadata.kind.is_scalar_like(): 82 msg = "Aggregations can't be applied to scalar-like expressions." ---> 83 raise InvalidOperationError(msg) 84 return self.__class__( 85 to_compliant_expr, self._metadata.with_kind(ExprKind.AGGREGATION) 86 ) InvalidOperationError: Aggregations can't be applied to scalar-like expressions.

Ok pyright should be fixed now. Ibis' type system is funky but did my best to set the correct types or cast to something that made sense

Much appreciated @rwhitten577!

It seems the casts might be unavoidable for now (#2000 (comment)) 😔

I had to do a lot of that in

fix(typing): Resolve all mypy & pyright errors for _arrow #2007

which eventually led to

Tracking: pyarrow-stubs upstream fixes #2124

That is to say, identifying the issues is a big step and could even lead to improvements down the line 😅

- narwhals-dev#2000 (comment) - narwhals-dev#2000 (comment)

See narwhals-dev#2140

dangotbanned · 2025-04-16T20:34:05Z

narwhals/_ibis/expr.py

+    def diff(self) -> Self:
+        def _func(window_inputs: WindowInputs) -> ir.NumericValue:
+            expr = cast("ir.NumericColumn", window_inputs.expr)
+            return expr - cast(
+                "ir.NumericColumn", expr.lag().over(ibis.window(following=0))
+            )
+
+        return self._with_window_function(_func)


@NickCrews it seems like some of our issues (#2000 (comment)) might be solved by changing the return type of some ibis methods to Self.
Just throwing this out there 🙂

Picking one at random, but say we start with:

expr: ir.NumericColumn

We first call:

https://github.com/ibis-project/ibis/blob/d7cd846627269b1f4c901cc3f0e380ee9024f796/ibis/expr/types/generic.py#L2707-L2711

op_1: ir.Column = expr.lag()

Then:

https://github.com/ibis-project/ibis/blob/d7cd846627269b1f4c901cc3f0e380ee9024f796/ibis/expr/types/generic.py#L817-L125

op_2: ir.Value = op_1.over(ibis.window(following=0))

But at that point we can't do the binary op, since the right-hand side isn't a ir.NumericValue:

https://github.com/ibis-project/ibis/blob/d7cd846627269b1f4c901cc3f0e380ee9024f796/ibis/expr/types/numeric.py#L646-L648

# Operator "-" not supported for types "NumericColumn" and "Value"Pylance[reportOperatorIssue] result = expr - op_2

Yep that would help greatly here to return Self from funcs like over, name, etc which currently lose the original column type.

rwhitten577 · 2025-04-17T13:15:21Z

if we're adding initial support, i'd advocate for keeping it simple and just raising if self._backend_version < (0,10) for some operations, we can always lower the minimum version later

Just added this guard for the few places ibis.cases is used

- narwhals-dev@5f3687f - narwhals-dev#2000 (comment)

dangotbanned · 2025-04-17T20:06:43Z

#2000 (comment)

@rwhitten577 I fumbled big time on the spelling, but I've used (#2000 (comment)) for these in

refactor: Use @require.backend_version

https://typing.python.org/en/latest/spec/generics.html#use-in-parameter-types

- https://github.com/narwhals-dev/narwhals/actions/runs/14524056463/job/40751595357?pr=2000 - narwhals-dev@2843b7c - narwhals-dev#2000 (comment)

dangotbanned · 2025-04-17T20:30:21Z

narwhals/_ibis/dataframe.py

    def __init__(
-        self: Self,
-        df: ir.Table,
-        *,
-        backend_version: tuple[int, ...],
-        version: Version,
+        self, df: ir.Table, *, backend_version: tuple[int, ...], version: Version
    ) -> None:
        self._native_frame: ir.Table = df
        self._version = version


FYI @MarcoGorelli

You might have noticed I've been gradually doing this where it isn't needed.

refactor(typing): Remove implicit Self

Now seemed like a good isolated time to remove them all for ibis 🙂

I've been leaving them on the public API, in case there was some benefit I didn't know about

https://typing.python.org/en/latest/spec/generics.html#use-in-parameter-types

We can test this now narwhals-dev#2000 (comment)

Some of https://github.com/narwhals-dev/narwhals/actions/runs/14524350214/job/40752476168?pr=2000

https://github.com/narwhals-dev/narwhals/actions/runs/14524350214/job/40752476168?pr=2000

dangotbanned · 2025-04-17T21:21:03Z

yeah CI check would at least need to be green before merging

we can't run this in every environment that we measure coverage on due to upper-bound constraits

they accepted my pr to remove upper-bound constraints, so we can probably include ibis in the coverage metrics

@MarcoGorelli we can do ibis coverage on 3.11 & 3.13 jobs

(ae79ca4)

I'm not sure how to disable it only on the 3.8 & 3.9 jobs 😞

rwhitten577 added 7 commits February 11, 2025 14:26

Initial copy of duckdb impl for ibis

a186e7d

Rename existing _ibis -> _ibis_old, update most of IbisNamespace

4a98c0b

Update ibis namespace

7b25227

Remove _ibis_old, update dataframe methods

bbee1d2

Update Expr methods

6c739d4

All ibis frame tests passing

70605ad

WIP ibis expr tests

56a5f08

rwhitten577 mentioned this pull request Feb 12, 2025

Request for contributions: Ibis support #1886

Open

dangotbanned mentioned this pull request Mar 13, 2025

[Bug]: type-checking from_native(spark_dataframe) broke between v1.22 and v1.23 #2194

Closed

Merge branch 'main' into feature/initial-ibis-lazyframe

82c41f8

rwhitten577 force-pushed the feature/initial-ibis-lazyframe branch from b0763ae to 82c41f8 Compare April 14, 2025 16:40

rwhitten577 added 5 commits April 14, 2025 13:45

Update ibis dataframe and groupby to match latest duckdb

04a41ec

Update ibis expr match latest duckdb

9e756e5

Update ibis expr dt, list, str to match latest duckdb

8ccff78

Update ibis namespace and selectors to match latest duckdb

6140962

All ibis frame tests passing

15e89c9

rwhitten577 force-pushed the feature/initial-ibis-lazyframe branch from 5b12e12 to 15e89c9 Compare April 14, 2025 21:15

rwhitten577 added 5 commits April 15, 2025 11:27

Fix more ibis tests, esp group_by, read/scan, and std/var ddof

81a62f0

All ibis tests except expr_and_series passing

e71d899

More ibis expr tests passing

4459ca7

Fix ibis type issues

8eb20f6

Merge branch 'main' into feature/initial-ibis-lazyframe

22026ac

dangotbanned reviewed Apr 16, 2025

View reviewed changes

dangotbanned and others added 12 commits April 16, 2025 16:50

the great unbanning

21dd5b4

- narwhals-dev#2000 (comment) - narwhals-dev#2000 (comment)

Fix ibis pyright issues on all but expr.py and namespace.py

6f489f1

Add branch for converting nw enum to ibis

af77027

Fix ibis pyright for namespace.py

a685837

refactor(typing): Simplify concat

1b07dfe

refactor: Also same error in concat

25b30e0

fix: Collect iterable before consuming

dceda64

refactor: Simplify concat some more

8f7f005

refactor: Use more chain.from_iterable

55240aa

See narwhals-dev#2140

a little lit

49e9918

Fix ibis pyright for expr.py

28b7ec5

Fix ibis pyright for dataframe.py

1ed691a

dangotbanned reviewed Apr 16, 2025

View reviewed changes

rwhitten577 added 2 commits April 17, 2025 09:04

Add ibis to pytest gh action

2843b7c

Add ibis >= 10.0 requirement around cases, fix interval casting

5f3687f

dangotbanned added 2 commits April 17, 2025 20:58

Merge remote-tracking branch 'upstream/main' into pr/rwhitten577/2000

5ade4b8

refactor: Use @require.backend_version

3248a50

- narwhals-dev@5f3687f - narwhals-dev#2000 (comment)

dangotbanned added 2 commits April 17, 2025 21:15

refactor(typing): Remove implict Self

5df6b65

https://typing.python.org/en/latest/spec/generics.html#use-in-parameter-types

ci: Make sure ibis is installed

ae79ca4

- https://github.com/narwhals-dev/narwhals/actions/runs/14524056463/job/40751595357?pr=2000 - narwhals-dev@2843b7c - narwhals-dev#2000 (comment)

dangotbanned reviewed Apr 17, 2025

View reviewed changes

dangotbanned added 4 commits April 17, 2025 21:34

ci: Include ibis in coverage

ea32978

We can test this now narwhals-dev#2000 (comment)

refactor: Use not_implemented

d0490d7

Some of https://github.com/narwhals-dev/narwhals/actions/runs/14524350214/job/40752476168?pr=2000

ci: omit _ibis.typing

89275e3

https://github.com/narwhals-dev/narwhals/actions/runs/14524350214/job/40752476168?pr=2000

more coverage

88e36c0

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: `IbisLazyFrame` support #2000

feat: `IbisLazyFrame` support #2000

rwhitten577 commented Feb 12, 2025 •

edited

Loading

NickCrews commented Feb 21, 2025

MarcoGorelli commented Feb 21, 2025

dangotbanned commented Mar 12, 2025

MarcoGorelli commented Mar 19, 2025

afrisgaard commented Mar 27, 2025

MarcoGorelli commented Mar 27, 2025

rwhitten577 commented Apr 2, 2025

MarcoGorelli commented Apr 2, 2025

dangotbanned commented Apr 16, 2025

MarcoGorelli commented Apr 16, 2025

dangotbanned Apr 16, 2025

dangotbanned Apr 16, 2025

rwhitten577 Apr 16, 2025

dangotbanned Apr 16, 2025

dangotbanned Apr 16, 2025

dangotbanned Apr 16, 2025

dangotbanned Apr 16, 2025 •

edited

Loading

MarcoGorelli Apr 16, 2025

rwhitten577 Apr 16, 2025

dangotbanned Apr 16, 2025

dangotbanned Apr 16, 2025 •

edited

Loading

rwhitten577 Apr 17, 2025

rwhitten577 commented Apr 17, 2025

dangotbanned commented Apr 17, 2025

dangotbanned Apr 17, 2025 •

edited

Loading

dangotbanned commented Apr 17, 2025

		from narwhals.utils import _FullContext


		class IbisExpr(LazyExpr["IbisLazyFrame", "ir.Expr"]):

	class NativeExpr(Protocol):
	"""An `Expr`-like object from a package with [Lazy-only support](https://narwhals-dev.github.io/narwhals/extending/#levels-of-support).

	Protocol members are chosen purely for matching statically - as they
	are common to all currently supported packages.
	"""

	def between(self, args: Any, *kwds: Any) -> Any: ...
	def isin(self, args: Any, *kwds: Any) -> Any: ...

	def median(self: Self, *, _return_py_scalar: bool = True) -> float:
	from narwhals.exceptions import InvalidOperationError

	if not self.dtype.is_numeric():
	msg = "`median` operation not supported for non-numeric input type."
	raise InvalidOperationError(msg)

	return maybe_extract_py_scalar(
	pc.approximate_median(self.native), _return_py_scalar
	)

feat: IbisLazyFrame support #2000

Are you sure you want to change the base?

feat: IbisLazyFrame support #2000

Conversation

rwhitten577 commented Feb 12, 2025 • edited Loading

What type of PR is this? (check all applicable)

Related issues

Checklist

If you have comments or can explain your changes, please do so below

NickCrews commented Feb 21, 2025

MarcoGorelli commented Feb 21, 2025

dangotbanned commented Mar 12, 2025

ibis.selectors

MarcoGorelli commented Mar 19, 2025

afrisgaard commented Mar 27, 2025

MarcoGorelli commented Mar 27, 2025

rwhitten577 commented Apr 2, 2025

MarcoGorelli commented Apr 2, 2025

dangotbanned commented Apr 16, 2025

MarcoGorelli commented Apr 16, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dangotbanned Apr 16, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dangotbanned Apr 16, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rwhitten577 commented Apr 17, 2025

dangotbanned commented Apr 17, 2025

dangotbanned Apr 17, 2025 • edited Loading

Choose a reason for hiding this comment

dangotbanned commented Apr 17, 2025

feat: `IbisLazyFrame` support #2000

feat: `IbisLazyFrame` support #2000

rwhitten577 commented Feb 12, 2025 •

edited

Loading

`ibis.selectors`

dangotbanned Apr 16, 2025 •

edited

Loading

dangotbanned Apr 16, 2025 •

edited

Loading

dangotbanned Apr 17, 2025 •

edited

Loading