Discussion: adopt datafusion-python crate

[ion-elgreco]

Description

Use Case
We can adopt the datafusion-python crate, for the deltalake-python. A couple benefits would be this:

pro's:

• Make sessionContext configurable from python
• Allows UDFs to be registered and used during operations
• Potentially allow plans to be returned as dry run and explained before running?

cons:

• Datafusion-python updates at a slower rate than datafusion, which could hold us back and make the update process which is already complicated even more complicated

Related Issue(s)

[ion-elgreco]

@timsaucer thoughts on this?

[timsaucer]

My general recommendation is to keep datafusion rust repo as your primary rust dependency and (if needed) to use datafusion-python as a python dependency. But I suspect I don't fully understand the use case of what you're trying to do here.

Can you give an example of what you're currently trying to do that is hampered by not using datafusion-python?

[ion-elgreco]

Some folks want to be able to configure the datafusion sessioncontext or register UDFs, neither is possible at the moment. But I also don't want to build this integration from the ground-up, therefore depending on datafusion-python crate would solve that

[roeap]

My gut feeling would be that our best bet is to expose record batch iterators ideally for both log and data. most modern engines / dataframe libraries should be able to consume these. The main challenge would be how to do predicate pushdown when the query would ideally only be issues at the "frontend" library.

For some engines native support is also on the way, and hopefully that happens more :).

When it comes to UDFs, I guess there only need to be available in the engine that does the downstream processing? But maybe we can expose some sort of python table provider (factory) that integrates with datafusion, hoping that this can be done w/o coupling the datafusion version with datafusion-python.

I'd be curious to learn, what the most requested config on the session is - do we have some insights on this?

[ion-elgreco]

My gut feeling would be that our best bet is to expose record batch iterators ideally for both log and data. most modern engines / dataframe libraries should be able to consume these. The main challenge would be how to do predicate pushdown when the query would ideally only be issues at the "frontend" library.

For some engines native support is also on the way, and hopefully that happens more :).

Yeah I wasn't looking at the reader side, but rather operations from py->rust. But I agree, Datafusion-python (python library, not the crate) has already native support. Polars also has somewhat native support, not the most optimal but it's already tons faster than pyArrow dataset

When it comes to UDFs, I guess there only need to be available in the engine that does the downstream processing? But maybe we can expose some sort of python table provider (factory) that integrates with datafusion, hoping that this can be done w/o coupling the datafusion version with datafusion-python.

I'd be curious to learn, what the most requested config on the session is - do we have some insights on this?

Yeah udfs in this case are only to be used during our Delta operations that use Datafusion. Then each operation in python can set the session context as well

[timsaucer]

If the case is that some users want to modify the session context or register functions, why do those users not get at this via datafusion python package?

In general, pulling in datafusion-python rust I think will cause you to re-export a lot of code and not be performant.

[ion-elgreco]

@timsaucer They were asking for the functions to be executed within the delta-rs operations (MERGE, write etc)

[rtyler]

I have been following this discussion and thinking about it a fair bit. @timsaucer I am curious to hear your perspective on what it would look like if we actually broke up the Python code from a monolithic deltalake package and instead released sub-packages similar to what we do on the 🦀 side?

I wonder if we made deltalake a bit more slim as a package, and then invested in the effort to make it a little easier for other Python packages to provide meaningful ways for dependencies to modify behavior such as a deltalake-datafusion 🐍 package or deltalake-hdfs and so on.

The idea is still a little fuzzy on how it might work in my head, and I have inflicted the most overhead on myself with the meta-/sub-crates architecture for 🦀 but I think our Python code is getting to the point where being more decomposable might be really helpful.

[ion-elgreco]

If it was pure python this would probably work well, but with this rust-py binding layer, and without FFI across these sub packages I am not sure if it's feasible

deltalake-hdfs would be nice, and I know there was someone from SAP who want to register an SAP objectstore in python, but this would require an FFI implementation of object store.