Separable Python and Rust components

[awhyte]

Is your feature request related to a problem or challenge? Please describe what you are trying to do.
I maintain a Rust library that uses Datafusion internally for certain calculations. The remainder of this library can be operated from Python via pyo3.

In order to control the behvariour of some of the internal Datafusion calculations, I would like to be able to pass datafusion-python Python objects like [Py]Expr and [Py]SortExpr into Rust as arguments.

I can include the datafusion_python crate with its incredibly helpful wrappers, structs and conversions. However, regular datafusion-python Python objects cannot be passed in because the datafusion Python package links to its own private Rust library rather than my independent Rust library.

pyo3 complains with:

TypeError: argument 'example_custom_expr': 'Expr' cannot be converted to 'Expr'.

This seems to be because the two Rust backends are not binary compatible.

Describe the solution you'd like
I would like to be able to point the datafusion-python Python frontend at a different Rust backend in order to make use of its great feature coverage and overall capability.

Would it be possible to override the default linking location?

Describe alternatives you've considered
The only way I can think to make this work would be to copy relevant chunks of Python code from datafusion-python package into my own Python code, and try to keep it synchronised.

Additional context
This would allow datafusion to be used as a component within more projects, especially as increasing backend work is being done by Rust with great end-user experience provided by Python.

[timsaucer]

I think this might be a duplicate of #853 which may need to be reopened.

I suspect your immediate problem is resolved by passing .expr on the calls you are trying to make or wrapping your PyExpr.

More generally I would also be careful about trying to share these binaries across different builds as there can be problems with the non stable ABI. There is additional information here: https://datafusion.apache.org/python/contributor-guide/ffi.html

[awhyte]

Thanks very much for the quick response, Tim.

Sadly passing col("a").expr as my_lib_function(col("a").expr) also fails with the same message. Perhaps the build configuration was too different - I'll take a look at that to check.

I'm not quite sure I understand what you mean by wrapping the PyExpr?

Thank you for the link and the great write-up there - I'm not sure how I missed that previously. Would a helpful contribution here be for me to write FFI_ and Foreign implementations for expressions? The current datafusion-ffi crate seems to handle expressions via protobuf serialisation from datafusion-proto (datafusion/ffi/src/udtf.rs) which doesn't seem so nice.

[timsaucer]

Ok, I think I see what is happening then. I do think this is a problem of the unstable ABI.

Are all of the extension points you're trying to get into python UDFs? If so it may be not too difficult. I just put up this repo for the purpose of demonstrating extension types, but I think it may be a good example for your use as well. Specifically as long as you use 48.0.0 (both rust and python) you should be able to expose your UDFs like in this library