Upgrade DataFusion Version to support more functions

[chaitnaya1391]

Is your feature request related to a problem? Please describe.
Yes, currently facing a challenge with limited functions supported on Cube store. We have a read-only Trino (thrift connectors) access that connects to multiple data sources. Our data joining happens at runtime and is exported to Cube. We would like to store this as the original SQL pre-aggregation but in cubestore. The documentation specifically mentions that we do not use an external source for this, but we do not have a solution to write back to Trino (Hive/Iceberg) - database data types incompatibility.

Describe the solution you'd like
Upgrade the cubestore arrow-datafusion to use the latest datafusion.

Describe alternatives you've considered
We tried external DB for pre-aggregations, but that is just too slow compared to cubestore. And with your warnings of that being deprecated in the future, it is not a solution that we would like to explore further.

[igorlukanin]

We would like to store this as the original SQL pre-aggregation but in cubestore.

Maybe it's just a typo but I would recommend on using the rollup pre-aggregation type, not original_sql.

Yes, currently facing a challenge with limited functions supported on Cube store.

I don't fully understand how is that related. Which functions do you think you need—and why?

FYI, if there are certain functions in sql parameters of your dimensions, you can lift them up to the sql of your cubes.

[chaitnaya1391]

Hey @igorlukanin, thank you for responding.

We tried the roll up but that creates a duplicate query on the DB and since the queries are duplicated, this seems like something that can be avoided if only Datafusion functions support comes in?

Function we were hoping to use: median (the simplest of an agg function) but original_sql doesn't support this as cubestore only uses Datafusion 4.0.0; by maybe decoupling datafusion from cubestore, your users would get a lot more functionality.

Our usecase: fully customised tenant specific dashboards and queries. A rollup preagg and original_sql preagg would run 2 queries leading to inefficient and slower turn-around time for full data export though dashboards.