meta: Port backends to new relational operators/sqlglot

[jcrist]

We currently have a long running branch porting backends to use the new relational operators (and sql backends to use sqlglot) introduced by:

• refactor(ir): split the relational operations #7752

This is a meta issue to track and organize remaining backends that need to be ported before that branch can be merged with main. Feel free to edit this list to keep it up to date.

SQL Backends

• bigquery (refactor(bigquery): port to sqlglot #7968)
• clickhouse (refactor(sql): add a base SQLGlot backend for DuckDB and ClickHouse #7796)
• druid (refactor(druid): port to sqlglot #8006)
• duckdb (refactor(sql): add a base SQLGlot backend for DuckDB and ClickHouse #7796)
• exasol (refactor(exasol): port to sqlglot #8032)
• flink (refactor(flink): port to sqlglot #8268)
• impala (refactor(impala): port to sqlglot #7993)
• mssql (refactor(mssql): port to sqlglot #7936)
• mysql (refactor(mysql): port to sqlglot #7926)
• oracle (refactor(oracle): port to sqlglot #8020)
• postgres (refactor(postgres): port to sqlglot #7877)
• snowflake (refactor(snowflake): port to sqlglot #7853)
• sqlite (refactor(sqlite): port to SQLGlot #8154)
• trino (refactor(trino): port to sqlglot #7871)
• risingwave (refactor(risingwave): port to sqlglot #8171)

Other Backends

• dask (refactor(dask): port the dask backend to the new execution model #8005)
• datafusion (refactor(datafusion): port to new sqlglot backend and compiler #7846)
• pandas (refactor(pandas): port the pandas backend with an improved execution model #7797)
• polars (refactor(polars): update the polars backend to use the new relational abstractions #7868)
• pyspark (refactor(pyspark): reimplement the backend using the new relational operations and sqlglot  #7950)

[jcrist]

I'll start working on the sqlite backend now to familiarize myself with the new sqlglot code path. @cpcloud also just pinged on zulip saying he's working on the mysql backend. AFAIK all other backends are still available for work.

[cpcloud]

For Impala, we should be able to use the hive sqlglot dialect and supplement by adding rules to its TRANSFORMS dict (see the DataFusion backend for a custom dialect subclass).

[cpcloud]

I am starting on MS SQL now.

[kszucs]

Working on PySpark. Since the tests are really slow it will take some time.

[cpcloud]

I will start on BigQuery today or tomorrow, depending on how much progress I can make with mssql

[gforsyth]

I'll start on Oracle

[cpcloud]

I'm starting on the impala backend today.

[cpcloud]

I'll start on Druid today.

[cpcloud]

I am working on the exasol port today.

[lostmygithubaccount]

Weekly Update:
2024-01-18

This is the primary focus of the team, intending to get all backends refactored to the new relational model (building on previous refactor work) and in the process migrating from SQLAlchemy to SQLGlot. We are on track to get this done in Q1, with most backends already ported and the rest in progress (excluding Flink, RisingWave, and perhaps one more).

This week we landed the following backends in the port:

• druid
• impala
• pyspark
• BigQuery is ported and awaiting a final review.

Work is started on these backends:

• exasol
• sqlite
• oracle
• dask

(copying over last week's update on behalf of Phillip)

[cpcloud]

Work has completed on:

• exasol
• oracle

In PR:

• dask

In progress:

• sqlite

Not yet started:

• flink
• risingwave

Reviews:

• risingwave (likely to go in this week)

Others:

• flink blog
• 8.0.0 release

[cpcloud]

Work has completed on:

• sqlite

In review:

• dask

In progress:

• flink

Not yet started:

• risingwave

Others:

• risingwave-sans-sqlglot was merged

We're close to merging the-epic-split into main, and have reorganized remaining work items into tes-required-for-merge and tes-required-for-release labels to help identify blockers for the respective steps.

[cpcloud]

Work to port backends has completed on:

• dask
• risingwave

In review:

• flink

We're VERY close!

We've tagged a bunch of follow ups using the labels mentioned in the previous comment.

[cpcloud]

All backends ported and merged into the-epic-split. Closing this out!

Tracking the epic rebase in #8310.

Thanks to all involved in porting and refactoring, this was a huge effort!

[lostmygithubaccount]

reopening and adding [DONE] in the title for keep in on the GH project

great work everyone!

[cpcloud]

@lostmygithubaccount Why do we need to keep these open?

[lostmygithubaccount]

so they aren't removed from the GH project after 2 weeks and our roadmap is empty

[cpcloud]

We should try to come up with a way to address the specific concern without having to "hack" github issues by adding conventions.

A roadmap's issues IMO should be closed. It indicates that work is complete. We should also close milestones when they are reached, similar to how we close out release milestones when the release is complete.

If our roadmap is empty, and we don't want it to be empty, which seems reasonable, then we need to add more things to it.

[lostmygithubaccount]

re-closing per discussion