feat(sql): support inserts with default constraints

[IndexSeek]

Description of changes

This change intends to support scenarios where a user needs to insert into a table where a table contains DEFAULT values on columns, meaning the incoming object may have fewer columns than the target table schema.

It's common to insert into tables with DEFAULT constraints; in many cases, these are sequences, and it is a bit tricky to provide the sequence "nextval" approach today.

This topic was initially brought up in Zulip.

Here is an example of this in practice:

import ibis
import pandas as pd

con = ibis.duckdb.connect()
con.raw_sql("CREATE TABLE example (a INTEGER DEFAULT 1, b VARCHAR NOT NULL);")

df = pd.DataFrame(
    {
        "a": [1, 2, 3],
        "b": ["foo", "bar", "baz"],
    }
)

Backend error

If the user excludes a required column (e.g., NOT NULL without a DEFAULT) from the container used for insert, the backend will error out with its respective error.

con.insert("example", df[["a"]])

Successful insert with excluded columns

con.insert("example", df[["b"]])
con.table("example").to_pandas()

a	b
1	foo
1	bar
1	baz

I'm still working through the tests for this, but wanted to put this out here in case anyone wanted to take a look before I can return to it.

[IndexSeek]

I have made more progress on the tests. I got the Oracle backend passing, but it required a hacky solution to ensure the identifier was being quoted.

import ibis.backends.sql.compilers as sc

quoted = getattr(sc, con.dialect.__name__.lower()).compiler.quoted

It works, but I wonder if there is a better way.

I will try to look into the remaining backends that are still failing, which are: Clickhouse, Trino, Druid, Exasol.

Edit:
I was able to get the Clickhouse query to compile by suffixing it with ORDER BY "a", but I noticed in another line that create table was not implemented, so I will mark this as notimpl as well.

[cpcloud]

Let's make a variable for source.schema and then use col in variable to do this lookup.

[IndexSeek]

I believe the change I have made will reduce the time complexity to avoid needing to check again against the source_cols list. I have rewritten things a bit here, using a variable named columns to refer to the column list to be used for the insert SQL expression.

[IndexSeek]

I swapped things around here to use issubset - my idea was that if the source contains fewer columns than the target, but the column names are the same, that's fine, as we can use the source's columns in the insert list.

issubset will also evaluate to True if the columns are the same in both source and target, regardless of order. In this scenario, we want to use the order of source, so that the SQL expression will be written properly.

So if my source schema is:

ibis.Schema {
  b  string
  a  int64
}

But my target schema is:

ibis.Schema {
  a  int64
  b  string
}

This is okay, because the underlying SQL expression come out to:

INSERT INTO target (b, a)
SELECT * 
FROM source;

and in the event that maybe the target schema has a default constraint on column "a", and our source schema looks like

ibis.Schema {
  b  string
}

the query will be written like so:

INSERT INTO target (b)
SELECT * 
FROM source;

[IndexSeek]

I have marked this one ready for review, but I have a couple of questions.

1. The Oracle backend is still failing, and I wasn't sure if there was a clean way to address that with a SQLGlot method to ensure quoting the identifiers in the Create expression from parse_one or if I should adjust the ct_sql variable to enclose the object identifiers in double-quotes. My concern is that the Snowflake backend may encounter a similar outcome when cloud runs are triggered.
2. Druid is marked "notyet," but fails because raw_sql is not supported. Should I include that in the list with exasol?

[cpcloud]

I'll add an xfail marker to the Druid backend, since it doesn't support CREATE TABLE.

[cpcloud]

Oh, looks like you did that already. Let me see what's failing then.

[cpcloud]

@IndexSeek I improved the implementation bit: we can use our Schema.keys() method (whose output behaves like a set) instead of constructing a set and calling issubset with a sequence (which itself will construct a set if the input argument isn't already a set).

This avoids constructing two throwaway sets, the effects of which can show up in wide-table use cases.

[IndexSeek]

@IndexSeek I improved the implementation bit: we can use our Schema.keys() method (whose output behaves like a set) instead of constructing a set and calling issubset with a sequence (which itself will construct a set if the input argument isn't already a set).

This avoids constructing two throwaway sets, the effects of which can show up in wide-table use cases.

This is a very nice improvement. Thank you for the assistance here and the explanation, @cpcloud!

[cpcloud]

Running the cloud backend test suite, then if that's all green this is good to merge!

[cpcloud]

I'm going to put up a separate PR to fix the reason why this is so gross.