Error when appending to DuckLake table

[arouel]

When I want to append many records to a DuckLake table with an appender, I get the following error when closing the DuckDBAppender instance. I used duckdb-java version 1.3.0.0.

java.sql.SQLException: INTERNAL Error: Calling GetStorage on a TableCatalogEntry that is not a DuckTableEntry
This error signals an assertion failure within DuckDB. This usually occurs due to unexpected conditions or errors in the program's logic.
For more information, see https://duckdb.org/docs/stable/dev/internal_errors
	at org.duckdb.DuckDBNative.duckdb_jdbc_appender_close(Native Method)
	at org.duckdb.DuckDBAppender.close(DuckDBAppender.java:102)
	at test.DuckLakeAppenderTest.fails_appender_ducklake(DuckLakeAppenderTest.java:32)
	at java.base/java.lang.reflect.Method.invoke(Method.java:565)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1604)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1604)

The following test cases may help to reproduce the error.

package test;

import static org.assertj.core.api.BDDAssertions.*;

import java.sql.DriverManager;
import java.sql.SQLException;

import org.duckdb.DuckDBConnection;
import org.junit.jupiter.api.Test;

class DuckLakeAppenderTest {

    @Test
    void fails_appender_ducklake() throws SQLException {
        try (
            var connection = (DuckDBConnection) DriverManager.getConnection("jdbc:duckdb:");
            var statement = connection.createStatement()
        ) {
            // not necessary with duckdb 1.3.1
            connection.setAutoCommit(false);

            statement.execute("""
                ATTACH 'ducklake:/tmp/test_fails_appender_ducklake.ducklake' AS my_ducklake;
                USE my_ducklake;
                CREATE TABLE IF NOT EXISTS test (a INT);
                """);

            try (var appender = connection.createAppender("main", "test")) {
                appender.beginRow();
                appender.append(124);
                appender.endRow();
                // fails: INTERNAL Error: Calling GetStorage on a TableCatalogEntry that is not a DuckTableEntry
            }

            try (var rs = statement.executeQuery("select * from test")) {
                rs.next();
                then(rs.getInt(1)).isEqualTo(124);
            }
        }
    }

    // workaround via temporary table
    @Test
    void succeeds_appender_ducklake_via_temporary_table() throws SQLException {
        try (
            var connection = (DuckDBConnection) DriverManager.getConnection("jdbc:duckdb:");
            var statement = connection.createStatement()
        ) {
            connection.setAutoCommit(false);

            statement.execute("""
                CREATE TEMP TABLE IF NOT EXISTS test_temp (a INT);
                ATTACH 'ducklake:/tmp/test_succeeds_appender_ducklake_via_temporary_table.ducklake' AS my_ducklake;
                USE my_ducklake;
                CREATE TABLE IF NOT EXISTS test (a INT);
                """);

            try (var appender = connection.createAppender("main", "test_temp")) {
                appender.beginRow();
                appender.append(124);
                appender.endRow();
            }

            statement.execute("""
                INSERT INTO test SELECT * FROM test_temp;
                DROP TABLE test_temp;
                """);

            try (var rs = statement.executeQuery("select * from test")) {
                rs.next();
                then(rs.getInt(1)).isEqualTo(124);
            }
        }
    }
}

Is there a plan to implement or fix it so that we can append directly the records to a DuckLake table?

[staticlibs]

Shall I use a temporary table until this is implemented/fixed?

Yes, temporary in-memory table + COPY, some more details are in duckdb/ducklake#46.

Also, the JDBC Appender is being reworked to make it closer to the one used in Golang's DuckDB client, but this change is not confirmed for 1.3.1. Current JDBC Appender is not thread-safe - needs to be used with caution.

[arouel]

My workload heavily relies on Appender. Any improvement is very appreciated.

[nickperrott]

My workload heavily relies on Appender. Any improvement is very appreciated.

Me too!

I am using duckdb to export large amounts of data (billions of rows) in short time periods by running 100s of concurrent export processes. Each process inserts data first into an in-memory duckdb tables using appenders and then exports the tables to Parquet.

I have tried changing the 2nd bit to COPY or INSERT INTO ducklake tables, but that is really slow.

It looks like I could speed it up by exporting first to Parquet, then loading the Parquet files into ducklake, but this is starting to get quite messy.

[staticlibs]

Appender improvements were added in reworked Appender in #295, closing the issue, please reopen if needed.