Does this extension support concurrent access to duckdb from multiple connections?

[chrisfw]

Hello,

I am very excited by the prospect of using this extension and combining two amazing projects!

That said, during experimentation with duckdb_fdw, I discovered that using multiple concurrent connections from separate processes was not supported. This seemed like an inherent problem with using an embedded db in a multi-process environment, so I am wondering if this will somehow be addressed architecturally by this extension implementation?

Thanks,
Chris Whelan

[JelteF]

Yes, supporting multiple concurrent connections is definitely something we support.

Because we currently don't support store any data in DuckDB, but only use its execution engine and read_csv/read_parquet functions. We simply create one DuckDB instance per connection.

This approach is something that we want to change though to make resource management easier, and because we want to support storing data in DuckDB itself. We plan on using a Postgres BackgroundWorker for this. Then we'll load DuckDB there and communicate with this process from the client processes using IPC (sockets and/or shared memory).

[chrisfw]

@JelteF , thank you for providing this info, that is great news! I am very excited to test this extension out and also to hear that in the future the integration will support storing data in duckdb as well!

Regards,
Chris