datafusion-go

datafusion-go provides a database/sql driver and Arrow APIs for Apache DataFusion. It is an unofficial community binding in datafusion-contrib.

Go API reference | User guide | Releases | Changelog | Contributing

Install

The package requires Go 1.26 or newer, a C toolchain, and cgo enabled. On Windows, use a MinGW/GNU toolchain for the x86_64-pc-windows-gnu Rust ABI.

Prebuilt native libraries support these platforms:

Operating system	Go platforms
macOS	darwin-arm64, darwin-amd64
Linux	linux-amd64, linux-arm64
Windows	windows-amd64

The shell examples use POSIX shell syntax.

If you do not have a Go module, run these commands in order:

mkdir datafusion-quickstart
cd datafusion-quickstart
go mod init example.com/datafusion-quickstart

From your module directory, install the package:

go get github.com/datafusion-contrib/datafusion-go

On first use, the driver downloads libdatafusion_go from the GitHub release for the installed module version. It verifies the library checksum. For library selection and download controls, see Native Runtime.

Use a tagged release for applications. Development snapshots from @main can lack native libraries with the same version.

Quick Start

Use the module directory from Install for this example.

1. Create trips.csv:

   printf 'city,trips\nnyc,3\nnyc,5\nsf,2\n' > trips.csv

2. Save this code as main.go in the same directory:

   package main
   
   import (
   	"context"
   	"database/sql"
   	"fmt"
   	"log"
   
   	_ "github.com/datafusion-contrib/datafusion-go"
   )
   
   func main() {
   	ctx := context.Background()
   
   	db, err := sql.Open("datafusion", "")
   	if err != nil {
   		log.Fatal(err)
   	}
   	defer db.Close()
   
   	_, err = db.ExecContext(ctx, `create external table trips
   		stored as csv location 'trips.csv'
   		options ('format.has_header' 'true')`)
   	if err != nil {
   		log.Fatal(err)
   	}
   
   	rows, err := db.QueryContext(ctx, `select city, sum(trips) as total
   		from trips group by city order by total desc`)
   	if err != nil {
   		log.Fatal(err)
   	}
   	defer rows.Close()
   
   	for rows.Next() {
   		var city string
   		var total int64
   		if err := rows.Scan(&city, &total); err != nil {
   			log.Fatal(err)
   		}
   		fmt.Printf("%s\t%d\n", city, total)
   	}
   	if err := rows.Err(); err != nil {
   		log.Fatal(err)
   	}
   }

3. Run the program:

   go run .

   The output is:

   nyc	8
   sf	2
   

More examples: simple queries, parameters, and Arrow.

User Guide

This guide describes the Go binding. For SQL syntax and session options, see DataFusion's SQL reference and configuration reference.

Go API	Guide
sql.Open / sql.OpenDB	Sessions, DSNs, and initialization
QueryContext / QueryRowContext	Parameters and type conversion
QueryArrowContext	Arrow batches
RegisterArrowReader / RegisterArrowReaderZeroCopy	Arrow tables and buffer ownership
RegisterFFITableProvider	Foreign table providers
ExecStatements	Multiple setup statements

For driver limits and deployment, see Semantics and Limits and Native Runtime.

Driver and Sessions

By default, connections from one connector share a DataFusion SessionContext. Catalog and configuration changes apply across pooled connections from the same sql.DB.

If each physical connection must have its own session state, use isolated sessions:

db, err := sql.Open("datafusion", "?datafusion.go.shared_session=false")

To select isolated sessions on a connector, use WithSharedSession(false):

connector, err := datafusion.NewConnectorWithInitContext(
	"",
	nil,
	datafusion.WithSharedSession(false),
)

When you close a *sql.Conn, its physical connection returns to the pool. On reuse, shared sessions keep their state. The driver resets isolated sessions and prepares cached statements again. Connection closure does not immediately release registered tables or Arrow batches that callers still hold.

By default, the memory pool for DataFusion queries has no limit. Set a memory budget in the connector initializer:

connector, err := datafusion.NewConnectorWithInitContext("",
	func(ctx context.Context, exec driver.ExecerContext) error {
		_, err := exec.ExecContext(ctx,
			"SET datafusion.runtime.memory_limit = '512M'", nil)
		return err
	},
)
// After checking err:
db := sql.OpenDB(connector)

The initializer applies the budget before queries run and after isolated-session resets. The budget limits only the query-execution memory that DataFusion tracks.

Set different budgets for registered in-memory tables, Arrow batches that callers keep, and Go allocations.

DSNs

The driver accepts these data source name (DSN) forms:

• An empty string, ""
• Options after a question mark, ?<options>
• The URL form, datafusion://
• The URL form with options, datafusion://?<options>.

The driver passes query parameters to DataFusion as session configuration options:

?datafusion.execution.batch_size=8192

Driver-owned options use the datafusion.go. prefix. The driver removes these options before it passes the remaining options to DataFusion:

?datafusion.go.shared_session=false

The driver rejects file paths, hosts, and other URL forms. For file tables, put the path in CREATE EXTERNAL TABLE SQL.

Initialization

If setup SQL is necessary before you use a pooled database, use NewConnector or NewConnectorWithInitContext:

connector, err := datafusion.NewConnectorWithInitContext(
	"",
	func(ctx context.Context, exec driver.ExecerContext) error {
		_, err := exec.ExecContext(ctx, "create view nums as select 1 as n", nil)
		return err
	},
)
if err != nil {
	return err
}
defer connector.Close()

db := sql.OpenDB(connector)
defer db.Close()

In shared-session mode, the initialization callback runs one time for each connector. In isolated-session mode, it runs for each connection and reset.

Multiple Setup Statements

DataFusion prepares one SQL statement at a time.

1. Split migration or setup scripts into individual SQL statements.

2. Execute the statements in order with ExecStatements:

   err := datafusion.ExecStatements(ctx, db, []string{
   	"create view one as select 1 as n",
   	"create view two as select 2 as n",
   })

The helper skips blank statements. If a statement fails, the helper includes its index in the error.

Context Cancellation

The driver connects Go query contexts to native cancellation. It checks for cancellation during query planning, stream creation, and record-batch reads:

ctx, cancel := context.WithTimeout(context.Background(), 2*time.Second)
defer cancel()

rows, err := db.QueryContext(ctx, "select * from some_large_table")

To identify native cancellation errors, use errors.Is(err, datafusion.ErrNativeCancelled).

SQL Parameters

The driver supports DataFusion SQL parameters through database/sql:

row := db.QueryRowContext(ctx, "select ? + 1, ?", int64(41), "x")

The driver accepts these ordinary parameter values:

• A null value, nil
• A Boolean value, bool
• Signed integer types in the int64 range
• Unsigned integer types as DataFusion UInt64
• Floating-point values as float64
• A string, string
• A byte slice, []byte
• A time value, time.Time, as Timestamp[ns]
• A duration, time.Duration, as Duration[ns].

The time.Time conversion keeps loadable IANA locations, such as America/New_York. Fixed-offset, local, and other non-loadable locations bind as UTC. To supply an explicit Arrow time zone, use TimestampWithTimeZone.

The database/sql conversion promotes float32 values to DataFusion Float64. Before native execution, CheckNamedValue rejects other value types.

If type inference is ambiguous or exact Arrow/DataFusion types are necessary, use typed wrappers:

row := db.QueryRowContext(
	ctx,
	"select $1, $2, $3, $4, $5",
	datafusion.DateFromTime(day),
	datafusion.TimeFromTime(clock),
	datafusion.DurationFromTime(2*time.Second),
	datafusion.DecimalString("123.45", 10, 2),
	datafusion.NullOf(datafusion.ParameterInt64),
)

The wrappers support UInt64, Date, Time, Timestamp, Duration, and Decimal. Bare nil binds as an untyped DataFusion null. If a concrete null type is necessary, use NullOf, NullDecimal, or NullTimestamp. For example, select $1 + 1 accepts NullOf(ParameterInt64).

Prepared statements report Stmt.NumInput for these parameter styles:

• Each ? occurrence counts as a separate positional parameter.
• Dollar-numbered parameters, such as $1 and $2, use positional counts.
• Each name, such as $name, counts as one parameter regardless of the number of occurrences.

The driver rejects statements that mix question-mark, dollar-numbered, and named parameter styles during preparation.

Positional statements require positional arguments. Named statements require sql.Named arguments with matching names. The driver rejects missing, extra, or duplicate supplied names before query execution.

Arrow-Native Usage

Query Arrow Batches

For exact schemas or values that database/sql cannot scan, use QueryArrowContext on a *sql.Conn:

conn, err := db.Conn(ctx)
if err != nil {
	return err
}
defer conn.Close()

reader, err := datafusion.QueryArrowContext(ctx, conn, "select $1", int64(42))
if err != nil {
	return err
}
defer reader.Close()

for {
	record, err := reader.Read()
	if err == io.EOF {
		break
	}
	if err != nil {
		return err
	}
	// Use record.
	record.Release()
}

Call Release on each record after use. Call Close on the reader to release its native stream resources. The finalizer does not guarantee immediate cleanup.

Register Arrow Tables

To register an Arrow record reader as an in-memory DataFusion table, use RegisterArrowReader:

rdr, err := array.NewRecordReader(schema, []arrow.RecordBatch{batch})
if err != nil {
	return err
}
defer rdr.Release()

if err := datafusion.RegisterArrowReader(ctx, conn, "events", rdr); err != nil {
	return err
}

RegisterArrowReader consumes the remaining batches from the reader. It transfers each batch as an Arrow IPC stream, then atomically registers the decoded Rust-owned batches. Only one encoded batch is staged at a time; the final in-memory table still retains all decoded batches. The copy lets the table outlive the cgo call. Ordinary Go Arrow arrays can contain Go-owned buffers that native code must not keep after that call.

RegisterArrowReaderZeroCopy exports the reader through the Arrow C Stream Interface without an IPC copy. Each exported buffer must stay valid for native use until table removal or closure of its session or connector.

Before you use RegisterArrowReaderZeroCopy, make sure that native code can keep all exported buffers for this period. Examples include buffers from Arrow Go's memory/mallocator package and other C or foreign allocators.

For Go-allocated buffers, obey the cgo pointer rules during the full period of native use. Keep these buffers valid for this period. If you cannot satisfy these requirements, use RegisterArrowReader.

Register Foreign FFI Table Providers

To register a datafusion-ffi FFI_TableProvider from a foreign library, use RegisterFFITableProvider:

// providerPtr is an *FFI_TableProvider handed to you by the producing library,
// and providerVersion is the datafusion version that library was built against.
table, err := datafusion.RegisterFFITableProvider(ctx, conn, "t", providerPtr, providerVersion)
if err != nil {
	return err
}
defer table.Deregister(ctx)

rows, err := db.QueryContext(ctx, `SELECT ... FROM t WHERE ...`)

Version and ownership

1. Get providerVersion from the library that supplies the provider.
2. Make sure that providerVersion equals this package's DataFusionVersion.

This release uses DataFusion 55.1.0. Rebuild foreign providers against datafusion-ffi 55.1.0 when upgrading from the 55.0.0-based driver. The Go-facing registration API is unchanged.

Do not substitute this package's DataFusionVersion for the version from the foreign library. The driver compares versions before it dereferences the provider pointer. A mismatch returns an error.

Exact version equality is stricter than the major-version ABI contract of datafusion-ffi.

The provider pointer must refer to memory that the foreign C or Rust library owns. Do not supply a pointer to Go heap memory. Native code retains cloned callback pointers after registration returns.

Registration clones the provider and increases its reference count. You retain ownership of the original pointer. After registration returns, you can free the original pointer through its library.

Library lifetime

The foreign library must stay loaded while registrations or dependent foreign objects exist. These objects include views, query plans, readers, and returned Arrow batches. They can invoke foreign callbacks after deregistration.

Before you unload the library, complete these steps:

1. Stop new queries.
2. Deregister the tables.
3. Remove dependent views.
4. Close the readers.
5. Release the Arrow batches.
6. Free the original provider through its library.
7. Release all other dependent foreign objects.

Deregistration

RegisterFFITableProvider returns a *RegisteredTable handle. While its *sql.Conn is open, call Deregister to remove the catalog entry.

In shared and isolated modes, connection closure normally returns the physical connection to the pool. It does not guarantee table release. Queries can retain foreign objects even after the connector closes. The driver does not deregister a table when you discard its registration handle.

Direct Sessions and Go Extensions

Applications that work directly with Arrow can use NewSession without a SQL connection pool. It uses the same query execution and cancellation implementation:

session, err := datafusion.NewSession("")
if err != nil {
    return err
}
defer session.Close()
reader, err := session.QueryArrowContext(ctx, "SELECT $1", int64(42))
// Check err, close reader, and release every returned batch as usual.

Session also exposes ExecContext, RegisterArrowReader, DeregisterTable, RegisterTableProvider, RegisterScalarFunction, and RegisterCatalog. The last three registration functions also accept *sql.Conn as package-level helpers, so existing database/sql applications can use the extensions directly.

• A TableProvider supplies its schema and a fresh Arrow reader per scan. Implement PushdownTableProvider to receive ScanOptions: projection must be exact, while filters are advisory and DataFusion rechecks them. A nil projection means all columns; an empty projection means zero columns with preserved row counts. A limit of -1 means no safe hint. Implement WritableTableProvider to consume INSERT input and explicitly accept or reject each InsertOp.
• A ScalarFunction declares argument types, return type, volatility and a context-aware batch evaluator. Its zero-value volatility is Volatile. Arguments are borrowed; the returned array transfers ownership to the engine. Row count is passed separately for zero-argument functions. Signatures match nested field metadata exactly. Use immutable/stable only when your function satisfies those optimizer assumptions.
• A CatalogProvider resolves (schema, table) names on demand. Each query resolves each referenced table once and retains that snapshot for planning. Later queries resolve afresh. Existing resolver-only catalogs keep this behavior. Optionally implement DiscoverableCatalogProvider with context-aware SchemaNames and TableNames methods to expose remote schemas/tables through information_schema and SHOW TABLES / SHOW COLUMNS. Names are literal identifiers, not SQL fragments. Nil lists are empty; duplicates are ignored; empty names and names containing NUL are rejected. Discovery does not provide transactional snapshots across a changing remote catalog.

Metadata queries refresh listings for each query. Schema listings do not list individual tables; table listings do not resolve providers or scan data. Column metadata resolves listed providers lazily, once per query, without calling Scan. SQL metadata filters are applied by DataFusion and are not pushed into these listing callbacks, so a column query may resolve tables beyond its filter. Ordinary data queries resolve only referenced tables and do not call discovery. Return an empty list for a missing schema, and (nil, nil) when a listed table has disappeared before resolution. Discovery errors propagate to the query.

ProjectReader(reader, indices) and LimitReader(reader, limit) help implement ScanWithOptions without copying Arrow buffers. Projection preserves field, schema and batch metadata, accepts duplicate indices, and distinguishes nil (all columns) from empty (zero columns). Limit accepts -1 for unlimited rows and 0 for no rows. Both helpers take ownership of the input only on success; release the input yourself on error. They do not evaluate filters. Apply a limit hint only after any source-side filtering; DataFusion still rechecks advisory filters.

Provider authors can call providertest.Run(t, factory) from github.com/datafusion-contrib/datafusion-go/providertest with a small, finite fixture. It checks repeated and concurrent scans, schemas, retained-batch ownership, and optional projection/limit behavior. It runs without a native library, including with CGO_ENABLED=0; add your own storage/value/write tests.

Providers and functions must support concurrent calls and honor their supplied contexts. Reader close and query cancellation cancel outstanding callback I/O; callbacks that ignore cancellation cannot be forcibly stopped. Reader release must complete promptly. A provider must not synchronously execute a mutating statement on its own session from a callback, because that statement may already hold the session's mutation lock.

Go-produced batches use the Arrow C Data interface with a single copy into C-owned buffers for supported layouts, so ordinary Go Arrow allocations remain safe. Sliced arrays are compacted before export. Dictionary, extension, view and other unsupported fast-path layouts retain the per-batch IPC compatibility path. Small multi-column batches also keep IPC to avoid per-array allocation overhead. Native UDF arguments cross through C Data without serialization. Providers remain lazy and result streams remain incremental. Native input to writable providers uses the Arrow C stream interface; that input reader is borrowed for the callback and released by the bridge. Retain batches explicitly when keeping them after a write returns. Check the reader's error and finish commit or rollback before returning: the driver does not retry failed writes or provide transactions for a provider.

Registration rejects existing names. Closing a session or deregistering a table does not invalidate active results. The callback implementation retains its owners until outstanding work finishes. File scans already supported by native DataFusion remain native; no cloud SDK or Iceberg dependency is added to the Go module. See the Go extensions example.

Opt-in Execution Tuning

NewSession and NewConnectorWithInitContext accept:

• WithPreparedStatementCache(true): retain parsed SQL syntax for prepared statements. Every execution still resolves current catalogs and creates fresh logical/physical plans. Changed parser settings bypass the retained syntax.
• WithRuntimeWorkers(n): use n Tokio worker threads. Zero preserves the default.
• WithSharedRuntime(true): share Tokio workers with other opting-in connectors configured with the same worker count. Catalogs, configuration and DataFusion memory pools remain separate. The runtime is released after its last user.

The corresponding DSN keys are datafusion.go.cache_statements, datafusion.go.runtime_workers (positive integer), and datafusion.go.shared_runtime. Explicit tuning options override their DSN keys, including false and zero. Caching and sharing default to false. Batch size, query memory limits and target partitions remain ordinary DataFusion settings.

Type Conversion

The database/sql row conversion supports these types:

Arrow type family	Go value
Null	nil
Bool	bool
Signed integers	int64
Unsigned integers	int64 when in range
Float16/Float32/Float64	float64
Utf8/LargeUtf8/StringView	string
Binary/LargeBinary/FixedSizeBinary/BinaryView	[]byte
Date/Time/Timestamp	time.Time
Duration	int64 nanoseconds
Decimal	string
Intervals	string

Where the Arrow schema provides precise information, the driver exposes column metadata through database/sql:

• Nullable columns use typed sql.Null* scan types where practical.
• Fixed-size binary columns report length.
• Decimal columns report precision and scale.
• Temporal and interval database type names include their Arrow unit or interval subtype.

Variable-width string and binary columns do not report declared lengths. The Arrow result schema does not preserve SQL declarations such as VARCHAR(32).

The driver converts time-only values to UTC time.Time values on the Unix epoch date. It converts durations to int64 nanoseconds because database/sql/driver.Value does not accept time.Duration. Interval strings keep the month, day, millisecond, and nanosecond components.

When schema information is available, row conversion rejects lists, structs, maps, unions, dictionaries, extensions, and run-end encoded values. For these types or exact batch data, use QueryArrowContext.

Semantics and Limits

• PrepareContext validates SQL syntax with the DataFusion parser. A prepared query must contain exactly one SQL statement.
• The driver does not support multiple result sets. Rows.NextResultSet reports no additional result sets.
• The connector serializes non-query statements across ExecContext, QueryContext, and QueryArrowContext. Concurrent queries and DDL can still have ordering effects.
• Connections implement driver.Validator. The driver reports closed connections as invalid before they return to the pool.
• Native errors carry machine-readable kinds across the C ABI. The driver exposes these kinds on *datafusion.Error.NativeKind.
• errors.Is matches the native sentinels ErrNativeCancelled, ErrNativeInvalidArgument, ErrNativeFailure, and ErrNativePanic.
• RowsAffected returns 0 by default. If DataFusion emits a single integer output column named count, rows_affected, or rowsaffected, the driver reports its sum.
• LastInsertId returns 0, nil. DataFusion does not expose insert IDs through this driver.
• The driver reuses statement handles for db.Prepare and conn.PrepareContext. DataFusion plans and executes each run. The driver does not cache physical plans. Parsed SQL syntax can be cached with the opt-in setting described above.
• Close is idempotent for connectors, connections, statements, rows, and Arrow readers.
• Transactions return explicit unsupported errors. For an already-canceled context, BeginTx returns the context error.

SQL executes with the filesystem and network permissions of the host process. Isolated sessions have independent catalogs. They do not sandbox hostile SQL or native FFI providers.

Run untrusted workloads in a different process with restricted operating-system permissions and resource limits.

Native Runtime

Default builds use cgo but do not link DataFusion at Go link time. At runtime, the driver searches for a shared libdatafusion_go library in this order:

1. The path in DATAFUSION_GO_LIBRARY, if set
2. The source-checkout directory internal/native/lib/<goos>-<goarch>/
3. The user cache, with automatic download from the GitHub release for the installed module version.

These environment variables control library selection and downloads:

Variable	Effect
DATAFUSION_GO_LIBRARY	Selects an explicit absolute path to a trusted shared library.
DATAFUSION_GO_NO_DOWNLOAD=1	Disables automatic release-asset downloads.
DATAFUSION_GO_DOWNLOAD_BASE	Selects an alternative HTTPS base URL for release downloads. Redirects must also use HTTPS.

Automatic source and cache resolution checks ownership and write permissions for the library and its ancestor directories. Keep these directories private to the service account or system administrators. An explicit library path is trusted configuration and bypasses automatic permission checks.

For setuid, setgid, or Linux file-capability processes, use datafusion_use_bundled or datafusion_use_source. These processes must link the library at build time. The driver disables runtime resolution for them.

Before you run driver examples or Go tests from a source checkout, run make bundle or make test. Each target builds the Rust shim and copies the native archive and shared library into internal/native/lib/<goos>-<goarch>/.

The package also has these link modes:

Build tag	Library source
datafusion_use_bundled	The static archive in internal/native/lib/<goos>-<goarch>/.
datafusion_use_source	The static archive from the Rust release build in the source checkout.
datafusion_use_static_lib	The same static archive as datafusion_use_source.
datafusion_use_lib	A system library that the linker finds through -ldatafusion_go.

Select one link mode with go build -tags=<build-tag>. For datafusion_use_lib, use CGO_LDFLAGS to add the library directory with -L. Configure the runtime loader to find the shared library on your operating system.

For native-build setup and tests, see CONTRIBUTING.md.

Troubleshooting

Problem	Action
datafusion-go requires cgo	Install a C toolchain. Set CGO_ENABLED=1.
Native library not found	Set DATAFUSION_GO_LIBRARY to a local library. For a source checkout, run make bundle. See Native Runtime.
Local checkout tests fail before the driver opens	Run make test from the repository root to build the native library before the Go tests.
DSN rejected	Use an empty DSN or session options.
database/sql cannot scan a result column	Use QueryArrowContext for complex Arrow values.
Windows build or test failures	Use a MinGW/GNU C toolchain for the x86_64-pc-windows-gnu Rust ABI.

Developing

For setup, test modes, version changes, and the release process, see CONTRIBUTING.md.

Before you submit code changes, run these commands from the repository root:

make lint
make test
make test.source
make rust.test

For questions and ordinary bug reports, use GitHub Issues. Code and documentation contributions are welcome through pull requests.

Versioning

Release tags encode the bundled DataFusion version as v<major>.<encoded-datafusion-version>.<patch>: DataFusion 53.1.0 encodes as 530100, so v0.530100.1 bundles DataFusion 53.1.0.

versions.toml contains the release metadata. For version changes and the release workflow, see CONTRIBUTING.md.

License

Licensed under the Apache License, Version 2.0.