Skip to content

kernel: distribute darwin/arm64 lib as nested per-platform module (go get, no build step) #440

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
msrathore-db wants to merge 3 commits into
base: main
Choose a base branch
from
Open

kernel: distribute darwin/arm64 lib as nested per-platform module (go get, no build step) #440

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
a9c3e8a
kernel: distribute darwin/arm64 lib as nested per-platform module
msrathore-db Aug 12, 2026
a11b8e2
ci: sync committed kernel C-ABI libs from kernel release
msrathore-db Aug 12, 2026
d743db3
ci: drop pull-model sync workflow in favor of kernel push
msrathore-db Aug 12, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 10 additions & 2 deletions .gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -67,7 +67,15 @@ _tmp*
__debug_bin
.DS_Store

# Kernel (SEA) backend build artifacts — produced by `make kernel-lib`, never committed.
# Kernel (SEA) backend build artifacts.
#
# The committed distribution model (see README) commits the small
# platform-independent C header (include/) and each platform's prebuilt archive
# under kernellib/<platform>/ (nested modules), so `go get` needs no build step.
# Those committed paths are intentionally NOT ignored.
#
# The paths below remain ignored: they are scratch dirs still used by
# `make kernel-lib` for platforms not yet committed (linux/windows) and for
# source builds — never committed.
/build/kernel-src/
/internal/backend/kernel/lib/
/internal/backend/kernel/include/
39 changes: 39 additions & 0 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@ standard `database/sql` interface.
## Contents

- [Quick start](#quick-start)
- [Cloning the repository](#cloning-the-repository)
- [Choosing a backend (Thrift vs SEA/kernel)](#choosing-a-backend-thrift-vs-seakernel)
- [Building](#building)
- [Connecting](#connecting)
Expand Down Expand Up @@ -46,6 +47,44 @@ defer rows.Close()
See [`doc.go`](./doc.go) for full package documentation or the Databricks documentation
for the [SQL Driver for Go](https://docs.databricks.com/dev-tools/go-sql-driver.html).

> **Using the driver in your own project?** You never clone this repository — you
> add it with `go get github.com/databricks/databricks-sql-go` and `go build`.
> `go get` fetches per-version module archives, not git history, and for a
> default Thrift build it pulls **no** kernel binaries at all. The guidance below
> is only for people who `git clone` this repo directly (contributors / CI).

## Cloning the repository

This repo commits a small number of **prebuilt kernel binaries** (per-platform
`libdatabricks_sql_kernel.a`, ~62 MB each) so that the SEA/kernel backend works
straight from `go get` with **no build step** (see
[SEA/kernel](#seakernel--cgo--a-linked-rust-static-library)). Committed binaries
cannot be delta-compressed by git, so a *full* clone accumulates their whole
history over releases.

**If you clone this repo directly, use a partial clone** so you download only the
binary versions you actually check out, not the entire history:

```bash
git clone --filter=blob:none https://github.com/databricks/databricks-sql-go
```

`--filter=blob:none` fetches commits and trees immediately and pulls file blobs
lazily, only when a checkout needs them. This keeps `.git` small and — unlike a
naive `git clone` — it does **not** grow with the number of releases (only your
current checkout's blobs are fetched). GitHub serves this by default. To also
avoid materializing other platforms' archives in your working tree, add
`--sparse` and select the paths you need:

```bash
git clone --filter=blob:none --sparse https://github.com/databricks/databricks-sql-go
cd databricks-sql-go
git sparse-checkout set --no-cone '/*' '!/internal/backend/kernel/kernellib' \
'internal/backend/kernel/kernellib/darwin_arm64' # keep only your platform
```

CI checkouts in this repo use `--filter=blob:none` for the same reason.

## Choosing a backend (Thrift vs SEA/kernel)

The driver has **two execution backends**, selected once per connection:
Expand Down
9 changes: 9 additions & 0 deletions go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -52,3 +52,12 @@ require (
github.com/rs/zerolog v1.28.0
golang.org/x/sys v0.45.0 // indirect
)

// Nested per-platform kernel library modules. Each carries one platform's
// prebuilt kernel static archive + its cgo link directive, so a build downloads
// only the archive for the platform it targets (and nothing at all for a
// pure-Go Thrift build). The replace pins them to the in-tree directories; when
// published, the require versions are what a `go get` consumer resolves.
require github.com/databricks/databricks-sql-go/internal/backend/kernel/kernellib/darwin_arm64 v0.0.0

@peco-review-bot peco-review-bot Bot Aug 12, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Medium — The nested module is wired with require .../kernellib/darwin_arm64 v0.0.0 + a local replace. This works for in-tree builds, but note two things that undercut the PR's "works straight from go get" goal for external consumers:

  1. replace is not transitive. A downstream project that does go get github.com/databricks/databricks-sql-go ignores this repo's replace directive entirely (replace is honored only in the main module). It sees only the bare require .../darwin_arm64 v0.0.0.

  2. Module-graph resolution is build-tag-independent. MVS must load the go.mod of every required module to build the graph, even for a pure-Thrift (CGO_ENABLED=0, no tag) build that never compiles a file from the nested module. Because v0.0.0 is not a published/tagged version of the nested module, that resolution would fail for all consumers — not just kernel builds — with an "unknown revision" error, once a release of this repo is cut carrying this go.mod.

The PR description acknowledges this ("when published, the require versions are what a go get consumer resolves"), so this is a known follow-up rather than a defect in the in-tree workflow. Flagging so the release that publishes this is gated on: tagging the nested module at a real version and updating the require to match. Until then, external go get of a tagged release would break even for Thrift-only users.

@peco-review-bot peco-review-bot Bot Aug 12, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟠 High — The nested module is pulled in with an unconditional require ... v0.0.0 plus a replace => ./internal/backend/kernel/kernellib/darwin_arm64. This resolves correctly for in-repo builds (make test, go build ./..., CI), because replace short-circuits the placeholder v0.0.0.

But replace directives are ignored in any module other than the main module (Go modules reference). So for an external consumer that does go get github.com/databricks/databricks-sql-go, the replace does not apply and Go must resolve .../kernellib/darwin_arm64@v0.0.0 from the proxy/VCS. Since the top-level require is unconditional, this module's go.mod is loaded for every consumer build graph — including a pure-Thrift, CGO_ENABLED=0 build that never compiles a kernellib file. Unless the tag internal/backend/kernel/kernellib/darwin_arm64/v0.0.0 actually exists in the published repo, go get/go build fails at module resolution (unknown revision v0.0.0) for all consumers, not just kernel users.

This directly contradicts the README section added in this PR ("works straight from go get with no build step") and the PR's headline claim. The PR text acknowledges "when published, the require versions are what a go get consumer resolves" — so the gap is known, but as it stands the merged+tagged state would break the default build for downstream consumers. Please confirm the publish/tagging plan makes v0.0.0 (or the eventual pinned version) resolvable before the README advertises frictionless go get, or gate the require so a Thrift build doesn't force resolution.

@peco-review-bot peco-review-bot Bot Aug 12, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Medium — The nested-module wiring here works only for in-tree builds — which is all the PR's verification actually exercised (go build ./... run from within this repo, where the replace applies). It does not deliver the headline "works straight from go get, no build step" for external consumers:

  • replace directives are ignored for dependencies. When someone adds databricks-sql-go as a dependency, only their main module's replace directives take effect; the replace on line 62 of this repo's go.mod is dropped. So a downstream go build -tags databricks_kernel for darwin/arm64 will try to resolve .../kernellib/darwin_arm64 v0.0.0 from the module proxy, not from ./internal/....
  • v0.0.0 is not a resolvable version. There is no internal/backend/kernel/kernellib/darwin_arm64/v0.0.0 tag, so that resolution fails for a consumer. go-duckdb's model (cited in the PR) requires the nested modules to be published as real tagged versions that the parent requires directly — not a v0.0.0 placeholder held together by an in-tree replace.

Net effect: the go get path the PR is built around is currently only proven for builds run inside this checkout. Recommend either (a) tempering the README/PR claim to "builds from a repo checkout" until the nested modules are published and required at real versions, or (b) documenting the publish+version-bump step as a hard prerequisite before the kernel backend is advertised as go-get-installable. The Thrift-build verification is unaffected (the import is build-tag-excluded), so this only concerns the kernel-tag darwin/arm64 consumer path.


replace github.com/databricks/databricks-sql-go/internal/backend/kernel/kernellib/darwin_arm64 => ./internal/backend/kernel/kernellib/darwin_arm64
26 changes: 11 additions & 15 deletions internal/backend/kernel/cgo_darwin.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,19 +2,15 @@

package kernel

// Link flags for darwin/arm64. NOTE: this platform is not yet exercised in CI
// (M0 is linux/amd64); the flags below are the intended shape but must be
// validated on a mac before darwin is enabled.
// darwin/arm64 link wiring. The kernel static archive and its `#cgo LDFLAGS`
// live in a NESTED per-platform module
// (internal/backend/kernel/kernellib/darwin_arm64) so that a `go get`/`go build`
// only downloads the darwin archive when actually building for darwin/arm64 with
// the databricks_kernel tag — see that module's link.go and the repo README.
//
// Two darwin-specific differences from linux:
// - Apple's ld64 does NOT accept the GNU `-l:<file>.a` extension, so the
// archive is passed as a positional input by absolute ${SRCDIR} path
// instead. Since only the .a is placed under lib/darwin_arm64 (see
// kernel-lib.sh), there is no .so to accidentally prefer.
// - -lc++ (not -lstdc++) is the macOS C++ runtime; @loader_path keeps any
// dynamic reference resolvable relative to the built binary.

/*
#cgo LDFLAGS: ${SRCDIR}/lib/darwin_arm64/libdatabricks_sql_kernel.a -lc++ -lm -Wl,-rpath,@loader_path
*/
import "C"
// This file's sole job is to import that module for its link side-effect: cgo
// collects `#cgo LDFLAGS` from every imported cgo package at final link time, so
// the blank import below is what pulls libdatabricks_sql_kernel.a into the
// binary. It carries the same build constraint as the nested link.go so the two
// are always selected (or excluded) together.
import _ "github.com/databricks/databricks-sql-go/internal/backend/kernel/kernellib/darwin_arm64"
Loading
Loading