Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
65 commits
Select commit Hold shift + click to select a range
6f9fd56
Integrated rust into the project via maturin
alxmrs Jul 15, 2025
fd6fe0e
Properly set up the rust project (with dummy table provide, can compi…
alxmrs Jul 16, 2025
4dcaa52
Phase 1 and 2 implementation of zarr store plan!
alxmrs Jul 16, 2025
3aef11a
Next step: only processes a single array in the zarr group as record …
alxmrs Jul 16, 2025
ff75797
Attempting to minimize copying.
alxmrs Jul 16, 2025
3881787
Had Claude refactor the code to introduce generics, in order to reduc…
alxmrs Jul 16, 2025
cbfc0e6
Better explained requirements for tables, claude wrote tests.
alxmrs Jul 16, 2025
97e7a3c
Applied automatic Zarr fixes from the compiler.
alxmrs Jul 16, 2025
baf4f3b
Core data var to table logic implemented, according to Claude. Needs …
alxmrs Jul 16, 2025
09251fb
Adding test_data to gitignore.
alxmrs Jul 16, 2025
df8debd
Added example and test scripts as sanity checks. might delete later.
alxmrs Jul 16, 2025
b55db35
Adding predicate pushdown logic, but I think it may need revision, wi…
alxmrs Jul 16, 2025
b6efef2
Generated integration tests to test sql functionality.
alxmrs Jul 17, 2025
9e6a0b0
Initial `from_zarr` implementation, which calls the rust TableProvide…
alxmrs Jul 17, 2025
1937393
First stage of multi-stage test refactoring from Claude: creating a t…
alxmrs Jul 17, 2025
6457dd8
Made some minor refactors.
alxmrs Jul 17, 2025
0777064
Implemented strategy 2 and 3.
alxmrs Jul 17, 2025
7a5700a
Import opt.
alxmrs Jul 17, 2025
624c3af
Tests for semantic equivalence to xarray operations.
alxmrs Jul 17, 2025
7c328e0
Moved test data creation scripts into examples.
alxmrs Jul 17, 2025
028e69b
Removing examples for now, because they add too much noise to the rev…
alxmrs Jul 17, 2025
b30bde6
Reformatted the python code with pyink.
alxmrs Jul 17, 2025
fff4203
Import optimization
alxmrs Jul 17, 2025
0dd67fd
Revert to simpler tests! These tests have gotten too complicated and …
alxmrs Jul 17, 2025
cd59d51
WIP, manually refactoring tests to test both from_zarr and from_dataset
alxmrs Jul 17, 2025
27baf67
I think I found an actual error with the Zarr logic.
alxmrs Jul 17, 2025
38f2796
Fixed signficant correctness bugs in the tabel provider impl related …
alxmrs Jul 17, 2025
fdf19bf
Test is incorrect; updating.
alxmrs Jul 17, 2025
8476f5f
Table provider passes tests, but it looks like it needs to be simplif…
alxmrs Jul 17, 2025
76196d6
Continuing my zarr vs dataset refactor.
alxmrs Jul 17, 2025
c2d95c2
Pinning down the first of a two part problem.
alxmrs Jul 17, 2025
acbb2bb
Added skips to tests to look into later.
alxmrs Jul 17, 2025
6e02906
Reformatted tests.
alxmrs Jul 17, 2025
c85d382
Attempt to fix minor warnings.
alxmrs Jul 18, 2025
42637b3
Revised manually! Now, they all test both from_dataset and from_zarr.…
alxmrs Jul 22, 2025
c059ee6
I think a more sane way to represent stations (station id is now a di…
alxmrs Jul 22, 2025
755190a
Manually revised what Claude wrote on the airplane.
alxmrs Jul 22, 2025
8b6beb5
Apply clippy suggestions and rustfmt to #69 (#70)
kylebarron Jul 24, 2025
6657a45
Skipping station tests -- we want to rethink this type of data in the…
alxmrs Jul 25, 2025
374f7b0
Claude implementation to get nearly all tests to pass to get feature …
alxmrs Jul 25, 2025
b196a9b
Supports Zarr strings as arrow strings. Also, handles empty zarr/data…
alxmrs Jul 25, 2025
40e3e5f
Fix to get tests to pass ("coordinate"-only Zarr arrays). This requir…
alxmrs Jul 25, 2025
4562c19
Omitting stale performance tests, which break maturin CI on windows.
alxmrs Jul 25, 2025
7b14abc
Called rust fmt (mimicking CI).
alxmrs Jul 25, 2025
2fe9be4
Passing rust lint checks and tests.
alxmrs Jul 25, 2025
e289bec
Reformatted python with pyink.
alxmrs Jul 25, 2025
4bded87
Renamed CI for clarity. Also, added a dev build step for python tests…
alxmrs Jul 25, 2025
e706eb6
Adding maturin as a dev dep
alxmrs Jul 25, 2025
ffc6ebf
Remove the s390x (IBM Mainframe) compiler target.
alxmrs Jul 25, 2025
7a3ca51
`from_zarr` is now only supported for Python 3.11+
alxmrs Jul 27, 2025
43e5046
pyink fmt
alxmrs Jul 27, 2025
92203fe
Try to fix linker problem for the maturin arm build.
alxmrs Jul 27, 2025
7e20540
Put in the wrong file
alxmrs Jul 27, 2025
29d132e
Upgrade ubuntu version to get gcc 14
alxmrs Jul 27, 2025
986b498
Attempting to fix IBM target via: https://github.com/rust-lang/stacke…
alxmrs Jul 27, 2025
ea6870a
Add test to create test datasets for rust tests.
alxmrs Jul 29, 2025
b7a8494
Adding integration tests from Claude.
alxmrs Jul 29, 2025
69f065a
Adding build step to run integration tests
alxmrs Jul 29, 2025
243015b
fmt int tests
alxmrs Jul 29, 2025
14b4ea2
Attempting to take a much simpler approach via an adapter.
alxmrs Sep 11, 2025
5ac86e4
New, simpler approach that makes use of `arrow-zarr`.
alxmrs Sep 12, 2025
82e54cc
Current WIP, got back to where I started :/
alxmrs Sep 26, 2025
7f24256
Fix: don't pass chunks.
alxmrs Sep 26, 2025
1d58461
Made a really important fix! The python datafusion version needs to m…
alxmrs Sep 27, 2025
76102d9
Updated xarray-sql branch of arrow-zarr.
alxmrs Sep 27, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
182 changes: 182 additions & 0 deletions .github/workflows/ci-maturin.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,182 @@
# This file is autogenerated by maturin v1.9.1
# To update, run
#
# maturin generate-ci github
#
name: ci-maturin

on:
push:
branches:
- main
tags:
- '*'
pull_request:
workflow_dispatch:

permissions:
contents: read

jobs:
linux:
runs-on: ${{ matrix.platform.runner }}
strategy:
matrix:
platform:
- runner: ubuntu-24.04
target: x86_64
- runner: ubuntu-24.04
target: x86
- runner: ubuntu-24.04
target: aarch64
- runner: ubuntu-24.04
target: armv7
- runner: ubuntu-24.04
target: s390x
- runner: ubuntu-24.04
target: ppc64le
env:
CFLAGS_s390x_unknown_linux_gnu: -march=z10
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: 3.x
- name: Build wheels
uses: PyO3/maturin-action@v1
with:
target: ${{ matrix.platform.target }}
args: --release --out dist --find-interpreter
sccache: ${{ !startsWith(github.ref, 'refs/tags/') }}
manylinux: auto
- name: Upload wheels
uses: actions/upload-artifact@v4
with:
name: wheels-linux-${{ matrix.platform.target }}
path: dist

musllinux:
runs-on: ${{ matrix.platform.runner }}
strategy:
matrix:
platform:
- runner: ubuntu-22.04
target: x86_64
- runner: ubuntu-22.04
target: x86
- runner: ubuntu-22.04
target: aarch64
- runner: ubuntu-22.04
target: armv7
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: 3.x
- name: Build wheels
uses: PyO3/maturin-action@v1
with:
target: ${{ matrix.platform.target }}
args: --release --out dist --find-interpreter
sccache: ${{ !startsWith(github.ref, 'refs/tags/') }}
manylinux: musllinux_1_2
- name: Upload wheels
uses: actions/upload-artifact@v4
with:
name: wheels-musllinux-${{ matrix.platform.target }}
path: dist

windows:
runs-on: ${{ matrix.platform.runner }}
strategy:
matrix:
platform:
- runner: windows-latest
target: x64
- runner: windows-latest
target: x86
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: 3.x
architecture: ${{ matrix.platform.target }}
- name: Build wheels
uses: PyO3/maturin-action@v1
with:
target: ${{ matrix.platform.target }}
args: --release --out dist --find-interpreter
sccache: ${{ !startsWith(github.ref, 'refs/tags/') }}
- name: Upload wheels
uses: actions/upload-artifact@v4
with:
name: wheels-windows-${{ matrix.platform.target }}
path: dist

macos:
runs-on: ${{ matrix.platform.runner }}
strategy:
matrix:
platform:
- runner: macos-13
target: x86_64
- runner: macos-14
target: aarch64
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: 3.x
- name: Build wheels
uses: PyO3/maturin-action@v1
with:
target: ${{ matrix.platform.target }}
args: --release --out dist --find-interpreter
sccache: ${{ !startsWith(github.ref, 'refs/tags/') }}
- name: Upload wheels
uses: actions/upload-artifact@v4
with:
name: wheels-macos-${{ matrix.platform.target }}
path: dist

sdist:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Build sdist
uses: PyO3/maturin-action@v1
with:
command: sdist
args: --out dist
- name: Upload sdist
uses: actions/upload-artifact@v4
with:
name: wheels-sdist
path: dist

release:
name: Release
runs-on: ubuntu-latest
if: ${{ startsWith(github.ref, 'refs/tags/') || github.event_name == 'workflow_dispatch' }}
needs: [linux, musllinux, windows, macos, sdist]
permissions:
# Use to sign the release artifacts
id-token: write
# Used to upload release artifacts
contents: write
# Used to generate artifact attestation
attestations: write
steps:
- uses: actions/download-artifact@v4
- name: Generate artifact attestation
uses: actions/attest-build-provenance@v2
with:
subject-path: 'wheels-*/*'
- name: Publish to PyPI
if: ${{ startsWith(github.ref, 'refs/tags/') }}
uses: PyO3/maturin-action@v1
env:
MATURIN_PYPI_TOKEN: ${{ secrets.PYPI_API_TOKEN }}
with:
command: upload
args: --non-interactive --skip-existing wheels-*/*
4 changes: 3 additions & 1 deletion .github/workflows/ci.yml → .github/workflows/ci-python.yml
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@
# See the License for the specific language governing permissions and
# limitations under the License.
# ==============================================================================
name: ci
name: ci-python

on:
# Triggers the workflow on push or pull request events but only for the main branch
Expand Down Expand Up @@ -47,5 +47,7 @@ jobs:
python-version: ${{ matrix.python-version }}
- name: Install xarray_sql
run: uv sync --dev
- name: Prepare bindings for tests
run: maturin develop
- name: Run unit tests
run: uv run pytest
52 changes: 52 additions & 0 deletions .github/workflows/ci-rust.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,52 @@
name: ci-rust

on:
push:
branches:
- main
pull_request:

concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true

jobs:
lint-test-all-features:
name: Lint and Test
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
with:
submodules: "recursive"
- name: Install uv
uses: astral-sh/setup-uv@v5
- name: Install Rust
uses: dtolnay/rust-toolchain@stable
with:
components: clippy
- uses: Swatinem/rust-cache@v2
- name: Clippy
run: cargo clippy --all-features --tests -- -D warnings
- name: Check
run: cargo check --all-features
- name: create test data
run: uv run examples/create_test_datasets.py
- name: Test
run: cargo test --all-features
- name: Integration test
run: cargo run --example sql_integration_tests

fmt:
name: Fmt
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
with:
submodules: "recursive"
- name: Install Rust
uses: dtolnay/rust-toolchain@master
with:
toolchain: nightly-2025-05-14
components: rustfmt
- name: Fmt
run: cargo +nightly-2025-05-14 fmt -- --check --unstable-features --config imports_granularity=Module,group_imports=StdExternalCrate
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@
# See the License for the specific language governing permissions and
# limitations under the License.
# ==============================================================================
name: lint
name: lint-python

on:
# Triggers the workflow on push or pull request events but only for the main branch
Expand Down
4 changes: 3 additions & 1 deletion .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -5,4 +5,6 @@ dist
__pycache__
.pytest_cache
.venv
.idea
.idea/
target/
test_data/
Loading
Loading