Skip to content

feat(io): allow a custom parquet writer for points elements - #1201

Closed
cornhundred wants to merge 2 commits into
scverse:mainfrom
cornhundred:feat/regular-grid-parquet-profile
Closed

feat(io): allow a custom parquet writer for points elements#1201
cornhundred wants to merge 2 commits into
scverse:mainfrom
cornhundred:feat/regular-grid-parquet-profile

Conversation

@cornhundred

Copy link
Copy Markdown

Adds an optional points_writer hook to SpatialData.write() and write_points(), letting a caller control the points.parquet layout (row-group boundaries, compression, number of files) while SpatialData still writes all element metadata. Default behaviour is unchanged when the hook is omitted.

This follows the existing pass-through option pattern on write() (shapes_geometry_encoding, raster_compressor,
convert_table_strings_to_categoricals) rather than introducing a new concept, and it is deliberately layout-generic: nothing about it is specific to any tiling scheme or viewer.

Motivation: producing spatially tiled row groups otherwise requires writing the points parquet once with the default writer and then rewriting it. On an 8M transcript Xenium dataset that measured 13.5s vs 8.5s, and it writes ~250MB of transient output that is immediately discarded -- a gap that widens on 300M-transcript panels.

Note the hook receives the dataframe after transformations are stripped from attrs (as the default writer requires), so a writer needing the coordinate transform must obtain it from the element beforehand.

Nicolas Fernandez and others added 2 commits September 6, 2026 14:10
Adds an optional points_writer hook to SpatialData.write() and write_points(),
letting a caller control the points.parquet layout (row-group boundaries,
compression, number of files) while SpatialData still writes all element
metadata. Default behaviour is unchanged when the hook is omitted.

This follows the existing pass-through option pattern on write()
(shapes_geometry_encoding, raster_compressor,
convert_table_strings_to_categoricals) rather than introducing a new concept,
and it is deliberately layout-generic: nothing about it is specific to any
tiling scheme or viewer.

Motivation: producing spatially tiled row groups otherwise requires writing the
points parquet once with the default writer and then rewriting it. On an 8M
transcript Xenium dataset that measured 13.5s vs 8.5s, and it writes ~250MB of
transient output that is immediately discarded -- a gap that widens on
300M-transcript panels.

Note the hook receives the dataframe after transformations are stripped from
attrs (as the default writer requires), so a writer needing the coordinate
transform must obtain it from the element beforehand.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@codecov

codecov Bot commented Sep 6, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 91.89%. Comparing base (ccf1ea0) to head (229f1a8).

Additional details and impacted files
@@           Coverage Diff           @@
##             main    #1201   +/-   ##
=======================================
  Coverage   91.89%   91.89%           
=======================================
  Files          53       53           
  Lines        7942     7947    +5     
=======================================
+ Hits         7298     7303    +5     
  Misses        644      644           
Files with missing lines Coverage Δ
src/spatialdata/_core/spatialdata.py 93.86% <ø> (ø)
src/spatialdata/_io/io_points.py 100.00% <100.00%> (ø)
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@cornhundred cornhundred closed this Sep 7, 2026
@cornhundred
cornhundred deleted the feat/regular-grid-parquet-profile branch September 7, 2026 00:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant