Skip to content

Unable to roundtrip sharded zarr #11460

Description

@taus

What happened?

When creating a sharded zarr data set, reopening the dataset looses the sharding information. Shard and chunk sizes are set using encoding in to_zarr. When the dataset is reopened the sharding information is disregarded and the zarr chunks are used instead.

This results in issues e.g. when reopening a dataset to enrich it with metadata.

See also #11429

What did you expect to happen?

Running the minimal example it is demonstratet that, when saving, chunks are (4,4,2) matching the shards. When reopening the chunks are (2,2,2,2,2)

I would expected the dataset to be opened with the same chunking as the stored dataset.

Minimal Complete Verifiable Example

# /// script
# requires-python = ">=3.11"
# dependencies = [
#   "xarray[complete]@git+https://github.com/pydata/xarray.git@main",
# ]
# ///
#
# This script automatically imports the development branch of xarray to check for issues.
# Please delete this header if you have _not_ tested this script with `uv run`!

import numpy as np
import xarray as xr


xr.show_versions()
store = "test.zarr"

chunks= {"x":2,"y":2}
shards= {"x":4,"y":4}
dataset_shape = {"x":10,"y":10}
try:
    rng = np.random.default_rng()
    ds = xr.Dataset(
        {
            "data": (("x", "y"),rng.random((dataset_shape["x"], dataset_shape["y"]))),
        },
        coords={
            "x": np.arange(dataset_shape["x"]),
            "y": np.arange(dataset_shape["y"]),
        }
    )

    # Make the xarray chunks match the zarr shards
    ds_sharded = ds.chunk(shards)

    encoding = {
        "data": {
            "chunks": (2, 2),
            "shards": (4, 4)
        },
        "x": {
            "chunks": (2,),
            "shards": (4,)
        },
        "y": {
            "chunks": (2,),
            "shards": (4,)
        },
    }

    try:
        ds_sharded.to_zarr(
            store,
            mode="w",
            encoding=encoding,
            zarr_format=3,
            consolidated=False,
        )
        print("Zarr store created without errors.")
    except ValueError as e:
        print(f"Error: {e}")

    #reopen dataset and store it back
    reopend_ds = xr.open_zarr(store,consolidated=False)
    print(f"ds_sharded.chunks: {ds_sharded.chunks}")
    print(f"reopened_ds.chunks: {reopend_ds.chunks}")
    #This throws an error
    reopend_ds.to_zarr(store, mode="a")

except Exception as e:
    print(f"Error: {e}")

Steps to reproduce

No response

MVCE confirmation

  • Minimal example — the example is as focused as reasonably possible to demonstrate the underlying issue in xarray.
  • Complete example — the example is self-contained, including all data and the text of any traceback.
  • Verifiable example — the example copy & pastes into an IPython prompt or Binder notebook, returning the result.
  • New issue — a search of GitHub Issues suggests this is not a duplicate.
  • Recent environment — the issue occurs with the latest version of xarray and its dependencies.

Relevant log output

Zarr store created without errors.
ds_sharded.chunks: Frozen({'x': (4, 4, 2), 'y': (4, 4, 2)})
reopened_ds.chunks: Frozen({'x': (2, 2, 2, 2, 2), 'y': (2, 2, 2, 2, 2)})
Error: Specified Zarr chunks encoding['chunks']=(4, 4) for variable named 'data' would overlap multiple Dask chunks. Please check the Dask chunks at position 1 and 2, on axis 0, they are overlapped on the same Zarr chunk in the region slice(None, None, None). Writing this array in parallel with Dask could lead to corrupted data. To resolve this issue, consider one of the following options: - Rechunk the array using `chunk()`. - Modify or delete `encoding['chunks']`. - Set `safe_chunks=False`. - Enable automatic chunks alignment with `align_chunks=True`.

Anything else we need to know?

No response

Environment

Details INSTALLED VERSIONS ------------------ commit: None python: 3.14.3 (main, Feb 12 2026, 00:42:54) [Clang 21.1.4 ] python-bits: 64 OS: Linux OS-release: 6.17.0-35-generic machine: x86_64 processor: x86_64 byteorder: little LC_ALL: None LANG: en_US.UTF-8 LOCALE: ('en_US', 'UTF-8') libhdf5: 1.14.6 libnetcdf: 4.9.3

xarray: 2026.7.1.dev5+gfa2c0ca9a
pandas: 3.0.3
numpy: 2.4.6
scipy: 1.18.0
netCDF4: 1.7.4
pydap: 3.5.10
h5netcdf: 1.8.1
h5py: 3.16.0
zarr: 3.2.1
cftime: 1.6.5
nc_time_axis: 1.4.1
iris: None
bottleneck: 1.6.0
dask: 2026.7.1
distributed: 2026.7.1
matplotlib: 3.11.1
cartopy: 0.25.0
seaborn: 0.13.2
numbagg: 0.9.4
fsspec: 2026.6.0
cupy: None
pint: None
sparse: 0.19.0
flox: 0.11.2
numpy_groupies: 0.11.3
setuptools: None
pip: None
conda: None
pytest: None
mypy: None
IPython: None
sphinx: None

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugneeds triageIssue that has not been reviewed by xarray team member

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions