What happened?
When creating a sharded zarr data set, reopening the dataset looses the sharding information. Shard and chunk sizes are set using encoding in to_zarr. When the dataset is reopened the sharding information is disregarded and the zarr chunks are used instead.
This results in issues e.g. when reopening a dataset to enrich it with metadata.
See also #11429
What did you expect to happen?
Running the minimal example it is demonstratet that, when saving, chunks are (4,4,2) matching the shards. When reopening the chunks are (2,2,2,2,2)
I would expected the dataset to be opened with the same chunking as the stored dataset.
Minimal Complete Verifiable Example
# /// script
# requires-python = ">=3.11"
# dependencies = [
# "xarray[complete]@git+https://github.com/pydata/xarray.git@main",
# ]
# ///
#
# This script automatically imports the development branch of xarray to check for issues.
# Please delete this header if you have _not_ tested this script with `uv run`!
import numpy as np
import xarray as xr
xr.show_versions()
store = "test.zarr"
chunks= {"x":2,"y":2}
shards= {"x":4,"y":4}
dataset_shape = {"x":10,"y":10}
try:
rng = np.random.default_rng()
ds = xr.Dataset(
{
"data": (("x", "y"),rng.random((dataset_shape["x"], dataset_shape["y"]))),
},
coords={
"x": np.arange(dataset_shape["x"]),
"y": np.arange(dataset_shape["y"]),
}
)
# Make the xarray chunks match the zarr shards
ds_sharded = ds.chunk(shards)
encoding = {
"data": {
"chunks": (2, 2),
"shards": (4, 4)
},
"x": {
"chunks": (2,),
"shards": (4,)
},
"y": {
"chunks": (2,),
"shards": (4,)
},
}
try:
ds_sharded.to_zarr(
store,
mode="w",
encoding=encoding,
zarr_format=3,
consolidated=False,
)
print("Zarr store created without errors.")
except ValueError as e:
print(f"Error: {e}")
#reopen dataset and store it back
reopend_ds = xr.open_zarr(store,consolidated=False)
print(f"ds_sharded.chunks: {ds_sharded.chunks}")
print(f"reopened_ds.chunks: {reopend_ds.chunks}")
#This throws an error
reopend_ds.to_zarr(store, mode="a")
except Exception as e:
print(f"Error: {e}")
Steps to reproduce
No response
MVCE confirmation
Relevant log output
Zarr store created without errors.
ds_sharded.chunks: Frozen({'x': (4, 4, 2), 'y': (4, 4, 2)})
reopened_ds.chunks: Frozen({'x': (2, 2, 2, 2, 2), 'y': (2, 2, 2, 2, 2)})
Error: Specified Zarr chunks encoding['chunks']=(4, 4) for variable named 'data' would overlap multiple Dask chunks. Please check the Dask chunks at position 1 and 2, on axis 0, they are overlapped on the same Zarr chunk in the region slice(None, None, None). Writing this array in parallel with Dask could lead to corrupted data. To resolve this issue, consider one of the following options: - Rechunk the array using `chunk()`. - Modify or delete `encoding['chunks']`. - Set `safe_chunks=False`. - Enable automatic chunks alignment with `align_chunks=True`.
Anything else we need to know?
No response
Environment
Details
INSTALLED VERSIONS
------------------
commit: None
python: 3.14.3 (main, Feb 12 2026, 00:42:54) [Clang 21.1.4 ]
python-bits: 64
OS: Linux
OS-release: 6.17.0-35-generic
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8
LOCALE: ('en_US', 'UTF-8')
libhdf5: 1.14.6
libnetcdf: 4.9.3
xarray: 2026.7.1.dev5+gfa2c0ca9a
pandas: 3.0.3
numpy: 2.4.6
scipy: 1.18.0
netCDF4: 1.7.4
pydap: 3.5.10
h5netcdf: 1.8.1
h5py: 3.16.0
zarr: 3.2.1
cftime: 1.6.5
nc_time_axis: 1.4.1
iris: None
bottleneck: 1.6.0
dask: 2026.7.1
distributed: 2026.7.1
matplotlib: 3.11.1
cartopy: 0.25.0
seaborn: 0.13.2
numbagg: 0.9.4
fsspec: 2026.6.0
cupy: None
pint: None
sparse: 0.19.0
flox: 0.11.2
numpy_groupies: 0.11.3
setuptools: None
pip: None
conda: None
pytest: None
mypy: None
IPython: None
sphinx: None
What happened?
When creating a sharded zarr data set, reopening the dataset looses the sharding information. Shard and chunk sizes are set using
encodingin to_zarr. When the dataset is reopened the sharding information is disregarded and the zarr chunks are used instead.This results in issues e.g. when reopening a dataset to enrich it with metadata.
See also #11429
What did you expect to happen?
Running the minimal example it is demonstratet that, when saving, chunks are (4,4,2) matching the shards. When reopening the chunks are (2,2,2,2,2)
I would expected the dataset to be opened with the same chunking as the stored dataset.
Minimal Complete Verifiable Example
Steps to reproduce
No response
MVCE confirmation
Relevant log output
Anything else we need to know?
No response
Environment
Details
INSTALLED VERSIONS ------------------ commit: None python: 3.14.3 (main, Feb 12 2026, 00:42:54) [Clang 21.1.4 ] python-bits: 64 OS: Linux OS-release: 6.17.0-35-generic machine: x86_64 processor: x86_64 byteorder: little LC_ALL: None LANG: en_US.UTF-8 LOCALE: ('en_US', 'UTF-8') libhdf5: 1.14.6 libnetcdf: 4.9.3xarray: 2026.7.1.dev5+gfa2c0ca9a
pandas: 3.0.3
numpy: 2.4.6
scipy: 1.18.0
netCDF4: 1.7.4
pydap: 3.5.10
h5netcdf: 1.8.1
h5py: 3.16.0
zarr: 3.2.1
cftime: 1.6.5
nc_time_axis: 1.4.1
iris: None
bottleneck: 1.6.0
dask: 2026.7.1
distributed: 2026.7.1
matplotlib: 3.11.1
cartopy: 0.25.0
seaborn: 0.13.2
numbagg: 0.9.4
fsspec: 2026.6.0
cupy: None
pint: None
sparse: 0.19.0
flox: 0.11.2
numpy_groupies: 0.11.3
setuptools: None
pip: None
conda: None
pytest: None
mypy: None
IPython: None
sphinx: None