Surprising behavior of DataArray.chunk when using automatic chunksize determination

[grahamfindlay]

What is your issue?

I have a DataArray da with dims (x, y), and additional coordinates such as x_coord on dim x. If I try to chunk this array using da.chunk(chunks={'x': 'auto'}), I end up with a situation where:

1. The data themselves are chunked along x with chunksize a.
2. The x coordinate itself is not chunked.
3. The x_coord coordinate on dim x is chunked, with chunksize b != a.

As far as I can tell, what is going on is that da.chunk(chunks={'x': 'auto'}) is autodetermining the chunksize differently for each "thing" (data, variable, coordinate, etc) on the x dimension. What I expected was for it to determine one chunksize based on the data in the array, then use that chunksize (or no chunking) to each coordinate as well. Maybe there could be an option to yield unified chunks by default.

I discovered this because after chunking, da.chunksizes raises a ValueError because of the mismatch between the data and x_coord, and the proposed solution -- calling da.unify_chunks() -- then results in irregular chunksizes on both the data and x_coord. To get the behavior that I expected I have to call da.chunk(da.encoding['preferred_chunks']), which also, incidentally, seems like what I would have expected from da.unify_chunks().

[andersy005]

Thank you for the detailed report, @grahamfindlay!

2. The x coordinate itself is not chunked.
3. The x_coord coordinate on dim x is chunked, with chunksize b != a.

coordinates in Xarray are typically not chunked. this is why you might observe that the x coordinate itself is not chunked, even though you've specified 'auto' chunking for the x dimension. this behavior is expected by design and isn't something you need to worry about unless you have a specific reason for wanting to chunk your coordinates. however, note that non-dimensional/auxiliary coordinates (i.e., coordinates that share at least one dimension with your data variables but are not dimensions themselves) like your x_coord can be chunked. This is likely why you see a chunk size for x_coord.

As far as I can tell, what is going on is that da.chunk(chunks={'x': 'auto'}) is autodetermining the chunksize differently for each "thing" (data, variable, coordinate, etc) on the x dimension. What I expected was for it to determine one chunksize based on the data in the array, then use that chunksize (or no chunking) to each coordinate as well.

the dtype of the data and coordinates can influence the chunk size. If the data and x_coord have different dtypes, it could lead to different chunk sizes. it's not clear from your description if this is the case.

Could you provide a minimal working example that reproduces this issue? This would be extremely useful for diagnosing the problem

[grahamfindlay]

Hi Anderson, thanks for your reply. Here is a minimal working example:

import numpy as np
import xarray as xr

nx = int(1e7)
ny = 10
da = xr.DataArray(
    np.zeros(shape=(nx, ny)),
    dims=("x", "y"),
    coords={"x": np.arange(nx), "y": np.arange(ny), "x_coord": ("x", np.arange(nx))},
).chunk({"x": "auto"})
print(f"Data chunks: {da.chunks}")
print(f"x_coord chunks: {da.x_coord.chunks}")
print(da.chunksizes)

Which produces the following output:

Data chunks: ((1677721, 1677721, 1677721, 1677721, 1677721, 1611395), (10,))
x_coord chunks: ((10000000,),)
...
ValueError: Object has inconsistent chunks along dimension x. This can be fixed by calling unify_chunks().

I understand that this is all the intended behavior, and nothing is broken. It is just extremely surprising behavior. In increasing order of surprise, I expected x and x_coord to either be

1. Not chunked at all.
2. Both chunked, with chunksizes that match the data.
3. x not chunked, x_coord chunked, with chunksizes that match the data.
4. x not chunked, x_coord chunked with chunksizes that do not match the data. (Actual behavior)