align_chunks not working for datasets

[josephnowak]

I forgot to pass the align_chunks to the to_zarr method on the datasets, which makes the feature useless for this kind of data structure. I added a specific test to cover this issue.

Now, the align_chunks works on the cases where the data is smaller than a single Zarr chunk (also, one test was added to cover this scenario).

I modified (again) the error message shown with the safe_chunks, now it includes information about the two chunks that overlap with a single Zarr chunk. From what I saw on the error 10501, the original message was not helping the users to understand what was happening.

• Closes to_zarr() via dask with silent data loss on append #10501
• Tests added
• User visible changes (including notable bug fixes) are documented in whats-new.rst

[josephnowak]

Looks like those tests failing are not related to this PR

[josephnowak]

Hi @max-sixty, sorry for bothering you. I'm not sure if you have some free time to review this PR.

I'm not sure how the reviewer is assigned on Xarray in this case, as no other person was involved in the issue, and probably no one else was notified to review this.

[lbesnard]

@josephnowak if that helps for the reviewing process, I can confirm that those changes work for me

[lbesnard]

@josephnowak do you think someone else could maybe look at this?

[josephnowak]

Hi @dcherian, sorry for bothering you, not sure if you have some free time to take a look on this, or if you know of someone else that have the time to review this PR would be awesome.

[josephnowak]

Hi @lbesnard,

Unfortunately, I don't know of anyone else who could review the PR. The previous time that I sent a PR to Xarray, it was reviewed in a couple of days, so probably most of the maintainers are busy during this month.

I think that as a temporary solution, you can copy and paste the function grid_rechunk that is on my branch and use it directly on your code as I did here. It is more problematic because you need to keep track of the actual chunk structure of your data, but it should help.

[lbesnard]

Oh i ve been using your hash commit which is great. But im using this in a production tool, which is using poetry, (tldr; pointing xarray package to this hash fails on my CICD pipeline), so Im a bit blocked at the moment.

Its just creating more work for you as you always have to rebase.. thanks a lot anyway!

However I'm surprised no one else seems to have notice this bug.