Roadmap involving zarr format support?

[cpelley]

From the comment #5399 (comment) there appears to be a suggestion that zarr might be on the roadmap. Is this correct?

Science post processing is planning to explore zarr as a fileformat for various scenarios that might be better and or more unified solution than other formats (NetCDF, parquet tables etc.). Future considerations like cloud compute for example might make zarr a preferred choice over NetCDF (TBD).

[cpelley]

ping @PaulAbernethy @mo-robert-purvis @SamGriffithsMO

https://ogcapi.ogc.org/edr/ (link provided by @mo-robert-purvis)

[trexfeathers]

Hi @cpelley. So far this is an ambition more than a plan. Zarr support would obviously be a beneficial feature, but it obviously gets weighed against other beneficial features since we can't deliver them all.

@pp-mo is working hard on #4994 and that is expected to deliver a medium-term solution to those needing Zarr - Zarr I/O via Xarray, with conversion to/from Iris if Iris features are needed. We expect this won't be good enough for all cases, e.g. mask handling will be less sophisticated than direct Iris I/O, so we'd love to hear: how far would a working Xarray bridge get you?

[cpelley]

Thanks for the quick reply, appreciated 👍
Gosh I hadn't considered the bridging to Xarray work, awesome!

@mo-robert-purvis will be exploring zarr use for post processing sometime in the new year. I'm sure he will be in touch with @pp-mo or team at some point to share thoughts/ideas.
Ah the familiar problem of mask handling hehe :)

Thanks again 🎅

[pp-mo]

BTW just noticed this again :
I have checked that we can load+store netcdf "as" Zarr by using ncdata as a bridge to Xarray.

Some representative fragments...

import xarray as xr
import ncdata.iris_xarray as ncix

cubes = iris.load(something)
xrds = ncix.cubes_to_xarray(cubes)
print(xrds)
print("\nsaving to zarr at :", testdir_path)
xrds.to_zarr(testdir_path)

readback_ds = xr.open_dataset(testdir_path, engine='zarr', chunks=-1)
print("\nRead back to xarray:")
print(readback_ds)

readback_cubes = ncix.cubes_from_xarray(readback_ds)
print(readback_cubes)

Not investigated in any depth, but it does basically "work"

[trexfeathers]

Options going forward:

1. Include the above example in the documentation
2. Encode the above example in a new Iris module/plugin) (would introduce Xarray and ncdata as optional dependencies)
3. Use nczarr with our existing NetCDF I/O
4. Interface directly with Zarr, similarly to how we already do FF/PP/GRIB

Some MO people are discussion on 2024-05-21, perhaps we will have a clearer idea after that.

[cpelley]

Thanks for laying out the options @trexfeathers 👍
Something I had been wondering myself. A factor to consider is whether Zarr would become a first class citizen or not of iris and how that should be exposed. My layman understanding is that Zarr has no real conventions as such, so xarray or iris (if 1st class citizen) would essentially encode their own. 'If I'm correct', then we would need to consider the implications of certain options on the taking ownership of such conventions.

I think there is potentially enough user interest that it could be worth considering a set of integration tests either way? - checking end-to-end save-load cycles for data with common characteristics (masked, unmasked, n-dimensioned etc.).
Do we have a test repository for such things? (i.e. testing of common user workflows, rather than of the iris library itself)

Such tests could then potentially be migrated to iris if such capability become exposed directly at a later date.

Just a thought.

[trexfeathers]

My layman understanding is that Zarr has no real conventions as such, so xarray or iris (if 1st class citizen) would essentially encode their own. 'If I'm correct', then we would need to consider the implications of certain options on the taking ownership of such conventions.

The Iris data model is married to CF¹, and we therefore apply a CF interpretation to all our loading - not just NetCDF but FF/PP/GRIB as well. Xarray has a much more flexible data model so it represents GRIB/NetCDF in a more 'raw' way, relying on the user to interpret it themselves. If Zarr has no 'conventions' then that fits easily into the Xarray model, but less so with Iris - I expect the closest analogy would be to look at how Iris loads NetCDF files that have no CF content.

Footnotes

1. IMO it should definitely remain this way. Some detailed context in Common agreement on loading CF non-compliant NetCDF files #5165. ↩

[trexfeathers]

Do we have a test repository for such things? (i.e. testing of common user workflows, rather than of the iris library itself)

We are aware of the immense value this would offer, especially given the limited resource of scientists making it difficult for our users to try out release candidates in time. It's not just tests that would be valuable but performance benchmarking too.

However the SciTools developers are spread pretty thin so we sadly just can't commit to this right now without harming our other offerings.

[cpelley]

The Iris data model is married to CF...

Indeed, as a first class citizen, the issue is how you represent CF data in zarr and the then implication that zarr data if provided to downstream customers implicitly require iris to correctly interpret it.
Anyway, I should refrain further until the workshop :) (I don't have direct knowledge for zarr)

...then that fits easily into the Xarray model, but less so with Iris...

Yes, that's my layman feeling so far. As you say this upcoming workshop should help.

...However the SciTools developers are spread pretty thin so we sadly just can't commit to this right now without harming our other offerings.

Appreciated but I think we need more community involvement in this. This upcoming workshop reflects that people are testing zarr usage with typical usecases and benchmarking already. Their work simply isn't being captured through a centralised, common framework/repository. scitools devs needn't necessarily own this framework either.

[pp-mo]

the issue is how you represent CF data in zarr

Actually I don't think that part is at all tricky. Since CF implies NetCDF, and there are existing fairly-standard solutions for mapping NetCDF to Zarr.

The only real catch is that Xarray invented their own "Zarr Encoding Specification" solution, whereas Unidata have since then (I think) defined their own "NCZarr" protocol for it.
Xarray can now also read NCZarr, but probably not write it (it may be possible via inbuilt support in the netcdf library, but I'm still not clear on this).
Which is more "popular" or useful in the wider world, I don't have much idea.

From the Iris POV, using ncdata+xarray, we can do what Xarray can here (without further effort!).

Practically I believe that the key problem to be solved is to associate the matching dimensions of different arrays, since Zarr doesn't do that (it stores arrays more-or-less independently of one another). The available datatypes, attributes and grouping concept all map pretty much directly.
Hence, the two solutions are functionally pretty much the same, but implemented a bit differently.

[pratiman-91]

@pp-mo Just want to know if iris is going to support "Zarr" or "Cloud Optimized GeoTIFF (COG)".

I am working on a single very large NetCDF file compressed around 100 GB. It is time consuming to write such as large single NetCDF files. As a convention for master files, we need a single NetCDF.

It would be great if iris can either support zarr or cog format which makes life easier for a lot of people.
Thanks!

[pp-mo]

We do have a long-term plan to support Zarr, but we haven't yet assigned that a definite timescale.
We don't think we would ever be supporting geotiff natively, as it isn't a good fit to the data model.

However, we do have ncdata as a "bridge to xarray", which efficiently translates Iris cubes to and from xarray datasets. This give us a route to other fileformats which xarray can handle, such as zarr (see below). We have no experience of COG.

I would say that zarr is probably a more natural match for storing netcdf-like data, if you just need a cloud-friendly storage format.
If you are looking for a format which supports parallel writing for better speed, that is hard to achieve with netcdf files, but may be easier with zarr.
Possibly xarray can do this ? (this seems to suggest parallel loading, not sure about writes)

For Zarr, our current solution, at its simplest, looks like this...

ncdata.threadlock_sharing.enable_lockshare(iris=True, xarray=True)  # probably needed?

xrds = xarray.open_dataset(zarr_input_path, engine="zarr", chunks={})
cubelist = ncdata.iris_xarray.cubes_from_xarray(xrds)
 . . .
xrds_out= nncdata.iris_xarray.cubes_to_xarray(my_cubelist)
xrds_out.to_zarr(output_path)

This delivers a reasonable interim solution, which at least preserves lazy data references for performance purposes.
A tested example can be seen here : https://github.com/pp-mo/ncdata/blob/v0.3.0/tests/integration/test_zarr_to_iris.py

Is this practical for your usecases ?

BTW we are intending to adopt ncdata into SciTools soon-ish, so it will be supported by the whole team and not my personal invention any more !

[pratiman-91]

@pp-mo Thank you for the detailed reply. It seems like for the time being I have to stick with netCDF as master files cannot be in Zarr unless iris fully supports it.

Possibly xarray can do this ? (this seems to suggest parallel loading, not sure about writes)

The does support parallel reading, however, writing is still single-threaded.