Will geozarr be interoperable outside of the xarray ecosystem?

[geospatial-jeff]

The medium articles linked in the README do a great job explaining the CDM data model, thanks for writing those! One paragraph in the "Geospatial Data Model Encoding" post (see here) drew my attention:

Since Zarr does not directly support some key CDM features — like named dimensions, scalar variables, or detailed metadata — NCZarr and Xarray provide their own encoding rules to keep the meaning and structure of the CDM.

Does this mean that GeoZarr (built on CDM) can only be interpreted by readers such as xarray that provide specific encoding rules that bridge the gap between Zarr and CDM? Can someone please provide more details on what these specific encoding rules are, and how they are implemented in xarray?

Thanks!

[maxrjones]

Will geozarr be interoperable outside of the xarray ecosystem?

I think the goal of developing a spec rather than just an implementation is to make it interoperable outside the xarray ecosystem.

Since Zarr does not directly support some key CDM features — like named dimensions, scalar variables, or detailed metadata — NCZarr and Xarray provide their own encoding rules to keep the meaning and structure of the CDM.

Just one caveat that Zarr specification 3 elevated dimension names to a part of the core spec. I would also be more specific that "Zarr does not directly interpret some key CDM features" because each of those components are "supported" in the sense that their inclusion does not break the Zarr implementation or specification.

Does this mean that GeoZarr (built on CDM) can only be interpreted by readers such as xarray that provide specific encoding rules that bridge the gap between Zarr and CDM?

Yes, currently. I believe @briannapagan did a very recent check of interoperability with QGIS via GDAL and found lacking interoperability due to it not understanding the proposed bridge between Zarr and CDM. This is also reflected by the last more comprehensive check over a year ago.

Some of us believe that extracting CF encoding from xarray into a separate library (e.g., https://github.com/xarray-contrib/cf-codecs) would really help for broader usage of CF metadata and structure and interoperability.

[geospatial-jeff]

Some of us believe that extracting CF encoding from xarray into a separate library (e.g., https://github.com/xarray-contrib/cf-codecs) would really help for broader usage of CF metadata and structure and interoperability.

This sounds like a great idea to me, could be a good reference example.

[geospatial-jeff]

the produced coordinate vs data vs auxiliary variables in a Xarray dataset differed depending on whether you used the path of NetCDF -> h5netcdf -> xarray.Dataset vs NetCDF -> Kerchunk -> fsspec + Zarr -> xarray.Dataset.

@maxrjones @sharkinsspatial pulling this back into this issue (from #76), can you provide the datasets / code you used here? I'm interested in exploring this more.

[maxrjones]

the produced coordinate vs data vs auxiliary variables in a Xarray dataset differed depending on whether you used the path of NetCDF -> h5netcdf -> xarray.Dataset vs NetCDF -> Kerchunk -> fsspec + Zarr -> xarray.Dataset.

@maxrjones @sharkinsspatial pulling this back into this issue (from #76), can you provide the datasets / code you used here? I'm interested in exploring this more.

This is another case where the discussions are split over many issues/PRs (sorry), but here's a couple references to get you started:

• open_virtual_dataset returns some coordinates as data variables VirtualiZarr#189
• Manipulation of coordinages do not materialize to kerchunk refs  VirtualiZarr#281 (comment)

Note that we still need to triage if those issues are solved by the major redesign of VirtualiZarr internals for v2.0, which included deferring designs about what constitutes a coordinate to https://github.com/zarr-developers/VirtualiZarr/blob/6fd8634b4a5f32ae17cab87ea886d3df3bed8a15/virtualizarr/xarray.py#L169-L172 if producing a virtual dataset or xarray directly if using VirtualiZarr to load data directly.

[dcherian]

Since Zarr does not directly support some key CDM features — like named dimensions, scalar variables, or detailed metadata — NCZarr and Xarray provide their own encoding rules to keep the meaning and structure of the CDM.

This is very Zarr V2-specific.

1. Zarr V3 [supports dimension_names](https://zarr-specs.readthedocs.io/en/latest/v3/core/index.html#dimension-names though they are optional
2. Scalar variables are supported in V2 and V3
3. Zarr V2 and V3 support arbitrary metadata (up to JSON limitations)

The key bit of CDM that is not enforced by Zarr is that the same dimension name referenced in multiple arrays in a group must be associated with the same size. E.g. the time dimension in arrays foo/var1 and foo/var2 (foo is group) must be of the same size.

extracting CF encoding from xarray into a separate library (e.g., xarray-contrib/cf-codecs) would really help for broader usage of CF metadata and structure and interoperability.

No. Xarray interprets CF metadata according to its own internal model. CF is much richer in recording relationships between variables, Xarray cannot represent that richness explicitly at least, the best it can do is propagate arbitrary attributes.

[maxrjones]

extracting CF encoding from xarray into a separate library (e.g., xarray-contrib/cf-codecs) would really help for broader usage of CF metadata and structure and interoperability.

No. Xarray interprets CF metadata according to its own internal model. CF is much richer in recording relationships between variables, Xarray cannot represent that richness explicitly at least, the best it can do is propagate arbitrary attributes.

Can you please expand upon this opinion in response to pydata/xarray#155 (comment)? It's not obvious why your statement that Xarray adopts a subset of the relationships described by CF makes it unhelpful to expose those relations as CF codecs usable by other libraries either through a public portion of the xarray API or a separate library.

[dcherian]

AFAICT pydata/xarray#155 (comment) is not relevant to the virtualizarr coordinates-handling issues listed in your comment. (certainly exposing DatetimeCoder would be useful for cftime handling but that doesn't affect the listed issues).

The "what is a coordinate" logic is here: https://github.com/pydata/xarray/blob/9c334ae4a505e52b1e5d4ba909fe252fe03ba02d/xarray/conventions.py#L429 and reflects Xarray's limited notion of "coordinate variables". A DataArray da is associated with a coordinate variablec iff set(c.dims) <= set(da.dims) (with one very new exception) while CF allows recording much more specific relationships through coordinates, ancillary_variables, grid_mapping etc. cf-xarray does do that extra parsing and interpretation and cfdm more fully interprets and tracks those relationships in its data model.

All I'm saying is that Xarray's limited interpretation (or "decoding") of "coordinates" should NOT limit what the spec allows users to express, which I think we all agree on. Also if you are using Xarray, then this stuff is exposed in public API: xr.decode_cf