refactor v3 data types #2874
+3,742
−1,337
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,2 @@ | ||
| Adds zarr-specific data type classes. This replaces the direct use of numpy data types for zarr | ||
| v2 and a fixed set of string enums for zarr v3. For more on this new feature, see the `data types user guide <https://zarr.readthedocs.io/en/stable/user-guide/data_types.html>`_ |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change | ||||||
|---|---|---|---|---|---|---|---|---|
| @@ -0,0 +1,110 @@ | ||||||||
| Data types | ||||||||
|
There was a problem hiding this comment. This file is a super useful read. I'm wondering what to do with it though. Were you thinking it would go under the There was a problem hiding this comment. No strong opinion from me. IMO our docs right now are not the most logically organized, so I anticipate some churn there in any case. |
||||||||
| ========== | ||||||||
|
|
||||||||
| Zarr's data type model | ||||||||
| ---------------------- | ||||||||
|
|
||||||||
| Every Zarr array has a "data type", which defines the meaning and physical layout of the | ||||||||
| array's elements. Zarr is heavily influenced by `NumPy <https://numpy.org/doc/stable/>`_, and | ||||||||
|
There was a problem hiding this comment.
Do you mean the data format, or Zarr-Python here? Would be good to clarify. |
||||||||
| Zarr-Python supports creating arrays with Numpy data types:: | ||||||||
|
|
||||||||
| >>> import zarr | ||||||||
| >>> import numpy as np | ||||||||
| >>> z = zarr.create_array(store={}, shape=(10,), dtype=np.dtype('uint8')) | ||||||||
| >>> z | ||||||||
| <Array memory:... shape=(10,) dtype=uint8> | ||||||||
|
|
||||||||
| Unlike Numpy arrays, Zarr arrays are designed to be persisted to storage and read by Zarr implementations in different programming languages. | ||||||||
| This means Zarr data types must be interpreted correctly when clients read an array. So each Zarr data type defines a procedure for | ||||||||
| encoding/decoding that data type to/from Zarr array metadata, and also encoding/decoding **instances** of that data type to/from | ||||||||
| array metadata. These serialization procedures depend on the version of the Zarr format used. | ||||||||
|
|
||||||||
| Data types in Zarr version 2 | ||||||||
| ----------------------------- | ||||||||
|
|
||||||||
| Version 2 of the Zarr format defined its data types relative to `Numpy's data types <https://numpy.org/doc/2.1/reference/arrays.dtypes.html#data-type-objects-dtype>`_, and added a few non-Numpy data types as well. | ||||||||
| Thus the JSON identifier for a Numpy-compatible data type is just the Numpy ``str`` attribute of that dtype:: | ||||||||
|
|
||||||||
| >>> import zarr | ||||||||
| >>> import numpy as np | ||||||||
| >>> import json | ||||||||
| >>> store = {} | ||||||||
|
There was a problem hiding this comment.
Suggested change
to break it up a bit? |
||||||||
| >>> np_dtype = np.dtype('int64') | ||||||||
| >>> z = zarr.create_array(store=store, shape=(1,), dtype=np_dtype, zarr_format=2) | ||||||||
| >>> dtype_meta = json.loads(store['.zarray'].to_bytes())["dtype"] | ||||||||
d-v-b marked this conversation as resolved.
Show resolved
Hide resolved
|
||||||||
| >>> assert dtype_meta == np_dtype.str # True | ||||||||
|
There was a problem hiding this comment. Instead of claiming this assert is true in a comment, it would be nicer just to show it explicitly
Suggested change
|
||||||||
| >>> dtype_meta | ||||||||
| '<i8' | ||||||||
|
|
||||||||
| .. note:: | ||||||||
| The ``<`` character in the data type metadata encodes the `endianness <https://numpy.org/doc/2.2/reference/generated/numpy.dtype.byteorder.html>`_, or "byte order", of the data type. Following Numpy's example, | ||||||||
| in Zarr version 2 each data type has an endianness where applicable. However, Zarr version 3 data types do not store endianness information. | ||||||||
|
|
||||||||
| In addition to defining a representation of the data type itself (which in the example above was just a simple string ``"<i8"``), Zarr also | ||||||||
| defines a metadata representation of scalars associated with that data type. Integers are stored as ``JSON`` numbers, | ||||||||
|
There was a problem hiding this comment.
Suggested change
I scratched my head for a bit wondering why a scalar representation was needed, before realising (I think I'm right?). I'm not sure my suggestion is very well written, but something similar to explain why here migth be nice? There was a problem hiding this comment. the scalar representation is only used for the fill value metadata, so I will say as much in the docs |
||||||||
| as are floats, with the caveat that `NaN`, positive infinity, and negative infinity are stored as special strings. | ||||||||
|
|
||||||||
| Data types in Zarr version 3 | ||||||||
| ----------------------------- | ||||||||
|
|
||||||||
| Zarr V3 brings several key changes to how data types are represented: | ||||||||
|
There was a problem hiding this comment. Might be nice to link out the relevant bit of the Zarr spec here? |
||||||||
|
|
||||||||
| - Zarr V3 identifies the basic data types as strings like ``int8``, ``int16``, etc. In Zarr V2 ``int8`` would represented as ``|i1``, ``int16`` would be ``>i2`` **or** ``<i2``, depending on the endianness. | ||||||||
| - A Zarr V3 data type does not have endianness. This is a departure from Zarr V2, where multi-byte data types would be stored in ``JSON`` with an encoding that included endianness. Instead, | ||||||||
| Zarr V3 requires that endianness, where applicable, is specified in the ``codecs`` attribute of array metadata. | ||||||||
| - Zarr V3 data types can also take the form of a ``JSON`` object like | ||||||||
| ``{"name": "foo", "configuration": {"parameter": "value"}}``. This structure facilitates specifying data types that take parameters. | ||||||||
|
There was a problem hiding this comment. Could you give an example of a data type that takes parameters? There was a problem hiding this comment. no I cannot! the zarr v3 spec does not define any data types that use the |
||||||||
|
|
||||||||
|
|
||||||||
| Data types in Zarr-Python | ||||||||
| ------------------------- | ||||||||
|
|
||||||||
| The two Zarr formats that Zarr-Python supports specify data types differently: | ||||||||
| data types in Zarr version 2 are encoded as Numpy-compatible strings, while data types in Zarr version | ||||||||
| 3 are encoded as either strings or ``JSON`` objects, | ||||||||
| and the Zarr V3 data types don't have any associated endianness information, unlike Zarr V2 data types. | ||||||||
|
|
||||||||
| To abstract over these syntactical and semantic differences, Zarr-Python uses a class called `ZDType <../api/zarr/dtype/index.html#zarr.dtype.ZDType>`_ to wrap native data types (e.g., Numpy data types) and provide Zarr V2 and Zarr V3 compatibility routines. | ||||||||
|
There was a problem hiding this comment.
Suggested change
Is there a reason to use 'native' (new jargon) instead of Numpy if native just means Numpy? There was a problem hiding this comment. part of the goal of this PR is to abstract over any array data type, including but not limited to numpy data types. |
||||||||
| Each data type supported by Zarr-Python is modeled by a subclass of ``ZDType``, which provides an API for the following operations: | ||||||||
|
|
||||||||
| - Wrapping / unwrapping a native data type | ||||||||
| - Encoding / decoding a data type to / from Zarr V2 and Zarr V3 array metadata. | ||||||||
| - Encoding / decoding a scalar value to / from Zarr V2 and Zarr V3 array metadata. | ||||||||
|
|
||||||||
|
|
||||||||
| Example Usage | ||||||||
| ~~~~~~~~~~~~~ | ||||||||
|
|
||||||||
| .. code-block:: python | ||||||||
|
|
||||||||
| from zarr.core.dtype.wrapper import Int8 | ||||||||
|
|
||||||||
| # Create a ZDType instance from a native dtype | ||||||||
| int8 = Int8.from_dtype(np.dtype('int8')) | ||||||||
|
|
||||||||
| # Convert back to native dtype | ||||||||
| native_dtype = int8.to_dtype() | ||||||||
| assert native_dtype == np.dtype('int8') | ||||||||
|
|
||||||||
| # Get the default value | ||||||||
| default_value = int8.default_value() | ||||||||
| assert default_value == np.int8(0) | ||||||||
|
|
||||||||
| # Serialize to JSON | ||||||||
| json_representation = int8.to_json(zarr_format=3) | ||||||||
|
|
||||||||
| # Serialize a scalar value | ||||||||
| json_value = int8.to_json_value(42, zarr_format=3) | ||||||||
| assert json_value == 42 | ||||||||
|
|
||||||||
| # Deserialize a scalar value | ||||||||
| scalar_value = int8.from_json_value(42, zarr_format=3) | ||||||||
| assert scalar_value == np.int8(42) | ||||||||
|
Comment on lines
+78
to
+102
There was a problem hiding this comment. This should be changed to a doctest like in other guides. Then the output would be shown/tested. |
||||||||
|
|
||||||||
| Custom Data Types | ||||||||
| ~~~~~~~~~~~~~~~~~ | ||||||||
|
|
||||||||
| Users can define custom data types by subclassing `ZDType` and implementing the required methods. | ||||||||
| Once defined, the custom data type can be registered with Zarr-Python to enable seamless integration with the library. | ||||||||
|
|
||||||||
| <TODO: example of defining a custom data type> | ||||||||
|
There was a problem hiding this comment.
Suggested change
I would get rid of this line, and open an issue to keep track of this. There was a problem hiding this comment. This is a super nice guide! |
||||||||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -8,6 +8,7 @@ User guide | ||
|
|
||
| installation | ||
| arrays | ||
| data_types | ||
| groups | ||
| attributes | ||
| storage | ||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Do you know why the number of bytes has changed here? Does that mean the data/bytes being stored has changed somehow?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think it's because the bytes codec no longer specifies endianness, so the JSON document is slightly smaller, but I haven't confirmed this.