zarr v3 does not accept stores of type FSMap

[malmans2]

Zarr version

v3.0.0

Numcodecs version

v0.14.1

Python Version

3.11.11

Operating System

Mac

Installation

using pip into conda environment

Description

With Zarr v2, I was able to pass a store of type fsspec.mapping.FSMap. However, with Zarr v3, I get the following error:

TypeError: Unsupported type for store_like: 'FSMap'

Is this a bug, a breaking change, or was passing mappers like FSMap not officially supported even in Zarr v2?
The snippet below only works with Zarr v2.

Steps to reproduce

import tempfile

import fsspec
import zarr

fs = fsspec.filesystem("file")
with tempfile.TemporaryDirectory() as tmpdir:
    mapper = fs.get_mapper(tmpdir)
    z = zarr.open(store=mapper, shape=(100, 100), chunks=(10, 10), dtype="f4")

Additional output

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
Cell In[7], line 9
      7 with tempfile.TemporaryDirectory() as tmpdir:
      8     mapper = fs.get_mapper(tmpdir)
----> 9     z = zarr.open(store=mapper, shape=(100, 100), chunks=(10, 10), dtype="f4")

File ~/miniforge3/envs/zarr/lib/python3.11/site-packages/zarr/_compat.py:43, in _deprecate_positional_args.<locals>._inner_deprecate_positional_args.<locals>.inner_f(*args, **kwargs)
     41 extra_args = len(args) - len(all_args)
     42 if extra_args <= 0:
---> 43     return f(*args, **kwargs)
     45 # extra_args > 0
     46 args_msg = [
     47     f"{name}={arg}"
     48     for name, arg in zip(kwonly_args[:extra_args], args[-extra_args:], strict=False)
     49 ]

File ~/miniforge3/envs/zarr/lib/python3.11/site-packages/zarr/api/synchronous.py:190, in open(store, mode, zarr_version, zarr_format, path, storage_options, **kwargs)
    152 @_deprecate_positional_args
    153 def open(
    154     store: StoreLike | None = None,
   (...)
    161     **kwargs: Any,  # TODO: type kwargs as valid args to async_api.open
    162 ) -> Array | Group:
    163     """Open a group or array using file-mode-like semantics.
    164
    165     Parameters
   (...)
    188         Return type depends on what exists in the given store.
    189     """
--> 190     obj = sync(
    191         async_api.open(
    192             store=store,
    193             mode=mode,
    194             zarr_version=zarr_version,
    195             zarr_format=zarr_format,
    196             path=path,
    197             storage_options=storage_options,
    198             **kwargs,
    199         )
    200     )
    201     if isinstance(obj, AsyncArray):
    202         return Array(obj)

File ~/miniforge3/envs/zarr/lib/python3.11/site-packages/zarr/core/sync.py:142, in sync(coro, loop, timeout)
    139 return_result = next(iter(finished)).result()
    141 if isinstance(return_result, BaseException):
--> 142     raise return_result
    143 else:
    144     return return_result

File ~/miniforge3/envs/zarr/lib/python3.11/site-packages/zarr/core/sync.py:98, in _runner(coro)
     93 """
     94 Await a coroutine and return the result of running it. If awaiting the coroutine raises an
     95 exception, the exception will be returned.
     96 """
     97 try:
---> 98     return await coro
     99 except Exception as ex:
    100     return ex

File ~/miniforge3/envs/zarr/lib/python3.11/site-packages/zarr/api/asynchronous.py:309, in open(store, mode, zarr_version, zarr_format, path, storage_options, **kwargs)
    280 """Convenience function to open a group or array using file-mode-like semantics.
    281
    282 Parameters
   (...)
    305     Return type depends on what exists in the given store.
    306 """
    307 zarr_format = _handle_zarr_version_or_format(zarr_version=zarr_version, zarr_format=zarr_format)
--> 309 store_path = await make_store_path(store, mode=mode, path=path, storage_options=storage_options)
    311 # TODO: the mode check below seems wrong!
    312 if "shape" not in kwargs and mode in {"a", "r", "r+", "w"}:

File ~/miniforge3/envs/zarr/lib/python3.11/site-packages/zarr/storage/_common.py:316, in make_store_path(store_like, path, mode, storage_options)
    314     else:
    315         msg = f"Unsupported type for store_like: '{type(store_like).__name__}'"  # type: ignore[unreachable]
--> 316         raise TypeError(msg)
    318     result = await StorePath.open(store, path=path_normalized, mode=mode)
    320 if storage_options and not used_storage_options:

TypeError: Unsupported type for store_like: 'FSMap'

[d-v-b]

accepting instances of FSMap was part of the v2 design (see this PR), so I'd say you are seeing a regression / bug.

I will let the fsspec experts weigh in on whether there's any good reason not to accept FSMap instances here -- this may very well be a case of "we didn't implement this yet", in which case we should implement it :)

[jhamman]

This was an intentional breaking change. The Fsspec's FSMap interface was used as a drop in for Zarr's previous store interface (i.e. MutableMapping).

If you have a fsspec FileSystem, you can create the store directly though. Something like this should work:

import fsspec
import zarr

fs = fsspec.filesystem("s3", asynchronous=True)
store = zarr.storage.FsspecStore(fs, path="bucket/foo/bar")
z = zarr.open(store=mapper, shape=(100, 100), chunks=(10, 10), dtype="f4")

Note that the file implementation in fsspec does not support async so it cannot be currently used in zarr (see #2533 for a wip pr).

[dstansby]

I'll label this as docs, since it would be good to put the above comment in our migration guide.

[rabernat]

This is going to bite a lot of people. 😬

It is relatively easy to convert an FSMap object to the type of store we need. We could consider adding some convenience layer for this.

[jhamman]

It is relatively easy to convert an FSMap object to the type of store we need. We could consider adding some convenience layer for this.

@martindurant will know best but remember that we're using Fsspec's async interface which is not active in most (any?) FSMap implementations. There was concern a while back about loop-in-loop issues if using filesystems with asynchronous=false.

[martindurant]

Rather than wrapping, passing the filesystem and path arguments directly seems the right thing to do, which was already suggested above.

I'm not sure what might happen if a (sync) FSMap was used at the (hidden) dict within a (async) MemoryStore. It would probably work and for local files not make a performance difference?

[jhamman]

👍 - very open to us finding a way to accept FSMap objects as store arguments - we'll need to convert them to an FSspecStore but it sounds like we can do that without much risk.