Zarr+obstore directory listing truncates names (likely str.lstrip bug), breaks AnnData/Zarr reads on Azure

[antoinegaston]

Zarr version

v3.1.5

Numcodecs version

v0.16.5

Python Version

3.14

Operating System

Mac

Installation

using uv

Description

When using obstore (Azure) through Zarr’s zarr.storage.ObjectStore, listing “directories” returns truncated child names (e.g. "resolution" becomes "olution", "names" becomes "ames", "logfoldchanges" becomes "gfoldchanges"). This causes Zarr to misinterpret valid keys as “not part of a Zarr hierarchy” and can ultimately lead to KeyError / FileNotFoundError when libraries like anndata traverse groups.

Steps to reproduce

# /// script
# requires-python = ">=3.10"
# dependencies = [
#   "zarr@git+https://github.com/zarr-developers/zarr-python.git@main",
# ]
# ///
#
# This script automatically imports the development branch of zarr to check for issues

import asyncio
import zarr.storage._obstore as zarr_obstore

async def fake_list_result():
    prefix = "uns/leiden_res_0_25/params"
    return {
        "common_prefixes": [
            f"{prefix}/resolution",
            f"{prefix}/random_state",
        ],
        "objects": [],
    }

async def main():
    items = [x async for x in zarr_obstore._transform_list_dir(
        fake_list_result(),
        "uns/leiden_res_0_25/params",
    )]
    print(items)

asyncio.run(main())

This minimal repro. script returns ['olution', 'om_state'] instead of ['resolution', 'random_state']

Workaround / fix

A one-line fix in the adapter is enough: replace obj.lstrip(prefix) with obj.removeprefix(prefix) (and similarly for the object-path handling).

[maxrjones]

Thanks for identifying this issue @antoinegaston. Would you be interested in submitting a PR with a proposed fix + unit test?

[d-v-b]

A one-line fix in the adapter is enough: replace obj.lstrip(prefix) with obj.removeprefix(prefix) (and similarly for the object-path handling).

This is the second or third time we have gotten bugs from using lstrip instead of removeprefix! I suspect lstrip is just poorly named, given what it does.

[maxrjones]

A one-line fix in the adapter is enough: replace obj.lstrip(prefix) with obj.removeprefix(prefix) (and similarly for the object-path handling).

This is the second or third time we have gotten bugs from using lstrip instead of removeprefix! I suspect lstrip is just poorly named, given what it does.

Yes, we should remove lstrip usage. I also think that it's been more of a whack-a-mole game than ideal due to the path handling happening in individual store implementations rather than at a higher abstraction.

[d-v-b]

we already have some path-manipulation utilities in _utils.py, I bet we could re-use these in stores, and add more as needed

[antoinegaston]

Thanks for identifying this issue @antoinegaston. Would you be interested in submitting a PR with a proposed fix + unit test?

Sure thing, although if you foresee a better implementation using existing utils let me know I can investigate on that.

[maxrjones]

Thanks for identifying this issue @antoinegaston. Would you be interested in submitting a PR with a proposed fix + unit test?

Sure thing, although if you foresee a better implementation using existing utils let me know I can investigate on that.

I think It's worth checking whether _relativize_path applies the appropriate logic if used in L266 and L267 here -

zarr-python/src/zarr/storage/_obstore.py

Lines 266 to 267 in 884a8c9

	prefixes = [obj.lstrip(prefix).lstrip("/") for obj in list_result["common_prefixes"]]
	objects = [obj["path"].removeprefix(prefix).lstrip("/") for obj in list_result["objects"]]

.