Skip to content

Commit f9331e8

Browse files
authored
fix: limit inventory by prefix (#43)
resolves #24 <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - **Refactor** - Streamlined the retrieval process for stored objects, enhancing clarity and consistency in cache updates while improving efficiency when interacting with storage. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
1 parent c78fa78 commit f9331e8

File tree

1 file changed

+7
-4
lines changed

1 file changed

+7
-4
lines changed

snakemake_storage_plugin_s3/__init__.py

Lines changed: 7 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,7 @@
44

55
import boto3
66
import botocore.credentials
7+
import os
78

89
from snakemake_interface_common.exceptions import WorkflowError
910
from snakemake_interface_storage_plugins.settings import StorageProviderSettingsBase
@@ -232,12 +233,14 @@ async def inventory(self, cache: IOCacheStorageInterface):
232233
# bucket has been inventorized before, stop here
233234
return
234235

235-
# check if bucket exists
236236
if not self.bucket_exists():
237237
cache.exists_in_storage[self.cache_key()] = False
238-
else:
239-
cache.exists_in_storage[self.get_inventory_parent()] = True
240-
for obj in self.s3bucket().objects.all():
238+
return
239+
240+
cache.exists_in_storage[self.get_inventory_parent()] = True
241+
prefix = os.path.dirname(self.key)
242+
if prefix:
243+
for obj in self.s3bucket().objects.filter(Prefix=prefix):
241244
key = self.cache_key(self._local_suffix_from_key(obj.key))
242245
cache.mtime[key] = Mtime(storage=obj.last_modified.timestamp())
243246
cache.size[key] = obj.size

0 commit comments

Comments
 (0)