-
Notifications
You must be signed in to change notification settings - Fork 96
Open
Description
Feature Proposal
Extend <filepath@store> to reference existing directories (folders with files), not just individual files.
Motivation
<filepath@store> currently only supports individual files. Users who need to reference existing directory structures in a store (e.g., multi-file recordings, dataset folders managed by acquisition software) must use <object@store>, which copies/uploads content — the opposite of filepath's "reference only" philosophy.
Directory support would be consistent with filepath's design: no copying, no lifecycle management, just a path reference.
Current Behavior
FilepathCodec.encode()hardcodesis_dir: FalseStorageBackend.exists()usesPath.is_file()for local stores, rejecting directories- Inserting a directory path raises
FileNotFoundError
Proposed Changes
1. storage.py — Add isdir(), widen exists()
- Add
isdir(remote_path)method:Path.is_dir()for local,fs.isdir()for cloud - Change
exists()fromPath.is_file()toPath.exists()for local protocol (safe — the only other caller,hash_registry, operates under_hash/paths which filepath rejects as reserved)
2. filepath.py — Detect directories in encode()
- After confirming path exists, call
backend.isdir(path)to detect type - For directories: compute
sizeas sum of contained files, trackitem_count - Set
is_dirdynamically instead of hardcodedFalse
3. objectref.py — Graceful _verify_folder() without manifest
_verify_folder() currently requires a .manifest.json (created by object@). Filepath directories won't have one. Degrade gracefully: if no manifest exists, just verify the directory itself exists.
4. What does NOT change
decode()— already returnsObjectRef.from_json()which readsis_dirfrom metadataObjectRefdirectory methods (listdir(),walk(),download()) — already implemented and gated onis_dirvalidate()— already accepts str/Path- Reserved section and
filepath_prefixvalidation — works the same for directories
Files to Modify
src/datajoint/storage.pysrc/datajoint/builtin_codecs/filepath.pysrc/datajoint/objectref.pytests/unit/test_codecs.py
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
No labels