Skip to content

Support directory references in <filepath@store> #1410

@dimitri-yatsenko

Description

@dimitri-yatsenko

Feature Proposal

Extend <filepath@store> to reference existing directories (folders with files), not just individual files.

Motivation

<filepath@store> currently only supports individual files. Users who need to reference existing directory structures in a store (e.g., multi-file recordings, dataset folders managed by acquisition software) must use <object@store>, which copies/uploads content — the opposite of filepath's "reference only" philosophy.

Directory support would be consistent with filepath's design: no copying, no lifecycle management, just a path reference.

Current Behavior

  • FilepathCodec.encode() hardcodes is_dir: False
  • StorageBackend.exists() uses Path.is_file() for local stores, rejecting directories
  • Inserting a directory path raises FileNotFoundError

Proposed Changes

1. storage.py — Add isdir(), widen exists()

  • Add isdir(remote_path) method: Path.is_dir() for local, fs.isdir() for cloud
  • Change exists() from Path.is_file() to Path.exists() for local protocol (safe — the only other caller, hash_registry, operates under _hash/ paths which filepath rejects as reserved)

2. filepath.py — Detect directories in encode()

  • After confirming path exists, call backend.isdir(path) to detect type
  • For directories: compute size as sum of contained files, track item_count
  • Set is_dir dynamically instead of hardcoded False

3. objectref.py — Graceful _verify_folder() without manifest

_verify_folder() currently requires a .manifest.json (created by object@). Filepath directories won't have one. Degrade gracefully: if no manifest exists, just verify the directory itself exists.

4. What does NOT change

  • decode() — already returns ObjectRef.from_json() which reads is_dir from metadata
  • ObjectRef directory methods (listdir(), walk(), download()) — already implemented and gated on is_dir
  • validate() — already accepts str/Path
  • Reserved section and filepath_prefix validation — works the same for directories

Files to Modify

  1. src/datajoint/storage.py
  2. src/datajoint/builtin_codecs/filepath.py
  3. src/datajoint/objectref.py
  4. tests/unit/test_codecs.py

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions