Failed to extract metadata: Indexing elements must be in increasing order

[chrishalcrow]

Hello,

I've made some nwb files and am trying to upload them to DANDI using dandi upload. I have three of them. Two upload just fine. The other one hits this error trace:

2026-05-28T16:06:11+0100 [WARNING ] dandi 923713:127939540670144 Failed to extract NWB metadata from /home/nolanlab/fromgit/nolanlab_nwb/001854/sub-M22/sub-M22_ses-D34OF2_ecephys+behavior.nwb: TypeError: Indexing elements must be in increasing order
2026-05-28T16:06:11+0100 [ERROR   ] dandi 923713:127939540670144 Error uploading /home/nolanlab/fromgit/nolanlab_nwb/001854/sub-M22/sub-M22_ses-D34OF2_ecephys+behavior.nwb:
Traceback (most recent call last):
  File "/home/nolanlab/fromgit/nolanlab_nwb/.venv/lib/python3.13/site-packages/dandi/upload.py", line 392, in process_path
    metadata = dfile.get_metadata(
               ~~~~~~~~~~~~~~~~~~^
        digest=file_etag, ignore_errors=allow_any_path
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    ).model_dump(mode="json", exclude_none=True)
    ^
  File "/home/nolanlab/fromgit/nolanlab_nwb/.venv/lib/python3.13/site-packages/dandi/files/bases.py", line 494, in get_metadata
    metadata = nwb2asset(self.filepath, digest=digest)
  File "/home/nolanlab/fromgit/nolanlab_nwb/.venv/lib/python3.13/site-packages/dandi/metadata/nwb.py", line 149, in nwb2asset
    metadata = get_metadata(nwb_path)
  File "/home/nolanlab/fromgit/nolanlab_nwb/.venv/lib/python3.13/site-packages/fscacher/cache.py", line 164, in fingerprinter
    ret = fingerprinted(*args, **kwargs_)
  File "/home/nolanlab/fromgit/nolanlab_nwb/.venv/lib/python3.13/site-packages/joblib/memory.py", line 607, in __call__
    return self._cached_call(args, kwargs, shelving=False)[0]
           ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/nolanlab/fromgit/nolanlab_nwb/.venv/lib/python3.13/site-packages/joblib/memory.py", line 562, in _cached_call
    return self._call(call_id, args, kwargs, shelving)
           ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/nolanlab/fromgit/nolanlab_nwb/.venv/lib/python3.13/site-packages/joblib/memory.py", line 832, in _call
    output = self.func(*args, **kwargs)
  File "/home/nolanlab/fromgit/nolanlab_nwb/.venv/lib/python3.13/site-packages/fscacher/cache.py", line 100, in fingerprinted
    return f(path, *args, **kwargs)
  File "/home/nolanlab/fromgit/nolanlab_nwb/.venv/lib/python3.13/site-packages/dandi/metadata/nwb.py", line 106, in get_metadata
    meta.update(_get_pynwb_metadata(r))
                ~~~~~~~~~~~~~~~~~~~^^^
  File "/home/nolanlab/fromgit/nolanlab_nwb/.venv/lib/python3.13/site-packages/dandi/pynwb_utils.py", line 279, in _get_pynwb_metadata
    session_duration = _get_session_duration(nwb)
  File "/home/nolanlab/fromgit/nolanlab_nwb/.venv/lib/python3.13/site-packages/dandi/pynwb_utils.py", line 350, in _get_session_duration
    start = float(np.min(np.r_[st_data[0], st_data[idxs[:-1]]]))
                                           ~~~~~~~^^^^^^^^^^^
  File "/home/nolanlab/fromgit/nolanlab_nwb/.venv/lib/python3.13/site-packages/hdmf/container.py", line 1002, in __getitem__
    return self.get(args)
           ~~~~~~~~^^^^^^
  File "/home/nolanlab/fromgit/nolanlab_nwb/.venv/lib/python3.13/site-packages/hdmf/common/table.py", line 61, in get
    return super().get(key)
           ~~~~~~~~~~~^^^^^
  File "/home/nolanlab/fromgit/nolanlab_nwb/.venv/lib/python3.13/site-packages/hdmf/container.py", line 1010, in get
    return self.data[args]
           ~~~~~~~~~^^^^^^
  File "h5py/_objects.pyx", line 56, in h5py._objects.with_phil.wrapper
  File "h5py/_objects.pyx", line 57, in h5py._objects.with_phil.wrapper
  File "/home/nolanlab/fromgit/nolanlab_nwb/.venv/lib/python3.13/site-packages/h5py/_hl/dataset.py", line 893, in __getitem__
    selection = sel.select(self.shape, args, dataset=self)
  File "/home/nolanlab/fromgit/nolanlab_nwb/.venv/lib/python3.13/site-packages/h5py/_hl/selections.py", line 82, in select
    return selector.make_selection(args)
           ~~~~~~~~~~~~~~~~~~~~~~~^^^^^^
  File "h5py/_selector.pyx", line 282, in h5py._selector.Selector.make_selection
  File "h5py/_selector.pyx", line 215, in h5py._selector.Selector.apply_args
TypeError: Indexing elements must be in increasing order

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/nolanlab/fromgit/nolanlab_nwb/.venv/lib/python3.13/site-packages/dandi/upload.py", line 396, in process_path
    raise UploadError("failed to extract metadata: %s" % str(e))
dandi.exceptions.UploadError: failed to extract metadata: Indexing elements must be in increasing order
2026-05-28T16:06:11+0100 [DEBUG   ] urllib3.connectionpool 923713:127941258118976 Starting new HTTPS connection (1): rig.mit.edu:443
2026-05-28T16:06:16+0100 [DEBUG   ] dandi 923713:127941258118976 Caught exception failed to extract metadata: Indexing elements must be in increasing order
Traceback (most recent call last):
  File "/home/nolanlab/fromgit/nolanlab_nwb/.venv/lib/python3.13/site-packages/dandi/upload.py", line 392, in process_path
    metadata = dfile.get_metadata(
               ~~~~~~~~~~~~~~~~~~^
        digest=file_etag, ignore_errors=allow_any_path
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    ).model_dump(mode="json", exclude_none=True)
    ^
  File "/home/nolanlab/fromgit/nolanlab_nwb/.venv/lib/python3.13/site-packages/dandi/files/bases.py", line 494, in get_metadata
    metadata = nwb2asset(self.filepath, digest=digest)
  File "/home/nolanlab/fromgit/nolanlab_nwb/.venv/lib/python3.13/site-packages/dandi/metadata/nwb.py", line 149, in nwb2asset
    metadata = get_metadata(nwb_path)
  File "/home/nolanlab/fromgit/nolanlab_nwb/.venv/lib/python3.13/site-packages/fscacher/cache.py", line 164, in fingerprinter
    ret = fingerprinted(*args, **kwargs_)
  File "/home/nolanlab/fromgit/nolanlab_nwb/.venv/lib/python3.13/site-packages/joblib/memory.py", line 607, in __call__
    return self._cached_call(args, kwargs, shelving=False)[0]
           ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/nolanlab/fromgit/nolanlab_nwb/.venv/lib/python3.13/site-packages/joblib/memory.py", line 562, in _cached_call
    return self._call(call_id, args, kwargs, shelving)
           ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/nolanlab/fromgit/nolanlab_nwb/.venv/lib/python3.13/site-packages/joblib/memory.py", line 832, in _call
    output = self.func(*args, **kwargs)
  File "/home/nolanlab/fromgit/nolanlab_nwb/.venv/lib/python3.13/site-packages/fscacher/cache.py", line 100, in fingerprinted
    return f(path, *args, **kwargs)
  File "/home/nolanlab/fromgit/nolanlab_nwb/.venv/lib/python3.13/site-packages/dandi/metadata/nwb.py", line 106, in get_metadata
    meta.update(_get_pynwb_metadata(r))
                ~~~~~~~~~~~~~~~~~~~^^^
  File "/home/nolanlab/fromgit/nolanlab_nwb/.venv/lib/python3.13/site-packages/dandi/pynwb_utils.py", line 279, in _get_pynwb_metadata
    session_duration = _get_session_duration(nwb)
  File "/home/nolanlab/fromgit/nolanlab_nwb/.venv/lib/python3.13/site-packages/dandi/pynwb_utils.py", line 350, in _get_session_duration
    start = float(np.min(np.r_[st_data[0], st_data[idxs[:-1]]]))
                                           ~~~~~~~^^^^^^^^^^^
  File "/home/nolanlab/fromgit/nolanlab_nwb/.venv/lib/python3.13/site-packages/hdmf/container.py", line 1002, in __getitem__
    return self.get(args)
           ~~~~~~~~^^^^^^
  File "/home/nolanlab/fromgit/nolanlab_nwb/.venv/lib/python3.13/site-packages/hdmf/common/table.py", line 61, in get
    return super().get(key)
           ~~~~~~~~~~~^^^^^
  File "/home/nolanlab/fromgit/nolanlab_nwb/.venv/lib/python3.13/site-packages/hdmf/container.py", line 1010, in get
    return self.data[args]
           ~~~~~~~~~^^^^^^
  File "h5py/_objects.pyx", line 56, in h5py._objects.with_phil.wrapper
  File "h5py/_objects.pyx", line 57, in h5py._objects.with_phil.wrapper
  File "/home/nolanlab/fromgit/nolanlab_nwb/.venv/lib/python3.13/site-packages/h5py/_hl/dataset.py", line 893, in __getitem__
    selection = sel.select(self.shape, args, dataset=self)
  File "/home/nolanlab/fromgit/nolanlab_nwb/.venv/lib/python3.13/site-packages/h5py/_hl/selections.py", line 82, in select
    return selector.make_selection(args)
           ~~~~~~~~~~~~~~~~~~~~~~~^^^^^^
  File "h5py/_selector.pyx", line 282, in h5py._selector.Selector.make_selection
  File "h5py/_selector.pyx", line 215, in h5py._selector.Selector.apply_args
TypeError: Indexing elements must be in increasing order

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/nolanlab/fromgit/nolanlab_nwb/.venv/lib/python3.13/site-packages/dandi/cli/base.py", line 126, in wrapper
    return f(*args, **kwargs)
  File "/home/nolanlab/fromgit/nolanlab_nwb/.venv/lib/python3.13/site-packages/dandi/cli/cmd_upload.py", line 113, in upload
    upload_(
    ~~~~~~~^
        paths,
        ^^^^^^
    ...<9 lines>...
        validation_log_path=companion,
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/home/nolanlab/fromgit/nolanlab_nwb/.venv/lib/python3.13/site-packages/dandi/upload.py", line 505, in upload
    raise upload_err
  File "/home/nolanlab/fromgit/nolanlab_nwb/.venv/lib/python3.13/site-packages/dandi/upload.py", line 396, in process_path
    raise UploadError("failed to extract metadata: %s" % str(e))
dandi.exceptions.UploadError: failed to extract metadata: Indexing elements must be in increasing order

I don't know what this means, or how to proceed.

Strange things:

• dandi validate runs fine.
• the metadata is generated in the same way for all nwb files, so I don' t understand why only one of them would fail.
• When I open the nwb file using pynwb, it looks fine and all the metadata looks reasonable.

Any advice very welcome!

[CodyCBakerPhD]

Looks like the error stems from the attempt to calculate the session duration:

  File "/home/nolanlab/fromgit/nolanlab_nwb/.venv/lib/python3.13/site-packages/dandi/metadata/nwb.py", line 106, in get_metadata
    meta.update(_get_pynwb_metadata(r))
                ~~~~~~~~~~~~~~~~~~~^^^
  File "/home/nolanlab/fromgit/nolanlab_nwb/.venv/lib/python3.13/site-packages/dandi/pynwb_utils.py", line 279, in _get_pynwb_metadata
    session_duration = _get_session_duration(nwb)
  File "/home/nolanlab/fromgit/nolanlab_nwb/.venv/lib/python3.13/site-packages/dandi/pynwb_utils.py", line 350, in _get_session_duration
    start = float(np.min(np.r_[st_data[0], st_data[idxs[:-1]]]))

Would you be able/willing to share this problematic file directly so I can investigate where the issue might be?

@yarikoptic How 'fail safe' should the metadata extraction like this be on the DANDI CLI side (should it just return None if there is a failure like this and not block upload?)

[chrishalcrow]

Thanks Cody, I found your e-mail address and have sent you the file.
Interesting. How does it compute this? There is an external video file, which might be unusual?

[CodyCBakerPhD]

The logic can be found at

dandi-cli/dandi/pynwb_utils.py

Line 288 in 8fa9188

def _get_session_duration(nwb: pynwb.NWBFile) -> float | None:

video timestamps / rate are still internal to the file; an indexing error would imply something about a table usually

[CodyCBakerPhD]

The file has been obtained and I am investigating

[chrishalcrow]

Hello, I found the bug - thanks for pointing me in the right direction @CodyCBakerPhD !

If st_data is a hdmf.common.table.VectorData, it must be indexed with elements in increasing order (the error message was super descriptive!). E.g. this is fine

st_data[np.array([1,2,3])

while this is not

st_data[np.array([2,2,3])

Why is this happening with my data? We sort three experimental sessions (openfield, then vr, then openfield) together, then split the results into three individual nwb files. So there's a unit which did not spike at all in the second openfield session (Note: probably an algorithm mistake or noise). idxs is a list of the index position of the first spike in each spike train. Since one spike train has zero spikes, two units have the same first spike index, leading to this issue.

Two possible fixes:

• I get rid of the zero spike unit before adding the the nwb file (I don't particularly want to do this - I like that we have explicitly kept the fact that "unit 32 has zero spikes" in the nwb file)
• You use np.unique to avoid this repetition issue. (ie line 352 in pynwb_utils becomes start = float(np.min(np.r_[st_data[0], st_data[np.unique(idxs[:-1])]]))) [EDIT: also need to replace idxs a few lines down: end = float(np.max(st_data[np.unique(idxs) - 1]))]

[CodyCBakerPhD]

Since the NWB Validator and Inspector both say it is OK to have empty rows (diff of indexes equal to zero), I would indeed agree this is an issue with this script - will send a PR soon

A few other items learned from the file you shared:

NWB Inspector flags the following:

1  BEST_PRACTICE_SUGGESTION
===========================

1.1  sub-M22_ses-D34OF2_ecephys+behavior.nwb: check_description - 'VectorData' object with name 'group'
       Message: Description ('No description.') is a placeholder.

1.2  sub-M22_ses-D34OF2_ecephys+behavior.nwb: check_col_not_nan - 'Units' object with name 'units'
       Message: Column 'spread' might have all NaN values. Consider removing it from the table.

1.3  sub-M22_ses-D34OF2_ecephys+behavior.nwb: check_timestamp_of_the_first_sample_is_not_negative - 'ImageSeries' object at location '/acquisition/top_capture'
       Message: Timestamps should not be negative. This usually indicates a temporal misalignment of the data. It is recommended to align the `session_start_time` or `timestamps_reference_time` to be the earliest time value that occurs in the data, and shift all other signals accordingly.

I feel less strongly than others about issue 1.3, though if you ever translate this to BIDS using the new BEP032 extension, I think it might be a bigger deal (negative timestamps over there are more prohibited as I recall) - so that might change as we begint to harmonize the two

Also, did you consider using https://github.com/rly/ndx-pose for the pose tracking rather than separate raw time series per node? I'd strongly recommend it

[CodyCBakerPhD]

Also, even if not adding any high-pass or LFP data, please consider adding some device metadata for how the ecephys was captured (we see spikes, but how did you collect/detect those?)

[CodyCBakerPhD]

@chrishalcrow Hi Chris, you should be able to download the latest version of the client to resolve the issue (pip install -U dandi)