Implementation of DICOMs3Client based on DICOMwebProtocol

[hackermd]

@pieper @chafey As discussed in #56, it would be great to implement a DICOMs3Client based on DICOMwebProtocol.

In general, I think we should consider the following aspects for the design of an S3 client:

1. Index of metadata for efficient query: DICOM attributes needed for response payloads of DICOMweb Search Transaction (Study Resource, Series Resource, and Instance Resource)
2. Offset to pixel data and individual frames for efficient frame retrieval: for natively encoded Pixel Data, the client can just compute the offset of individual frames, but for an encapsulated Pixel Data element, the client would require the Basic Offset Table or Extended Offset Table
3. Read and write: DICOMweb Store Transaction

(1) I've been thinking that we could put the metadata into the S3 Object Metadata. The PUT request header is limited to 8 KB in size, but that should be sufficient for the metadata. This could also include the offset to the Pixel Data, Float Pixel Data, or Double Float Pixel Data element (see 2).

(2) If the Pixel Data element contains a Basic Offset Table item, then the client could just read its value and cache the offsets locally. Otherwise the client would have to determine the offset of frame items by reading the header of each frame item. That could result in a lot of HTTP calls, so ideally we would make sure that the Basic Offset Table gets included into image data sets.

(3) To support writes (including parallel writes), we shouldn't store any aggregate study- or series-level information such as Number of Study Related Series in the object metadata, but dynamically compute it client-side.

What do you think?

[hackermd]

cc @cgorman @CPBridge

[chafey]

It isn't clear to me what you are trying to do here - can you ELI5? An architecture diagram would help (sorry I am not familiar with this project at all)

[chafey]

As @pieper mentioned, I am working on a DICOMweb based archive format as well as a serverless DICOMweb implementation. Here are some links:
https://github.com/chafey/dicomweb-archive-direct
https://github.com/chafey/dicomp10-to-dicomweb-js
https://github.com/chafey/dicomweb-dump

[ntenenz]

@chafey, (AFAIK) this would be a python client for fetching data from a serverless DICOMweb implementation (rather than specifying paths directly).

[chafey]

@ntenenz i am still confused - this is a client library which speaks dicomweb - shouldn't it work as is with any dicomweb implementation regardless of its architecture? I feel like I am missing something here

[ntenenz]

I don't have enough context on the proposal to answer your question intelligently (does it match the existing DICOMweb spec and simply leverage a different form of implementation via static S3 blobs?). @pieper referenced it here: #56 (comment), so perhaps he's best positioned to discuss what he's thinking?

[chafey]

The concept is still coming together, but the general idea is to leverage object storage as much as possible for scalability. Storing pregenerated DICOMweb responses in object storage enables many access use cases and layering on serverless compute (lambda, dynamodb, object lambda) to implement the other DICOMweb functionality. This has been a regular topic of the monday morning meetings: https://github.com/chafey/medical-imaging-community

[ntenenz]

I've only been able to attend a couple meetings thus far due to a standing conflict. That being said, if the proposal ends up being API-compatible with the existing DICOMweb standard (i.e., a new method of implementation), then the existing DICOMwebClient would be sufficient. If not however, a new client would be required.

[chafey]

In terms of this specific issue/request, I suppose another approach could be to implement the existing API, but have it run directly use AWS (S3, maybe other services). I am not sure if this is really worth it though as we should be able to get most if not all of DICOMweb implemented using servless as described above..

[pieper]

I think the pre-computed metadata works for archives (rarely changing data, like IDC where we plan monthly versions).

@hackermd mentioned he wanted to use this client in a dynamic environment and was concerned about keeping static content in sync.

[chafey]

We do want to trigger rebuilding of the metadata any time the source data changes so that shouldn't be a problem. Another option is to store DICOM P10 in S3 with wado-uri paths and then implement DICOMweb using serverless on top of it. It wouldn't be as fast, but it should be just as scalable.

[hackermd]

@chafey the DICOMwebClient class is a Python client for interfacing with a DICOMweb server over HTTP. The new DICOMfileClient allows users to access DICOM data sets that are stored locally as DICOM Part10 files via the same Python API (DICOMwebProtocol). As a next step, I would like to extend this concept to DICOM data sets stored in an S2 bucket.