Allowing decoupling of decoding and demuxing for images

[padenot]

There are a number of container formats that can hold a variety of different codecs’ coded data as well as the possibility of the same coded data appearing in a variety of containers.

If a demuxing API is later considered (as it's not infrequently brought up during calls and discussions with developers, #24 for tracking, certainly less urgent than decoders!), this could be handled by desugaring the image decoding API maybe ?

[dalecurtis]

Are you aware of any containers here that are not ISOBMFF? If not, AFAIK images are always stored muxed, only multi-frame content based on a multi-frame codec like HEVC, AV1, or H264 is stored demuxed -- but those are served by VideoDecoder.

So my take is that we probably don't need any changes to the ImageDecoder API, just a demuxing API and clients will use VideoDecoder for things that demuxed codecs and ImageDecoder for things that are still complete image files.

[padenot]

@baumanj, if you can maybe explain more precisely what you had in mind with this?

[baumanj]

WebP uses RIFF, which is not ISOBMFF. Also, since there's lots of useful information that can be derived from containers without ever doing decoding, it seems useful to make a separation to allow for greater forward compatibility. Add that to the fact that IP restrictions make formats like HEIC undecodable on many platforms, but the same container interpretation code could give access to useful metadata. Finally, I don't assume that ISOBMFF, which is fairly complex, not-free and heavyweight will be the only container any new formats ever use, so separating the ability to decode the image data one may store within it today would increase flexibility and innovation in the future.

[dalecurtis]

To me that sounds like you're just advocating for a containers API, which I agree is a nice value add. However, I can't think any concrete changes that we'd make to the ImageDecoder API which would help this use case. Can you give some examples of what you're thinking about?

I.e., in a world with a browser provided RIFF, ISOBMFF, $future_format demuxer that vends demuxed samples, what would we add to the ImageDecoder API that's not there already? Since we take a raw array buffer + mime type, I think we can decode whatever may come if it's appropriate to home it on ImageDecoder instead of say VideoDecoder.

[baumanj]

I'm not suggesting this needs to be part of the ImageDecoder API itself, but rather that implicitly including demuxing as a black box inside the decoder API makes it less useful and forward-looking. Wouldn't it be better to keep the decoder API smaller and simpler and break out the demuxing concerns to an API which is appropriately focused on the details relevant to containers?

For one, the ability to get metadata about the image(s) without doing the decode seems like a natural one. Given containers with the ability to have multiple images encoded at different sizes, color spaces, etc., there are many questions that could be asked of a container that I don't think it makes sense to add to a decoder interface which is more appropriately streamlined to f(coded image) → pixel data. Also, the same container could conceivably contain multiple coded image types only some of which the platform has decoder support for. How would the Image Decoding API handle that?

[dalecurtis]

Images and their containers are highly coupled in the by-far most common cases (GIF, JPEG, etc), so anything that separates those processes seems bound to incur more complexity and not less. I don't think it makes sense in the common case to force authors to go through a separate demuxing and decoding phase for images. Aside from the complexity this feels like a performance issue in the single image case. I talk about this in the explainer.

I 100% agree that a containers API and metadata extraction is very useful and something we should consider going forward, but I think it's orthogonal to what we have in the ImageDecoder API. I'm absolutely on board with exposing what metadata makes sense through the tracks API that ImageDecoder has. In your example we'd only expose the tracks the platform can decode.

[baumanj]

Images and their containers are highly coupled in the by-far most common cases

I definitely agree that is the case with the legacy formats, but since WebP, it looks like things are moving in a different direction. All the formats developed in the past decade likely to see broad use that I'm aware of (HEIC, AVIF, JPEG-XL, etc.) decouple the container from the codec. Also, it looks like images are tending to become more video-like in their development, so I would expect to see more of the mixing and matching of container and codec that's so common in that space.

I don't think it makes sense in the common case to force authors to go through a separate demuxing and decoding phase for images

I agree. There should definitely be a simple path for the common case of inputting the whole container and getting out something renderable. I'm just advocating for the inclusion of container-level awareness as a first class abstraction here since I believe it will smooth adoption of new formats.

[dalecurtis]

It sounds like you're just in favor of issue #24 then. Do you have any concrete proposals for how we should change ImageDecoder? If not I think we should close this issue and move discussion to #24.

Per #205 (comment) I think ImageDecoder can already do everything we'd want to do in a post-containers API world and VideoDecoder cover any gaps in the demuxed packets case.

[baumanj]

Do you have any concrete proposals for how we should change ImageDecoder?

I think the addition of a metadata query would be very useful. That wouldn't strictly be part of a container API since it's also germane to JPEG, PNG and other highly coupled formats like you mentioned, but going forward, that should almost certainly be a container-level operation to extract metadata like dimensions, bitdepth, colorspace, existence of alpha, exif data, etc.

VideoDecoder cover any gaps in the demuxed packets case

Can you elaborate?

[dalecurtis]

I think the addition of a metadata query would be very . That wouldn't strictly be part of a container API since it's also germane to JPEG, PNG and other highly coupled formats like you mentioned, but going forward, that should almost certainly be a container-level operation to extract metadata like dimensions, bitdepth, colorspace, existence of alpha, exif data, etc.

As mentioned above. I'm in total agreement on adding such metadata, I designed the ImageTrack interface for such things. Today it's just frame count and other simple data, but I envision it to hold all the things you're talking about and more. Metadata is decoded automatically in the current spec language. It is indeed a separate step from decoding.

So apologies, but I'm still confused on what your request is for the current API shape. Are you just suggesting those metadata fields? That seems a minor addition and less of a fundamental shape thing. Can you elaborate more?

VideoDecoder cover any gaps in the demuxed packets case

Can you elaborate?

I.e., in some future world where we have a WebContainers API (which we're all in favor of, just maybe not as part of WebCodecs or at least not in v1), you'd pass a bytestream to said container API and after some track selection, in addition to metadata, you'd get demuxed packets of codec XYZ. If that codec happens to be a video one, you can use the VideoDecoder API as it stands today. If it's an image one, there's no reason the current API shape can't accept those as a ReadableStream of bytes or typed chunks.

[baumanj]

So apologies, but I'm still confused on what your request is for the current API shape. Are you just suggesting those metadata fields? That seems a minor addition and less of a fundamental shape thing. Can you elaborate more?

I agree the shape is reasonable (providing metadata via the ImageTrack interface), but I didn't realize that was the intention because it doesn't have most of the kind of metadata I'd assumed would be included (dimensions, color space, transforms, etc.). Is there a reason that can't be included now?

[dalecurtis]

Sorry that wasn't clearer! The only reason is that we were trying to be conservative in what was exposed currently. We dropped exif rotation for now since we couldn't agree on the best way to expose it. Dimensions seems like an easy and non-controversial one to add now. Are there any others you'd prefer would be in WebCodecs v1?

Color space and transforms will need to wait until we figure out the right language for describing them, which may not be a part of v1 since we'll probably need to consider interplay between the new canvas color spaces and such.

[chcunningham]

Triage note: tentatively marking 'extension' as recent discussion proposes new attributes/metadata.