When is encoder conversion necessary?

[Yahweasel]

The spec specifies that an implementation MUST support pixel format conversion for rendering and MUST support sample format conversion at least to float32. The spec thus makes it clear that everything has to be convertible to float32 (samples) and RGBA (pixels). But, all that VideoEncoder and AudioEncoder have to say on the matter is «Attempt to use [[codec implementation]] to encode frameClone according to options.». In practice, Chrome will happily convert anything to the pixel/sample format used internally by the selected codec. It'll even happily scale for you! But, it won't resample or mix/split tracks? Poking different AudioDatas and VideoFrames into it, I can get it to do some things and not to do others, but it's not at all clear what it will and won't do.

The question is, what aspects of this internal conversion during encoding are MUST, SHOULD, or implementation-specific? Right now, it seems like it's all implementation-specific, and if that's so, it would be nice to document that fact in the spec. But, surely it MUST at least support pixel and sample format conversion, as those are both codec-specific. The spec should make this clear.

(Apologies for all the issue posts lately, but the upshot is that there are now two implementations of the spec, so that's a good thing, right?)

[sandersdan]

The spec tries to impose minimal requirements. We require the ability to render but not necessarily RGBA, although it's likely that RGBA or BGRA will be supported in any UA that would be able to implement WebCodecs due to the inclusion of CanvasImageSource APIs.

Scaling is perhaps that largest requirement WebCodecs imposes on encoder implementations, but it's implied by SVC support and we didn't have a VideoFrame.scale() API design ready.

There was a plan to add a pixel format feature detection ability (something like isConfigSupported() for VideoFrame) but so far there has not been demand and therefore little feedback to guide the design. Similar to AudioData, we can also add a format parameter on copyTo().

Conversions for encoding has gone through a few iterations due to late discovery of some flaws in the design of color space configuration. The current text allows encoders to perform conversions but makes no requirements, encoders are allowed to reject unsupported conversions.

In practice the supported codec profiles are all currently YUV 4:2:0 and I expect all implementations to be able to convert to that. We would like to add at least 4:4:4 and 4:2:2 but there is not yet a proposed design for the VideoEncoderConfig.

Audio conversion to f32 is required for compatibility with WebAudio AudioBuffer. I don't know if we'll increase that requirement but I do expect Chrome to support a full set of conversions eventually as there does seem to be demand.

[Yahweasel]

I feel that's a bit loose, and the spec could stand to demand this simple, minimum requirement: encode must not reject any valid sample/pixel format, and thus must perform sample/pixel format conversion as needed for the codec. This seems particularly important since no part of the configuration communicates to the user what internal formats are used, so you would have to know from external expertise what will and won't work. And, Chrome is already doing it anyway, so codifying that in the spec doesn't require code changes :)

I believe it would technically be spec compliant for me to say "What, this frame is RGB? No way!" and demand the user convert everything to YUV420, without even communicating to said user in any way that YUV420 is the right format to use...

(FYI, libavjs-webcodecs-polyfill is currently more generous than Chrome. It will rescale, transform sample/pixel formats, and resample and mix or split channels.)

[sandersdan]

I believe it would technically be spec compliant for me to say "What, this frame is RGB? No way!"

That's correct.

encode must not reject any valid sample/pixel format, and thus must perform sample/pixel format conversion as needed for the codec

This requirement is reasonable, it makes sense that an implementation should be able to encode in any format it advertises codec support for, and should be able to do so with any frame it can render.

As a first step I'd recommend working on the feature detection side so that a polyfill can accurately fill only the missing features and easily support multiple browser versions / implementations.

[Yahweasel]

As a first step I'd recommend working on the feature detection side so that a polyfill can accurately fill only the missing features and easily support multiple browser versions / implementations.

I largely agree, and the degree of communication back to the user is a bit lackluster right now (there's isn't even a way to know what pixel format a video encoder is using...), but for the titular issue, just mandating the suggested baseline (must be able to convert from any valid format to whatever is used by the codec) would free a polyfill of needing to fill anything in :) . I suspect that any basically sensible implementation would perform these conversions anyway, because it's absurdly impractical not to.

[chcunningham]

@sandersdan and I chatted about this just now.

the spec could stand to demand this simple, minimum requirement: encode must not reject any valid sample/pixel format, and thus must perform sample/pixel format conversion as needed for the codec.

We're supportive. Generally we like the contract that: if you can construct a VideoFrame, you can render it and you can encode it.

The rendering half of that is already true in Chrome, but our encoders currently reject I444 and I422 frames. At a glance we don't see any blocking issues to add support for those (or converting where needed). We'll investigate it more thoroughly before proposing a spec change.

Aside: for VideoFrame speicifically, the current (implied) requirement is that all implementers must support construction of VideoFrame's with any VideoPixelFormat. As we add new formats, there will be implementations that are behind the spec, but that is easily discoverable (failed construction). For now we don't see a need to add a VideoFrame.isFormatSupported() type API, but we can fallback to that if needed.

[chcunningham]

@dalecurtis @Djuffin

[Yahweasel]

Aha! I didn't know there was some conversion that Chrome didn't support. So much for my effort to figure out what is and isn't there 🤣

Whatever is ultimately landed on, greater clarity on this in the spec would be appreciated. Even a "MAY convert" :)

[chcunningham]

Whatever is ultimately landed on, greater clarity on this in the spec would be appreciated. Even a "MAY convert" :)

Agreed.

[Yahweasel]

Aside: for VideoFrame speicifically, the current (implied) requirement is that all implementers must support construction of VideoFrame's with any VideoPixelFormat. As we add new formats, there will be implementations that are behind the spec, but that is easily discover-able (failed construction). For now we don't see a need to add a VideoFrame.isFormatSupported() type API, but we can fallback to that if needed.

Just to respond to this, I agree; the constructor throwing an exception is plenty for feature detection. I am a non-contributing external whiner, so my agreement is neither needed nor solicited, but I agree nonetheless ;)

[sandersdan]

I am a non-contributing external whiner

FWIW, I'm very interested in your experience building a polyfill and absolutely count it as a WebCodecs contribution, even if it is external.

[youennf]

What about resolution mismatch between video frames and video encoder configuration?
It seems Chrome is doing some kind of conversion, is that expected? Should it error instead for simplicity?
If there is desire to support conversion, it seems very important to specify how the conversion happens: scaling, cropping, letterboxing...

[sandersdan]

I'm surprised to see that isn't in the spec since it was discussed at length. The consensus at that time was that it would be very common to want to scale frames, we didn't have an efficient scaling API, and we expected encoders to already have access to the most efficient scaling methods.

Chrome's implementation scales the visible rect of the source frame to the requested frame size of the encoder. There is no letterboxing, it is assumed that the configured size is the intended display aspect ratio.

Edit: There may be a legitimate argument for letterboxing; it's what WebRTC does and in theory both the source frame and the encoder have configured pixel aspect ratios. It does require making more decisions on behalf of the app, though.

[youennf]

There is no letterboxing, it is assumed that the configured size is the intended display aspect ratio.

In which case, if aspect ratio does not match, Chrome would error the encoder, or dump the frames?

There may be a legitimate argument for letterboxing; it's what WebRTC does and in theory both the source frame and the encoder have configured pixel aspect ratios.

I am unsure what WebRTC does here. It could rescale or reconfigure the encoder.