Need for modifying metadata

[alvestrand]

In some applications there is the need to modify the metadata, but the spec only allows a GetMetadata operation.
This is inconvenient if the transform operation means that the metadata really should change.

A PutMetadata function would make it simple.

[nisse-work]

Another attribute that would be very nice to have is some kind of id that can be matched to unencoded frames processed upstream using the insertable streams api. It seems one reasonable candidate for an id (if we don't want to add a new one) is VideoFrame.timestamp (https://developer.mozilla.org/en-US/docs/Web/API/VideoFrame). It would be nice if this value was propagated through the encoder and attached to the RTCEncodedVideoFrame or RTCEncodedVideoFrameMetadata.

Example use: Updating csrc list (part of RTCEncodedVideoFrameMetadata) to match any mixing/compositing done when processing this particular frame upstream to the encoding.

[nisse-work]

Hmm, maybe I was confused; I thought we only had RTP timestamp in RTCEncodedVideoFrame, but it also has this timestamp field: https://w3c.github.io/webrtc-encoded-transform/#dom-rtcencodedvideoframe-timestamp

It's poorly documented, but if it's specified to equal the VideoFrame.timestamp of the corresponding unencoded frame, it's should be good enough, I think.

[fippo]

have you seen https://chromium-review.googlesource.com/c/chromium/src/+/3654171 which clarified things a bit?
(it is a mess...)

[nisse-work]

Confusing indeed. Seems this was discussed previously on #116.

Exposing capture time should be reasonably easy on the send side (prototype cl to do that for video: https://webrtc-review.googlesource.com/c/src/+/265403).

I don't really understand the issues in getting that kind of timestamp on the receive side. If it can't easily be set to useful value on receive side, making it optional would be an improvement for use cases I have in mind.

[fippo]

we can't change the definition of timestamp now :-(
👍 for exposing an optional absoluteCaptureTimestamp (present if the abs-capture-time header extension is used?) on RTCEncodedVideoFrameMetadata though (it looks like some code surrounding your CL already does something similar?)

[nisse-work]

So what if we add two new attributes to RTCEncodedVideoFrameMetadata:

• rtpTimestamp (required, representing the uint32_t RTP timestamp)

• timestamp (optional, intended to match VideoFrame.timestamp. It's clear to me how that should work on the send side. Less clear on the receive side, could perhaps be conditionally populated depending on presence of absolute capture time extension).

Does that make sense? We could then consider deprecating the current timestamp attribute which isn't in the "metadata".

[fippo]

Less clear on the receive side, could perhaps be conditionally populated depending on presence of absolute capture time extension

I'd go for captureTime -- similar to how it works in requestVideoFrameCallback - @drkron can explain the details. Downside of this is that it is best effort and requires SR or rtrr for clock offset estimation so it won't be available on every frame.

[nisse-work]

I created a pull request, see #137. It seems I may have to create some w3c account, to get it on track for the process?

[dontcallmedom]

indeed @nisse-work although in practice the high-level bit will be your employment situation and its impact on IPR. Please get in touch at dom@w3.org if you want more details.

[fippo]

or talk to @alvestrand ;-)

[nisse-work]

Back to setMetadata. I can make a pull request to add that, but I need some guidance about how it should work.

1. Newbie question: The type dictionary RTCEncodedVideoFrameMetadata listen in the spec, is that a plain javascript dictionary to the application, so that, e.g., it could be constructed as {"width" : 100} ? It seems clear that this should be the argument type for setMetadata, for consistency with the getMetadata method.
2. Return value: The method needs a way to indicate success or failure (might fail, e.g, if application tries to set unknown keys, or invalid values such as width < 0). Should this simply be a bool (true for success), or something more complex? Or should it be an asynchronous operation with promise/callback?
3. Should caller be required to pass a dictionary with all defined attributes present (typical pattern would then be getMetadata, modify one or two attributes, setMetadata)? Or is it fine to pass a partially populated dictionary, and then leave all attributes not mentioned unchanged?
4. Most of the attributes seem rather inappropriate for the application to change. E.g., changing the spatial index or dependency list will likely break decoding at the remote end. Setting csrcs (the use case I have in mind) is the only one that looks reasonably safe. Maybe it's better to define a specific method to change cscrs only? I'm thinking of the case where a webrtc encoder is used; in case the application has it's own encoder (implemented in js or webasm), then the situation is different and the application has to know, and populate, all the attributes of the encoded frame.

[alvestrand]

1. Yes, WebIDL dictionaries are constructible from JS dictionaries. The construction process ignores unknown keys, so unknown keys "can't happen".
2. My suspicion is that setMetadata could be synchronous, but I'm mosty in the "when in doubt, return a promise" camp. I'd like to hear from @guidou whether he could think of a reason for making it async. Failure in JS is mostly signaled by throwing an exception or rejecting a promise, so the function should return "undefined".
3. I'd suggest that the setMetadata fully replaces the current metadata. So get/modify/set would be the normal pattern. Sending in a dictionary with only the changes makes it hard to describe that you want to delete an attribute.
4. If we move a frame between PeerConnections, there may be good reasons for changing the dependency descriptors - for instance, if we need to renumber frames, then DD indexes will need to change too. But if the result doesn't make sense, SetMetadata should throw an error.

[guidou]

Implementing setMetadata requires going deep into the webrtc stack and deal with a number of objects living in different threads. Might be doable synchronously (as getMetadata is), but it would be prudent to try to sketch an implementation before committing to a sync interface.

[youennf]

To sketch this API, we need to identify many use cases, especially if we go with setMetadata.
Going with a dedicated csrc/ssrc mutator might be easier to sell if the specific use case is convincing.

As of sync vs. async, if we have to hop to threads, we usually go async.
But I think we should first look at the actual model.
It seems to me the natural model is that the metadata is only used after the frame is enqueued: at the time of enqueueing the frame, we clone the metadata and this gets used down the pipe. This seems to call for a sync API.
The question of bad metadata is something to take into account.
We should also learn from WebCodec decoders and metadata approach.

[nisse-work]

Implementing setMetadata requires going deep into the webrtc stack and deal with a number of objects living in different threads. Might be doable synchronously (as getMetadata is), but it would be prudent to try to sketch an implementation before committing to a sync interface.

I had a quick look at the webrtc implementation of getMetadata, and it appears to return a reference to internal state, and it appeared to make the assumption that it's immutable (and then I suppose chromium makes a copy when packaging it for js access). Making it mutable and properly synchronized looked a bit tricky at first look.

So maybe we shouldn't require synchronous operation.