Interaction with Congestion Control

[aboba]

The Virtual Reality Gaming use case may potentially involve adding metadata to the encoded frame. The metadata could be substantial (hundreds of bytes).

Similarly, there are accessibility scenarios (captioning) in which the captions might be sent along with the frames.

So the question arises as to the interaction with congestion control in these scenarios. When adding to (or even substracting from?) the size of the encoded frame, is there a way to properly interact with congestion control?

[alvestrand]

The streams mechanism allows pushback to avoid buffer overrun, but doesn't allow any other mechanism for congestion signalling.
Such feedback is one of the core usages for a feedback channel as requested in #90

[youennf]

Two potential sub-issues:

• Make sure that implementors handle size increase triggered by the transform.
• Should the transform itself be able to understand how much space it can use. If so, we might need to expose some values about encoder bitrate, encoder+transform allocated bitrate

[alvestrand]

This is even more intense when you consider single-ended use cases as in #106.