Explicit frame drop signal from VideoEncoder
Djuffin opened this issue · comments
Currently the spec doesn't specify any way for VideoEncoder to signal that a frame has been dropped due to realtime
mode constraints.
I think it would be useful to have such a signal.
Possible options:
- the output callback should be called with an empty encoded chunk
- the output callback should be called with a new flag set in EncodedVideoChunkMetadata
This came up after WPT test change that accounts for frame drops.
On the other hand, 2 existing implementations drop frames silently, so using the regular output callback as is will be a breaking change. If we ever do this we'll need a VideoEncoderConfig option to turn it on.
Let's wait and see if it actually needed by people building RTC apps.
At least Meet is interested in drop frame notifications.
Currently when a real time encoder drops a frame clients don’t get any signal from VideoEncoder. If they really want to know that a frame was dropped they need to look at the timestamps of the encoded video chunks.
This issue can be addressed by a new flag in the encoder config that would cause the encoder to call EncodedVideoChunkOutputCallback for dropped frames with an empty data buffer in the encoded chunk.
partial dictionary VideoEncoderConfig {
boolean notifyAboutFrameDrops = false;
}