w3c / webcodecs

WebCodecs is a flexible web API for encoding and decoding audio and video.

Home Page:https://w3c.github.io/webcodecs/

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Explicit frame drop signal from VideoEncoder

Djuffin opened this issue · comments

Currently the spec doesn't specify any way for VideoEncoder to signal that a frame has been dropped due to realtime mode constraints.
I think it would be useful to have such a signal.
Possible options:

  1. the output callback should be called with an empty encoded chunk
  2. the output callback should be called with a new flag set in EncodedVideoChunkMetadata

This came up after WPT test change that accounts for frame drops.

On the other hand, 2 existing implementations drop frames silently, so using the regular output callback as is will be a breaking change. If we ever do this we'll need a VideoEncoderConfig option to turn it on.
Let's wait and see if it actually needed by people building RTC apps.

At least Meet is interested in drop frame notifications.

Currently when a real time encoder drops a frame clients don’t get any signal from VideoEncoder. If they really want to know that a frame was dropped they need to look at the timestamps of the encoded video chunks.

This issue can be addressed by a new flag in the encoder config that would cause the encoder to call EncodedVideoChunkOutputCallback for dropped frames with an empty data buffer in the encoded chunk.

partial dictionary VideoEncoderConfig {
	boolean notifyAboutFrameDrops = false;
}