Giters

w3c / webcodecs

WebCodecs is a flexible web API for encoding and decoding audio and video.

Home Page:https://w3c.github.io/webcodecs/

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Receiving Uncompressed Webcam Data without Browser Compression

abdikaiym01 opened this issue · comments

commented

Can raw, unaltered data be obtained directly from a webcam without the browser introducing additional compression?

I've attempted to use code like this
""" const stream = await navigator.mediaDevices.getUserMedia( { video: true } );
const track = stream.getVideoTracks()[0]
const processor = new MediaStreamTrackProcessor( track );
const reader = processor.readable.getReader();
reader.read().then( ({ done, value }) => {
let buffer = new Uint8Array(value.allocationSize());
let layout = value.copyTo(buffer);
} """
to access video frames in 'I420' format from my webcam. I want to confirm if the video frame (buffer) retrieved directly from the webcam remains unmodified by the browser's rendering or any built-in web codecs.

Possibly, within the stream, video frames might have undergone encoding processes such as various video codecs before arriving in the buffer as frames in the I420 format?

commented

This is unrelated to Web Codecs, and is about https://github.com/w3c/mediacapture-transform/issues/. Generally, if you don't use a VideoEncoder or VideoDecoder, and there's no EncodedVideoData, there's no compression involved. MediaStreamTrackProcessor only deals with uncompressed data (because that's what you want for processing).

However, there are no guarantees that a camera will produce frames in yuv420 format (e.g. sometimes it's in NV12), but the code snippet you wrote guarantees that you get the frames as raw as possible, in the format the browser uses. You will need to handle other formats if your code is to be run without control of the specifics of the OS / camera.

commented

FYI some cameras can use lossy compression internally.

commented

It's frequent yes (even mjpeg, sometimes "real" video compression), but there is no Web API to be able to consume the "compressed" frames from a camera using getUserMedia(...).

commented

Thank you for your response @padenot, @Djuffin!

How does the process of streaming a webcam to a browser screen typically function, as per my understanding?

  1. Initially, the image arrives in an encoded format, like MJPEG.
  2. Subsequently, the browser decodes the image and initiates its stream, projecting it onto the screen.

There is a significant suspicion that the browser's stream undergoes codec processing to facilitate frame transportation, or there might be other underlying reasons. However, based on your response, it appears that the browser doesn't encode frames within its stream; rather, it transmits decoded frames directly from the camera.

my other question, does the conversion of a frame to a canvas via drawImage keep the exact values of the original frame, or does the transformation potentially introduce modifications through the browser's rendering process?

commented

Canvas typically work in RGB8 format in sRGB color space (except when using e.g. 'p3' and so on), so format conversion and color space conversion can happen if this is not what the camera is outputting frames in. If you draw a video frame that is (for example) in YUV420 to a canvas, pixel format conversion and color space conversion will happen.

HTMLMediaElement, encoders, decoders, etc. can handle lots of different formats, and the specifics of the conversion depends on the implementation. It's fairly plausible that if you do getUserMedia(...) -> HTMLMediaElement, and your camera outputs in e.g. NV12, no conversion is performed between what the camera outputs and what you see on the screen. But I don't think we can draw a general rule there.

If you see something strange, I suggest you gather all the information you can (browser, OS, camera, maybe have a test case and screenshots, etc.) and open a ticket at the browser implementer issue tracker

This issue tracker really is about Web Codecs, and this is a bit off-topic, but we usually know a bit about the rest of the browser -- however this seem to be an implementation issue and not a spec issue.