Provide a cropping/viewport IPC

[Huan Truong]

What will be a really useful feature for the webcam would be the ability to provide automatic panning/centering to the face. Face recognition is a rather simple task.

My recent poking around shows that access to the camera is not exclusive. Given that is true, we can run a opportunistic face recognition daemon on the pi independent of uvc-gadget and determine the frame that we should crop. Then all we need to do then is a IPC mechanism to communicate between uvc-gadget and the daemon to pan the frame.

[Bruno Bronosky]

I've been studying OpenCV face detection for just this purpose. I'm not sure if it is this simple, but it seems to me that today https://github.com/peterbay/uvc-gadget/blob/master/uvc-gadget.c#L821 does:

CSI_INPUT --video-stream--> uvc-gadget --video-stream--> USB_OUTPUT

What we want to do is:

            /--video-stream--> OpenCV --padded-face-cordinates--> \
CSI_INPUT --                                                       \
            \--video-stream--> -----------------------------------> --> video_cropper_and_scaler --video-stream--> uvc-gadget --video-stream--> USB_OUTPUT

As silly as it sounds, that video_cropper_and_scaler may actually be ffmpeg.

I doubt the cropping would be done after uvc-gadget, but I haven't really grokked the source and destination drivers yets. I'm just starting this journey. It might be able to be done "in it".

Here are some references I need to study when I have had some sleep.

• v4l
  • https://www.kernel.org/doc/html/v4.11/media/v4l-drivers/soc-camera.html?highlight=vidioc_s_fmt#vidioc-s-crop-and-vidioc-s-fmt-behaviour
• v4l2loopback
  • https://github.com/umlaeute/v4l2loopback/wiki/FFmpeg#using-the-raspberry-pi-camera-as-a-network-webcam
  • https://unix.stackexchange.com/questions/582621/how-to-create-a-v4l2-device-that-is-a-cropped-version-of-a-webcam
  • https://video.stackexchange.com/questions/4563/how-can-i-crop-a-video-with-ffmpeg

[Petr Vavrin]

In branch new-code https://github.com/peterbay/uvc-gadget/tree/feat/new-code is now possible to send video stream from ffmpeg/gstreamer to uvc-gadget.