developsessions / camera-streamer

High-performance low-latency camera streamer for Raspberry PI's

Yet another camera streamer

There's a number of great projects doing an UVC/CSI camera streaming on SBC (like Raspberry PI's).

This is yet another camera streamer project that is primarly focused on supporting a fully hardware accelerated streaming of MJPEG streams and H264 video streams for minimal latency.

This supports well CSI cameras that provide 10-bit Bayer packed format from sensor, by using a dedicated ISP of Raspberry PI's.

Take into account that this is a draft project, and is nowhere as complete and well supported as (awesome ustreamer)[https://github.com/pikvm/ustreamer]. This project was inspired by mentioned ustreamer.

Validate your system

This streamer does only use hardware, and does not support any software decoding or encoding. It does require system to provide:

1. ISP (/dev/video13, /dev/video14, /dev/video15)
2. JPEG encoder (/dev/video31)
3. H264 encoder (/dev/video11)
4. JPEG/H264 decoder (for UVC cameras, /dev/video10)
5. At least LTS kernel (5.15) for Raspberry PIs

You can validate the presence of all those devices with:

uname -a
v4l2-ctl --list-devices

The 5.15 kernel is easy to get since this is LTS kernel for Raspberry PI OS:

apt-get update
apt-get dist-upgrade
reboot

Ensure that your /boot/config.txt has enough of GPU memory (required for JPEG re-encoding):

# Example for IMX519
dtoverlay=vc4-kms-v3d,cma-128
gpu_mem=128 # preferred 160 or 256MB
dtoverlay=imx519

# Example for Arducam 64MP
gpu_mem=128
dtoverlay=arducam_64mp,media-controller=1

# Example for USB cam
gpu_mem=128

Compile

git clone https://github.com/ayufan-research/camera-streamer.git --recursive
apt-get -y install libavformat-dev libavutil-dev libavcodec-dev libcamera-dev liblivemedia-dev v4l-utils pkg-config xxd build-essential cmake libssl-dev

cd camera-streamer/
make
sudo make install

Use it

There are three modes of operation implemented offering different compatibility to performance.

Use a preconfigured systemd services

The simplest is to use preconfigured service/camera-streamer*.service. Those can be used as an example, and can be configured to fine tune parameters.

Example:

systemctl enable $PWD/service/camera-streamer-arducam-16MP.service
systemctl start camera-streamer-arducam-16MP

If everything was OK, there will be web-server at http://<IP>:8080/.

Error messages can be read journalctl -xef -u camera-streamer-arducam-16MP.

High-compatibility via libcamera

This script uses libcamera to access camera and provide manual and automatic brightness and exposure controls. The settings for those are configurable via described below controls.

This due to extra overhead has worse latency than direct decoding via ISP.

tools/libcamera_camera.sh -help
tools/libcamera_camera.sh -camera-format=YUYV

High-performance mode via ISP for CSI

This script uses high-performance implementation for directly accessing sensor feeds and passing it via DMA into bcm2385 ISP. As such it does not implement brightness control similarly to libcamera.

# This script uses dumped IMX519 parametrs that are feed into bcm2385 ISP module
# This does not provide automatic brightness control
# Other sensors can be supported the same way as long as ISP parameters are adapted
tools/imx519_camera.sh -help
tools/imx519_camera.sh -camera-format=RG10 ...

This mode allows to provide significantly better performance for camera sensors than described in specs. For example for Arducam 16MP (using IMX519) it is possible to achieve:

1. 120fps on 1280x720, with latency of about 50ms, where-as documentation says 1280x720x60p
2. 60fps on 1920x1080, with latency of about 90ms, where-as documentation says 1920x1080x30p
3. 30fps on 2328x1748, with latency of about 140ms, where-as documentation says 2328x1748x15p

The above was tested with camera connected to Raspberry PI Zero 2W, streaming MJPEG over WiFi with camera recording https://www.youtube.com/watch?v=e8ZtPSIfWPc from desktop monitor. So, it as well measured also a desktop rendering latency.

High-performance mode via direct decoding for USB

This script uses direct decoding or passthrough of MJPEG or H264 streams from UVC camera into re-encoded stream provided over web-interface.

tools/usb_camera.sh -help
tools/usb_camera.sh -camera-format=H264 ...
tools/usb_camera.sh -camera-format=MJPEG ...

HTTP web server

All streams are exposed over very simple HTTP server, providing different streams for different purposes:

• http://<ip>:8080/ - index page
• http://<ip>:8080/snapshot - provide JPEG snapshot (works well everywhere)
• http://<ip>:8080/stream - provide M-JPEG stream (works well everywhere)
• http://<ip>:8080/video - provide H264 in-browser muxed video output (using https://github.com/samirkumardas/jmuxer/, works only in Desktop Chrome)
• http://<ip>:8080/video.mp4 or http://<ip>:8080/video.mkv - provide remuxed mkv or mp4 stream (uses ffmpeg to remux, works as of now only in Desktop Chrome and Safari)

Resolution

Camera capture and resolution exposed is controlled by threee parameters:

• -camera-width and -camera-height define capture resolution
• -camera-video.height - define height for an aspect ratio scaled resolution for /video and /webrtc (H264) output - this might require rescaller and might not always work
• -camera-stream.height - define height for an aspect ratio scaled resolution for /stream (MJPEG) output - this might require rescaller and might not always work
• -camera-snapshot.height - define height for an aspect ratio scaled resolution for /snapshot (JPEG) output - this might require rescaller and might not always work

Any video, stream and snapshot might not work as this requires usually decoding, scaling, and encoding to achieve the desired resolution.

This works ONLY BEST when using libcamera, the support for USB will varry and might require configuring /boot/config.txt to set enough of GPU memory to be able to re-encode JPEG.

RTSP server

The camera-streamer implements RTSP server via live555. Enable it with:

• adding -rtsp-port: will enable RTSP server on 8554
• adding -rtsp-port=1111: will enable RTSP server on custom port

The camera-streamer will expose two stream (if low res mode is enabled):

• rtsp://<ip>:8554/stream.h264 - high resolution stream (always enabled if H264 stream is available directly or via encoding)
• rtsp://<ip>:8554/stream_low_res.h264 - low resolution stream if low res mode is configured via -camera-low_res_factor

List all available controls

You can view all available configuration parameters by adding -log-verbose to one of the above commands, like:

tools/libcamera_camera.sh -log-verbose ...

Depending on control they have to be used for camera, ISP, or JPEG or H264 codec:

# specify camera option
-camera-options=brightness=1000

# specify ISP option
-camera-isp.options=digital_gain=1000

# specify H264 option
-camera-h264.options=bitrate=10000000

# specify JPEG option
-camera-jpeg.options=compression_quality=60

List all available formats and use proper one

You might list all available capture formats for your camera:

v4l2-ctl -d /dev/video0 --list-formats-ext

Some of them might be specified to streamer:

tools/*_camera.sh -camera-format=RG10 # Bayer 10 packet
tools/*_camera.sh -camera-format=YUYV
tools/*_camera.sh -camera-format=MJPEG
tools/*_camera.sh -camera-format=H264 # This is unstable due to h264 key frames support

Camera support

Arducam 16MP

The 16MP sensor is supported by default in Raspberry PI OS after adding to /boot/config.txt. However it will not support auto-focus nor manual focus due to lack of ak7535 compiled and enabled in imx519. Focus can be manually controlled via i2c-tools:

# /boot/config.txt
dtoverlay=imx519,media-controller=0
gpu_mem=160 # at least 128

# /etc/modules-load.d/modules.conf
i2c-dev

# after starting camera execute to control the focus with `0xXX`, any value between `0x00` to `0xff`
# RPI02W (and possible 2+, 3+):
i2ctransfer -y 22 w4@0x0c 0x0 0x85 0x00 0x00

# RPI4:
i2ctransfer -y 11 w4@0x0c 0x0 0xXX 0x00 0x00

Latency according to my tests is due to the way how buffers are enqueued and processing delay, this is for triple buffering in a case where sensor is able to deliver frames quick enough. Depending on how you queue (on which slope) and when you enqueue buffer you might achieve significantly better latency as shown in the ISP-mode. The libcamera can still achieve 120fps, it is just slightly slower :)

2328x1748@30fps

# libcamera
$ ./camera_streamer -camera-path=/base/soc/i2c0mux/i2c@1/imx519@1a -camera-type=libcamera -camera-format=YUYV -camera-fps=120 -camera-width=2328 -camera-height=1748 -camera-high_res_factor=1.5 -log-filter=buffer_lock
device/buffer_lock.c: http_jpeg: Captured buffer JPEG:capture:mplane:buf2 (refs=2), frame=63/0, processing_ms=101.1, frame_ms=33.1
device/buffer_lock.c: http_jpeg: Captured buffer JPEG:capture:mplane:buf0 (refs=2), frame=64/0, processing_ms=99.2, frame_ms=31.9
device/buffer_lock.c: http_jpeg: Captured buffer JPEG:capture:mplane:buf1 (refs=2), frame=65/0, processing_ms=99.6, frame_ms=34.8

# direct ISP-mode
$ ./camera_streamer -camera-path=/dev/video0 -camera-format=RG10 -camera-fps=30 -camera-width=2328 -camera-height=1748 -camera-high_res_factor=1.5 -log-filter=buffer_lock
device/buffer_lock.c: http_jpeg: Captured buffer JPEG:capture:mplane:buf1 (refs=2), frame=32/0, processing_ms=49.7, frame_ms=33.3
device/buffer_lock.c: http_jpeg: Captured buffer JPEG:capture:mplane:buf2 (refs=2), frame=33/0, processing_ms=49.7, frame_ms=33.3
device/buffer_lock.c: http_jpeg: Captured buffer JPEG:capture:mplane:buf0 (refs=2), frame=34/0, processing_ms=49.7, frame_ms=33.4

2328x1748@10fps

# libcamera
$ ./camera_streamer -camera-path=/base/soc/i2c0mux/i2c@1/imx519@1a -camera-type=libcamera -camera-format=YUYV -camera-fps=10 -camera-width=2328 -camera-height=1748 -camera-high_res_factor=1.5 -log-filter=buffer_lock
device/buffer_lock.c: http_jpeg: Captured buffer JPEG:capture:mplane:buf2 (refs=2), frame=585/0, processing_ms=155.3, frame_ms=100.0
device/buffer_lock.c: http_jpeg: Captured buffer JPEG:capture:mplane:buf0 (refs=2), frame=586/0, processing_ms=155.5, frame_ms=100.2

# direct ISP-mode
$ ./camera_streamer -camera-path=/dev/video0 -camera-format=RG10 -camera-fps=10 -camera-width=2328 -camera-height=1748 -camera-high_res_factor=1.5 -log-filter=buffer_lock
device/buffer_lock.c: http_jpeg: Captured buffer JPEG:capture:mplane:buf1 (refs=2), frame=260/0, processing_ms=57.5, frame_ms=99.7
device/buffer_lock.c: http_jpeg: Captured buffer JPEG:capture:mplane:buf2 (refs=2), frame=261/0, processing_ms=57.6, frame_ms=100.0
device/buffer_lock.c: http_jpeg: Captured buffer JPEG:capture:mplane:buf0 (refs=2), frame=262/0, processing_ms=58.0, frame_ms=100.4

1280x720@120fps for Arducam 16MP

# libcamera
$ ./camera_streamer -camera-path=/base/soc/i2c0mux/i2c@1/imx519@1a -camera-type=libcamera -camera-format=YUYV -camera-fps=120 -camera-width=1280 -camera-height=720 -log-filter=buffer_lock
device/buffer_lock.c: http_jpeg: Captured buffer JPEG:capture:mplane:buf0 (refs=2), frame=139/0, processing_ms=20.1, frame_ms=7.9
device/buffer_lock.c: http_jpeg: Captured buffer JPEG:capture:mplane:buf1 (refs=2), frame=140/0, processing_ms=20.6, frame_ms=8.8
device/buffer_lock.c: http_jpeg: Captured buffer JPEG:capture:mplane:buf2 (refs=2), frame=141/0, processing_ms=19.8, frame_ms=8.1

# direct ISP-mode
$ ./camera_streamer -camera-path=/dev/video0 -camera-format=RG10 -camera-fps=120 -camera-width=1280 -camera-height=720 -log-filter=buffer_lock
device/buffer_lock.c: http_jpeg: Captured buffer JPEG:capture:mplane:buf0 (refs=2), frame=157/0, processing_ms=18.5, frame_ms=8.4
device/buffer_lock.c: http_jpeg: Captured buffer JPEG:capture:mplane:buf1 (refs=2), frame=158/0, processing_ms=18.5, frame_ms=8.3
device/buffer_lock.c: http_jpeg: Captured buffer JPEG:capture:mplane:buf2 (refs=2), frame=159/0, processing_ms=18.5, frame_ms=8.3

WebRTC support

The WebRTC is accessible via http://<ip>:8080/video by default and is available when there's H264 output generated.

WebRTC support is implemented using awesome libdatachannel library.

The support will be compiled by default when doing make.

License

GNU General Public License v3.0