V4L2 parameters ignored on M2M codec device (e.g. unable to set bitrate)

[Adam Nielsen]

Describe the bug
If I want to change the H264 bitrate or other hardware encoder options with h264_v4l2m2m, it does not seem possible.

To reproduce

# Check default values
$ v4l2-ctl -d /dev/video11 --get-ctrl=h264_i_frame_period --get-ctrl=repeat_sequence_header --get-ctrl=video_bitrate
h264_i_frame_period: 60
repeat_sequence_header: 0
video_bitrate: 10000000

# Change values
$ v4l2-ctl -d /dev/video11 --set-ctrl=h264_i_frame_period=30 --set-ctrl=repeat_sequence_header=1 --set-ctrl=video_bitrate=5000000

# Check again - nothing was changed
$ v4l2-ctl -d /dev/video11 --get-ctrl=h264_i_frame_period --get-ctrl=repeat_sequence_header --get-ctrl=video_bitrate
h264_i_frame_period: 60
repeat_sequence_header: 0
video_bitrate: 10000000

Expected behaviour
I expected the updated value to take effect but they do not, and the default values seem to remain active.

Actual behaviour
The values set did not apply. Running the encoder still seems to use the default values instead of those set.

System

• Which model of Raspberry Pi? Pi1B
• Which OS and version (cat /etc/rpi-issue)? Arch Linux ARM 32-bit ARMv6
• Which firmware version (vcgencmd version)?

Aug 11 2021 14:01:51 
Copyright (c) 2012 Broadcom
version 5c8758dbac5b27911d2384c5dcd5db843528b606 (clean) (release) (start_x)

• Which kernel version (uname -a)? Linux 5.10.52-8-ARCH #1 SMP Sat Aug 14 15:10:14 UTC 2021 armv6l GNU/Linux

Logs
No messages appear in dmesg when running the above commands.

Additional context
I am assuming /dev/video11 is the correct device as this is the one ffmpeg selects:

[h264_v4l2m2m @ 0x5d1180] Using device /dev/video11
[h264_v4l2m2m @ 0x5d1180] driver 'bcm2835-codec' on card 'bcm2835-codec-encode' in mplane mode
[h264_v4l2m2m @ 0x5d1180] requesting formats: output=YU12 capture=H264

[6by9]

https://www.kernel.org/doc/html/latest/userspace-api/media/v4l/dev-mem2mem.html

Memory-to-memory devices function as a shared resource: you can open the video node multiple times, each application setting up their own properties that are local to the file handle, and each can use it independently from the others. The driver will arbitrate access to the hardware and reprogram it whenever another file handler gets access. This is different from the usual video node behavior where the video properties are global to the device (i.e. changing something through one file handle is visible through another file handle).

v4l2-ctl -d /dev/video11 --set-ctrl=h264_i_frame_period --get-ctrl=h264_i_frame_period should return that the value has been set, but v4l2-ctl then closes the file handle and gives a new instance on next open.
The encoder, decoder, and isp V4L2 devices all support multiple simultaneous instances, each of which will have their own set of properties.

[6by9]

bitrate etc therefore have to be set by FFmpeg, rather than trying to preconfigure it with v4l2-ctl.

[Adam Nielsen]

Ahh I see, many thanks for the quick reply! I couldn't find anything in the ffmpeg docs about how to set these options so I figured this must be the way, but it looks like it might be a feature missing from ffmpeg.

I've asked on the ffmpeg mailing list so I'll update this if I find out how it's done, to help anyone coming across this via Google in the future.

[6by9]

Use the normal -b:v 6000k to set the video bitrate.

i frame period there may be a mismatch at present.
https://www.kernel.org/doc/html/latest/userspace-api/media/v4l/ext-ctrls-codec.html

V4L2_CID_MPEG_VIDEO_H264_I_PERIOD (integer)
Period between I-frames in the open GOP for H264. In case of an open GOP this is the period between two I-frames. The period between IDR (Instantaneous Decoding Refresh) frames is taken from the GOP_SIZE control. An IDR frame, which stands for Instantaneous Decoding Refresh is an I-frame after which no prior frames are referenced. This means that a stream can be restarted from an IDR frame without the need to store or decode any previous frames. Applicable to the H264 encoder.

The encoder currently only takes V4L2_CID_MPEG_VIDEO_H264_I_PERIOD, but I'd need to check whether we encode I or IDR frames. FFmpeg only sets GOP_SIZE

REPEAT_SEQUENCE_HEADER isn't handled. There was a thread recently covering that where FFMpeg had been patched to always set it. raspberrypi/linux#4520 and https://www.raspberrypi.org/forums/viewtopic.php?p=1780707

[Adam Nielsen]

Ah interesting. The video bitrate does indeed appear to work. I will have to have a look at the ffmpeg patch as the repeating sequence header seems to be what I need, as currently I can only watch the streamed video if I start the player before the video begins. If the video encoded by h264_v4l2m2m is already streaming when the player is started, it sits there forever waiting for a frame that tells it what the video resolution is.

On a somewhat related note, has anything similarly changed with the main /dev/video0 device and the V4L2 controls? Previously on a years old kernel, I was running this before streaming the video:

v4l2-ctl -v width=1296,height=972,pixelformat=H264 \
        --set-ctrl=exposure_dynamic_framerate=1 \
        --set-ctrl=video_bitrate=2000000 \
        --set-ctrl=scene_mode=8 \
        --set-ctrl=h264_i_frame_period=30 \
        --set-ctrl=repeat_sequence_header=1

Ignoring the fact that some of these have now moved off /dev/video0 as discussed above, the scene_mode and exposure values no longer seem to work. I am now running:

v4l2-ctl \
        -d /dev/video0 \
        --set-ctrl=exposure_dynamic_framerate=1 \
        --set-ctrl=scene_mode=8 \
        -v width=1280,height=972,pixelformat=YU12

The width is slightly less as a workaround due to #1608, but the scene and exposure values don't seem to have any effect on the video. They set correctly and are read back correctly, but if I stream some video with the above options set, the exposure is only adjusted at the beginning of the stream and then remains unchanged throughout. If it gets dark I have to stop and restart the stream to get the exposure to change. I also don't seem to be able to get a really low framerate low light mode as before.

Do these values now also need to be set through the application rather than v4l2-ctl or should they still work? Is the problem because I am retrieving video from the Pi camera in a raw format instead of letting it use the hardware encoder for me? The only reason for requesting raw video is because I am using ffmpeg to write the time and some other text onto the frame before then using h264_v4l2m2m to encode it.

[6by9]

/dev/video0 is the device node for the Pi camera module which can also include H264 encoding. Being a camera source and therefore a "traditional" V4L2 device, it has one set of global state for all clients. That still exists should you be using the Pi camera module.
There is a slight mismatch between V4L2 and MMAL in differentiating stills vs video capture. With H264 or MJPEG it's fairly obviously video. With JPEG it's fairly obviously stills. With raw formats it's harder to distinguish, and the pipeline switches based on resolution. Default switch point is 1280x720, but can be configured via module parameters. Certainly if you're still in video mode then scene_mode etc should be operating in exactly the same way.

[Adam Nielsen]

Ahh that was the problem! The larger raw video capture was indeed being treat as stills and the scene mode and other parameters were being ignored.

I temporarily changed the module parameter to see if this was the problem:

echo 2048 > /sys/module/bcm2835_v4l2/parameters/max_video_height

And when this worked, I created /etc/modprobe.d/pi-video.conf with these contents:

# Need to enlarge this otherwise the Pi assumes larger raw capture sizes are
# still images so doesn't auto-adjust the exposure.
options bcm2835_v4l2 max_video_width=4096 max_video_height=2048

That did the trick, now scene_mode is correctly applied during a raw video capture, even after rebooting.

Many thanks for your help!