Framerate Issues RPi 5 + Camera Module 3
GuyPerets106 opened this issue · comments
Hi , I'm using Raspberry Pi 5 with 8GB of RAM + Camera Module 3 NoIR Wide.
When running the command rpicam-vid --mode 1536:864:10:P --width 640 --height 400 --hflip 1 --vflip 1 --framerate 120 --frames 1200 -n --hdr off --level 4.2 --lens-position 6 --denoise off -o test.h264 and view the h264 file output I see the video in slow motion. It still 120fps (Checked) but slowed. When trying to convert to mp4 with ffmpeg I get a fast-moving video , also when specifying the framerate in the conversion.
Something I noticed is when recording I see:
Output #0, h264, to 'test.h264': Stream #0:0: Video: h264, yuv420p(tv, smpte170m/smpte170m/bt709), 640x400, q=2-31, 120 fps, 120 tbr, 1000k tbn
but when running ffprobe -i test.h264 I see:
Input #0, h264, from 'test.h264': Duration: N/A, bitrate: N/A Stream #0:0: Video: h264 (High), yuv420p(tv, smpte170m/smpte170m/bt709, progressive), 640x400, 1000k tbr, 1200k tbn
The differences in tbr (which I assume should represent fps) goes from 120 in the rpicam-vid text output to 1000k (or 1000000) in ffprobe.
There is something I'm doing wrong?
Thanks!
H.264 elementary streams do not have any timestamping information in them. So when you play them back in any video application, it assumes a rate of 30fps. You need to use a container such as mkv or mp4 when saving the encoded file so that timestamps are embedded correctly.
In your command above, replace -o test.h264 with -o test.mp4 and that should then work correctly.
H.264 elementary streams do not have any timestamping information in them. So when you play them back in any video application, it assumes a rate of 30fps. You need to use a container such as mkv or mp4 when saving the encoded file so that timestamps are embedded correctly.
In your command above, replace
-o test.h264with-o test.mp4and that should then work correctly.
Oh ok. I actually used this command with my RPi 4 (when it was libcamera-vid) and then converted to mp4 with ffmpeg and it worked fine. So I thought it was a RPi 5 problem.
There's any performance differences with using -o <filename>.mp4 compared to -o <filename>.h264 ?
I'm really trying to push the limit of stable framerate with this device. Should I see an increase in stable framerate with the same sensor as going from RPi 4 to 5?
There's any performance differences with using
-o <filename>.mp4compared to-o <filename>.h264?
There is a very minimal overhead to using a mp4 container. With everything else running in the system during video encode, you likely won't even be able to measure the difference.
I'm really trying to push the limit of stable framerate with this device. Should I see an increase in stable framerate with the same sensor as going from RPi 4 to 5?
Yes you should see better encode performance on the Pi 5. You also have the ability to dial up/down the encoder CPU usage to match your application if you want.
I'll mark this as complete now. If there is anything else you need to query, feel free to reopen.
There's any performance differences with using
-o <filename>.mp4compared to-o <filename>.h264?There is a very minimal overhead to using a mp4 container. With everything else running in the system during video encode, you likely won't even be able to measure the difference.
I'm really trying to push the limit of stable framerate with this device. Should I see an increase in stable framerate with the same sensor as going from RPi 4 to 5?
Yes you should see better encode performance on the Pi 5. You also have the ability to dial up/down the encoder CPU usage to match your application if you want.
I'll mark this as complete now. If there is anything else you need to query, feel free to reopen.
Thanks , How can I dial up the encoder CPU usage?
Thanks , How can I dial up the encoder CPU usage?
You can use the --libav-video-codec-opts command line arguments to set various ffmpeg codec options to set encode features. However, I would leave this as-is unless you really need to.
Thanks , How can I dial up the encoder CPU usage?
You can use the
--libav-video-codec-optscommand line arguments to set various ffmpeg codec options to set encode features. However, I would leave this as-is unless you really need to.
Ok , thanks. I will try different options. What does the 'preset=ultrafast' do? and how can I incorporate i in a rpicam-vid command?
My ultimate goal is to get 120fps recording in 2 cameras simultaneously , hopefully using the non-cropped mode (2304x1296) , I currently use the cropped 1536x864 mode with 100fps in each camera. You think my goal is feasible?
You can find details on the presets here: https://dev.beandog.org/x264_preset_reference.html.
Dual camera encode at high framerate is going to be quite restrictive I'm afraid. You should easily be able to get something like 800x600 120fps working, but I'm not sure you are going to get much higher in resolution. Perhaps MJPEG might be better, it's less CPU intensive, but it does take up approx. 10x more disk space.
You can find details on the presets here: https://dev.beandog.org/x264_preset_reference.html.
Dual camera encode at high framerate is going to be quite restrictive I'm afraid. You should easily be able to get something like 800x600 120fps working, but I'm not sure you are going to get much higher in resolution. Perhaps MJPEG might be better, it's less CPU intensive, but it does take up approx. 10x more disk space.
Oh I didn't mention - I'm only using this resolutions as the camera modes that control the crop , but I specify --width 640 --height 400 for my videos.
In which case, the default encoder settings ought to just work for you for dual 640x400 120fps encode, including the full FoV sensor mode.
What could be the reason for which I specify --mode 2304:1296 and still get Selected sensor format: 1536x864-SBGGR10_1X10 - Selected CFE format: 1536x864-PC1B?
I think you want to use --mode 2304:1296:10:P?
I think you want to use
--mode 2304:1296:10:P?
Yes of course. Same problem.
Full command is rpicam-vid --mode 2304:1296:10:P --width 640 --height 400 --camera 0 --framerate 120 --frames 600 -n --hflip 1 --vflip 1 --level 4.2 --hdr off --denoise off -o test.mp4 if I did a mistake somewhere else.
The full output is:
[1:09:32.898817920] [27965] INFO Camera camera_manager.cpp:284 libcamera v0.1.0+118-563cd78e
[1:09:32.918802060] [27968] INFO RPI pisp.cpp:653 libpisp version v1.0.2 fa44a258644a 22-11-2023 (21:59:22)
[1:09:32.936962103] [27968] INFO RPI pisp.cpp:1112 Registered camera /base/axi/pcie@120000/rp1/i2c@88000/imx708@1a to CFE device /dev/media0 and ISP device /dev/media3 using PiSP variant BCM2712_C0
[1:09:32.937057618] [27968] INFO RPI pisp.cpp:653 libpisp version v1.0.2 fa44a258644a 22-11-2023 (21:59:22)
[1:09:32.946745857] [27968] INFO RPI pisp.cpp:1112 Registered camera /base/axi/pcie@120000/rp1/i2c@80000/imx708@1a to CFE device /dev/media1 and ISP device /dev/media4 using PiSP variant BCM2712_C0
[1:09:32.948887756] [27965] WARN V4L2 v4l2_pixelformat.cpp:338 Unsupported V4L2 pixel format Y16
[1:09:32.948917311] [27965] WARN V4L2 v4l2_pixelformat.cpp:338 Unsupported V4L2 pixel format RGB6
[1:09:32.948925273] [27965] WARN V4L2 v4l2_pixelformat.cpp:338 Unsupported V4L2 pixel format BGR6
[1:09:32.948934143] [27965] WARN V4L2 v4l2_pixelformat.cpp:338 Unsupported V4L2 pixel format PC1M
Mode selection for 2304:1296:10:P(120)
SRGGB10_CSI2P,1536x864/120.135 - Score: 2400
SRGGB10_CSI2P,2304x1296/56.0255 - Score: 127949
SRGGB10_CSI2P,4608x2592/14.3536 - Score: 212193
Stream configuration adjusted
[1:09:32.949179949] [27965] INFO Camera camera.cpp:1183 configuring streams: (0) 640x400-YUV420 (1) 1536x864-RGGB16_PISP_COMP1
[1:09:32.949251927] [27968] INFO RPI pisp.cpp:1396 Sensor: /base/axi/pcie@120000/rp1/i2c@88000/imx708@1a - Selected sensor format: 1536x864-SRGGB10_1X10 - Selected CFE format: 1536x864-PC1R
[libx264 @ 0x55566e446580] using cpu capabilities: ARMv8 NEON
[libx264 @ 0x55566e446580] profile High, level 4.2, 4:2:0, 8-bit
[libx264 @ 0x55566e446580] 264 - core 164 r3095 baee400 - H.264/MPEG-4 AVC codec - Copyleft 2003-2022 - http://www.videolan.org/x264.html - options: cabac=1 ref=1 deblock=1:0:0 analyse=0x3:0x3 me=dia subme=0 psy=1 psy_rd=1.00:0.00 mixed_ref=0 me_range=16 chroma_me=0 trellis=0 8x8dct=1 cqm=0 deadzone=21,11 fast_pskip=1 chroma_qp_offset=0 threads=6 lookahead_threads=1 sliced_threads=0 slices=1 nr=0 decimate=1 interlaced=0 bluray_compat=0 constrained_intra=0 bframes=1 b_pyramid=0 b_adapt=1 b_bias=0 direct=1 weightb=0 open_gop=0 weightp=0 keyint=120 keyint_min=12 scenecut=0 intra_refresh=0 rc=crf mbtree=0 crf=23.0 qcomp=0.60 qpmin=0 qpmax=69 qpstep=4 ip_ratio=1.40 pb_ratio=1.30 aq=1:1.00
Output #0, mp4, to 'testWideFOV.mp4':
Stream #0:0: Video: h264, yuv420p(tv, smpte170m/smpte170m/bt709), 640x400, q=2-31, 120 fps, 120 tbr, 1000k tbn
Oh, ok sorry I misunderstood what you were trying to do. Since you are specifying --framerate 120 on your command line, it is overriding the --mode 2304:1296:10:P option and selecting the 1536x864 mode. This is the only mode that can run at 120fps since the camera is still only operating with 2 CSI-2 lanes. If you use the full FoV 2304:x1296 mode, you can only get a maximum of 56 fps unfortunately. This is down to the sensor and 2-lane operating limitations.
Also on Pi 5, you no longer need to specify --denoise off for fast farmerate operation.
Oh, ok sorry I misunderstood what you were trying to do. Since you are specifying
--framerate 120on your command line, it is overriding the--mode 2304:1296:10:Poption and selecting the 1536x864 mode. This is the only mode that can run at 120fps since the camera is still only operating with 2 CSI-2 lanes. If you use the full FoV 2304:x1296 mode, you can only get a maximum of 56 fps unfortunately. This is down to the sensor and 2-lane operating limitations.Also on Pi 5, you no longer need to specify
--denoise offfor fast farmerate operation.
Oh wow , that's unfortunate to hear. Since the Module 3's are relatively new , I thought it could really take advantage of the new hardware of the RPi 5 and not have the same limitations. :/
Thanks anyway for all the clarifications!