stream resolution higher than 1080p?
bb5d0 opened this issue · comments
Using raspiCam with my RPI v2 Camera the maximum streaming format seems to be 1440x1080
But using a different streamer program I'm able to stream @ 1640x1232 (see Hardware Specification)
Hey @bb5d0,
During the implementation process, I experimented with various resolutions using my HQ camera. However, I discovered that the h264 encoder has a limitation of 1080p, as mentioned in this issue on GitHub: raspberrypi/rpicam-apps#23
For instance, when I attempted to set the resolution to 2028x1520 using the command
libcamera-vid --width 2028 --height 1520 --timeout 10000 --output /home/pi/raspiCam/photos/20230707084819.h264
ERROR: *** failed to start output streaming ***
Nevertheless, your suggested resolution of 1640x1232 worked fine and i added it to the presets. Thank you for that.
In addition, I have made adjustments to accommodate higher resolutions. Now, the maximum stream and video resolutions have been increased to 2304x1520, which is the maximum for high quality (HQ) mode. It's important to note that these resolutions are only compatible with the mjpeg codec.
You can find the changes in the latest release 2.0.9
https://github.com/Lillifee/raspiCam/releases/tag/v2.0.9
That was quick! I just tested the new version. Streaming and recording @ 1640x1232 with h264 works! Thanks for that!
Two issues:
When starting the user interface without preexisting settings there was no video panel to configure them. I had to create the settings file for video manually.
Higher framerates please! At the moment the maximum is 30fps , 41fps should work too with my camera ;-)
Hey @bb5d0,
Thanks for testing! I made some changes based on your feedback. Initially, I updated the video panel's appearance to match the photo/video mode in a recent release.
However, I reverted the changes and now both the video and photo panels are always displayed. I also increased the the maximum framerate to 120fps.
You can find the latest release here:
https://github.com/Lillifee/raspiCam/releases/tag/v2.0.10
My RPI v2 camera seems to hit its limits with 33 fps @ 1640x1232. When I set it higher up to 41 the stream flickers etc.
But video recording with 41 fps @ 1640x1232 works! (I'm not sure if it's really 41 fps though, but I can live with that!)
The video panel now works intuitively without preexisting settings, thanks! But I had to set the framerate of the video record manually. Did I overlook some thing?!
One more thing thats bothering me: When the browser tab looses its focus, e.g. when I switch between browser tabs the stream fades in/out. Is there a way to deactivate this? I'd like to monitor camera streams of multiple devices at the same time without having a fade-in delay.
Otherwise, this program is becoming really great ... now I can't wait to buy a RPI v3 :-)
Hey @bb5d0,
You could also consider using a different player or the mjpeg codec as alternatives.
For h264 streams, you can enable stream statistics in the application settings to view the frames per second rendered. However, please note that these values should not be taken too seriously as they represent the average frames per second and may have a slight deviation.
The framerate can be adjusted in the video settings resolution expander. If no specific value is set, libcamera will use the default framerate, typically set to 25 frames per second.
When it comes to the fade in-out effect, I've already explored the settings in Chrome, but unfortunately, I couldn't determine why the stream stops when switching or minimizing the tab. I've experimented with the memory and energy saver options, as well as sifted through numerous flags available in chrome://flags/ but none of them seemed to address the issue directly. It appears that the MJPEG codec doesn't stop. Perhaps you could give it a shot and see if it helps resolve the problem.
I would require additional time to investigate the reasons behind Chrome freezing or sleeping disabled tabs, as well as explore possible solutions to prevent chrome from it.
Thanks for the explanations!
I truly overlooked the framerate setting in the video panel while I saw it in the stream panel, lol sorry, my bad! That was written too hastily by me.
And you're also right about MJPEG, no fade-in/out behavior with it. The same with the other web-streamer I tried, but its more rudimentary h264-stream shows fade-in/out behavior only after a certain amount of time, sometimes not at all. Generally I accustomed myself too fast with h264-stream-only of raspiCam, because it works better than the other and generally uses a lot less bandwidth than MJPEG which is very nice otherwise.
Normally I use different browsers than Chrome, mostly Firefox-based ones. I tried different browsers and yes, several setting configurations including the stream statistics display, in all cases it never surpasses 33fps. All problems regarding flickering etc. vanish when it is set back to 30fps. This framerate issue seems to be hardware-related at that resolution combined with h264 and I'm okay with that. My guess is that internally the framerate produced by the camera is somewhat variable anyhow and seldom reaches 41fps or above 30 for that matter. But I'm eagerly interested in the performance of the RPI v3 Camera module. Sadly in every available shop it's still priced 1.5x of the manufacturer's price.
One major difference I noticed with the other web-streamer I mentioned, is the higher CPU-load of raspiCam, especially when using MJPEG which maxxes out all four cores of my RPI3A+ @1,4GHz. With the other one it is a barely noticeable CPU-load. You might want to compare your implementation with it. It's the only-streamer camera-streamer which also uses libcamera, but it has no high-res photo capture or any video recording functionality like yours.
So far your implementation has all the functionality that I need, but the other one is a more perfect streamer, especially using MJPEG. It seems I can't yet abandon it completely. That's why I also asked a question over there (see). No answer so far. I would like to pass on this question to you if it's possible to create a switch to turn the libcamera process on/off.
Until recently I used two raspi-lagacy-camera-stack http-server programs at the same time. One of them to stream, the other to capture/record photos/videos. I used these functions on-demand, so that only one of them at a time would access the camera resources.
Now migrating to libcamera ... I really like your no-nonsense http streaming interface. But I would also like to use a different program to capture/record without having to stop/start the linux service. Would it be possible to start the http server with a switch to turn the underlying libcamera process on/off?
I hope I did not miss anything again or some other stupid mistake ;-)
Thanks for all the help so far, that's it for the day!
Hey @bb5d0,
I apologize for the late response. I haven't tested streaming with a higher framerate before, so I can't provide a definitive answer. The performance depends on various factors such as resolution, bitrate, and so on. For example, when streaming at 720p and 60fps, I typically achieve around 52-55fps. However, I've noticed that the jmuxer player has some issues and drops frames every second. It's possible that manually setting the framerate could help, but if the expected framerate (60fps) doesn't match the actual stream framerate (52fps), we might encounter the opposite problem. In my experience, the broadway player handles higher framerates better.
Regarding performance, you can check the processes on your Raspberry Pi by using the top command and monitoring the CPU and memory usage.
For h264, here's an image showing the resource usage:
And for mjpeg, here's the corresponding image:
Most of the resources seem to be utilized by libcamera-vid, especially for mjpeg. The node process (raspicam) might consume more memory to buffer the video for a short time. Could you try running the libcamera-vid command from the command line without raspicam and observe the performance?
But you have to replace the --output - to --output ./test.h264 to record it on the hard disk.
The command should look like this for h264
libcamera-vid --nopreview --width 1280 --height 720 --framerate 25 --bitrate 6000000 --codec H264 --timeout 0 --profile baseline --inline --output ./test.h264
or for mjpeg
libcamera-vid --nopreview --width 1280 --height 720 --framerate 25 --bitrate 6000000 --codec MJPEG --timeout 0 --profile baseline --inline --output ./test.mjpeg
Additionally, please check if there are any warnings in the output related to client connection problems. Here's an example:
From what I've observed, the camera streamer appears to be using WebRTC. Although I initially intended to investigate WebRTC further, I ended up focusing on the two web players (jmuxer and broadway) for now. Exploring WebRTC might be one of the next topics, but it could turn into a larger undertaking.
Oh, and could you also please check the CPU and memory usage of the camera streamer with similar settings? I'm really interested in a comparison, but I haven't had the time to install and test it yet.
When I set the framerate to 60 for h264, I get
60fps @ 960x720
47fps @ 1440x1080
And like I said
33fps (unstable) @ 1640x1232 / 41 framerate
30fps (stable) @ 1640x1232 / 30 framerate
and that's okay for me, I just didn't expect it to be so low because of the official hardware specification.
Using JMuxer or Broadway doesn't seem to matter fps-wise, but as I mentioned there are differences in browser behaviour. I use JMuxer in my preferred browser.
These are the results for your two test commands
h264:
mjpeg:
I didn't get any client connection problems and I mostly don't get any using my preferred configurations. For me it seems to happen while having wifi issues or at times of h264 fade-ins/outs, not otherwise.
The streamer-comparison of streams using MJPEG @1640x1232 looks like this
raspiCam:
camera-streamer:
And as I mentioned, the camera-streamer CPU load for h264 is also somewhat lower compared to raspiCam.
I haven't looked into the code myself, but as far as I understand it, camera-streamer doesn't use libcamera-vid, but directly interfaces with the camera hardware resources which leads to its significantly lower CPU load.
WebRTC is optional in camera-streamer and I don't use it at all (I even deactivate it in all my browsers).
Hey @bb5d0
Thanks a lot for testing!
Regarding the framerate, i found a section (High framerate capture) in the raspberry pi documentation:
https://www.raspberrypi.com/documentation/computers/camera_software.html#high-framerate-capture
High framerate capture:
- The H.264 target level must be set to 4.2 with the --level 4.2 argument.
- Software colour denoise processing must be turned off with the --denoise cdn_off argument.
- For rates over 100 fps, disabling the display window with the -n option would free up some additional CPU cycles to help avoid frame drops.
- It is advisable to set force_turbo=1 in /boot/config.txt to ensure the CPU clock does not get throttled during the video capture.
- Adjust the ISP output resolution with --width 1280 --height 720 or something even lower to achieve your framerate target.
- On a Pi 4, you can overclock the GPU to improve performance by adding gpu_freq=550 or higher in /boot/config.txt.
example: libcamera-vid --level 4.2 --framerate 120 --width 1280 --height 720 --save-pts timestamp.pts -o video.264 -t 10000 --denoise cdn_off -n
It is possible that some arguments are missing for MJPEG. I will need to experiment with libcamera to find a solution. As you can see, I am currently using the built-in libcamera-vid with the corresponding arguments. However, I cannot do much if I continue to use libcamera.
I haven't had the chance to try camera-streamer yet, but I'm curious about the difference in CPU usage compared to the libcamera-vid version. Is it a lot for h264? I also noticed that the default settings in the camera-stremer service are set to 639x480. (https://github.com/ayufan/camera-streamer/blob/main/service/camera-streamer-raspi-v2-8MP.service)
Can you send me your camera-streamer startup settings.
The streamer-comparison of streams using h264 @1640x1232 looks like this
raspiCam:
camera-streamer:
I have changed my camera-streamer service for my RPI v2 camera. I always use it like this:
ExecStart=/usr/local/bin/camera-streamer \
--camera-path=/base/soc/i2c0mux/i2c@1/imx219@10 \
--camera-type=libcamera \
--camera-format=YUYV \
--camera-width=3280 \
--camera-height=2464 \
--camera-fps=30 \
--camera-nbufs=4 \
--camera-snapshot.options=compression_quality=100 \
--camera-snapshot.height=1440 \
--camera-video.height=1232 \
--camera-stream.options=compression_quality=88 \
--camera-stream.height=1232
Im not familiar with camera-streamer, so I decided to start with the settings provided on the control page (http://myip:8080/control).
-
However, I noticed that the refresh rate of the mjpeg stream was slightly slower compared to libcamera-vid. On the positive side, the CPU usage was really impressive. I'm still trying to figure out what the issue might be with libcamera-vid.
-
Next, I attempted to use webrtc. However, I noticed that the video quality seemed compressed, and the bitrate appeared to be much lower than in libcamera-vid. Additionally, the framerate felt significantly slower.
-
I'm curious about how to access the video stream. When I open it using a browser or VLC player using the URL http://myip:8080/video, I experience a delay of a few seconds, and the video settings are the same as in webrtc (compressed and low frame rate).
Do i miss something here?
Hope i find time to test a little bit more....
Because I use these streamer via browser, the fact that camera-streamer doesn't offer a video player like raspiCam (JMuxer/Broadway) and especially because the CPU load for MJPEG streaming is so low with camera-streamer, I mostly use it (i.e. /stream) there and not the video one (/video).
I'm not really sure what you mean with 'refresh rate'.
And like I said, I don't use WebRTC at all.
I don't care about the delay when I seldomly use its h264 stream (/video) via a media player like VLC or MPC.
In contrast to the then somewhat unstable h264-stream-playback of raspiCam, setting the framerate to 41 @1640x1232 for the camera-streamer (with --camera-fps=41) lets work both its streams overall (MJPEG/h264) without a difference to a 30 setting, while 41 only works stably for recording and MJPEG streaming with raspiCam. Media players show 41fps respectively, no problem with that.
I don't know if there other are framerate/compression settings. Setting it with --camera-fps=30/41 and compression_quality=... works for me.
Hey @bb5d0 ,
Regarding the MJPEG issue, I have created a report on the libcamera-apps page. Unfortunately, libcamera-vid currently relies on a software JPEG encoder (libjpeg/libjpeg-turbo), and they don't have any specific plans to implement hardware JPEG encoding. This limitation prevents me from making significant changes without switching to another library.
You can find the issue I created at this link: GitHub Issue Link.
I apologize for the confusion earlier when mentioning "refresh rate." I was actually referring to the frames per second (FPS). I noticed that with the camera-streamer, the FPS felt like it was around 10, regardless of whether I used the MJPEG or H264 stream. It's possible that I made a mistake in my configuration, but regardless, I haven't been able to achieve the same level of performance with libcamera-apps.
However, I will close this issue since it was related to the stream resolution. Let's create additional issues to address specific problems more effectively.