Performance and Progress Reporting Improvements
w-barath opened this issue · comments
I've authored a script for doing ffmpeg batch encoding. FastFlix does most of what my script does, but its performance and progress reporting don't quite match up due to two things:
-
Progress Reporting: when doing pass 1 of a 2 pass encode, FastFlix doesn't copy the audio channels. FFMPEG uses the audio channel to determine its location during the encode, so it reports frame zero for the entire first pass. So when doing the first pass, it's good to add
-c:a copy -
unsure why but FastFlix is using 100% of one core, competing with FFMPEG, throughout the encoding process, yet FastFlix is doing no actual work during this time. I'm guessing this is due to FastFlix doing a tight loop to poll FFMPEG's output instead of using blocking IO, which would sleep when there's been no new input. So introduce a usleep() call into that tight loop.
Dunno if can help in any way, but @porcino's Av1ador parallel encoder may inspire for a better implementation:
AV1/HEVC/VP9/H264 parallel encoder GUI for FFmpeg with preview/comparison while transcoding.
Further, when you suspend a Fastflix encode, it keeps using 100% of one core, to do nothing
Hi @w-barath thanks for the heads up about high CPU usage! Will get that fixed for next release.
For the progress report, what encoder and version of FFMpeg are you using? I am able to see progress on first pass with a few I tried (SVT-AV1, x265)
CPU performance improved in 5.5.7 https://github.com/cdgriffith/FastFlix/releases/tag/5.5.7
The other performance option I should have mentioned - Handbrake has a "turbo first pass" option and FFMPEG's own howto docs on their Track site recommends doing a real-time first pass for codecs that support it. While the quality difference is non-zero, it cuts the encoding time basically in half for high-effort presets, and they claim the quality difference is imperceptible.
Good to know, could you point me to that documentation for how to? I don't see that covered in either x264 or x265 two pass sections:
https://trac.ffmpeg.org/wiki/Encode/H.265#Two-PassEncoding
https://trac.ffmpeg.org/wiki/Encode/H.264#twopass
While the quality difference is non-zero, it cuts the encoding time basically in half for high-effort presets, and they claim the quality difference is imperceptible.
As suggested in the past, it would be cool to perform an HW-accelerated 1st pass, IMHO.
Good to know, could you point me to that documentation for how to?
I can't for the life of me find it. I get too many false positives when I use search terms around "pass" "preset" "quicker" "faster" etc...
The best I came across was this: https://www.streamingmedia.com/Articles/ReadArticle.aspx?ArticleID=154825
Note that I tested with -cpu-used 8 in the first pass and -cpu-used 4 in the second. That’s because the quality used in the first pass doesn’t impact overall quality.
They are using --cpu-used 8 / --cpu-used 4 for pass 1 and 2 respectively in a professional streaming setting, with input from Google engineers, so maybe they know what they're talking about 😉, but I get it would be nice if I could come up with actual research showing bd-rate for different first-pass presets.
Ah here's some references from Meta's video streaming engineers as well:
https://engineering.fb.com/2023/02/21/video-engineering/av1-codec-facebook-instagram-reels/
In previous studies we found that we could use the high-speed preset for first-pass encoding and to produce the convex hull, and then take a second pass to encode the selected (resolution, CRF) points with the high-quality preset. Even though this approach requires additional encoding, it’s faster because the first pass can be done much more quickly. (Coding efficiency drops only slightly.)
Here in context of their studies the "drop" is well under 1%, and by "faster" they mean the same perceptual quality at the same bitrate in less time, using 2-pass, than with the best competing (ABR + CRF + preset) combination.
Hope that helps.
Looks like with x265 it's a flag need to set no-slow-firstpass=1 in x265 params https://x265.readthedocs.io/en/master/cli.html#cmdoption-slow-firstpass
Also good to know can modify svt-av1 cpu settings.
What I don't want to do is blindly pick a faster preset, as that could modify an array of settings / sub parameters that could create straight up bad conversions without knowing what is safe to switch. Notice the list of things modified for x265 first pass is a lot shorter than all the options that a certain preset covers https://x265.readthedocs.io/en/master/presets.html so changing the wrong ones could be bad.