Giters

mifi / lossless-cut

The swiss army knife of lossless video/audio editing

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

šŸ˜ Implement "smart cut"

mifi opened this issue Ā· comments

commented

We cannot cut between keyframes, it's a technical limitation of modern video codecs. So ffmpeg will choose the nearest keyframe, which will not be the exact time you requested.

Might be able to use avcut for h264 although it is experimental. See #372 (comment)

See also discussion in #13

Remaining issues

Inaccurate seeking

Seeking is not always accurate, causing glitches and/or jumps in the stitch points. This is often because I haven't found a way to consistently seek to and cut on an exact frame. Sometimes it could be caused by the "audio samples" problem below. If you want to experiment, you can try to remove the audio track and cut only video to see if it makes a difference.

Audio samples problem

Cut time accuracy/glitches when audio samples are large? (try without audio and see if it makes a difference). A workaround seems to be to cut audio/video separately and then merge them, with the tradeoff of potential audio/video de-sync:

Other improvements

  • Investigate using aomav1 instead libsvtav1 instead of aomav1 (ffmpeg default) (codec needs to compiled for mac in ffmpeg-build-script) #1825
  • custom bit rate #1997
  • #1824
  • If there is no keyframe in a particular segment, re-encode the whole segment instead of erroring with Cannot find any keyframe after the desired start cut point, see ##1190 (comment)

Done

  • Implement a smart cut that re-encodes (lossy) the part between the cutpoint and next keyframe. (done)
  • #1606

Update

Have now made an initial implementation of this. This experimental feature will re-encode the part of the video from the cutpoint until the next keyframe in order to attempt to make a 100% accurate cut while losslessly copying the rest of the segment.

Limitations

  • Only works on some files:
    • I've had success with some h264 files, and only a few h265 files seem to work.
    • Other codecs may or may not work.
  • Only supports cutting a single video stream (track). All other video streams have to be disabled. We could still probably do multi-passes and then recreate a file with all the streams.
    • Thumbnail stream will be dropped
  • We scan up to 60 seconds to find a keyframe.
  • It tries to auto-detect bitrate from input stream, however if it fails, then it will calculate bitrate from the whole file's size/duration (may be wrong)
  • Tries to copy timebase from source to destination
commented

Hi guys,

Interesting that it is a limitation of the codecs. Reason I say this is that I can do a frame specific "lossless" cut using Apple's (very old) Quicktime Player 7 in Pro mode. For the cut to take at the specific frame it has to export to a new file (rather than edit the existing file). But you can do this just passing through the source data with out any form of recoding (both video and audio tracks).

Not sure exactly how it does it, but seems that som,ehting could be possible. Save time isn't too bad either as it just seems to shuttle the data into a new file.

Happy to send you a before and after file of a minute or so duration showing what it can do, if you'd like that.

This is about the only thing that I would need to replace the ancient Quicktime Player 7 with lossless-cut as my mainstay tool. Other enhancemnets wouldbe nice, but this one is pretty essential for frame level edits.

Hope this helps.

Cheers,

Ian

commented

Have you tried quicktime pro with a large file? Like 1gb, does it still take only a couple of seconds?
If it's quick then I'm thinking maybe they do a smart cut, re-encoding only the inter-keyframe portion

commented

This would be awesome really :)

commented

I more or less grew up with SvenOver's DVBcut which did implement "keyhole surgery" editing. He never updated it for HD (H.264 and etc) and, in the end, I shifted to VideoReDo which handles top-and-tailing and advert removal very efficiently and reliably using "smart cut" techniques and FFmpeg as the internal engine. Alas, it's Windows-only, not open-source and non-free (but relatively cheap) but it's a good benchmark to follow. So ... I'm absolutely enthralled if you're planning to add the facility to your LosslessCut ... all power to your elbow!

In the meantime though, I wonder if you might consider the following: There has been some discussion here already on "cut at the natural frame-break before or after the desired cut-point " (so apologies if the matter has already been settled). How about a pair of extra buttons adjacent to the < and > buttons that skip to the natural break respectively before or after? That way the user can choose (to taste) whether to exclude the last (intrusive) advert frame or include the first (treasured) movie frame. At the moment, it's not obvious which sacrifice one will make :-).

I'm enjoying getting to know your creation - it's a valuable addition to the linux toolkit. Thank you!

David

commented

I'd like to point out that if you use -ss to cut at a keyframe after the input, the resulting video doesn't seem to play the first segment correctly. Setting it to a tenth of a second before the keyframe does work, but it does go to show that using -ss before the input is superior.

It's a quirk to keep in mind when implementing smart cut.

commented

So ffmpeg will choose the nearest keyframe

Does this ever lose frames? If there are key frames at frames 10 and 15, and your trim starts at frame 13, does it take the *nearest keyframe?

Do you lose the first 2 frames of video? Or does it look for the first keyframe before/after your in/out points?

commented

Seeing as @ilaw138 didn't reply, I can. I've been using QTP7 and MPEG Streamclip for the past 10 years to manage the media files of a weekly show at work. It's possible to do a perfect cut on any size file, here are the steps:

  • Trim to the exact position you want using the in/out markers in QTP7
  • Save it as a separate file (you can use QTP7's reference file option though so it will save a new file however fast your computer can write to disk and the size of the file)
  • Open the file in MPEG Streamclip and select File > Save As
  • In the save dialogue box change the dropdown to be mp4 and Save.
    Done, no reencoding, exact to the frame and plays as expected. Feel free to ask me to provide you anything you need that might help you to solve this issue.
commented

Hi. Thanks for the info. I'm not really sure how QuickTime Pro does this, and I don't really understand why you need to use Streamclip in addition to quicktime pro?

commented
commented

... and I don't really understand why you need to use Streamclip in addition to quicktime pro?

Quicktime doesn't let you save a clipped file as a .mp4, only a .mov, Streamclip allowed me to get it back to .mp4 without recompressing. The only other way to get it out of QTP7 as a .mp4 was to export it but then it would compress it to those export settings.

commented

QTP7 didnā€™t really like large files in any event - at least in my experience
Have found if I save file from QTP7 (as opposed to export) it didnā€™t cut the frames exactly - so assume using key-frame - but export does.

Not meaning to contradict you Ian but my experience working with QTP7 for 10 years pretty much every day was totally different. It regularly handled the 5, 10 or even 50G files I threw at it and its accuracy was spot on to the frame each time. If it wasn't I wouldn't have been able to use it as the transitions between segments in the program had to be edited out to the frame and QTP7 worked flawlessly each time. I also used the info panels to find out the duration to the millisecond (see the screenshot of a video's details, note the Current Time)

Screen Shot 2020-02-15 at 2 24 29 am

If it's helpful I can do a screencast of myself using it and how accurate it was... if it's going to help.

commented
commented

By the way I think normal quicktime that comes with mac os (not pro) can aslo trim losslessly. I just tried a relatively big file and it was instant and accurate, so im thinking it must be lossless. However it only works on mov/mp4

commented

In my opinion this feature would be a key feature for lossless-cut. Maybe 'avcut' would be a good solution. I have been using it for some time now for smart rendering in my project.

Here you can find the repository, and here is a blog about it.

If you like, you can test a static build (Linux 64Bit) of mine.

Even if only a few frames (before and after the keyframes) have to be rendered, this editing variant still needs much more CPU resources. If one could build avcut/ffmpeg in combination with hardware acceleration (vaapi / qsv), it would probably almost eliminate this disadvantage.

commented

I tried starting to implement a keyframe accurate cutting with re-rendering the part outside the keyframes, similar to avcut, but I was not able to make it work consistently. When I cut exactly on keyframe timestamp, sometimes the beginning of the output file gets corrupted, sometimes not. depending on the video file I test on.

I managed to figure out how to render key frames effectively on the timeline however, so in the newest version you will have keyframes show up on the timeline, and you can zoom in to seek closely. Also now seeks should align with frames based on frame rate.

commented

This issue along with #372 should be a pinned issue.

commented

Sorry I didn't read the whole thread. But can't the out point cut be precise, as long as it's NOT on an i-frame?

The next frame (beginning of the next segment) would be an I-frame?

commented

I just used handbrake it is precisely cutting but does re-encoding as well. Maybe you can implement like them?

commented

@mattack I think for most files, the out cut is indeed precise.

commented

This seems like a very interesting feature and upon checking I found this answer on StackOverflow
https://stackoverflow.com/a/63604858

Basically the idea when having a start frame not on a keyframe seems to be
-doing a (minimal) reencoding between the exact start frame and the first keyframe
-copy without reencoding between the first keyframe and the end frame.
-concatenate both segments

I tried myself and seems to be working. Do you see any problem with this procedure, for example with audio?

By the way, your program is awesome @mifi

commented

@pldavid2 That's how I'm thinking it would work also. The biggest problems I think are:

  1. need to find out where to set the start cut point for the lossless cut. Sometimes when cutting at or near a keyframe, ffmpeg will instead cut at the frame before that, depending on the file.
  2. I believe the re-encoded part needs to have the exact same parameters like encoding profile, resolution, color space, FPS, bit depth, bitrate, and possibly more parameters. And this all differs from codec to codec.
  3. Audio, subtitles etc needs to be cut separately and we would probably need to do the same with some audio codecs? (e.g. re-encode some part of it with exact same parameters)
  4. Then need to re-merge the separately cut audio/video/other tracks and hopefully everything will stay in sync.

We could add a smart cut like this as an experimental option for selected codecs like h264, h265, aac, mp3, but there is still a lot of work to implement that unfortunately.

commented

For cutting commercials from a movie, there's still no better tool than QTP7. I'm keeping an old iMac running just for this purpose.

I have found that the only usable option to save the video is Save. The Export option, even with passthrough set for video and audio, removes all extra audio tracks, e.g. additional language tracks.

commented

You might be able to steal some code to do this from https://github.com/ozmartian/vidcutter.

commented
commented

It's probably the most wanted feature for my current needs. Thanks a lot!
Can it also work for merging videos? Sometimes I have one of two videos in a different format ā€” and I want to convert only those videos, leaving the main flow intact (and lossless).
It's technically the same as "re-encoding the part between the cutpoint and next keyframe" (as long as those videos have the same resolution and frame ratio and only codec is the difference).

commented

Looslesscut is a very handy tool i like much.
However my current need is accurate cuts.

So I make a piece of code that allows me to have smart cuts, but by continuing using Losslesscut: https://github.com/strawhatgami/smart-cut-llc

commented

I wish lossy mode was available in addition to normal & keyframe cut modes. Compared to other tools, losslesscut is lightweight and easy to use, but I often end up reencoding with something like ffmpeg -i input.mp4 -ss 00:00:01.600 -preset slow -b:v 5000k to remove extra (milli)seconds from the resulting file.

commented

@strawhatgami cool that you made a script that solves your problem! If I understand correctly, your script will re-encode the whole video. AFAIK that is actually the same as the lossy cut mode that I plan to implement in the not-too-far future, see #372

commented

@mifi Why can't the LLC reconstruct the frame where you want to cut from the most recent key frame and all intermediate partial frames? Isn't that exactly what every video player does when you play a video?

commented

@Droyk yes, but losslesscut doesn't operate on frames, it just operates on timestamps, and the player is doing that internally. AFAIK ffmpeg doesn't have any built in way to reconstruct (re-encode) only the part until the key frame, so losslesscut has to implement that, using exact same codec parameters as the codec in question.

commented

Have now made an initial implementation of this.

This experimental feature will re-encode the part of the video from the cutpoint until the next keyframe in order to attempt to make a 100% accurate cut while losslessly copying the rest of the segment.

Note:

  • Only works on some files:
    • I've had success with some h264 files, and only a few h265 files seem to work.
    • Other codecs may or may not work.
  • Currently only works on Linux/Windows. I'm working on Mac support (MacOS H264/H265 hardware encoder (h264_videotoolbox) seems to produce corrupted files when merged with the original, need to use x264, x265)
  • Only supports cutting a single video stream. All other video streams have to be disabled (in the tracks panel). You can still probably do multi-passes and then recreate a file with all the streams.
  • Only scans up to 60 seconds to find a keyframe.
  • It tries to auto-detect bitrate from input stream, however if it fails, then it will calculate bitrate from the whole file's size/duration (may be wrong)
  • Tries to copy timebase from source to destination

Inspired by smart-video-cutter.sh fronm #330 (comment)

Thanks @fernandoherreradelasheras !

commented

Hi there @mifi ,
I am very pleased to have found this tool and commend you on your efforts to implement a method of cutting a stream at a precise location. However, I would like to re-iterate earlier comments that QuickTime 7 Pro is able to edit at any frame, and then save without doing any re-encoding (to the best of my knowledge; certainly the save process seems too fast for there to be re-encoding happening).

I understand the very basics of video codecs but don't really know much about the finer points. Let's say there's a keyframe at frames 45, 55 and 65, and the user cuts out frames 50 to 60 inclusive. Is it possible that when QuickTime does the edit, the saved file ditches the intermediate frames from 50 to 54, but retains the keyframe at frame 55, and then puts something in the stream that instructs the player to skip from frame 49 to 61 when playing back?

I know this offer was made earlier, but it wasn't responded to directly, so I would like to ask again: would it be instructive for you to inspect a video file that has been edited using QuickTime 7? If you provide a source file, I would be happy to edit it at specific points as specified by you and then send back for you to inspect. I feel strongly that it would be a real coup to figure out exactly how QuickTime 7 does its trick!

EDIT: seems like it probably is as I described above, using "edit lists/atoms":
https://forum.videolan.org/viewtopic.php?t=115190
https://developer.apple.com/library/archive/documentation/QuickTime/QTFF/QTFFChap2/qtff2.html#//apple_ref/doc/uid/TP40000939-CH204-BBCCFBEF

EDIT 2:
Scott Lamb, who appears to be active on GitHub, claims to use Edit Lists in a project of his:
https://news.ycombinator.com/item?id=16889403
Perhaps you could get in touch?

commented

Have now made an initial implementation of this.

This experimental feature will re-encode the part of the video from the cutpoint until the next keyframe in order to attempt to make a 100% accurate cut while losslessly copying the rest of the segment.

Note:

  • Only works on some files:

    • I've had success with some h264 files, and only a few h265 files seem to work.
    • Other codecs may or may not work.

  • Currently only works on Linux/Windows. I'm working on Mac support (MacOS H264/H265 hardware encoder (h264_videotoolbox) seems to produce corrupted files when merged with the original, need to use x264, x265)

  • Only supports cutting a single video stream. All other video streams have to be disabled (in the tracks panel). You can still probably do multi-passes and then recreate a file with all the streams.

  • Only scans up to 60 seconds to find a keyframe.

  • It tries to auto-detect bitrate from input stream, however if it fails, then it will calculate bitrate from the whole file's size/duration (may be wrong)

  • Tries to copy timebase from source to destination

Inspired by smart-video-cutter.sh fronm #330 (comment)

Thanks @fernandoherreradelasheras !

I tested it on macOS with a full HD recording from Zoom, it worked flawlessly. The encoding took less than a minutes, since it's a screen sharing recording, there weren't many key frames, using smart cut is a must.

commented

@HarryDymond that's very interesting research. I'm also guessing that's how Apple does it. Although I assume it only works for MOV files, it would indeed be an interesting alternative smart cut implementation. The problem is that I also have the same basic understanding of codecs as you have, so I don't know how to implement it. I'm not going to implement a MP4 writer in C/C++ so we would need a MOV atom editor software library or tool that supports Edit List Atoms, and I don't know of any library that can do that.

I'll try to mention @scottlamb here, and see if he'd be interested in sharing how he implemented writing Edit List Atoms to mov files.

commented

I'll try to mention @scottlamb here, and see if he'd be interested in sharing how he implemented writing Edit List Atoms to mov files.

Sure. Yeah, my software uses edit lists for something similar.

Let's say there's a keyframe at frames 45, 55 and 65, and the user cuts out frames 50 to 60 inclusive.

I'll assume simplicity for the moment: key frame = IDR frame, and each non-IDR frame is a P frame using the previous frame as a reference, forming a chain, so reconstructing a frame needs everything up to and including the last key frame. There are other scenarios possible though. B frames (which I think are common in Blurays or the like but never used by my security cameras). Some of my IP security cameras have a "SVC" mode which makes half of the frames non-reference frames. Some support more intricate patterns of reference frame use.

Here's how the .mp4 could be written in this scenario: omit frames 50ā€“54, write frames 55 onward, and have an edit list that makes frames 55ā€“60 not get displayed (by skipping over their time range). No encoding necessary. Frames 55ā€“60 are still in the file and can be used as references by frames 61ā€“64 even though they're not displayed. Now, this means that someone can later remove the edit list and see those frames, so this approach is only valid if that's acceptable privacy-wise.

My software isn't using ffmpeg, so I don't know ffmpeg's options to achieve this.

commented

Thanks for your detailed explanation Scott, that's very helpful. Do you use an open source library or tool to write the edit lists to the .mp4, or some proprietary code or tool?

commented

It's my own open source code. It's specialized to what I'm doing (serving byte range requests for my HTTP API from my database schema, trimming away just the start and end of a "segment" as defined in my glossary) rather than a general-purpose library.

https://github.com/scottlamb/moonfire-nvr/blob/841e06e35458db1e40e16c72730cd2959cfdc5e6/server/src/mp4.rs#L1372

commented

I've tried the smart cut mode and there's a question.
I saw the commandline llc invoked was like this:

/usr/share/losslesscut/resources/ffmpeg \
    -hide_banner \
    -i /home/xx/Videos/test/input.mp4 \
    -ss 2255.48540 \
    -t 2.01427 \
    -map 0:0 \
    -c:0 h264 \
    -b:0 2627472 \
    -map 0:1 \
    -c:1 copy \
    -ignore_unknown \
    -video_track_timescale 16000 \
    -f mp4 \
    -y \
    /home/xx/Videos/test/input-smartcut-segment-encode-0.mp4

This took more than 3 minutes to finish, despite there's only 2 seconds of content to re-encode. I've previously seen on this thread about the position of -ss and -to/-t argument affecting ffmpeg behavior, so I went ahead and manually ran this:

/usr/share/losslesscut/resources/ffmpeg \
    -hide_banner \
    -ss 2255.48540 \
    -t 2.01427 \
    -i /home/xx/Videos/test/input.mp4 \
    -map 0:0 \
    -c:0 h264 \
    -b:0 2627472 \
    -map 0:1 \
    -c:1 copy \
    -ignore_unknown \
    -video_track_timescale 16000 \
    -f mp4 \
    -y \
    /home/xx/Videos/test/1.mp4

which is exactly the same as the one llc invoked, except I moved the -ss and -t before -i. This command finished in less than 2 seconds. This resulted in a file with different size, but it has the same duration and I think it's literally the same as the one produced by llc's command. I've attached their corresponding mediainfo result.

I wonder if you could adjust the argument position to speed up smart cut?

llc-smartcut-mediainfo.txt
custom-encode-mediainfo.txt

commented

Thanks for researching. Can you see if the files are exactly the same? Are the frames the same around the cutpoint in the output?

I did add a comment in the code when I implemented it, that having -ss before -i will lead to issues:

lossless-cut/src/ffmpeg.js

Line 826 in 9ddcd85

// cannot use -ss before -i here (will lead to issues)

...but I don't remember what kind of issues. So it's something to look more into.

commented

It's hard to tell. The file sizes are different, but according to qctools:

  • both file contains 60 frames
  • the video part seems to be identical (based on color statistics)
  • but the audio part slightly differs (by roughly 1 or 2 frames)

It's a livestream recording without much movement, so I can't tell much from the picture movement whatsoever, and the original file is too large to share. I'll try some other samples once I get the time.

commented

I tested again with -ss before -i for the smart cut encoding, and I can see that output files get a black section in the beginning. However what seems to work is -ss -i input.mp4 -ss 0, for some strange reason. No black section at the beginning, and it's fast too. So I will try that for the next version of llc.

commented

I tried the latest version (v3.45.0) and it indeed is fast! Cutting the same amount of video range only took several seconds (instead of several minutes). The result is quite promising too - for h.264 files (I've not got any h.265 samples so can't test that). Thank you very much for the quick update!

commented

So ffmpeg will choose the nearest keyframe

Does this ever lose frames? If there are key frames at frames 10 and 15, and your trim starts at frame 13, does it take the *nearest keyframe?

Do you lose the first 2 frames of video? Or does it look for the first keyframe before/after your in/out points?

I seem to only have reliable results in my program if I use a timecode that is in the "middle" of a frame's lifetime. In other words, to keep a frame, use its starting timecode plus 1/2 of the frame delay (1/60 of a second if video is 60fps). Here's some of my C# code: https://github.com/poikilos/RetroEngine-cs/blob/433773855ab8e792f9756ed0b39c80d4976a097a/RConvert.cs#L257 and below I've converted it to Python:

from __future__ import division
# ^ make sure only `//` (not `/`) can force floor division in Python 2
import math

def frame_to_ffmpeg_timecode(frame_number, fps):
    fps = float(fps)
    frame_remainder = float(frame_number)
    second_to_hour = 60.0 * 60.0
    second_to_minute = 60.0
    hour = int(frame_remainder / (fps * second_to_hour))
    frame_remainder -= float(hour) * (fps * second_to_hour)
    minute = int(frame_remainder / (fps * second_to_minute))
    frame_remainder -= float(minute) * (fps * second_to_minute)
    second = int(frame_remainder / fps)
    frame_remainder -= float(second) * fps
    sec_per_frame = 1.0 / fps
    millisecond = int(frame_remainder * (1000.0 / fps) + 1000.0 * sec_per_frame / 2.0)
    # ^ add sec_per_frame / 2.0 to get to the "middle" of the frame--so as not to undershoot!
    return "{}:{}:{}:{:0=3d}".format(hour, minute, second, millisecond)

If you know why my code wasn't frame accurate without this function, feel free to ignore it, but if you're having trouble with frame accuracy in FFMPEG I suggest you use this code :edit: or just millisecond = int(frame_remainder * (1000.0 / fps) + 1000 * sec_per_frame / 2.0) [code fixed June 20, 2022] if you can fit that into your code. I understand this is a keyframe issue, but using a timecode that is at the exact beginning of a frame (possibly messed up by float storage) seems to require this workaround in ffmpeg.

commented

I have a doubling of the first few frames with smart cut, not with lossless methods. Anyone else have this issue?

I'm cutting a webm file with VP90 codec

commented

@Poikilos that is interesting. I think I experimented with something like this before, but I think I couldn't find any value to add that will work consistently for all files I was testing with. (sometimes +1/2 a frame duration, sometimes +3/4 a frame duration was needed). I wonder why counting frames multiplied by fps is beneficial. I would think it would be the same as using the time values given by ffprobe, and maybe even less accurate because then we would have to assume that fps is an accurate number and assume that the file doesn't use variable fps or custom frame PTS.

commented

@mifi I had some trouble with other aspects of the coding back then but I've been getting back to making a rotoscoping program, so the point is to only store modified frames and a minimal cache for markers. Adding only a tiny fraction of a frame is what seemed consistent (I was doing SPF/2 instead of 1000*SPF/2), so float accuracy may be all I was overcoming. It seemed good for NTSC dropped frame (29.97 fps). Adding a tiny fraction was a bug in my code I fixed here, so less than half a frame may be better. The one liner could be replaced by adding an offset to a timecode. I'm not sure if the code will behave differently on start times vs end times but that's something to consider as well. Another factor I didnt know back then is that 29.97 and 23.98 and 59.94 aren't the true values. Dropped frame video actually uses a fraction. :edit: The best starting point seems to be to use ffmpeg's own formulas I just found: https://ffmpeg.org/doxygen/2.2/timecode_8c_source.html#l00084 but if using Python cast to int using JavaScript use Math.floor() to match where C stores a result as int.

commented

Ok. I'd really like to get to the bottom of the accurate seeking issue, but I tried many times and I never found something that works consistently. At least I have one more thing to try next time

commented

I doubt you'll make it very far. As much as I'd love for there to be a solution, until ffmpeg supports indexing by frame number, this may not be possible. See https://ffmpeg.org/ffmpeg.html#:~:text=-ss%20position%20(input%2Foutput)

By all means, prove me wrong :)

commented

With lossless cutting the right timecode may not enough but still seems correct and may avoid issues. Another situation where it isn't enough is if -noaccurate_seek is put in with PR 13 so it seems like that argument should be optional. If all that isn't enough, it will at least be correct, and if there is some third factor (the second being 1/4 or whatever offset if that is still necessary after that, possibly no offset for the end) that could be explored.

commented

@Poikilos -noaccurate_seek only exists in that PR (not in losslesscut codebase), and that PR is not merged and probably never will be. It would be nice if someone who knows ffmpeg internals well could explain exactly how seeking works with regards to timestamps, frames and keyframes. But I'm afraid the reason that it's not documented anywhere is because it's not really consistently implemented across formats/codecs/params in ffmpeg so if someone were to document it, it would be wrong.

This stuff is not really related to smart cut though, so I created a new issue where we can discuss this core problem of how to actually seek in ffmpeg.

commented

Have you guys taken a look at StaxRip? It is able to do lossless cuts in between keyframes. I think it uses Avi/Vapour Synth to do so.

commented

Reporting my experience with the smart cut feature. For my files, recorded with the default settings of OBS it never works, the output is a file composed by:

  • the re-encoded, precisely cut part
  • the not re-encoded part but it does not start at the cut point, much earlier

Reading online I thought it was a limitation of ffmpeg, but I tried to do what I think lossless cut is doing under the hood, directly with ffmpeg, and it worked.

My file had a very long black part in the beginning that I wanted to cut, but lossless cut was unable to do it, it either kept all of it, or cut too much (or present the weird behavior described above with smart cut). With this command I discarded the first 9.5 seconds and re-encode from there to the nearest keyframe.

.\ffmpeg.exe -i "input.mkv" -ss 9.5 -to 10.468 -o -c:a copy "part1.mkv"

Then I created this txt file for the concat command:

file 'part1.mkv'
file 'input.mkv'
inpoint 10.468

And merge the re-encoded part and the original file with:

.\ffmpeg.exe -f concat -i "input.txt" -map 0:v -map 0:a -c:a copy -c:v copy "output.mkv"
Here are the original file's media info 
General
Unique ID                                : 329460729124114956463710958428421807938 (0xF7DBD4102F9C88DDD61B13ADED82DF42)
Format                                   : Matroska
Format version                           : Version 4
File size                                : 1.53 GiB
Duration                                 : 47 min 30 s
Overall bit rate                         : 4 608 kb/s
Writing application                      : Lavf59.16.100
Writing library                          : Lavf59.16.100
ErrorDetectionType                       : Per level 1

Video
ID                                       : 1
Format                                   : AVC
Format/Info                              : Advanced Video Codec
Format profile                           : High@L4
Format settings                          : CABAC / 4 Ref Frames
Format settings, CABAC                   : Yes
Format settings, Reference frames        : 4 frames
Codec ID                                 : V_MPEG4/ISO/AVC
Duration                                 : 47 min 30 s
Width                                    : 1 920 pixels
Height                                   : 1 080 pixels
Display aspect ratio                     : 16:9
Frame rate mode                          : Constant
Frame rate                               : 23.976 (24000/1001) FPS
Color space                              : YUV
Chroma subsampling                       : 4:2:0
Bit depth                                : 8 bits
Scan type                                : Progressive
Default                                  : Yes
Forced                                   : No
Color range                              : Limited
Color primaries                          : BT.709
Transfer characteristics                 : BT.709
Matrix coefficients                      : BT.709

Audio
ID                                       : 2
Format                                   : AAC LC
Format/Info                              : Advanced Audio Codec Low Complexity
Codec ID                                 : A_AAC-2
Duration                                 : 47 min 30 s
Channel(s)                               : 2 channels
Channel layout                           : L R
Sampling rate                            : 48.0 kHz
Frame rate                               : 46.875 FPS (1024 SPF)
Compression mode                         : Lossy
Delay relative to video                  : -67 ms
Title                                    : simple_aac_recording
Default                                  : Yes
Forced                                   : No
commented
commented

Is this supposed to be working? I'm trying to smart cut a small segment but I'm only getting an audio output and not video

commented

@derickso see my OP. try with different file with a different codec (like h264) and see if it works

commented

Where is the option to Smart Cut?

Thanks

commented

I'm having the same issue as flagg19 with the latest release. Using ffmpeg directly works

commented

Is it possible to have the re-encode be lossless?
Every time I using it, the re-encode is really low quality.

commented

Is it possible to have the re-encode be lossless? Every time I using it, the re-encode is really low quality.

No, re-encode can not be lossless for most codecs. However if you're seeing really low quality on the re-encoded part, maybe there is something wrong with the code that detects bit rate from the input:

lossless-cut/src/smartcut.js

Line 50 in 125112e

let videoBitrate = parseInt(videoStream.bit_rate, 10);

in your dev tools do you see the warning message Unable to detect input bitrate?

commented

No, re-encode can not be lossless for most codecs.

Wouldn't crf 0 mean lossless? Or at least close enough to lossless?

in your dev tools do you see the warning message Unable to detect input bitrate?

My mistake, just tried it with a different file and I see the Unable to detect input bitrate error.

commented

Wouldn't crf 0 mean lossless? Or at least close enough to lossless?

close to lossless yes, but also very high bit rate. I imagine it could cause problems if the bitrate of the re-encoded part is much higher than the rest of the video. that's why the code tries to match the bitrate to the existing video's bitrate.

maybe the ffprobe-reported bit rate of your video stream is wrong (too low), and that's why you're getting the low quality encode. Not sure what's the best way to solve it

commented

Add a way to override the re-encode bitrate in the settings? Or a default lower bound could be calculated from file format & image size?

commented

Where is the option to Smart Cut?

Thanks

commented

Where is the option to Smart Cut?

Thanks

It shows up in the Export screen:

image

commented

in your dev tools do you see the warning message Unable to detect input bitrate?

Sorry, made a mistake, I do have this error.
I thought it appears when the video was added not when exporting, my bad šŸ˜†

smartcut.js:52 Unable to detect input bitrate
py @ smartcut.js:52
processTicksAndRejections @ node:internal/process/task_queues:96
await in processTicksAndRejections (async)
processTicksAndRejections @ node:internal/process/task_queues:96
await in processTicksAndRejections (async)
(anonymous) @ useFfmpegOperations.js:369
(anonymous) @ index.js:57
await in (anonymous) (async)
m @ index.js:54
(anonymous) @ index.js:59
await in (anonymous) (async)
processTicksAndRejections @ node:internal/process/task_queues:96
await in processTicksAndRejections (async)
m @ index.js:54
(anonymous) @ index.js:59
processTicksAndRejections @ node:internal/process/task_queues:96
await in processTicksAndRejections (async)
processTicksAndRejections @ node:internal/process/task_queues:96
await in processTicksAndRejections (async)
m @ index.js:54
(anonymous) @ index.js:59
processTicksAndRejections @ node:internal/process/task_queues:96
await in processTicksAndRejections (async)
m @ index.js:54
(anonymous) @ index.js:59
await in (anonymous) (async)
m @ index.js:54
(anonymous) @ index.js:74
e.exports @ index.js:12
(anonymous) @ useFfmpegOperations.js:436
await in (anonymous) (async)
(anonymous) @ App.jsx:1297
onClick @ ExportConfirm.jsx:242
Pe @ react-dom.production.min.js:54
Ge @ react-dom.production.min.js:54
(anonymous) @ react-dom.production.min.js:55
Fr @ react-dom.production.min.js:105
Br @ react-dom.production.min.js:106
(anonymous) @ react-dom.production.min.js:117
cc @ react-dom.production.min.js:274
Re @ react-dom.production.min.js:52
Yr @ react-dom.production.min.js:109
Wt @ react-dom.production.min.js:74
Ht @ react-dom.production.min.js:73
commented

After cutting several hundreds of h.264 videos with smart cut, I noticed that an average of over 50% of them does not work very well.

I thought I know the basic workflow when the starting point is not a keyframe:

  1. LLC re-encodes a small segment - from the keyframe before the starting point, to the first keyframe after the starting point - and cut from the starting point to the end of the segment;
  2. it then keyframe-cut - from the first keyframe after the starting point, to the exact ending point;
  3. it then concatenates these two parts;

This process has two issues:

  1. It requires at least one keyframe in your cutting range.
  2. The resulting video has a slight audio mismatch at the combining point (the first keyframe after the starting point). This it what I meant by "over 50% of them does not work very well".

Issue 1 is not really a big deal. My samples are almost always key-framed per 5 seconds, and 5% of my time I need to cut a very small piece from within the 5-second range. I can workaround this issue by simply cutting a range from a keyframe to the next keyframe, then manually cut and re-encode the range I need with raw ffmpeg commandline.

Issue 2 is what annoys me the most. I have no idea what is happening. I could only guess that probably the ending time of the re-encode segment is not accurate enough. It does not always happen, though.

commented

I hope this isn't related, but I'd thought I should share it: The video may have a bad timecode (a decimal framerate that isn't precise): https://blog.frame.io/2017/07/17/timecode-and-frame-rates/. Example:

  • 29.97 != (30 / 1.001) frame/sec (which is really 29.97002997002997002997) fractional

Other interesting notes from the article:

  • 23.98 is not fractional? Can someone weigh in on this? It is apparently 23.98 exactly--since there is no number of frames to drop that makes it align perfectly, it is just slowed down rather than using drop frames. Also, there is something called non-drop-frame (NDF) 29.97 besides drop-frame (DF) 29.97fps:
    • Video timecode after 1hr @ 23.98fps = 01:00:03:14
    • Audio timecode after 1hr @ 24fps or 29.97fps DF = 01:00:00:00
    • Audio timecode after 1hr @ 29.97fps NDF = 01:00:03:18 (NDF, so slowed down rather than using drop-frames)
  • timecode and framerate are not the same thing: "Timecode is a way to label frames in a recording and frame rate is the speed" So maybe there is a choice that has to be made? Which does ffmpeg do?
commented

Hello guys,
I'm working since many years in a company that provide equipment made to generate Live TV, especially in TS output format (Sat, Terrestrial, IPTV, ...). So I know this format quite well... ;)
I was an audio engineer, now working in the Sales Engineering department.

I'm a newbie on LossLessCut, that I start using to edit my Terrestrial movie recordings.
LossLessCut + SmartCut feature is really a great combination. Many thanks @mifi for working on this !

When I read @JokerQyou issue report, something immediately pops-up in my mind : as you know, a live TS stream is made of successive video and audio frames, but these frames have generally not the same duration, so we could say they are "keyframed" at different (asynchronous) positions.
As a result, when a video KeyFrame is selected on LossLessCut for cut/merge there's indeed high chances that the transition occurs inside an audio frame. A common result is that when you cut a file, a portion of the segment-to-be-removed remains in the start/end audio frame. Another consequence is that "gluing" two audio packets will cause audio "glitches" at playback, as they contain unwanted audio samples.

@mifi, maybe you already know this.
Do you have implemented a strategy to manage "portions" of audio frames at cutting points ?

Again, many thanks,
Best regards, and Happy New Year 2023 !

commented

I'm not aware of how exactly this works. And i'm not aware how ffmpeg implements cutting when it comes to this TS issue, nor how to work around it. What if you remux the TS to something else like MP4 or MKV? Does the glueing issue go away then?

commented

How is ffmpeg even able to losslessly cut a video+audio file if the audio can't be split at a video keyframe? This question seems tangentially related but it's only just crossed my mind after years of using LosslessCut...

Do the common encoders force the audio to be separable at video keyframes?

commented

Correct me if I'm wrong but I think audio units (keyframe intervals) are much shorter than video, so it generally works out, but there might be some glitches as described above when merging two such cuts?

Also I think ffmpeg will offset the timestamps of the resulting (cut) audio/video streams to make sure they match up correctly. (e.g. to make sure the player will delay or advance the audio by a few ms when playing back)

commented

Why does Smart Cut change the framerate to variable when original is constant 60 fps?

Smart cut:
Frame rate mode : Variable
Frame rate : 59.421 FPS
Minimum frame rate : 14.925 FPS
Maximum frame rate : 60.000 FPS
Original frame rate : 60.000 FPS

Non-smart cut:
Frame rate mode : Constant
Frame rate : 60.000 FPS

This seems to happen with every clip. I've seen it report a 50 fps video as 250fps and I've even seen in metadata it claiming to be 1.5 million fps.
This tends to make advancing frame by frame backwards insanely buggy, atleast in MPC-HC.
Like with the 50 fps becoming 250, when skipping frame by frame backwards, it should go 20ms but instead does 4ms, because it really thinks it's 250 fps. But only when skipping backwards for some reason, forward still goes 20ms.

Smart Cut also seems to often repeat a section of the video for some reason instead of doing what it's supposed to, but the repeated section is often also significantly lower quality.
Also even when that bug doesn't happen, Smart Cut is very often much lower quality than the parts within the keyframes. Please give us an option like Smurfix asked #126 (comment)
Or atleast give us an option to set CRF manually, because whatever math Smart Cut is doing isn't working out. Nor should it try to match the bitrate, because a re-encode will always have lower quality unless you increase the bitrate.

commented

smart cut uses ffprobe to detect the fps it will use when encode. you could try to run ffprobe and see which fps it reports for your file.
Maybe when detecting bitrate, we should slightly increase the smart cut encoded bitrate to counteract the loss of quality when re-encoding. I agree it also makes sense to allow customising the bit rate of smart cut

commented

May I ask why does it detect the bitrate of some files but not others?
I've tried smart cut on a lot of different videos but its always unsuccessful when detecting the bitrate of the file and the re-encoded part is always lower in quality.

commented

if you can provide the text from "report an error" in the help menu, then i can look into it

commented

Thanks. It seems that your file is missing the bit_rate metadata in the video stream. This is why you're getting the error.

We fallback to estimating bitrate based on file size, but it seems that the estimate could be too low, see comment above: #126 (comment)

commented

could you try to open your file in another tool like for example MediaInfo and see if it reports a bitrate for the video stream? if so, maybe it's an issue with ffprobe

commented

Of course.

Screenshot_20230706_203614

I noticed in the logs that it does show bitrate

    "mainFileFormatData": {
      "filename": "/run/media/metabox/bcbf4852-097a-4fc3-9106-3b9e7fdae001/yt-dlp/[OZC-Live]OOO Ā· Den-O Ā· All Riders - Let's Go Kamen Riders [720p][NAFmux] [A7564FE8].mkv",
      "nb_streams": 7,
      "nb_programs": 0,
      "format_name": "matroska,webm",
      "format_long_name": "Matroska / WebM",
      "start_time": "0.000000",
      "duration": "5585.664000",
      "size": "2664904698",
      "bit_rate": "3816777",
      "probe_score": 100,
      "tags": {
        "encoder": "libebml v1.4.2 + libmatroska v1.6.4",
        "creation_time": "2022-03-04T20:43:10.000000Z"
      }
commented

Yes, but thatā€™s the bitrate of the whole file, not the video stream. (Probably same as dividing file size by duration). If a file has multiple streams (esp video), it will be incorrect.

could you try ffprobe version 6 on your file and see if it is able to fetch the bit rate? Something like ffprobe -show_streams file.mkv

commented

Sure thing.
Using the system ffprobe not the one included with losslesscut, it's Arch so it's up to date.

ffprobe.txt

commented

I have to say that my first experience with Lossless Cut has been quite rough because of this and other UX issues.

  • When I first exported, I noticed that some parts I had removed were being exported anyway (default "Keyframe cut" was enabled)
  • Then I tried converting the video from mkv to mp4 and using the same segments, but the import/export functionality is weird (export will silently drop a JSON file in the working directory, but then importing it for a file with a different name will silently not do anything)
  • Exporting the project to other formats silently didn't do anything (no logs or error messages)
  • I finally edited the JSON file manually to apply the segments to the mp4 file, same result
  • I tried "smart cut", and it failed, both with mkv and mp4 (fine, it's experimental)
  • I then tried disabling the "Keyframe cut" to use the "normal cut", and the resulting file could not be read by some players or had weird playback issues

At this point I gave up and just gave the raw file to my stakeholder, apologizing for the extra seconds.

commented

@astrojuanlu I believe most of these issues are not related to the Smart Cut feature (which is experimental).

Please read https://github.com/mifi/lossless-cut/blob/master/issues.md and if you're still having issues, then you may submit a new discussion or issue with more of the requested details about your system and reproduction steps, to help me help you.

commented

On macOS, I run into the issue that the smart cut feature works only if I remux the recording in OBS from mkv to mp4. This is fine, though it would be nice if it would just work with MKVs. Right now I'm not using any NKV-specific functionality such as multiple video or audio tracks, though maybe I want to do that in the future. With the MKV file I run into the issue of degraded video quality, and that the cut is not done correctly.

commented

I'm using the "smart cut" a lot, with very good result. Except when using it on a video which is x265 encoded, then the result is distorted. Would it be possible to support the "smart cut" also for x265 encoded video's ?

commented

smart cut does work on some x265 videos, but I don't know which codec parameter combinations are causing it to fail for some files, while others are fine. it's a quite complex issue.

commented
commented

Unfortunately Smart Cut fails also on H264 files.

Sample: https://anonymfile.com/437/sample.mkv
First 8 sec of original file: https://anonymfile.com/3aa/sampleorg.mkv

This is really strange: It starts with the correct frame, then a part of the cut out frames is shown and then it starts again.
Goal was to cut out the SC ad.

A also cut something at the end. There it seems to include one frame too much. I set the cut out at the last frame before the Sony logo and that was still included.

VidCutter can handle the file but will only cut one audio.

commented

So far, EVERY file failed with smartcut. It seems, the cut out segment is one keyframe too early.

I usually cut something at the beginning. So the reencoded part is placed first, then there is a part of one keyframe to the next from the part that should have been cut out and then it resumes normally. I can cut out the additional part in AviDemux but I would like to save that step.

commented

The issue I have may be a little odd, so let me first give you a screenshot of the timeline:

image

(Don't mind the 204, I deleted the segment previously to see if that helps and then used "Fill gaps")

As you can see I need to cut between the two keyframes. The waveform is showing it pretty clear, there's a small piece of silence and I want/need to cut on that.
The exported video (with smart cut of course) starts at the exact right time, i.e. the video starts quiet. Perfect!
But oh wait, it's not, because at Keyframe 2 it starts inserting everything from Keyframe 1 onwards, effectively inserting the cut-away part and duplicating the wanted part. So the video goes:

smart-cut to keyframe 2 | keyframe 1-2 | keyframe 2-end segment

I'm not sure how to handle this šŸ¤”

Upon further experimenting I noticed something more. So I split Segment 19 1 frame after the keyframe (as per known issues.. shifting segments and whatnot), and then exported 19 and (the newly created) 20. Theoretically 19 should only contain the cut part, correct? But what it's doing is the same as I had in the full video: it inserts from the smart cut to keyframe 2 (now end of the segment), and then inserts everything from Keyframe 1 to 2 again šŸ¤” I'm not sure why it's inserting anything after the end of the segment at all šŸ¤”

Segment 19 export looks like this:
image

Segment 1 is what smart cut produced, Segment 2 is everything from Keyframe 1 to 2

Workaround:
image

  1. Export segment 19 with smart-cut
  2. shift segments
  3. Export segment 20 without smart-cut
  4. Open up segment 19 file
  5. Split at offending keyframe
  6. Export segment 1 without smart-cut
  7. Merge segment 1 (of segment 19) and segment 20
  8. ... aaaannnd ddoonnee

... but actually not really because for some reason this produces 2 black frames at the cut point (music keeps going), and when playing back (in mpv) it actually seems even longer then that šŸ˜ž
I should mention the "smart" portion of the cut also has visibly degraded quality even in the full Segment 19 file :/

commented

Seems like the same issue I described the two entries before.

commented

I just found out that even if I remove the duplicate part we will miss one frame. So in EVERY cut mode we have an off-by-one error.

commented

Investigate using aomav1 instead
aomav1 is already the default in ffmpeg when encoding as -c:v av1

using a real world section I trimmed earlier today that would have used smart cut (103 frames long @ 4K 8000kb/s) this is the encode times on my 5900x:

  • aomav1: 19m28s @ 98.4953VMAF
  • aomav1 -usage realtime: 5s @ 93.8847VMAF
  • aomav1 -cpu-used 8: 31s @ 98.1873VMAF
  • libsvtav1 (no custom arguments): 8s @ 96.0291VMAF

When av1 is detected when using smart cut it should either apply -cpu-used 8 to increase speed by 37 times or set the encoder to be libsvtav1 for an increase of 146 times

you could use -usage realtime for an even faster encode but the quality is pretty large compared to the other two options

commented

I'm using the latest development version of SmartCut and it works well, but I notice that the ffmpeg command to make the -smartcut-segment-encode file does not appear in Last ffmpeg commands, even though the commands to make -smartcut-segment-copy and then concatenate the two files afterward do appear.

commented

I'm also using smart cut with av1 and it's very slow. It should not take multiple minutes to encode (on a Ryzen 5700X) what adds up to a few seconds at most. Is there a way to manually change the encoder used in smart cut?

commented

I'm also using smart cut with av1 and it's very slow. It should not take multiple minutes to encode (on a Ryzen 5700X) what adds up to a few seconds at most. Is there a way to manually change the encoder used in smart cut?

Even if you were to import the source file into Handbrake and select those frames/seconds to encode, you might find it takes a similarly long time. It doesn't just encode the selected portion; it moves through the file to that portion to find it. That is, it doesn't just jump to it and start encoding, but rather works its way to it. As for changing the encoder, I'd imagine it just automatically uses the same one the source file is encoded in, as using another would cause incompatibility, though I have no idea how it chooses the various settings for the encode beyond that. Given the chance to allow the user to customize this, that would provide the ability for using incompatible settings (not saying this shouldn't be done, just that it requires caution, and I've discussed this in another issue). I'd suggest importing your source file into Handbrake, selecting the same codec and as similar as possible quality settings as the source file, and having it encode the same portion of the file that you're trying to do with LC (i.e. the bit(s) at the beginning/end of the clip(s) you're editing. I'd wager that if you do this for the beginning and end of each clip, it's going to take a while, as for each one it's going to seek through the video up to that point before actually starting to encode. And while I can't say for sure, as obviously LC and HB are two different programs, I'm guessing they both use similar mechanisms to perform these operations, and that's likely why encoding a few pieces that are only a second or two, which seems like it should be very quick, are taking a while, because it's not the actual encoding that's taking so much time, but the encoder moving through the file to get to that point (or those points).

commented

Well, I can see through task manager that a lot of the time of the total video processing is just disk operations which makes sense as LC is making a copy of a large video file, but I can also see when the disk operations go down to 0% and the CPU usage goes up which is where the smart cut takes place. When I say it takes too long I mean just the smart cut portion, as the rest is justified. I also tested it with shorter video files and the smart cut takes many times longer than running the entire video through handbrake AV1-SVT encoding (although I don't know if the handbrake settings are comparable, but still). I tested the same 10 second file converted with Handbrake from AV1 to 264/265 too. Smart cut goes through 264/265 files in like a second or two but takes minutes on AV1.

commented

I'm also using smart cut with av1 and it's very slow. It should not take multiple minutes to encode (on a Ryzen 5700X) what adds up to a few seconds at most. Is there a way to manually change the encoder used in smart cut?

This is an issue I have brought up as well, Lossless cut uses whatever the default av1 implementation of ffmpeg is, this happens to be the original aomav1 encoder and that encoders default settings are molasses slow,
I've suggested using libsvtav1 in it's place because of the speed increade.

See my previous comment here for some numbers.

Hopefully this can be implemented some time soon as it would increase the speed of these encodings by a good amount especially for much slower machines

commented

I also tested it with shorter video files and the smart cut takes many times longer than running the entire video through handbrake AV1-SVT encoding

And this is with just one clip? Even with that, it's encoding two pieces, and for each one it has to run through the file up to that point and then encode it, so for each clip it has to do this twice, and that will take more time. But if you're saying it takes that much longer to do just one clip/segment vs encoding the entire video in HB, then that does sound wrong. I wonder if it's LC or ffmpeg (or an external encoder?) taking that long. I suppose you could watch the output files and see how long each one takes to complete and see what part of the process is so slow.