Maybe a seeking error after 2.40

Question

Maybe a seeking error after 2.40

vxzms opened this issue 3 years ago · comments

@misakikasumi get a seeking error when using ffms2_api4, I also reproduced this problem in 2.40, 2.23.1 looks fine.

When seeking a video, there is a high probability that the wrong frame will be returned in test.
The test environment is Windows + Vapoursynth / Aegisub + ffms2 / lsmashsource. Here is the test (sample video link):

Dump video to y4m by vspipe dump_y4m.vpy seek-test.y4m
dump_y4m.vpy: core.ffms2.Source("seek-test.mkv").set_output()
Make a comparison of these outputs at same frame:
a. core.ffms2.Source("seek-test.mkv")
b. core.lsmas.LWLibavSource("seek-test.mkv")
c. core.ffms2.Source("seek-test.y4m")
d. core.lsmas.LWLibavSource("seek-test.y4m")
e. Aegisub open seek-test.mkv use the same ffms2 version as VS

In the test, a and e results are always the same.
outputs of b, c and d are always the same.
a and c are likely to be different.

AkarinVS · Answer 1 · Mon Sep 13 2021 03:00:26 GMT+0800 (China Standard Time)

regression introduced by 13d9f.

Derek Buitenhuis · Answer 2 · Mon Sep 13 2021 06:15:15 GMT+0800 (China Standard Time)

Can you be a bit more specific on what types of files this affects? You've only said "a video" - and it definitely isnt all videos, (not e.g. MP4 AFAICT). Is it specific to Matroska? H.264 in Matroska? Blu-ray H.264 in Matroska?

Derek Buitenhuis · Answer 3 · Mon Sep 13 2021 06:21:16 GMT+0800 (China Standard Time)

Also can you test with master, particularily after: 5da8f49

Deleted user · Answer 4 · Mon Sep 13 2021 07:50:50 GMT+0800 (China Standard Time)

It is not specific to Matroska. In fact, I originally spot this with several Blu-ray m2ts files. The sample mkv file is actually a m2ts remux.

AkarinVS · Answer 5 · Mon Sep 13 2021 08:04:37 GMT+0800 (China Standard Time)

It should affect all avc video files. The latest master branch (rev 9525fcd) also exhibits this issue. I wrote a script to test ffms2 against lsmas. https://github.com/AkarinVS/weird-scripts/blob/master/ffms2/issue394.vpy

Derek Buitenhuis · Answer 6 · Mon Sep 13 2021 08:22:14 GMT+0800 (China Standard Time)

It definitely does not affect all AVC files, at least not via the C API - I've had petabytes of videos run via that + tests (albeit almost totally only MP4s) running though it since that commit when in. I've not tested via the VS plugin, though.

I have a few possible suspicions:

It's posible it may not affect MP4 so I didn't notice (all the unit tests are MP4s, and I mostly only send those though).
May be specific to the kinds of flags you'd get in a Blu-ray (fields, or fake fields).
It's possible FFmpeg's behavior changed since 2018 and made that commit no longer work - particularily if the delay calculation is no longer valid, it could trip the PAFF check.

I am leaning towards the third possibility, since I would have expected this to come up sooner in the previous 3 years since that commit, otherwise. That also means it will be a giant headache to debug...

I will check into it this week - any further info is of course welcome 🙂 .

vxzms · Answer 7 · Mon Sep 13 2021 08:30:00 GMT+0800 (China Standard Time)

I used Vapoursynth-R54, ffms3000 and lsmas to reproduce it many times, both avc and hevc, mkv and mp4.

Derek Buitenhuis · Answer 8 · Mon Sep 13 2021 08:41:04 GMT+0800 (China Standard Time)

I definitely suspect an FFmpeg behavior change, then.

AkarinVS · Answer 9 · Mon Sep 13 2021 10:23:04 GMT+0800 (China Standard Time)

I built the ffmpeg release 3.3.7 (commit 93e2cb4511417d265f3644c122167678fe4d3161, dated Apr 13 2018, which is about the same time frame as ffms2 rev 13d9f01) After fixing building ffms2 rev 13d9f01 with that ffmpeg version, I can reproduce the issue as well (using seek-test.mkv as in this issue). (And the previous revision doesn't exhibit the issue. In fact, I can compare these two ffms2 releases, and confirm the off-by-1 seeking as well. As both versions are built with the same ffmpeg revision, this should completely rule out the possibility of a regression in ffmpeg) Please note that regular testing won't catch this kind of seeking issues. To catch it, a dedicated test must be created: (option 1) first sequentially hashes each frame and then randomly seeks and compares frame hashes; or (option 2) compare against another frame-accurate source (lsmas probably isn't ready yet in 2018) Because when ffms2 gives wrong frames after random seeking, it's usually a frame off-by-one, so it's fairly easy to miss. I think this could help explaining why it hasn't been noticed till now. The fact that there are no official releases between 2016 and 2020 might also play a part in this.

Derek Buitenhuis · Answer 10 · Mon Sep 13 2021 15:09:53 GMT+0800 (China Standard Time)

Option 1 is how the tests I generally run are generated, interestingly enough.

Also, thanks for clarifying it is off-by-one, and testing. I'll try and dig into this.

TbtBI · Answer 11 · Mon Sep 13 2021 19:08:39 GMT+0800 (China Standard Time)

It should affect all avc video files. The latest master branch (rev 9525fcd) also exhibits this issue. I wrote a script to test ffms2 against lsmas. https://github.com/AkarinVS/weird-scripts/blob/master/ffms2/issue394.vpy

You can try this build. I have no issues with it. Here quick test between the different version. It would be great if the latest upstream builds fix the regressions.

Derek Buitenhuis · Answer 12 · Mon Sep 13 2021 20:19:40 GMT+0800 (China Standard Time)

@TbtBI that is interesting, since that is a very recent build - long after the supposed source of seeking issues...

I notice the fork of ffmpeg used in that build has: HomeOfAviSynthPlusEvolution/FFmpeg@95dfde1

TbtBI · Answer 13 · Mon Sep 13 2021 22:19:42 GMT+0800 (China Standard Time)

@TbtBI that is interesting, since that is a very recent build - long after the supposed source of seeking issues...

I notice the fork of ffmpeg used in that build has: HomeOfAviSynthPlusEvolution/FFmpeg@95dfde1

There are included patches too in the link of that build. It seems reverted back some commits:
https://forum.doom9.org/showthread.php?p=1879358#post1879358
https://forum.doom9.org/showthread.php?p=1921589#post1921589

Derek Buitenhuis · Answer 14 · Mon Sep 13 2021 22:33:23 GMT+0800 (China Standard Time)

Some of those contain revert, but that big diff contains a bunch of changes that seem to come form nowhere, without context? That's... kind of a mess.

Derek Buitenhuis · Answer 15 · Mon Sep 13 2021 23:41:56 GMT+0800 (China Standard Time)

I think this should fix it - if anyone of @vxzms @misakikasumi @AkarinVS can test it on problematic files.

diff --git a/src/core/videosource.cpp b/src/core/videosource.cpp
index 006099e..f838380 100644
--- a/src/core/videosource.cpp
+++ b/src/core/videosource.cpp
@@ -630,7 +630,7 @@ bool FFMS_VideoSource::DecodePacket(AVPacket *Packet) {
     // H.264 (PAFF) and HEVC can have one field per packet, and decoding delay needs
     // to be adjusted accordingly.
     if (CodecContext->codec_id == AV_CODEC_ID_H264 || CodecContext->codec_id == AV_CODEC_ID_HEVC) {
-        if (!PAFFAdjusted && DelayCounter > Delay && LastDecodedFrame->repeat_pict == 0 && Ret != 0) {
+        if (!PAFFAdjusted && DelayCounter > Delay && LastDecodedFrame->interlaced_frame == 1 && LastDecodedFrame->repeat_pict == 0 && Ret != 0) {
             int OldBFrameDelay = Delay - (CodecContext->thread_count - 1);
             Delay = 1 + OldBFrameDelay * 2 + (CodecContext->thread_count - 1);

Also, do you happen to have any files you know hit this bug that are less copyrighted? I want to add a unit test, but seek-test.mkv is copyrighted, obviously.

vxzms · Answer 16 · Wed Sep 15 2021 09:40:30 GMT+0800 (China Standard Time)

Also, do you happen to have any files you know hit this bug that are less copyrighted? I want to add a unit test, but seek-test.mkv is copyrighted, obviously.

~~https://samples.mplayerhq.hu/MPEG-4/CDR-Dinner_LAN_800k.mp4 also can reproduce when use ffms3000.~~

After a reminder from TbtBI, I build ffms2 master that looks fine when use vcpkg’s ffmpeg before test your diff. I'm not sure ffmpeg that myrsloik and akarin used, you can wait for them test it if you have patience.

@dwbuiten

Sorry, it’s my mistake, I used the wrong repo to create the wrong binary.

FFSM2 mater branch can reproduce the error, I test your diff, it works fine on my test video.

But I get error when test https://samples.mplayerhq.hu/MPEG-4/MPEGSolution_stuart.mp4.

Deleted user · Answer 17 · Wed Sep 15 2021 10:38:01 GMT+0800 (China Standard Time)

After a reminder from TbtBI, I build ffms2 master that looks fine when use vcpkg’s ffmpeg before test your diff. I'm not sure ffmpeg that myrsloik and akarin used, you can wait for them test it if you have patience.

Gosh… Everyone builds his own ffmpeg… Totally messed up… 😿

arch1t3cht · Answer 18 · Tue Jun 07 2022 04:20:58 GMT+0800 (China Standard Time)

I also experienced this issue (and know many other people who did), especially with hevc videos encoded with open-gop=1. I tested at least 3 different videos that are showing this bug on the current master, and the proposed change fixed it for all of them.

Also, do you happen to have any files you know hit this bug that are less copyrighted? I want to add a unit test, but seek-test.mkv is copyrighted, obviously.

I made a test video out of Big Buck Bunny 3D, which is licensed under the Creative Commons Attribution License. The subtitle file has different lines synchronized with all the video cuts. In my case (using ffms2 from Aegisub - specifically this fork, with a custom ffms2 subproject), seeking to the end of Line 3 already shows the third frame of the following shot, instead of the last frame of the current shot. Many later lines behave similarly. The proposed patch fixes all of them.

So, is there a chance of getting this fix committed to master?

Derek Buitenhuis · Answer 19 · Tue Jun 07 2022 22:13:18 GMT+0800 (China Standard Time)

OK, I'll look into doing that this week.