Ffmpeg Decoding Error

[Yu-Wu]

Hi,

I am trying to extract frames following readme (the docker command).

docker run --gpus "device=0"
-it
--rm
-v "$PWD:/workspace"
willprice/nvidia-ffmpeg
-hwaccel cuvid
-c:v hevc_cuvid
-i /workspace/videos/P02_01.MP4
-vf 'scale_npp=-2:256:interp_algo=super,hwdownload,format=nv12'
-qscale:v 4
-r 50 /workspace/rgb/P02_01/frame_%010d.jpg

However, I found there is something wrong with the decoder used in ffmpeg. Error log:

[hevc_mp4toannexb @ 0x3f16f40] Invalid NAL unit type in extradata: 8
Error while opening decoder for input stream #0:0 : Invalid data found when processing input

After debug, I found ffmpeg works only if I change the decoder from hevc_cuvid to h264_cuvid.

But I checked the extracted frames using h264_cuvid. I found the color is a little bit different from yours. It looks more gray/white. The examples are uploaded to Google Drive.

My questions are,

1. Do you use the same docker ffmpeg command (shown above) to extract frames for video P02_01.MP4 (which is an old video from EPIC-55)? If not, could you provide the command that used to reproduce the official frames?

2. Is the difference from the different decoders? If so, is there any wrong about my environment? My machine is Intel(R) Xeon(R) Gold 6148 CPU + Tesla V100-SXM2 GPU. The OS system is CentOS 6.10.

[Dima Damen]

Sure, there are different commands for EPIC-KITCHENS-55 and the extended videos. These are:

We have used different cameras and different frame rates, as explained in the paper. If you'd like more details on this, email us at: [e-mail]

2018:
ffmpeg
-hwaccel cuvid
-c:v h264_cuvid
-i PXX_YY.MP4
-vf 'scale_npp=-2:256,hwdownload,format=nv12'
-q:v 4
-r 60
PXX_YY/frame_%010d.jpg

2020:

ffmpeg
-hwaccel cuvid
-c:v hevc_cuvid
-i PXX_YY.MP4
-vf 'scale_npp=-2:256:interp_algo=super,hwdownload,format=nv12'
-q:v 4
-r 50
"PXX_YY/frame_%010d.jpg"

(Different scaling algorithm that results in crisper frames and frames are 50 FPS instead of 60 FPS)

[Yu-Wu]

@dimadamen

Thanks for your reply.

I have tried to use the provided command to extract frames, but I found my extracted frames looks different from the official frames provided by you. My frames looks more while/gray than yours.

Examples are shown in https://drive.google.com/file/d/16_OnraMglQ0O1V_Yk5tiQGVqJGP1OGJM/view?usp=sharing

Could you have a look at the difference? Any possible reasons? The videos were downloaded recently.

[Dima Damen]

Are you using the docker provided? willprice/nvidia-ffmpeg

Can you give us the exact command you've used to extract the frames?

[Yu-Wu]

The command is,

nvidia-docker run
-it
--rm
-v "$PWD:/workspace"
willprice/nvidia-ffmpeg
-hwaccel cuvid
-c:v h264_cuvid
-i /workspace/P02_01.MP4
-vf 'scale_npp=-2:256,hwdownload,format=nv12'
-qscale:v 4
-r 60 /workspace/frame_%010d.jpg

I am using the provided docker willprice/nvidia-ffmpeg.
I have also tried different FFmpeg version (both CPU and GPU). Their results are very similar.

[Dima Damen]

I see a difference between your interpolation algorithm and the one we recommend here

https://github.com/epic-kitchens/epic-kitchens-100-annotations/blob/master/README.md#erratum

scale_npp=-2:256:interp_algo=super

This could justify the difference.

[Yu-Wu]

Thanks for your reply.

Since the video is from EPIC 2018, so I used the command you provided here. #7 (comment)

So the interpolation algorithm should be the same with yours on the EPIC-KITCHENS 2018 dataset.

In addition, I have tried to add the interpolation algorithm in my command, and I found it does not have a clear affect on the color. In other words, the interpolation algorithm is not the reason for the color difference.

[Dima Damen]

I'm sorry. I don't find a difference between what you've tried and how we extracted the frames. I'll ask around, but nothing obvious I can help you with,

[Yu-Wu]

Thanks for your help. I will also search and try to find the reason.