Native support for codecs/formats (TS,H265,mpeg2++)

Question

Native support for codecs/formats (TS,H265,mpeg2++)

mifi opened this issue 6 years ago · comments

Due to losslesscut using chromium's html5 video tag which only supports certain container formats and codecs, we currently need to "convert to supported format" when trying to open exotic formats or codecs. This could be improved with the help of ffmpeg.

Workarounds

Compile custom electron

https://github.com/electron/build-tools

Convert to supported format

Currently it's possible to cut those files by first using the File -> Convert to supported format (slow) function, however this is a re-encoding (although fast) of the whole file, just to be able to view it in the HTML5 video player (note that the original and cut versions will not be re-encoded and are still lossless.)

Improve current proxy/dummy code

Currently a <canvas> element is used, and ffmpeg will stream low-resolution jpeg images to that canvas.

Drawbacks:

Slow, laggy, low resolution, low fps
No audio

This could be improved:

Electron (Chromium) might never support h265

It seems like Chrome and most other browsers are not going to be supporting H265 as it stands now, and since we are using Chromium (Electron), this will be a problem.
It might be possible to enable a flag to enable H265 decoding, but it requires some modding of electron.

Update: Chromium now supports h265 (hardware decoding only)

killaz47 · Answer 1 · Sun May 05 2019 15:47:20 GMT+0800 (China Standard Time)

heres the solution from the chrnoium developers
https://github.com/henrypp/chromium/blob/master/hevc_support.md
https://github.com/henrypp/chromium/blob/master/hevc_support_new.md

killaz47 · Answer 2 · Sun May 05 2019 15:48:36 GMT+0800 (China Standard Time)

Now that more and more cameras (including DJI Mavic 2) are starting to support H265, it would be nice to have support for this here as well.
Currently it's possible to cut those files by first using the File -> Convert to friendly format (slow) function, however this is a re-encoding (although fast) of the whole file, just to be able to view it in the HTML5 video player (note that the original and cut versions will not be re-encoded and are still lossless.)

It seems like Chrome and other browsers are not going to be supporting H265 as it stands now, and since we are using Chromium (Electron), this will be a problem.
It might be possible to enable a flag to enable H265 decoding, but it requires some modding of electron.

If anyone has some bright ideas, shout out!

https://stackoverflow.com/questions/36382568/h-265-hevc-web-browser-support

webtorrent/webtorrent-desktop#561

electron/libchromiumcontent#198

https://github.com/strukturag/libde265.js

heres the solution from the chrnoium developers
https://github.com/henrypp/chromium/blob/master/hevc_support.md
https://github.com/henrypp/chromium/blob/master/hevc_support_new.md

Mikael Finstad · Answer 3 · Mon May 06 2019 23:40:52 GMT+0800 (China Standard Time)

I'm not sure how to do this with electron. (LosslessCut uses Electron which uses Chromium.) Feel free to open an issue at electron to see if this can be integrated. I think it is not a trivial task, so I don't have time to look at this now.

TechManiacHD · Answer 4 · Thu May 30 2019 13:45:15 GMT+0800 (China Standard Time)

Lately I swap my gear and see, that recording in HEVC format gives me better quality over H264 even with lower bitrate. I would love to switch to HEVC for all my projects although only Lossless Cut stops me from doing that - https://i.imgur.com/avu5AQE.png (after converting to friendly format).

Is there a chance this will work at some point or should I still sit on old format? Thanks.

Esa-Matti Suuronen · Answer 5 · Thu May 30 2019 15:32:54 GMT+0800 (China Standard Time)

I'm seeing that too with GoPro 7 HEVC videos and fast convert. The slow one works.

TechManiacHD · Answer 6 · Fri May 31 2019 08:05:31 GMT+0800 (China Standard Time)

You're right! It works with slow one (That quality xD). Enough for cutting no worries :)
Thanks, that will do for time being.

Mikael Finstad · Answer 7 · Tue Aug 13 2019 05:45:31 GMT+0800 (China Standard Time)

Great! the reason for the low quality is to make it as fast as remotely possible, and only to be used as a preview.

Antonio · Answer 8 · Sat Aug 31 2019 21:39:22 GMT+0800 (China Standard Time)

As LossLess-cut is build in electron do you ever considered to use mpv as player component? I've just found that a project to do that it's already there... https://github.com/Kagami/mpv.js/ with also some samples implementation

Mikael Finstad · Answer 9 · Mon Sep 02 2019 06:14:25 GMT+0800 (China Standard Time)

Thanks for the tip! I didn't know about mpv.js, but if it's easy to integrate then that's definitely an option.
Kagami/mpv.js#27

Mikael Finstad · Answer 10 · Mon Nov 04 2019 20:07:49 GMT+0800 (China Standard Time)

In 1.5.0 I have implemented a hack that lets us cut H265 videos, but scrubbing and playing only works at a very low fps and currently without audio.

Nico Mexis · Answer 11 · Thu Dec 05 2019 06:00:15 GMT+0800 (China Standard Time)

Getting this error when trying to merge 2 H.265 videos together. Is there something I can do about this?

PeterH · Answer 12 · Fri Mar 27 2020 01:14:49 GMT+0800 (China Standard Time)

Is it possible to add support for grabbing a frame at 4k/hevc to jpg even though the preview is not native?

Mikael Finstad · Answer 13 · Sat Mar 28 2020 22:51:36 GMT+0800 (China Standard Time)

Is it possible to add support for grabbing a frame at 4k/hevc to jpg even though the preview is not native?

I have implemented this now

PeterH · Answer 14 · Sun Mar 29 2020 00:23:02 GMT+0800 (China Standard Time)

Nice, thanks! Can't wait to test it in the next release!

Mikael Finstad · Answer 15 · Sun Mar 29 2020 19:27:44 GMT+0800 (China Standard Time)

Check out newest version now

PeterH · Answer 16 · Sun Mar 29 2020 23:31:05 GMT+0800 (China Standard Time)

Thanks, tested 3.17.9. It's working great:)

Nico Mexis · Answer 17 · Sun Mar 29 2020 23:49:13 GMT+0800 (China Standard Time)

@mifi Is there any news on supporting H265 natively?

Mikael Finstad · Answer 18 · Mon Mar 30 2020 18:17:05 GMT+0800 (China Standard Time)

No. I think it needs a lot of work because have to compile a custom electron with a custom chromium that uses a custom ffmpeg. Not something i will prioritize unless this work gets funded somehow, or if we can find the work of someone else who did this already.

desertpogona · Answer 19 · Sat Dec 12 2020 05:10:01 GMT+0800 (China Standard Time)

This is still a problem in the current build and I was wondering if progress is being made on it as video standards move forward

Mikael Finstad · Answer 20 · Sat Dec 12 2020 05:17:36 GMT+0800 (China Standard Time)

AFAIK chromium/electron still doesn't support H265 and they refuse to support it due to expensive licensing: https://bugs.chromium.org/p/chromium/issues/detail?id=684382
So we still need to build a custom electron with a custom chromium to get this native support

Graham McMicken · Answer 21 · Thu Feb 17 2022 15:18:55 GMT+0800 (China Standard Time)

I came across this guide "it’s not as difficult as it might sound" Electron with custom Chromium build

and then the instructions for HEVC support in chromium

Also noticed this gem from the chromium forums that is currently being implemented Issue 1286132: Implement HEVC for video playback and video conferencing on Windows

Linnshet · Answer 22 · Fri Feb 18 2022 00:16:37 GMT+0800 (China Standard Time)

Any updates on native HEVC implementation? The solution of making a dummy proxy is pretty brilliant, but it would be nice to see my videos in full res.

Mikael Finstad · Answer 23 · Fri Feb 18 2022 00:24:24 GMT+0800 (China Standard Time)

@gmcmicken nice finds. I've not taken on the task of building a custom chromium. However I do have an idea with the https://developer.mozilla.org/en-US/docs/Web/API/MediaSource to hopefully allow for more smooth streaming from ffmpeg to the player. It may also allow playing audio. Note that it still needs to encode the video while playing but I think h264 encoding can be done very fast these days. Could also set the duration which would remove the need for the html5ified dummy file.

romadd64 · Answer 24 · Fri Jul 22 2022 19:31:59 GMT+0800 (China Standard Time)

There seems to be some good news: Sida Zhu has implemented support for hw decoding for HEVC in chromium v104.0.5084.0, disabled by default and enabled with the PlatformHEVCDecoderSupport feature.
(https://github.com/StaZhu/enable-chromium-hevc-hardware-decoding) .
chromium 104 needs electron nighty build 21 (https://github.com/electron/releases) .
Mikael, could you build losslesscut updated to this electron version and with feature enabled? Thanks a lot!

Mikael Finstad · Answer 25 · Fri Jul 22 2022 19:58:42 GMT+0800 (China Standard Time)

that's fantastic! Then I think once electron 20 becomes stable we can upgrade to that. Note that upgrading losslesscut to newest electron is quite a big job. But I plan on doing it soon, now that electron>v8 slowness on macos has been fixed #714

romadd64 · Answer 26 · Sat Jul 23 2022 18:10:42 GMT+0800 (China Standard Time)

You are right, also latest electron 20 beta has chromium 104, with the 8-weeks release cycle we should hopefully have a stable 20 soon. I understand that going from electon 8 to 20 could be a big jump, I'll donate for your effort. Thanks.

romadd64 · Answer 27 · Sun Sep 25 2022 18:14:37 GMT+0800 (China Standard Time)

Hi Mikael,
do you have some progress on this improvement?

Mikael Finstad · Answer 28 · Mon Sep 26 2022 02:14:56 GMT+0800 (China Standard Time)

Hi. Unfortunately not, as ive been very busy with other projects. But, assuming electron now supports natively, it will be solved once Llc upgrades electron, which is one of the first next big tasks

dellhem · Answer 29 · Wed Sep 28 2022 04:52:25 GMT+0800 (China Standard Time)

Hi. Unfortunately not, as ive been very busy with other projects. But, assuming electron now supports natively, it will be solved once Llc upgrades electron, which is one of the first next big tasks

I will definitely buy the app store version to subsidise a couple of cups of coffee for your effort ;)

Mikael Finstad · Answer 30 · Tue Nov 22 2022 23:03:07 GMT+0800 (China Standard Time)

I'm testing with Electron v21 now and with app.commandLine.appendSwitch('enable-features','PlatformHEVCDecoderSupport') h265 seems to be working! so expect an alpha release of losslesscut soon

Loogaru · Answer 31 · Wed Nov 23 2022 23:48:12 GMT+0800 (China Standard Time)

OMG, It works wonderfully! I'm gonna cry now)) Quick, precise, responsive, original quality, no preview conversion... this makes my workflow much comfortable (I'm shooting long scripted 4k videos, and use LosslessCut to markdown them, cut good takes, and it really helps me save much storage space). Thank you, thank you, thank you, @mifi!

Spaarpott · Answer 32 · Thu Nov 24 2022 16:18:48 GMT+0800 (China Standard Time)

Yep. In v3.48.2 HEVC works like a charm!

Tommi Saltiola · Answer 33 · Sat Nov 26 2022 01:18:10 GMT+0800 (China Standard Time)

HEVC cutting

Works like a charm on my Fujifilm X-T4 10 bit HEVC!

Im going to cut and delete more than half of my 14 TB storage now and save a ton from storage space costs now and in the future. Will toss a coin to my @mifi

Augusto Barreto · Answer 34 · Sat Nov 26 2022 02:07:09 GMT+0800 (China Standard Time)

This is great. Is the app in Microsoft Store going to be updated also? Thanks

dom · Answer 35 · Sat Nov 26 2022 03:14:43 GMT+0800 (China Standard Time)

thank you so much for working on this app and integrating this feature :) this will improve my housekeeping tremendously

Mikael Finstad · Answer 36 · Sat Nov 26 2022 09:59:32 GMT+0800 (China Standard Time)

@titobf yes, but i want to iron out bugs first before releasing to stores. already a few bugs coming in.

samahmoo96 · Answer 37 · Sat Nov 26 2022 12:07:05 GMT+0800 (China Standard Time)

I'm having an issue. I can't seem to convert h.265 to the player in 3.48.2. It just yells at me that the file is not natively supported, I try to convert, it does, and then it yells again. Here's the media info I pulled from VLC. I could send the file as well if you needed to reproduce the bug on your end.

Stream 0
Codec: MPEG-H Part2/HEVC (H.265) (hevc)
Type: Video
Video resolution: 1920x1080
Buffer dimensions: 1920x1088
Frame rate: 29.970030
Decoded format: Planar 4:2:0 YUV 10-bit LE
Orientation: Top left

Stream 1
Codec: A52 Audio (aka AC3) (a52 )
Language: English
Type: Audio
Channels: Stereo
Sample rate: 48000 Hz
Bits per sample: 32

Stream 2
Codec: Text subtitles with various tags (subt)
Language: English
Description: English CC
Type: Subtitle

EDIT: Nevermind, fixed it. New bug though. The program freaks out how I mentioned above if you're accessing a file off a network share. When I moved the file to the same PC, however, it was able to load up perfectly fine. So it has some issue with network shares (I'm currently using windows' built-in drive sharing if you wanted to replicate it yourself)

Alex Ong · Answer 38 · Mon Nov 28 2022 06:22:23 GMT+0800 (China Standard Time)

Just banding onto everyone else that is thanking you for HEVC support. My previous workflow was converting 12-13 hevc files to h264 using nvenc which took 25~ minutes before i could start any work.

This is excellent!

Vitalii Shvetsov · Answer 39 · Wed Dec 07 2022 09:04:51 GMT+0800 (China Standard Time)

https://chromestatus.com/feature/5186511939567616

Enables support for decoding HEVC video on platforms where hardware (e.g., GPU, media accelerator, etc) for decoding HEVC is available (Android 5.0+, macOS 11+, with supported hardware on Windows 8+ and ChromeOS).

I wonder how it will work on macOS 10.15 for example? Regarding this commit

@mifi

Doug Nelson · Answer 40 · Wed Dec 07 2022 15:43:03 GMT+0800 (China Standard Time)

Issue: can't see HEVC video in the main window -- just see black

Environment: Mac, Mojave (10.14.6), LC 3.48.2. Footage shot on GoPro 11 in HEVC format.

Interestingly, if I choose "Show thumbnails" the thumbnails do render. Audio plays correctly, and I can perform cuts. I just don't see the video in the main window. H.264 video does display.

UPDATE 22.12.10 - Upgrading to Mac OS 11 (Monterey) solved the problem. Thanks nosferatu500.

Vitalii Shvetsov · Answer 41 · Wed Dec 07 2022 16:12:18 GMT+0800 (China Standard Time)

@JerriAtric

HEVC is available (Android 5.0+, macOS 11+, with supported hardware on Windows 8+ and ChromeOS)

Vitalii Shvetsov · Answer 42 · Wed Dec 07 2022 16:15:15 GMT+0800 (China Standard Time)

I think the simplest solution here is to split the array for supported and unsupported systems with respect to HEVC.

Mikael Finstad · Answer 43 · Wed Dec 07 2022 16:26:26 GMT+0800 (China Standard Time)

Oh I assumed it works everwhere.
@JerriAtric do you see any error message in the developer tools?
I wonder how we can detect if hardware hevc support is enabled from inside losslesscut. Maybe we just need to try it and check if the video tag gives an error? And if it gives ane rror then fallback to generate preview

edit: maybe we need to use this api:
https://github.com/StaZhu/enable-chromium-hevc-hardware-decoding#mediacapabilities

Vitalii Shvetsov · Answer 44 · Wed Dec 07 2022 16:38:01 GMT+0800 (China Standard Time)

Source

let obj = document.createElement('video');
console.log(obj.canPlayType('video/mp4;codecs="hev1.1.6.L120.90"'));

David Allen · Answer 45 · Sun Dec 18 2022 15:42:53 GMT+0800 (China Standard Time)

Same issue as JerriAtric above, but on Win11 22621.963. Exact same behavior:

Issue: can't see HEVC video in the main window -- just see black
Interestingly, if I choose "Show thumbnails" the thumbnails do render. Audio plays correctly, and I can perform cuts. I just don't see the video in the main window. H.264 video does display.

Mikael Finstad · Answer 46 · Sun Dec 18 2022 16:24:59 GMT+0800 (China Standard Time)

@dpallen are you running newest llc beta? you can work around this for now by either downgrading, or file -> convert to supported format

David Allen · Answer 47 · Mon Dec 19 2022 02:04:39 GMT+0800 (China Standard Time)

Thanks for the quick reply, but I resolved my issue after some more testing. My footage used 4:4:4 chroma subsampling. Using 4:2:0 HEVC footage works perfectly in llc 3.48.2 beta.

Mikael Finstad · Answer 48 · Fri Dec 23 2022 18:31:46 GMT+0800 (China Standard Time)

So to summarize, in order to detect if the player/device supports a file, (for example HEVC) we need to:

const mediaConfig = {
  type: 'file',
  video: {
    contentType : 'video/mp4;codecs="hev1.1.6.L120.90"',
    width: 1920,
    height: 1080,
    bitrate: 10000, 
    framerate: 30,
  }
}

const { supported } = await navigator.mediaCapabilities.decodingInfo(mediaConfig);
// result.supported indicates whether or not the video with given profile, width, and height can played well on the browser

The problem is to find the contentType parameter. It is described here: https://developer.mozilla.org/en-US/docs/Web/Media/Formats/codecs_parameter but it varies by format/codec. Need to find a library where someone implemented this logic into a library for all formats/codecs. I would think that the output from ffprobe could be transformed into this codec parameter.

Potential implementations for each format/codec:

AV1 spec implementation
MP4

mifi / lossless-cut

Native support for codecs/formats (TS,H265,mpeg2++)

Workarounds

Compile custom electron

Convert to supported format

Improve current proxy/dummy code

Related

Electron (Chromium) might never support h265

HEVC cutting