Native support for codecs/formats (TS,H265,mpeg2++)
mifi opened this issue · comments
Due to losslesscut using chromium's html5 video tag which only supports certain container formats and codecs, we currently need to "convert to supported format" when trying to open exotic formats or codecs. This could be improved with the help of ffmpeg.
Workarounds
Compile custom electron
Convert to supported format
Currently it's possible to cut those files by first using the File
-> Convert to supported format (slow)
function, however this is a re-encoding (although fast) of the whole file, just to be able to view it in the HTML5 video player (note that the original and cut versions will not be re-encoded and are still lossless.)
Improve current proxy/dummy code
Currently a <canvas>
element is used, and ffmpeg will stream low-resolution jpeg images to that canvas.
Drawbacks:
- Slow, laggy, low resolution, low fps
- No audio
This could be improved:
- MediaSource API & https://www.electronjs.org/docs/latest/tutorial/message-ports
- Web Audio API: https://developer.mozilla.org/en-US/docs/Web/API/Web_Audio_API
- https://github.com/strukturag/libde265.js
- https://stackoverflow.com/questions/55376781/seeking-with-ffmpeg-at-the-end-of-an-rtsp-stream
- https://stackoverflow.com/questions/61672575/rtsp-to-webrtc-live-video-streaming
- https://stackoverflow.com/questions/41965793/rtsp-h-264-in-google-chrome-browser
- https://github.com/Streampunk/beamcoder
- https://github.com/GleasonK/WebRTC-YouTubeTogether
- https://phoboslab.org/log/2013/09/html5-live-video-streaming-via-websockets
- https://github.com/phoboslab/jsmpeg
Related
Electron (Chromium) might never support h265
It seems like Chrome and most other browsers are not going to be supporting H265 as it stands now, and since we are using Chromium (Electron), this will be a problem.
It might be possible to enable a flag to enable H265 decoding, but it requires some modding of electron.
- https://stackoverflow.com/questions/36382568/h-265-hevc-web-browser-support
- electron/libchromiumcontent#198
Update: Chromium now supports h265 (hardware decoding only)
heres the solution from the chrnoium developers
https://github.com/henrypp/chromium/blob/master/hevc_support.md
https://github.com/henrypp/chromium/blob/master/hevc_support_new.md
Now that more and more cameras (including DJI Mavic 2) are starting to support H265, it would be nice to have support for this here as well.
Currently it's possible to cut those files by first using theFile
->Convert to friendly format (slow)
function, however this is a re-encoding (although fast) of the whole file, just to be able to view it in the HTML5 video player (note that the original and cut versions will not be re-encoded and are still lossless.)It seems like Chrome and other browsers are not going to be supporting H265 as it stands now, and since we are using Chromium (Electron), this will be a problem.
It might be possible to enable a flag to enable H265 decoding, but it requires some modding of electron.If anyone has some bright ideas, shout out!
heres the solution from the chrnoium developers
https://github.com/henrypp/chromium/blob/master/hevc_support.md
https://github.com/henrypp/chromium/blob/master/hevc_support_new.md
I'm not sure how to do this with electron. (LosslessCut uses Electron which uses Chromium.) Feel free to open an issue at electron to see if this can be integrated. I think it is not a trivial task, so I don't have time to look at this now.
Lately I swap my gear and see, that recording in HEVC format gives me better quality over H264 even with lower bitrate. I would love to switch to HEVC for all my projects although only Lossless Cut stops me from doing that - https://i.imgur.com/avu5AQE.png (after converting to friendly format).
Is there a chance this will work at some point or should I still sit on old format? Thanks.
I'm seeing that too with GoPro 7 HEVC videos and fast convert. The slow one works.
You're right! It works with slow one (That quality xD). Enough for cutting no worries :)
Thanks, that will do for time being.
Great! the reason for the low quality is to make it as fast as remotely possible, and only to be used as a preview.
As LossLess-cut is build in electron do you ever considered to use mpv as player component? I've just found that a project to do that it's already there... https://github.com/Kagami/mpv.js/ with also some samples implementation
Thanks for the tip! I didn't know about mpv.js, but if it's easy to integrate then that's definitely an option.
Kagami/mpv.js#27
In 1.5.0 I have implemented a hack that lets us cut H265 videos, but scrubbing and playing only works at a very low fps and currently without audio.
Is it possible to add support for grabbing a frame at 4k/hevc to jpg even though the preview is not native?
Is it possible to add support for grabbing a frame at 4k/hevc to jpg even though the preview is not native?
I have implemented this now
Nice, thanks! Can't wait to test it in the next release!
Check out newest version now
Thanks, tested 3.17.9. It's working great:)
@mifi Is there any news on supporting H265 natively?
No. I think it needs a lot of work because have to compile a custom electron with a custom chromium that uses a custom ffmpeg. Not something i will prioritize unless this work gets funded somehow, or if we can find the work of someone else who did this already.
This is still a problem in the current build and I was wondering if progress is being made on it as video standards move forward
AFAIK chromium/electron still doesn't support H265 and they refuse to support it due to expensive licensing: https://bugs.chromium.org/p/chromium/issues/detail?id=684382
So we still need to build a custom electron with a custom chromium to get this native support
I came across this guide "it’s not as difficult as it might sound" Electron with custom Chromium build
and then the instructions for HEVC support in chromium
Also noticed this gem from the chromium forums that is currently being implemented Issue 1286132: Implement HEVC for video playback and video conferencing on Windows
Any updates on native HEVC implementation? The solution of making a dummy proxy is pretty brilliant, but it would be nice to see my videos in full res.
@gmcmicken nice finds. I've not taken on the task of building a custom chromium. However I do have an idea with the https://developer.mozilla.org/en-US/docs/Web/API/MediaSource to hopefully allow for more smooth streaming from ffmpeg to the player. It may also allow playing audio. Note that it still needs to encode the video while playing but I think h264 encoding can be done very fast these days. Could also set the duration which would remove the need for the html5ified dummy file.
There seems to be some good news: Sida Zhu has implemented support for hw decoding for HEVC in chromium v104.0.5084.0, disabled by default and enabled with the PlatformHEVCDecoderSupport feature.
(https://github.com/StaZhu/enable-chromium-hevc-hardware-decoding) .
chromium 104 needs electron nighty build 21 (https://github.com/electron/releases) .
Mikael, could you build losslesscut updated to this electron version and with feature enabled? Thanks a lot!
that's fantastic! Then I think once electron 20 becomes stable we can upgrade to that. Note that upgrading losslesscut to newest electron is quite a big job. But I plan on doing it soon, now that electron>v8 slowness on macos has been fixed #714
You are right, also latest electron 20 beta has chromium 104, with the 8-weeks release cycle we should hopefully have a stable 20 soon. I understand that going from electon 8 to 20 could be a big jump, I'll donate for your effort. Thanks.
Hi Mikael,
do you have some progress on this improvement?
Hi. Unfortunately not, as ive been very busy with other projects. But, assuming electron now supports natively, it will be solved once Llc upgrades electron, which is one of the first next big tasks
Hi. Unfortunately not, as ive been very busy with other projects. But, assuming electron now supports natively, it will be solved once Llc upgrades electron, which is one of the first next big tasks
I will definitely buy the app store version to subsidise a couple of cups of coffee for your effort ;)
I'm testing with Electron v21 now and with app.commandLine.appendSwitch('enable-features','PlatformHEVCDecoderSupport')
h265 seems to be working! so expect an alpha release of losslesscut soon
OMG, It works wonderfully! I'm gonna cry now)) Quick, precise, responsive, original quality, no preview conversion... this makes my workflow much comfortable (I'm shooting long scripted 4k videos, and use LosslessCut to markdown them, cut good takes, and it really helps me save much storage space). Thank you, thank you, thank you, @mifi!
Yep. In v3.48.2 HEVC works like a charm!
HEVC cutting
Works like a charm on my Fujifilm X-T4 10 bit HEVC!
Im going to cut and delete more than half of my 14 TB storage now and save a ton from storage space costs now and in the future. Will toss a coin to my @mifi
This is great. Is the app in Microsoft Store going to be updated also? Thanks
thank you so much for working on this app and integrating this feature :) this will improve my housekeeping tremendously
@titobf yes, but i want to iron out bugs first before releasing to stores. already a few bugs coming in.
I'm having an issue. I can't seem to convert h.265 to the player in 3.48.2. It just yells at me that the file is not natively supported, I try to convert, it does, and then it yells again. Here's the media info I pulled from VLC. I could send the file as well if you needed to reproduce the bug on your end.
Stream 0
Codec: MPEG-H Part2/HEVC (H.265) (hevc)
Type: Video
Video resolution: 1920x1080
Buffer dimensions: 1920x1088
Frame rate: 29.970030
Decoded format: Planar 4:2:0 YUV 10-bit LE
Orientation: Top left
Stream 1
Codec: A52 Audio (aka AC3) (a52 )
Language: English
Type: Audio
Channels: Stereo
Sample rate: 48000 Hz
Bits per sample: 32
Stream 2
Codec: Text subtitles with various tags (subt)
Language: English
Description: English CC
Type: Subtitle
EDIT: Nevermind, fixed it. New bug though. The program freaks out how I mentioned above if you're accessing a file off a network share. When I moved the file to the same PC, however, it was able to load up perfectly fine. So it has some issue with network shares (I'm currently using windows' built-in drive sharing if you wanted to replicate it yourself)
Just banding onto everyone else that is thanking you for HEVC support. My previous workflow was converting 12-13 hevc files to h264 using nvenc which took 25~ minutes before i could start any work.
This is excellent!
https://chromestatus.com/feature/5186511939567616
Enables support for decoding HEVC video on platforms where hardware (e.g., GPU, media accelerator, etc) for decoding HEVC is available (Android 5.0+, macOS 11+, with supported hardware on Windows 8+ and ChromeOS).
I wonder how it will work on macOS 10.15 for example? Regarding this commit
Issue: can't see HEVC video in the main window -- just see black
Environment: Mac, Mojave (10.14.6), LC 3.48.2. Footage shot on GoPro 11 in HEVC format.
Interestingly, if I choose "Show thumbnails" the thumbnails do render. Audio plays correctly, and I can perform cuts. I just don't see the video in the main window. H.264 video does display.
UPDATE 22.12.10 - Upgrading to Mac OS 11 (Monterey) solved the problem. Thanks nosferatu500.
HEVC is available (Android 5.0+, macOS 11+, with supported hardware on Windows 8+ and ChromeOS)
I think the simplest solution here is to split the array for supported and unsupported systems with respect to HEVC.
Oh I assumed it works everwhere.
@JerriAtric do you see any error message in the developer tools?
I wonder how we can detect if hardware hevc support is enabled from inside losslesscut. Maybe we just need to try it and check if the video tag gives an error? And if it gives ane rror then fallback to generate preview
edit: maybe we need to use this api:
https://github.com/StaZhu/enable-chromium-hevc-hardware-decoding#mediacapabilities
let obj = document.createElement('video');
console.log(obj.canPlayType('video/mp4;codecs="hev1.1.6.L120.90"'));
Same issue as JerriAtric above, but on Win11 22621.963. Exact same behavior:
Issue: can't see HEVC video in the main window -- just see black
Interestingly, if I choose "Show thumbnails" the thumbnails do render. Audio plays correctly, and I can perform cuts. I just don't see the video in the main window. H.264 video does display.
@dpallen are you running newest llc beta? you can work around this for now by either downgrading, or file -> convert to supported format
Thanks for the quick reply, but I resolved my issue after some more testing. My footage used 4:4:4 chroma subsampling. Using 4:2:0 HEVC footage works perfectly in llc 3.48.2 beta.
So to summarize, in order to detect if the player/device supports a file, (for example HEVC) we need to:
const mediaConfig = {
type: 'file',
video: {
contentType : 'video/mp4;codecs="hev1.1.6.L120.90"',
width: 1920,
height: 1080,
bitrate: 10000,
framerate: 30,
}
}
const { supported } = await navigator.mediaCapabilities.decodingInfo(mediaConfig);
// result.supported indicates whether or not the video with given profile, width, and height can played well on the browser
The problem is to find the contentType
parameter. It is described here: https://developer.mozilla.org/en-US/docs/Web/Media/Formats/codecs_parameter but it varies by format/codec. Need to find a library where someone implemented this logic into a library for all formats/codecs. I would think that the output from ffprobe
could be transformed into this codec parameter.
Potential implementations for each format/codec:
- AV1 spec implementation
- MP4
See also
hate to bear the bad news, but that decoder implementation is incorrect, long story short? the color-space decoding is fucked,
I went into detail here: Alex313031/thorium#70
I probably still recomment uzing StaZhu's or Muril-o patches, as they both seem to do color-space conversion correctly, I'll see about merging StaZhu's patches into Muril-o's as they additionally support more codecs, not just HEVC
Not sure why that's a problem. the hevc decoder is only used for previewing. When exporting, the decoder will not be used
Not sure why that's a problem. the hevc decoder is only used for previewing. When exporting, the decoder will not be used
And when you save a frame as .jpeg?
That’s a point. We could have a setting for whether to use <video>
tag or ffmpeg to capture frame.
I'm not a programmer, so I wouldn't know. As long as the image colors are correct in a export whether it is a .jpeg or a movie file.
I'm not sure how to detect if the player shows black screen. Could someone who sees a black screen with a HEVC file, please try to
- press "play",
- check Developer Tools log and look for the warning line
onVideoError
- copy paste this whole line here?
next version will allow choosing ffmpeg or video-tag for capturing frame, as well as captured image quality
next version will allow disabling the hevc support (for people who have problem with blank screen)
it will also include a basic check for hevc support
In my case, my hardware can not read H265. So if the support option is enabled, I have sound but blank screen. But if I disable the option, I have image but no sound.
I have no problem to read H265 in other players.
I'm having a strange issue with HEVC. My graphics chip support decoding it, and most HEVC videos play fine on LosslessCut, however videos specifically recorded from OBS then converted into HEVC with ffmpeg (either with libx265 or hevc_amf) into an mp4 container will turn into a black screen on LosslessCut. They play fine everywhere else including chrome.
I've attached a problematic mp4 file so you can play around with it:
2023-06-30.18-30-01_hevc.mp4
A workaround for now is converting to HEVC into an mkv container instead then remuxing into an mp4, or if I already have a problematic mp4, remux into mkv then back into mp4.
hmm, this video doesn't play in chrome for me. audio only
@mifi In my chrome version the file of Delphox plays fine. Chrome version: 114.0.5735.199
However, i don't hear any audio, nor in Chrome, nor in VLC, but it has a audio track.
It shows indeed a black screen in LosslessCut v3.54.
When I multiplex the file to MKV with mkvtoolnix, it opens in LosslessCut fine.
It has an empty audio track yea, as I recorded it in OBS with no audio playing (the sample video had no audio track at all). Removing the audio track makes it play on LosslessCut, which is strange, and why I left the audio track on.
Ok I just upgraded my chrome from 111 to 114 and now I can also play the video. So it seems like it was a bug in chrome(chromium). Some time in the future once we upgrade electron to the latest version in LosslessCut, the video will probably just start working in losslesscut too.
coming up in a pr #1844 an improved playback of unsupported files, using the MediaSource API. this allows playback of virtually all formats/codecs with audio as well