Test proposed shared buffer PR in JavaFX with VLCJ
hohonuuli opened this issue · comments
JavaFX has a new pull request to support native rendering by supporting WritableImages backed by NIO bytebuffers. This would allow JavaFX to avoid copying data from native buffers to a WritableImage, greatly reducing CPU load. The details of the PR are at https://mail.openjdk.java.net/pipermail/openjfx-dev/2019-June/023347.html.
I thought VLCJ would be provide a good test for this PR.
This is interesting, thanks for the tip!
I had a bit of time to make this work:
https://github.com/caprica/vlcj-javafx/tree/pixelbuffer-test
Feedback welcome, especially if anyone runs performance tests or if a JavaFX expert has any more tips for a more optimal solution.
Awesome! I'm not a JavaFX expert. Still, I hope to carve out a moment tonight to take your test for spin. Thanks for putting that together.
I went to run the demo and I'm getting this compile error in JavaFXDirectRenderingTest:
Error:(224, 86) java: cannot find symbol
symbol: method getNativeBuffers()
location: variable videoSurface of type uk.co.caprica.vlcj.javafx.test.JavaFXDirectRenderingTest.JavaFxVideoSurface
The offending line is:
pixelBuffer = new PixelBuffer(bufferWidth, bufferHeight, videoSurface.getNativeBuffers()[0], pixelFormat);There's no method getNativeBuffers() ... any tips?
OK making progress, I went to modify the vlcj source to add a method CallbackVideoSurface.getNativeBuffers(). But when I run mvn install I get an error:
Caused by: java.lang.UnsatisfiedLinkError: Unable to load library 'vlc':
dlopen(libvlc.dylib, 9): image not found
dlopen(libvlc.dylib, 9): image not found
Native library (darwin/libvlc.dylib) not found in resource path
I have VLC installed. What's the incantation to get libvlc on the lib path for vlcj's maven build?
Nevermind ... I didn't pay attention that libvlc was only used for tests. I was able to compile with:
mvn install -Dmaven.javadoc.skip=true -Dmaven.test.skip=true
I ran a quick test with the following:
tl;dr: Best rending I've seen from vlcj on a Mac. 👍
I have some 4k ProRes files I'd like to run tests with, but I'm off on vacation for a week. I'll run those tests when I get back.
Video Source
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'V4066_20170822T135738Z_h264.mp4':
Metadata:
major_brand : isom
minor_version : 512
compatible_brands: isomiso2avc1mp41
creation_time : 2017-08-22T13:57:38.000000Z
encoder : Lavf56.22.100
Duration: 00:15:00.42, start: 0.000000, bitrate: 26185 kb/s
Stream #0:0(und): Video: h264 (Main) (avc1 / 0x31637661), yuv420p, 1920x1080 [SAR 1:1 DAR 16:9], 26175 kb/s, 29.97 fps, 29.97 tbr, 30k tbn, 59.94 tbc (default)
Metadata:
creation_time : 2017-08-22T13:57:38.000000Z
handler_name : VideoHandler
Stream #0:1(und): Audio: aac (LC) (mp4a / 0x6134706D), 48000 Hz, stereo, fltp, 3 kb/s (default)
Metadata:
creation_time : 2017-08-22T13:57:38.000000Z
handler_name : SoundHandler
JDK
openjdk version "12.0.1" 2019-04-16
OpenJDK Runtime Environment AdoptOpenJDK (build 12.0.1+12)
OpenJDK 64-Bit Server VM AdoptOpenJDK (build 12.0.1+12, mixed mode, sharing)
OS
Test profiling
The only change I a had to make to vlcj was:
--- a/src/main/java/uk/co/caprica/vlcj/player/embedded/videosurface/CallbackVideoSurface.java
+++ b/src/main/java/uk/co/caprica/vlcj/player/embedded/videosurface/CallbackVideoSurface.java
@@ -79,26 +85,33 @@ public class CallbackVideoSurface extends VideoSurface {
libvlc_video_set_callbacks(mediaPlayer.mediaPlayerInstance(), lock, unlock, display, null);
}
+ public ByteBuffer[] getNativeBuffers() {
+ return nativeBuffers.buffers();
+ }
For grins, I ran a test with the new Shenandoah GC ... note the memory usage!
I'll make a corresponding branch in vlcj some time later today to include that change I forgot, thanks.
And thanks for the feedback.
You need the 4.2.0-SNAPSHOT that (at time of writing) is the current vlcj master.
Use:
mvn -Dmaven.test.skip=true clean install
You must use that snapshot build rather than a release build (as per the comments above).
So that profile chart posted by @hohonuuli shows 9Gb heap usage which is insane, but interestingly the CPU load seems to about 8% on average which is somewhat remarkable.
9Gb heap usage which is insane
Just to clarify ... that was when I tested with the new, experimental Shenandoah garbage collector. Otherwise memory usage was less than 300MB using the default Java 12 GC settings.
Shenandoah can be explicitly enabled using -XX:+UnlockExperimentalVMOptions -XX:+UseShenandoahGC. I was pretty shocked by the memory usage of it. I'm sure there's a way to tune it down though.
See #886 for the current state.
The vlcj master branch now contains the necessary changes - note that it was done differently (better) than what was described here, the changes should be very easy to accommodate.
Regarding the Shenandoah garbage collector, I saw similar behaviour on Linux, upper heap size looked like 7Gb+ IIRC. I did many forced GC while the video was playing back and it certainly looked like no pauses/stuttering in the video when the GC was running.
It's somewhat hilarious because the demo app runs in about 20Mb of heap ordinarily.
So now I really hope this new PixelBuffer becomes a part of the next JavaFX release.
I would now be very surprised if this JavaFX solution does not out-perform the Swing/Java2D direct-rendering solution.
Link to the pull request for convenient reference: javafxports/openjdk-jfx#472
Reminder to compare performance with ARGB buffer format since it seems it will be supported if this gets included in JFX.
New EA release build:
https://mail.openjdk.java.net/pipermail/openjfx-dev/2019-July/023438.html
It looks like this should make it into JavaFX 13.
Hi
This is a very simple example of a video player which uses the new WritableImage
of JavaFX with support for Buffers. The idea is to let VLC directly render into
this buffer and use the image directly in an ImageView without any explicit rendering
into a canvas or such thing. Only this brings the desired performance boost.
I did this because I think the examples I have seen here are conceptionally wrong.
Please correct me if I should have missed something.
What I have not considered yet is any kind of synchronization.
I think an extension of the PixelBuffer to support some kind of double-buffering
would be the right thing to do but it also runs smoothly without it. I also don't
need any timers in this example.
Looks very interesting, thanks.
I was hoping someone who knew JavaFX would offer something like this.
I'll look it over in detail in due course, but I have an immediate question - with your approach, is it still easy to paint an overlay on top of the video?
@mipastgt Thanks for posting your demo! @caprica I think since he's using a StackPane (line 75) It should be easy to throw another AnchorPane or Canvas to draw overlays on top of the ImageView that's rendering the video. When I get a few minutes, I'll give it a try and see how it performs.
@caprica Yes, you can use the video image inside an ImageView like any other Node. In a previous version of this demo I had a rotating Label on top of the video inside an AnchorPane. No problem. That's actually the whole purpose of this aproach, to integrate an arbitrary image source into the normal scene graph. You could even use the image as a texture inside a triangle mesh in 3D. (But I haven't tried that myself yet.)
Here is the same example with a rotating label on top of the video.
WritableImageVideoDemo2.java.zip
You can also resize the window and even maximize it and the video will be rescaled on the fly.
Thanks a lot for this.
I'd be interested in some performance comparisons. I tried playing a 4k movie trailer with my code on an old MacBook Pro from 2012 and the result was still smooth, the cpu load moderate even with the rotating label on top of it.
I have a 4k sample that had high CPU load on my Linux PC, I will do some comparisons at the weekend.
So this example is really nice. It is a bit simpler than what we already have.
I ran some very brief performance comparisons between this and the version currently in the vlcj-javafx project.
When playing 4K video the CPU usage was broadly comparable. This new version was no better or no worse on average.
I did see mild tearing and stuttering when playing 4K, in both versions as it turns out.
So, in short:
- I noticed no performance difference
- This new example is simpler, and is probably more "correct" in terms of how the native buffer should be used (with inherent rescaling, no specific painting and so on).
I still think there needs to be a proper solution to sync renders with vblank, and maybe ideally there is buffer switching to incorporate too. But I am unclear on all of that with JavaFX.
Performance (I mean relative performance between the current example and this new one) seemed slightly better with HD video rather than 4K, but the difference wasn't much.
I obtained the following results on my old MacBook Pro from 2012 with a FullHD video:
Old solution: 85-95% CPU
New solution: 45-55% CPU
VLC native: 12% CPU (just as a reference)
(100% would mean 1 core fully utilized.)
I did not observe any stuttering for any of the solutions. (Did you use VLC 3.0.8?)
I agree that double-buffering would be nice. How would that be configured in VLC and
how would the API tell me which buffer is currently in use? On the JavaFX side this can
probably only be implemented by providing a special version of PixelBuffer which can only be
done internally of JavaFX due to a lot of private API in there. But I could have a look at it.
I also tried to get the examples running on Windows 10 because I have attached an LG 5k/2k monitor their, but I could not get any of the examples working. I did not get past a black window.
I used an 8-core i7 with a mid-range GPU (at least at the time it was bought, a few years ago), on Linux with native nVidia drivers.
I don't see stuttering or tearing with full HD, only with 4K.
I guess that that the Linux code path is not as well optimised as OSX or Windows as I see double the CPU usage you report using either solution.
I see no significant difference when comparing one solution over the other, and certainly nowhere near a difference of nearly 100% that you are reporting. I don't doubt your figures, by the way. Performance is very sensitive to the execution environment of course.
As to the details on how to make multi-buffer work, I have no idea really.
Regarding your problems in Windows - I seem to remember reading something in the VLC development list that this new LibVLC code path was not yet functional on Windows.
Just some further data: My machine is a
Intel Core i7, 2.6 GHz, 4 cores, 8 GB RAM, NVIDIA GeForce GT 650M 1 GB, 2880 x 1800 Retina
With my new solution I can also play a 4k video without stuttering but have to reduce the window size so that it fits on my smaller screen. The CPU usage goes up to 80-90% then. With the previous solution I cannot play this video at all. I just see an initial frame as a still image and I can hear the sound but that's it.
As for Windows: The native VLC player works without problem.
Linux performance definitely appears to be sub-par. On paper my CPU specs look slightly better, I don't know about the GPU, but even then my CPU usage appears higher. I'm no expert at performance analysis but I expected lower CPU.
Comparing the native VLC player is not a like-for-like comparison, it is using a completely different code path to render video than what we're talking about here.
Well, thanks for your feedback, it was very interesting.
I like your simplified implementation a lot, and it's likely the "right" way to do it. I'm just not seeing big improvements with my own setup which I'm kinda sad about.
@mipastgt Thank you so much for the demo source code. I just have run it with Java 11 using OpenJFX 13-ea+14b and VLCJ 4.2.0 and seems to be running fine on my macbook.
CPU Usage was around 30%, compared to 7% if I play the same video on VLC directly. Is this what to be expected or should CPU usage differ less ?
So my view... this is a good solution, it's simple, it performs well, but it will never match the performance of VLC's optimal native rendering. I don't know how close to par your 30% CPU usage is, but I would be interested to know just how close to native performance could be achieved with this approach.
Honestly though, I have more hope for LibVLC 4.x's upcoming OpenGL rendering callbacks, IMO that should give superior performance. But how to integrate that into a JavaFX app I don't know.
So in short, I didn't really answer your question, sorry, but what is shown here is likely to be the "best" approach for using vlcj with JavaFX, no matter what it's performance characteristics are you probably won't find any other solution that's better.
@wimdeblauwe Yes, I think that is the expected performance. Its exact percentage depends on the general performance of your maching but the relative performance is pretty much what I would expect.
@caprica I agree with you as far as the status quo goes. But if you have the oportunity in the future to render a video directly in OpenGL, then there may be an even better solution. Have a look at this project: https://github.com/eclipse-efx/efxclipse-drift and here https://tomsondev.bestsolution.at/2018/12/04/announcing-efxclipse-driftfx-integrating-native-rendering-pipelines-into-javafx/ I would consider this project being in its early stages. I have seen the demo in Zürich myself and it was very impressive. When I tried the software myself however it was not yet really stable (the Java 11 branch) but that will probably change after some more time. If it works, this will be the ultimate solution with probably the best performance you can get in JavaFX. I will follow this project and see how it evolves.
For the code in #883 (comment) -> It uses a trick to work around the Java module system. For those that want to avoid that, you just need to put the code in a package of your liking, remove the workaround and add a module-info.java file with something like this:
module myapplication {
requires javafx.controls;
requires javafx.fxml;
requires javafx.web;
requires vlcj;
exports com.company.app;
}
Where myapplication is any name you like to call your "module" (which is your application) and com.company.app is the package where you have put the demo code.
To have it running easily, this is a Maven pom.xml that works:
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>mygroup</groupId>
<artifactId>myartifact</artifactId>
<version>1.0-SNAPSHOT</version>
<packaging>jar</packaging>
<name>My App</name>
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<java.version>11</java.version>
<javafx.version>13-ea+14b</javafx.version>
<maven.compiler.source>${java.version}</maven.compiler.source>
<maven.compiler.target>${java.version}</maven.compiler.target>
</properties>
<dependencies>
<dependency>
<groupId>org.openjfx</groupId>
<artifactId>javafx-controls</artifactId>
<version>${javafx.version}</version>
</dependency>
<dependency>
<groupId>org.openjfx</groupId>
<artifactId>javafx-fxml</artifactId>
<version>${javafx.version}</version>
</dependency>
<dependency>
<groupId>org.openjfx</groupId>
<artifactId>javafx-web</artifactId>
<version>${javafx.version}</version>
</dependency>
<dependency>
<groupId>uk.co.caprica</groupId>
<artifactId>vlcj</artifactId>
<version>4.2.0</version>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.8.1</version>
<configuration>
<release>${java.version}</release>
</configuration>
</plugin>
</plugins>
</build>
</project>
Whilst that is very useful, so thanks for posting it, it just shows to me the needless baggage introduced with the Java Module System. I understand the claimed benefits but in reality I don't see the point.
re: Java Module System ... tl;dr yes, it's a huge PITA. But there's no way around it now.
Unfortunately, the javapackager tool provided in JDK 8 has been dropped. It was a handy tool for generating "native" looking java apps. It's in-development replacement, jpackage, uses jlink under-the-hood to build the native apps. I've been porting several JavaFX apps from JDK 8 to 11 and packaging them using jpackage. One lesson learned is that library developers need to be at least module-aware and follow some best practices, especially around reflection, package names, and the naming of resource directories in the jar. Otherwise, at best, it requires developers to do funky contortions to get a library to work or, at worst, the pain of using a library is no longer worth it. I had to rip out a number of 3rd party libraries during the migration from 8 to 11.
@hohonuuli I fully agree with your opinion about the module system but what exactly are your problems? I am building a non-modular 100+ jar-file application with jpackage and do not have any problem with the module system because I strictly avoid it. You can use jpackage and jlink without fully going modular and suffering from all the pain.
@mipastgt Yeah, I just went all in with the modules which probably was a mistake. Some example pain points were:
- Two different modules had resources in
/images... that was a no-no. - Guice ... not worth the struggle of making it work in a modular project
- slf4j ... jlink couldn't decide if the module name was
slf4j.apiororg.slf4j. Took way to long to work through that. Also, logback has a dependency on the servlet api ... the workaround is either to explicitly include it or manually specify what goes in the merged module. - A module had a swing component that uses an image in it's own module. Another module subclassed that component but then couldn't access the image causing instantiation to fail.
- Different modules importing the same classes from different jars (e.g. javax.activation)
- Some 3rd party jars that used JavaFX would include both mac and linux javafx dependencies. These had to be manually excluded or jlink would fail.
- etc.
Yes, that is exactly the kind of problems which I also experienced when I tried to go modular. As I also didn't see any benefit in the module system which could not also be achieved in a less painfull way, I decided to completely avoid it and that worked for me so far and with the GraalVM at the horizon the module system becomes even less important.
Running tests using @mipastgt code. I've packaged that code up at https://github.com/hohonuuli/vlcfx to make it simple for folks to play with. Here's two test cases. Both videos are from one of our autonomous underwater vehicles running video transects at around 700m below the ocean's surface. The vehicle is running relatively fast while looking at very small animals, so any stuttering in video playback is very noticeable. Both videos are stored on the same external FireWire SSD. A typical video frame looks like:
Video 1 (2048x1080, 60fps)
Video looks great. Like really great. The playback is very smooth. On my Mac computer there's no subjective difference between this and playback from QuickTime player.
Codec info:
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from '/Volumes/LBD1/M3/master/i2MAP/2019/07/20190709/i2MAP_20190709T175042Z_701m_F031_9.mov':
Metadata:
major_brand : qt
minor_version : 537199360
compatible_brands: qt
creation_time : 2019-07-09T17:50:41.000000Z
Duration: 00:06:35.56, start: 0.000000, bitrate: 505949 kb/s
Stream #0:0(eng): Audio: pcm_s24le (in24 / 0x34326E69), 48000 Hz, 1 channels, s32 (24 bit), 1152 kb/s (default)
Metadata:
creation_time : 2019-07-09T17:50:41.000000Z
handler_name : AJA Sound Media Handler
Stream #0:1(eng): Audio: pcm_s24le (in24 / 0x34326E69), 48000 Hz, 1 channels, s32 (24 bit), 1152 kb/s (default)
Metadata:
creation_time : 2019-07-09T17:50:41.000000Z
handler_name : AJA Sound Media Handler
Stream #0:2(eng): Video: prores (HQ) (apch / 0x68637061), yuv422p10le(tv, unknown/reserved/reserved, progressive), 2048x1080, 502855 kb/s, SAR 1:1 DAR 256:135, 59.94 fps, 59.94 tbr, 60k tbn, 60k tbc (default)
Metadata:
creation_time : 2019-07-09T17:50:41.000000Z
handler_name : AJA Video Media Handler
encoder : Apple ProRes 422 (HQ)
timecode : 17:50:41:12
Stream #0:3(eng): Data: none (tmcd / 0x64636D74)
Metadata:
rotate : 0
creation_time : 2019-07-09T17:50:41.000000Z
handler_name : AJA Time Code Handler
reel_name : 082
timecode : 17:50:41:12
Profiling info
Video 2 (4096x2160, 30fps)
Noticeable stuttering and dropped frames. Video is certainly usable, but the playback is decidedly not smooth. 4K playback isn't quite there yet.
Codec info:
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from '/Volumes/LBD1/M3/master/i2MAP/2019/05/20190503/i2MAP_20190503T202328Z_600m_F031_13.mov':
Metadata:
major_brand : qt
minor_version : 537199360
compatible_brands: qt
creation_time : 2019-05-03T20:23:28.000000Z
Duration: 00:06:34.43, start: 0.000000, bitrate: 902929 kb/s
Stream #0:0(eng): Audio: pcm_s24le (in24 / 0x34326E69), 48000 Hz, 1 channels, s32 (24 bit), 1152 kb/s (default)
Metadata:
creation_time : 2019-05-03T20:23:28.000000Z
handler_name : AJA Sound Media Handler
Stream #0:1(eng): Audio: pcm_s24le (in24 / 0x34326E69), 48000 Hz, 1 channels, s32 (24 bit), 1152 kb/s (default)
Metadata:
creation_time : 2019-05-03T20:23:28.000000Z
handler_name : AJA Sound Media Handler
Stream #0:2(eng): Video: prores (HQ) (apch / 0x68637061), yuv422p10le(tv, unknown/reserved/reserved, progressive), 4096x2160, 900193 kb/s, SAR 1:1 DAR 256:135, 29.97 fps, 29.97 tbr, 30k tbn, 30k tbc (default)
Metadata:
creation_time : 2019-05-03T20:23:28.000000Z
handler_name : AJA Video Media Handler
encoder : Apple ProRes 422 (HQ)
timecode : 20:23:28:13
Stream #0:3(eng): Data: none (tmcd / 0x64636D74)
Metadata:
rotate : 0
creation_time : 2019-05-03T20:23:28.000000Z
handler_name : AJA Time Code Handler
reel_name : 082
timecode : 20:23:28:13
Profiling info
@hohonuuli Thanks for the tests. Maybe I should have mentioned that I also uploaded my code on GitHub :-) https://github.com/mipastgt/JFXToolsAndDemos
I tried to separate the demo code from the actual functionality to make it a bit more usable.
@hohonuuli You have used the Apple ProRes 422 (HQ) codec for the video. So I would not say that 4k in general is not yet usable but with this codec I am even astonished that it did work at all because this codec is very bandwidth-hungry :-) I doubt that I could even play that with Quicktime on my machine.
@mipastgt Yep, it's a pretty monster codec. We do have reasons for it though. For the type of science we do, our researchers have found that compression artifacts from video codecs can make it hard to do their work. We have a pretty fancy video target, with lots of different lines, grids, patterns, etc, to test for artifacts produced by different cameras and codecs. ProRes HQ was the winner, and we use it as a master codec. We do write mezzanine and proxy versions of each video too though.
We're going to be doing a lot of machine learning work on our video next year. For that, I really need a good toolkit for drawing on top of video. I'm rooting for VLCJ so I don't have to go write something in Swift/AVFoundation or C/C++
Very interesting, thanks.
@hohonuuli by the way, your project sounds amazing!
Do either of you think there's any further scope to optimise this from the vlcj side? I'm not sure what else can be done.
I'm not sure. I've been running tests with @mipastgt sample code against different files and codecs. Right now I'm trying different hevc (h265) encoded 4K-ish files. It's tough finding good 4K samples, I had to work through a bunch to find a sample online that was encoded without errors. I grabbed a 4k (3840x2160, 24fps) test file from https://x265.com/hevc-video-files/, specifically BigBuckBunny. It's only 10 seconds long, so maybe not the best test, but it plays using vlcj without any stuttering, flickering or tearing. It looks really good actually. (I hope it's not obnoxious to keep posting notes about my tests in this issue ...)
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'BigBuckBunny_2000hevc.mp4':
Metadata:
major_brand : iso4
minor_version : 1
compatible_brands: iso4hvc1
creation_time : 2014-08-25T21:13:56.000000Z
Duration: 00:00:10.00, start: 0.083333, bitrate: 2013 kb/s
Stream #0:0(und): Video: hevc (Main) (hvc1 / 0x31637668), yuv420p(tv), 3840x2160, 2010 kb/s, 24 fps, 24 tbr, 24k tbn, 24 tbc (default)
Metadata:
creation_time : 2014-08-25T21:13:56.000000Z
handler_name : hevc:fps=24@GPAC0.5.1-DEV-rev4807
"format": {
"filename": "BigBuckBunny_2000hevc.mp4",
"nb_streams": 1,
"nb_programs": 0,
"format_name": "mov,mp4,m4a,3gp,3g2,mj2",
"format_long_name": "QuickTime / MOV",
"start_time": "0.083333",
"duration": "10.000000",
"size": "2516468",
"bit_rate": "2013174",
"probe_score": 100,
"tags": {
"major_brand": "iso4",
"minor_version": "1",
"compatible_brands": "iso4hvc1",
"creation_time": "2014-08-25T21:13:56.000000Z"
}
}
}
Also ran a test with https://www.base-n.de/webm/outc.webm, both as a download and via it's URL. It's 3840x2160 @ 30fps encoded as VP9. It plays flawlessly. :-). Here's the file details:
Input #0, matroska,webm, from 'BigBuckBunny_VP9.webm':
Metadata:
encoder : Lavf55.19.104
Duration: 00:00:28.91, start: 0.074000, bitrate: 9563 kb/s
Stream #0:0: Video: vp9 (Profile 0), yuv420p(tv), 3840x2160, SAR 1:1 DAR 16:9, 30 fps, 30 tbr, 1k tbn, 1k tbc (default)
"format": {
"filename": "BigBuckBunny_VP9.webm",
"nb_streams": 1,
"nb_programs": 0,
"format_name": "matroska,webm",
"format_long_name": "Matroska / WebM",
"start_time": "0.074000",
"duration": "28.907000",
"size": "34555984",
"bit_rate": "9563353",
"probe_score": 100,
"tags": {
"encoder": "Lavf55.19.104"
}
}
}
And another test with an AV-1 encoded 4K file from https://www.elecard.com/storage/video/Stream3_AV1_4K_13.9mbps.webm. The bitrate on this is a bit higher than the others. the AV1 plays pretty well, but there are some noticeable slowdowns. My suspicion is that this is due to my Macbook pro having hardware decoding built in for hevc and vp9, but not for the av1 codec. The playback is fine in VLC itself though. Notably, neither MPV and QuickTime will play the AV1 file at all. here's the AV1 file details:
Input #0, matroska,webm, from 'Stream3_AV1_4K_13.9mbps.webm':
Metadata:
encoder : libwebm-0.2.1.0
Duration: 00:03:01.52, start: 0.000000, bitrate: 13882 kb/s
Stream #0:0(eng): Video: av1 (Main), yuv420p(tv), 3840x2160, SAR 1:1 DAR 16:9, 25 fps, 25 tbr, 1k tbn, 1k tbc (default)
"format": {
"filename": "Stream3_AV1_4K_13.9mbps.webm",
"nb_streams": 1,
"nb_programs": 0,
"format_name": "matroska,webm",
"format_long_name": "Matroska / WebM",
"start_time": "0.000000",
"duration": "181.520000",
"size": "314996122",
"bit_rate": "13882596",
"probe_score": 100,
"tags": {
"encoder": "libwebm-0.2.1.0"
}
}
}
@hohonuuli It's far from obnoxious, I'm very interested in your results.
@caprica In order to answer your question above: I think there are two things that can still be done. The first thing is a better synchronization between JavaFX and VLC and the second is real OpenGL rendering into a texture without using a memory buffer. The first could be prepared now on the VLCJ side, the other is a long term goal which needs a lot of further investigation.
For 1: VLC already provides two callbacks for synchronization but as far as I have seen these are not available in the VLCJ facade. This could be changed on the VLCJ side. I then might have a look at the synchronization on the JavaFX side. I already had a look at it, but it goes down very deep into the guts of JavaFX so this is not easy. But without the prerequisites on the VLCJ side this would be in vain anyway. These are just my two €Cent. Otherwise I don't see any improvement potential at the moment.
@mipastgt what do you mean by synchronisation? in earlier versions of vlcj i used to provide methods to protect the shared memory buffer while VLC was populating it vs Java rendering it, but honestly I never saw a problem without that buffer synchronisation so I questioned whether it was even necessary or not, that's why it's not in vlcj currently. Is that what you're talking, or something else?
OpenGL rendering into a texture is already possible, at least with a pre-release VLC. I have this working really nicely in an example, but it's not JavaFX obviously - it's an actual Java OpenGL application that happens to use vlcj.
I did look into that nascent OpenGL integration for JavaFX, I think it's mentioned in a comment above somewhere, but it seemed a bit difficult to integrate that with vlcj/LibVLC when I had a quick look. I didn't spend a great amount of time on it admittedly.
The point about LibVLC's OpenGL integration is that basically you just provide a way for VLC to access the OpenGL primitives, and VLC controls all the rendering itself, you don't have to do anything in your Java application. I basically got those new OpenGL callbacks working after about only 10 minutes of work.
- Synchronization: Yes, this is what I meant. I have also not spotted any obvious need for it yet but technically it would be the cleaner solution. But maybe we should be pragmatic and postbone this feature until someone proves that there is a need for it.
- OpenGL: Yes, I have mentioned that above and also provided the link. The author has recently updated the project to be compatible with Java 11+ but when I tried the demo it did not fully work. This may have been my own fault but the author did not provide any build instructions and so it is a bit difficult to use and judge it. I'll probably see the author in December. Maybe I can convice him to provide some more info.
As I said already, I otherwise don't see any real potential for further improvements.
I'm not sure if this is helpful, but MPV has an example where it looks like MPV is drawing to some native surface and that surface is just passed to the Stage in JavaFX (via JNA) where it's rendered. This is less idea than using a WritableImage though as you're bypassing JavaFX's scene graph ... I don't think you can draw overtop the video. https://github.com/mpv-player/mpv-examples/tree/master/libmpv/java
This is just another data-point as I've been doing some more casual testing on Linux.
Let me state first that I am not criticising this JavaFX solution, I think it's great and the best we have so far.
But I have an OpenGL implementation that is using around 1/4 the CPU that the JavaFX version uses - it's something like 20-30% CPU vs 110-140%.
Having said that, the performance of the JavaFX version is still very acceptable on my hardware.
And the JavaFX version does seem to perform better on OSX than Linux, which I'm a bit disappointed about.
Don't get me wrong. When I say that "I don't see any real potential for further improvements", then I am only talking about a solution which is fully integrated into the JavaFX framework. If you can also accept other solutions with heavy weight components and direkt rendering, then of course a better performance can be achieved. I have no problem with that. It is just a matter of priorities and your requirements.
+1 for integrating fully with JavaFX. It is really powerful to be able to integrate a video node in the scene graph.
The other thing I would say is that this current solution for JavaFX is extremely simple to understand and use, so those are big plus points even if the performance is not top-tier - and whilst it may not be top-tier the performance is still very good.
I fear it would be much more difficult trying to get OpenGL and JavaFX working together, unless it's completely transparent which I doubt is possible. That's just my suspicion.
I think this issue has fulfilled its purpose so I will close it now.