NDI HX support on Android (Quest 2)
inspired27 opened this issue · comments
Hi,
I have created a 3D URP project in Unity 2021.3.8f1 with a plane as the video screen, I have installed the KlakNDI plugin and setup and streaming from NDI Screen Capture HX on a laptop which I have confirm as working with NDI Monitor. The NDI video feed from the laptop is working in the editor and when playing in the editor in realtime but when I build for Quest 2 the apk loads but the plane is blank with error "NDI - Video decoder not found. Please visit ndi.tv/formats for help and information"
Do you have any idea why this is happening?
My setup details below:
Win10
NVidia RTX 3080
Unity 2021.3.8f1
KlakNDI 2.0.3
XR Interaction Tools 2.1.1
XR Plugin Management 4.2.1
Oculus XR Plugin 3.0.2
OpenXR Plugin 1.4.2
Build settings for Android devices in [Player Settings]-[Other Settings] are as follows.
[Rendering]-[Color Space]: Linear
Auto Graphics API: OFF
Graphics APIs: Vulkan
[Configuration]-[Scripting Backend]: IL2CPP
Target Architectures: ARM64
Static Batching: ON
Graphics Jobs: OFF
Texture compression format: ASTC
How about other senders? Does the receiver on Quest 2 work with simple sender apps like NDI Test Patterns or the KlakNDI sample project?
Hi, I installed NDI for OBS and the video stream worked in Quest 2, however when I moved my head with the Quest 2 on the video tearing was terrible even micro movements caused bad tearing.
I will try the KlakNDI sample project and NDI Test Patterns to see if they are any better.
Any ideas with the tearing?
Unfortunately, I don't have experience on VR devices, so I have no idea.
Hi,
I have tested the NDI video source that I will be using, inside Unity editor it works fine as an Android build the video shows, inside the Quest 2 I get the message "NDI - Video decoder not found. Please visit ndi.tv/formats for help and information" with the NDI logo screen. I have a feeling that this is a NDI | HX source does your plugin need to be updated to include this decoder?
Using NDI Monitor receives the NDI stream okay on my PC also.
The NDI receiver functionality on Android/iOS is minimal (they just added essential parts of it in the recent versions; They didn't support it at all in the previous versions).
Do you know any other app that supports HX on Android?
This kind of information is not documented clearly, so I only can do some guesswork.
Hi,
They have a NDI HX camera app?
The NDI SDK includes the android Libndi.so would replacing the one in your repo with the newest version work?
Sorry I am not a programmer.
They have a NDI HX camera app?
Yes, but it only uses the sender functionality -- What you need is the receiver functionality.
The NDI SDK includes the android Libndi.so would replacing the one in your repo with the newest version work?
I have no idea. It might work.
You have to put the package directory in the Packages directory to modify the contents.
It's easier to test by cloning this repository and replace the file in it if you're not sure how to develop a custom package.
Thanks will try
I'm also running into this same issue. Have created a Quest native app for 360 and 180 video playback, but the bandwidth requirements are a bit insane, so looking to use their HX encode/decode to improve performance. @crazyfrog-github did you make any progress with this? Maybe I can help.
I'm also running into this same issue. Have created a Quest native app for 360 and 180 video playback, but the bandwidth requirements are a bit insane, so looking to use their HX encode/decode to improve performance. @crazyfrog-github did you make any progress with this? Maybe I can help.
Hi, I have an encoder that puts out NDI HX over wifi and I want to view in the Quest 2 headset. Keijiro plugin does not work as a NDI receiver with NDI HX encoded streams, it does work for NDI but not NDI HX.
I am super keen to get NDI HX to work in Quest 2 and would appreciate working together on this, have you had a look at the NDI Advanced Android SDK documentation on the receiver side of things?
There must be a way to resolve the Video decoder not found issue on android Quest 2 and get it to work as it works fine on Windows, NDI monitor and in Unity editor no problem but does not work when build to Quest 2. NDI Tools 5.5 Advanced Android SDK where release recently which I have read does support NDI receiver capability not sure if it is for NDI HX | HX2.
I am getting snippets from around which might help, is it drivers or something that need to be used?
http://www.sienna-tv.com/ndi/knowhowhx2.html
https://forums.newtek.com/threads/ndiv4-hx-vs-hx2-confusion.164595/
Some further info about versions, @keijiro what version of the NDI SDK did you use for the NDI receiver plugin?
General NDI-HX rules for software applications
NDI 3.0 and above is required for HX1 along with the installation of the NDI-HX1 Driver (included with NDI Tools). This HX1 driver is required for higher NDI versions as well.
NDI 4.0 and above is required for HX2. If the product is mentioned as using the 'HX driver', that is HX1.
NDI 5.0 and above is required for HX3. Some HX3 devices can operate using HX2, use this mode for backwards compatibility.
I assume that the NDI 5.0 SDK operates with HX2
I upgraded to NDI 5.0 a year ago.
I upgraded to NDI 5.0 a year ago.
Hmm okay, why does NDI HX2 not work may be NDI 5.5 included support for Android receiver?, did you develop based on the doco attached?
Do you already know that it is not possible to create a NDI HX2 receiver on Android?
Crazyfrog, what version of the sender are you using? I believe this issue started for me after upgrading the tools to 5.5. They seem to have broken some backwards compatibility. I can confirm, Keijiro, that receiving an HX signal using version 2.0.3 on android does not work, however the original high bandwidth signals still decode fine.
…
-- LIGHT SAIL VR MATTHEW CELIA | CREATIVE DIRECTOR o: 213.221.1153 | m: 323.203.7659 lightsailvr.com
Lightsailvr, I checked with the company that sell the wifi encoder sender that I have and they said they used NDI 4.0 SDK on Linux and the encoded output is HX1 and HX2.
Yes the high bandwidth signals do decode fine. I used NDI Tools Screen Capture (not HX the original high bandwidth signal) from a laptop and it decoded and played fine in the Quest 2 so the issue is related to HX 1 and HX 2 decoding.
There are some considerations with Android receivers in the doco might be worth a read.
@keijiro I just tried your NSM NDI Simple Monitor and get the same results using NDI receiver on Samsung Galaxy S21 Android 11 device, original high bandwidth works fine but when I switch to HX sender missing decoder. This eliminates Quest 2 from the equation and is decoding related.
@keijiro could you look into the document and see if there is something you can do to fix, upgrade to 5.5?
Image1: NDI sender original high bandwidth
Image2: NDI sender HX
could you look into the document and see if there is something you can do to fix, upgrade to 5.5?
They rarely update the document. As I noted before, information about mobile support is not clearly documented. That's still one of the biggest headache for me about NDI.
My conclusion at the moment is that NDI on mobile (arm architecture) doesn't support HX.
could you look into the document and see if there is something you can do to fix, upgrade to 5.5?
They rarely update the document. As I noted before, information about mobile support is not clearly documented. That's still one of the biggest headache for me about NDI.
My conclusion at the moment is that NDI on mobile (arm architecture) doesn't support HX.
Thanks, have you read sections about reveiver in the NDI 5.5 Advanced SDK for Android document I attached above it explains about receiver and Android and other places in the doc where it mentions about Andorid considerations etc.
I have reached out to NDI.tv and asked if supported, await response and also sienna TV hopefully get something back.
NDI support have provided an initial response below, is there anything specific that you guys want to find out?
"This error is just indicating you don't have a decoder to decode video, simply just missing the codec. NDI usually relies on the systems codecs to work and these are normally provided in Windows/Mac. Linux will not have any already installed and Android might or might not have them already, but NDI doesn't look or communicate with the native Android codecs. When not working on Windows/Mac, it is up to the user to be able to handle the logic needed to properly decode these themselves. We wouldn't be able to provide any support for this specifically as its more of an Android issue. People have got this to work but it was due to their own knowledge, experience, and logic that made it possible. Sorry we won't be able to help here, it isn't the first time we have been asked about this."
It appears that it can be done but @keijiro do you have any ideas what would be required to make work on Android?
That answer makes much sense. Thanks for the information.
what would be required to make work on Android?
It's doable but requires time and budget. I think it's not feasible for a spare time project like this.
I'm closing this issue now as "by design". Please reopen it for further problems.
I'm trying to incorporate ffmpeg libraries to decode. I think we basically need libavcodec and it's dependencies complied for arm64 android included in the unity plugin as that's what is required to get hx decoding to work on linux. I imagine the principal is the same and in theory not that complicated.
@wsteelenyc My understanding is that NewTek was purged from the ffmpeg community due to the license violation. Is there any other way to utilize it for NDI decoding?
It's not really ffmpeg that we need, just the libavcodec libraries to decode h264 . For whatever reason the system codecs on android don't seem to be compatible, but on linux NDI HX H264 is decoded by libavcodec. I'm thinking the same could work for android since linux and android are so similar. The easiest way to get the shared object files for android would be to build ffmpeg for arm64 android.
Update...
It looks like HX decoding isn't at all supported on the regular NDI 5 SDK. Per the response from support...
"HX decoding is supported on Advanced SDK. The regular SDK are able to support Android applications however, they will not be decoded in HX but in our SHQ codec. The Advanced SDK allows developers to program how the receivers decodes either in H.264, HEVC, or SHQ. The Advanced SDK is fully supported with the libraries and the codec to help developers do so. "
At one point, probably NDI SDK 4.X you could use libavcodec with the regular SDK to decode NDI HX, at least on linux and in theory android too. Presumably that changed when Newtek was bought by VizRT. I applied for the advanced NDI SDK and am waiting to hear back. With the Advanced SDK I imagine I could just use Android's mediacodec api to decode HX. I hope you find this information useful. Thanks!
Hopefully you manage to get the Advanced SDK and try a few things out, happy to colab on this.
Hopefully you manage to get the Advanced SDK and try a few things out, happy to colab on this.
Yes I have access to the latest NDI Advanced SDK. Assuming you are the same guy I messaged on the NDI community. I sent you my contact info. If not, direct message me on here and I'll pass along my email.