Giters

UltraStar-Deluxe / Play

Free and open source singing game with song editor for desktop, mobile, and smart TV

Home Page:https://ultrastar-play.com

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Support more audio and video file formats by integrating LibVLC or ffmpeg

achimmihca opened this issue · comments

commented

Actual behaviour

Unity supports a limited, platform dependent set of audio and video file formats.
This is causing issues with existing UltraStar songs and is a recurring issue for user discussions (#344, #246).

Plus, Unity has a bug when decoding mp3 with variable bitrate (#323).

Expected behaviour

By integrating a third party lib like LibVLC or ffmpeg, it should be possible to support the plethora of file formats that users expect.

ffmpeg

There are C# bindings for ffmpeg:

and some attempts to integrate it in Unity:

However, most of these attempts to integrate ffmpeg with Unity seem like an early proof-of-concept to me. I did not find anything polished for desktop and nothing for Android or iOS.

LibVLC

There are C# bindings for LibVLC:

and an integration for Unity:

The LibVLC wiki names all important platforms for UltraStar Play:

Run on every platform, from desktop (Windows, Linux, Mac) to mobile (Android, iOS) and TVs (AppleTV, Android TV, Tizen) and Consoles (PS4/5, Xbox)

Plus, LibVLC seems to support network file systems, which could solve #347

Network browsing for distant filesystems (SMB, FTP, SFTP, NFS...) and servers (UPnP, DLNA)

In theory, it should be possible to make LibVLCSharp and its Unity integration support the same features.

License considerations

LibVLC and ffmpeg are both licensed under LGPL 2.1 or later.
From my understanding, this is OK for us (at least on desktop platforms) because Unity stores the libs as dynamically linked library (DLL) in the app directory (in Windows they are in ./UltraStar Play_Data/Managed, similar for macOS and Linux, not sure about iOS and Android though).
Thus, users are able to swap the implementation of the bundled libs.

From my understanding, this allows us to distribute the rest of the app (or "combined work" as the license calls it) "under terms of your choice", in our case under MIT license.

Closed source license considerations

For closed source projects, ffmpeg does not provide any other licensing.

Note that FFmpeg is not available under any other licensing terms, especially not proprietary/commercial ones, not even in exchange for payment.

In contrast, LibVLCSharp is also available under paid commercial license by videolabs.

Conclusion

The Unity integration of LibVLC seems more mature and the option for a commercial license offers more flexibility also for closed source projects.
Further, it seems LibVLC has been developed with all relevant platforms in mind and its distant filesystems feature could be handy as well.

Thus, I will try to integrate LibVLC for UltraStar Play.

commented

The free trial version of LibVLC for Unity works like a charm on Windows, Android is also supported.

Question is how to make it run on Linux, macOS, and iOS.

commented

About license considerations

I asked on the Unity forums and VLC for Unity GitLab whether mixing LGPL libs with an app that uses another license is possible in general on Windows / Linux / macOS / iOS / Android.

It seems to be possible in general because by now, all of these platforms support dynamic linking.
Furthermore, there should be plenty of example apps that are not under LGPL themselves but use ffmpeg, libvlc, or Qt.

One needs to verify this and provide instructions how to swap a lib in an app built with Unity on Android / iOS / tvOS. But at least it seems to be possible somehow.

commented

One needs to verify this and provide instructions how to swap a lib in an app built with Unity on Android

  • Get apk file from Android device using adb ( https://stackoverflow.com/questions/4032960/how-do-i-get-an-apk-file-from-an-android-device )
    • Determine the package name of the app: adb shell pm list packages
    • Get the full path name of the APK file for the desired package: adb shell pm path com.example.someapp
    • Using the full path name from Step 2, pull the APK file from the Android device to the development box: adb pull /data/app/com.example.someapp-2.apk path/to/desired/destination
  • Unpack the apk, swap lib, pack the apk using apktool
    • unpack: apktool d test.apk
    • swap lib. The DLLs can be found in the extracted apk in assets\bin\Data\Managed
    • pack: apktool b test
  • Afterwards, resign the apk
    • create keystore: keytool -genkey -v -keystore my.keystore -keyalg RSA -keysize 2048 -validity 10000 -alias app
    • sign the apk that was generated by apktool: apksigner sign --ks-key-alias alias_name --ks my.keystore my-new.apk
  • Install new apk
    • uninstall old apk: adb uninstall com.example.someapp
    • install new apk: adb install my-new.apk

This process allows to swap a dynamically linked lib. Thus, I think using an LGPL lib on Android is not an issue.

During my research I found that Android and its ecosystem are open, transparent, and versatile. I appreciate the platform even more than before.

And on the other side, there is Apple... closed, intransparent, and restricted by arbitrary regulations.

I can't even upload an app to my device. Apple does not allow to install current macOS on my macbook, does not allow to install XCode on the old macOS version, does not allow to build and upload the app to iOS 16 with old XCode.
WTF?! Even if the XCode version can only build for iOS 15, is the iOS 16 device incompatible so badly? Like, every new iOS version requries a rebuild of every app there is?

Probably not, there are projects to leverage macOS restrictions (https://github.com/dortania/OpenCore-Legacy-Patcher), which makes me assume that Apple just wants to sell new stuff by focing existing stuff into obsolescence.
Easiest way to solve this is to buy a more recent device, which - oh surprise - costs at least twice as much.

In conclusion, iOS may take some more time until I get to it.
Sorry for the rant.

EDIT: I was now able to install recent macOS thanks to the OpenCore Legacy Patcher project. Further, I was able to install Linux next to it. It was a quite tricky but now it works like a charm and I can build and test for every platform.

So yeah, of course the imposed restrictions by Apple are just arbitrary...

commented

With respect to iOS, one can

  • Get the installed ipa file from the device using Apple Configurator
  • unzip the ipa file. This yields a folder "Payload" that contains the .app folder

But then you will find that the individual C# DLLs are not present in the .app folder. There is a folder "Data/Managed", which is similar to the apk. But in contrast to the apk, the ipa conterpart does not contain individual libs.

This was expected.
So, I think this is an issue for using an LGPL lib in a closed source iOS app.

However, iOS seems to support some dynamic linking mechanism (dylib files).
Question is whether this can be used with Unity. I don't know but I will not dig deeper.

commented

I wonder whether using a LGPL lib in an MIT licensed project is an issue in the first place.

Actually, I don't think so because the MIT licensed project offers the source code and everything to relink and rebuild the application. I think this satisfies the LGPL.

LGPL 4d0:

  1. Convey the Minimal Corresponding Source under the terms of this License, and the Corresponding Application Code in a form suitable for, and under terms that permit, the user to recombine or relink the Application with a modified version of the Linked Version to produce a modified Combined Work, in the manner specified by section 6 of the GNU GPL for conveying Corresponding Source.

As I understand it

  • we (as MIT licensed project) include the LGPL licensed lib as-is ("under the terms of this License")
  • and we convey "the Corresponding Application Code in a form suitable for, and under terms that permit, the user to recombine or relink the Application with a modified version of the Linked Version to produce a modified Combined Work..."

I just don't understand the part "in the manner specified by section 6 of the GNU GPL for conveying Corresponding Source". Section 6 of GPL is about "Conveying Non-Source Forms". But an MIT licensed project even conveys the "source form".

Only issue I see is everything we include under the terms of Unity. I guess this might be an issue.

Phew! I am not a lawyer and English is not my native language so I am lost ;)
As it is now, I would not include libVLC in an iOS build.

Any further thoughts on the topic?

commented

Thus, I think using an LGPL lib on Android is not an issue

Except it can still conflict with additional terms of app stores and distribution channels.
More specifically, the Google Play terms do not allow modifying any content you've obtained through the store. You can see this clause under Section 4 (Rights and Restrictions).

Thanks for pointing this out on the Unity forums

So, I am back where I started.
LGPL is most likely not an issue on Desktop platforms but it most likely is an issue on mobile platforms.

I am unsure what to do.

Better not include libVLC in the first place?

The supported media file formats are already platform specific.
libVLC could at least unify this for desktop platforms.
Using libVLC would make mp4 available on Linux, such that mp4 would work on all platforms.

commented

Thank you so much for the work to clear up the legal side. My personal opinion / suggestion would be, to use LibVLC only for the desktop-targeting platforms for now. That should already alleviate a majority of difficulty for the average users. Having h264 on mp4 support for all platforms gives us universal support for the most widely used format among the usdx files creating community. This also gives wider formats support for Android targets that are not Google-walled, including various "cheap" TV connectors/boxes.

commented

SOunds awsome tha towrk is being done on this. I was just about to post about it. But im glad to see that support for this is being worked on.

commented

@achimmihca I'd love to see this feature included in Play! Is there any way to help you implementing it?

commented

Is there any way to help you implementing it

What is needed is a good integration of ffmpeg or libVLC in Unity.
This should be done in a dedicated repo, independent of USPlay.

Some thoughts so far

I integrated ffmpeg for Unity in Melody Mania. It nearly works but it is unstable.
I will try libVLC for Unity next, but it is only available for Windows, which is not a solution in the long run.

What I would like to see

commented

@g3n35i5 , see above "What I would like to see"

commented

@g3n35i5 , see above "What I would like to see"

Thanks for your answer! I hope that in the near future I will find time to deal with it.

commented

There are ffmpeg integrations for other game engines such as Godot

Maybe their approach can also be used to integrate ffmpeg in Unity.
Of course, the implementation will differ for Unity.