PACKR Issue building on Catalina
WalkerWhite opened this issue · comments
PROBLEM: Modern PACKR releases (4.0, 3.0)+ do not create functional (unsigned) LibGDX apps for Catalina
ENVIRONMENT: We can reproduce this on builds from multiple Macs (Catalina and Big Sur) as well as from Windows 10
WORKAROUND: We have found an older (pre-2020) version of PACKR that works correctly
Summary
Unsigned LibGDX applications built by the current version of PACKR on Catalina do not run. They quit immediately with the crash error
# Problematic frame:
# C [AppKit+0x3e5abc] -[NSOpenGLContext setView:]+0xe5
This is a known issue with Catalina where the OpenGL context is accessed outside the main frame, and introduced by how Catalina manages security. This is a problem introduced by PACKR, since the original JAR files run normally.
We did not have this problem last year (I use PACKR for a course). Indeed when we revert to our older version of PACKR, it successfully creates applications for Catalina. I have no idea which version of PACKR that is, but when I expand the JAR file I see that it was built on March 19, 2019. So I suspect it may be 1.2.
From research on the Internet (and the build date of the older version of PACKR), I think the problem is with the Mac SDK used to build the executables used by PACKR. I refer you to this thread for a similar issue with JOGAMP:
https://jogamp.org/bugzilla//show_bug.cgi?id=1398
In that case error was introduced by using the default Catalina SDK 'macosx10.15'. Use of an older SDK, such as 'macosx10.11' (even on Catalina or Big Sur Macs) caused the error to disappear.
We have not verified whether or not this problem goes away when you sign the app with a MacOS Developer Certificate. But ideally this should not be required for informal LibGDX distribution.
Hey Walker,
Sorry your having trouble with the latest packr. The issue JOGAMP issue you link is a known macOS idiosyncrasy. Have you tried specifying -XstartOnFirstThread as a VM argument? https://github.com/libgdx/packr#limitations
Unfortunately that replaces the problem with a different one. The application bounces in the Dock forever and does not launch. This is true both when I launch it as a .app bundle and when I run the contents executable from the command line. It appears to be hung.
I have seen problems in previous years that require the -XstartOnFirstThread solution before, but those problems are typically related to JAR issues. Once again, the JAR is fine and the 2019 PACKR creates a perfectly fine executable. It is only the newer PACKRs that cause a problem.
Some of our players are having the same issue with Mac OS X 10.15.7. Stack trace: https://paste.ofcode.org/KqEnW7pBJQkizAxBsLtQFi
The old packr workaround cannot be used if the game requires a later version of Java like 11. Old packr does not support Java 11.
jpackage is an alternative but it requires a Mac to build a package for Mac.
@WalkerWhite Are you using LWJGL2 or LWJGL3? I'm having the same issue with LWJGL2 but the user cannot run it with java -jar either in his case.
EDIT: After more testing, the issue happens with either LWJGL 2 or 3 and either with or without packr for our player.
EDIT2: It seems making the game compatible with Java 8 makes it working again (with java -jar) but not with packr. I imagine your java source compatibility was set to 1.8 or earlier as well.
EDIT3: After more remote debugging by sending builds to the player that had this issue, I found the only safe configuration is
LWJGL 2 + Java 8 + Old packr
Anything else basically just breaks. I could not get any LWJGL 3 builds to run on his machine, with or without XstartOnFirstThread and with or without packr. If the Java code source isn't compatible with Java 8 it refuses to run (even without packr, just with a jar). Then finally, if the game is LWJGL 2 + Java 8 compatible (Just like WhiteWalker had quite probably) then only Old packr works to bundle the JRE with it.
Does this only happen when you mix AWT/Swing and LWJGL?
@WalkerWhite Are you using LWJGL2 or LWJGL3? I'm having the same issue with LWJGL2 but the user cannot run it with
java -jareither in his case.
This was in the February release of LibGDX which appears to now be built on top of LWJGL3 (they were on a LWJGL2 for a long time until the new management this past year). We were able to get old packr to work with Java 8-15 (the students were spread all over the place).
EDIT: By old PACKR, I mean a snapshot that we had archived back in May of 2020. The old PACKRs we can find online seem to be worse with the problems you describe.
Can either of you make a as small as possible example that reproduces this issue?
Either fork and edit https://github.com/libgdx/packr/blob/master/PackrAllTestApp/src/main/java/com/badlogicgames/packrtestapp/PackrAllTestApplication.java or copy that example project.
I run packr, jpackage with LWJGL, and Swing with no issues on macOS and the current integration test (mentioned above) runs with no issues.
@WalkerWhite I see thanks for the info. I got it working for my player using the 1.2 release of this repo, good to know we can go all the way up to May 2020.
@karlsabo What is your macOS version?
It's difficult for me to run tests as I'm indirectly asking a player to run things. I cannot run a unit test on his computer but I could try to produce a minimal executable. Given the the amount of issues he has (including jagrosh/DiscordIPC#19) the possibility of it being an unsual problem with the kernel is quite high.
@WinterAlexander macOS 11.3.1 is what I have to test on. I'm trying to create a minimal libGDX project.
@WalkerWhite Are you using LWJGL2 or LWJGL3? I'm having the same issue with LWJGL2 but the user cannot run it with
java -jareither in his case.This was in the February release of LibGDX which appears to now be built on top of LWJGL3 (they were on a LWJGL2 for a long time until the new management this past year).
As part of said "new management", I can safely say this is incorrect. libGDX can use either LWJGL2 or LWJGL3, and the official gdx-setup defaults to LWJGL2 (it actually only offers LWJGL2). I make the third-party setup tool gdx-liftoff, which defaults to LWJGL3 and can generate projects that use either or both (as separate modules). libGDX isn't built on LWJGL2 or LWJGL3; it has both as backends. This is why we need to know, are you using LWJGL2 or LWJGL3? You can tell by looking at your DesktopLauncher class, if you have one; if it contains import com.badlogic.gdx.backends.lwjgl.LwjglApplication;, or another import from the com.badlogic.gdx.backends.lwjgl package, then you're using LWJGL2. If you only have an Lwjgl3Launcher class, you're using LWJGL3 of course, but this is also true if DesktopLauncher was modified to use LWJGL3. In that last case, it would contain import com.badlogic.gdx.backends.lwjgl3.Lwjgl3Application; or another import from the com.badlogic.gdx.backends.lwjgl3 package.
I created as simple of a libGDX + LWJGL3 example as I could and it ran on macOS 11.3.1.
I can't track down or think of any reason you all are running into this issue.
After loads of Googling and testing, I have come to the conclusion that you must be initializing AWT/Swing before GLFW.
I have updated the test with the breakage, if you try and do anything (maybe even importing) with AWT/Swing before GLFW.glfwInit() has been called, it will hang.
I have used AWT/Swing w/ libGDX just fine but this thread tipped me off: https://hub.jmonkeyengine.org/t/macos-nswindow-drag-regions-should-only-be-invalidated-on-the-main-thread/43523. I figured AWT initialization must be coming into play before GLFW does it's thing.
OpenGL calls must be made from the 'main' thread, e.g. on macOS use -XstartOnFirstThread as a VM arg and do not do anything AWT/Swing related until glfwInit and the window is created.
@karlsabo This may explain what happens with LWJGL 3 (sending a build to my tester free from AWT imports to confirm) but this doesn't explain the original issue with LWJGL 2.
WalkerWhite's stack trace demonstrate he is also using LWJGL 2. (There are no failure on [AppKit+0x3e5abc] -[NSOpenGLContext setView:] with LWJGL 3 as the stack trace is different). We still have no explanation as to why LWGL 2 builds only work with old packr and not a newer version of packr. (Or why it only works with java source compatibility set to 8 and nothing newer).
EDIT: Not having any luck with LWJGL3 Might be totally my fault and still have some hidden unknown awt imports in my codebase but I can't find any. It just seems safer to me to stick to LWJGL2
@WinterAlexander I created a test branch for libGDX w/ LWJGL2 as the backend, I had no issues on macOS 11.3.1.
When using LWJGL2 don't use -XstartOnFirstThread, libGDX creates a new thread to run the game loop on.
I had no issues mixing AWT and libGDX (LWJGL2).
When using LWJGL2 don't use
-XstartOnFirstThread, libGDX creates a new thread to run the game loop on.
Even if I am running LibGDX on LWJGL2, this advice is wrong with the latest LibGDX builds. Choosing this option causes the program to hang when run in Java (let alone PACKR). This used to be a work-around for LibGDX, but with latest releases makes the program unusable.
As an aside, the lwgl3 backend for LibGDX is no longer in the docs. That is why I was confused about the state of LWJGL. The JavaDocs only include com.badlogic.gdx.backends.lwjgl in the latest releases.
EDIT: Can confirm. Adding this line
import com.badlogic.gdx.backends.lwjgl3.Lwjgl3Application;
Causes a compilation error in LibGDX.
error: package com.badlogic.gdx.backends.lwjgl3 does not exist
Are you sure you are talking about the latest LibGDX build.
@WalkerWhite If you want to use LWJGL3 you need to switch your gradle dependencies https://gist.github.com/crykn/eb37cb4f7a03d006b3a0ecad27292a2d
@karlsabo I have no doubt it works great on your mac. It seems to be a Catalina issue mostly. Or perhaps it's a very specific problem with the setup of my tester that only affects 1% of Catalina users. The main problem is we have absolutely no idea and thus no idea what % of mac users are able to run it. If we at least a dev who runs Catalina who can run the unit tests it would help more.
At the moment I don't have much to contribute to this issue but speculating. If I ever get my hand on a Catalina (hackintosh or real) I'll make sure to test things.
@WalkerWhite as you know, libGDX is maintained by volunteers and the documentation would definitely benefit from someone with your expertise. You have a unique insight into all the getting started challenges that your students face every semester.
@WinterAlexander true, it could be a Catalina issue. I don't have the resources to investigate.
@kalsabo There is no need for a special project to test this issue. I can reproduce all of this with the default (my-gdx-game) project generated by the setup app. I am worried that I am coming off more combative in this thread than I mean to be, so I have just finished several experiments to confirm and isolate the problem I am experiencing (results below with files to download).
I think part of the problem is that we are on different OSs. My development machine is running Catalina 10.15.7, and Catalina has introduced a lot of unpredictable behavior. However, it is possible for me to identify what is an issue with LibGDX and what is an issue with Catalina. On my hard drive I have 12+ LibGDX projects a year, from every year since 2013. Each year's worth of projects uses the LibGDX snapshot from January of that year (except 2021, which uses the February snapshot).
That is why I am confident about -XstartOnFirstThread (confirmed by experiments below). That was required for projects in 2019 (and I can still verify that), but for reasons I do not fully understand was not required in 2020 and 2021 (e.g. post-Catalina). I assumed at the time it was due to a change in LWJGL. I have a side project that modified the OpenAL backend and I noticed some major LWJGL api changes at the time. But apparently you are correct that this is still LWJGL2, so I was wrong.
But the fact remains that not only is -XstartOnFirstThread not required for LWJGL2, it will actively cripple the application. As you are likely aware, Catalina has made it a hard requirement that all OpenGL calls must be made on the main thread (because of how OpenGL is implemented on top of Metal). Applications the violate this will either crash or hang, depending on error handling. This is required for all applications, including Java applications. So it is simply not possible that LWJGL2 runs its OpenGL calls off the main thread. The default LibGDX setup would not run on Catalina if that were true.
I have no knowledge of the architecture of LWJGL2, but one thing that would explain what I am seeing is if LWJGL2 forked its rendering pipeline off in a separate thread, away from the primary LWJGL thread. In that case, putting LWJGL on the main thread would kick the rendering pipeline off the main thread, and cause the behavior I am seeing. But again, this is all speculation.
Okay, on to the experiments.
Experimental Setup
All experiments are on the default my-gdx-game app, with the classic BadLogic icon in the corner. I have removed all sub-builds other than Desktop. I have also removed Box2d, to strip this down as minimal as possible. I am using the default Java (11.0.5) that now comes with Catalina to build the application. My computer is running macOS 10.15.7.
I have created two different versions of the project. One uses LWJGL2 and the other uses LWJGL3. They differ only in the DesktopLauncher, per the standard LWJGL3 instructions.
In the experiment I compare PACKR 4.0 vs PACKR 1.2 (May 2020). For PACKR 4.0 I am using OpenJDK15U-jre_x64_mac_hotspot_15.0.2_7.tar.gz. I was unable to get OpenJDK 11 JRE to work at all in the experiments -- all tests crashed. But OpenJDK 15 appears to work fine on my copy of Catalina.
Unfortunately, I had to use a different JDK for PACKR 1.2. That version of PACKR apparently wants a classic folder structure with a folder called jre in the Contents folder; current OpenJDK builds do not provide this folder structure. So I am having to use a copy of jdk-9.0.4.mac.zip that we have used successfully in the past. This is an official Sun/Oracle JRE downloaded from the days before the restricted licensing (but the license restrictions now apply to it, so I would like to no longer use it).
LWJGL 2
I have created all the following files, which are free to download and test.
- demo-lwjgl2.jar
- demo-lwjgl2-packr1.2-plain.app
- demo-lwjgl2-packr1.2-thread.app
- demo-lwjgl2-packr4.0-plain.app
- demo-lwjgl2-packr4.0-thread.app
demo-lwjgl2.jar is the base jar. It runs normally with a double click launch and from the command line. Adding the option -XstartOnFirstThread causes the application to hang, and you must force quit it.
demo-lwjgl2-packr1.2-plain.app is created with PACKR 1.2 and does not include -XstartOnMainThread. This app runs fine with no problems.
demo-lwjgl2-packr1.2-thread.app is created with PACKR 1.2 and does include -XstartOnMainThread. This app hangs on launch, bouncing forever. You must force-quit the app.
demo-lwjgl2-packr4.0-plain.app is created with PACKR 4.0 and does not include -XstartOnMainThread. Launching this app seg faults with the Report to Apple error window.
demo-lwjgl2-packr4.0-thread.app is created with PACKR 4.0 and does include -XstartOnMainThread. This app hangs on launch, bouncing forever. You must force-quit the app.
LWJGL 3
I have created all the following files, which are free to download and test.
- demo-lwjgl3.jar
- demo-lwjgl3-packr1.2-plain.app
- demo-lwjgl3-packr1.2-thread.app
- demo-lwjgl3-packr4.0-plain.app
- demo-lwjgl3-packr4.0-thread.app
demo-lwjgl3.jar is the base jar. Double clicking it, or it running normally from the command line, causes a segmentation fault with the Report to Apple error window. Adding the option -XstartOnFirstThread causes the application to run normally (shame this cannot be added to the manifest).
demo-lwjgl3-packr1.2-plain.app is created with PACKR 1.2 and does not include -XstartOnMainThread. Launching this app seg faults with the Report to Apple error window.
demo-lwjgl3-packr1.2-thread is created with PACKR 1.2 and does include -XstartOnMainThread. This app runs fine with no problems.
demo-lwjgl3-packr4.0-plain.app is created with PACKR 4.0 and does not include -XstartOnMainThread. Launching this app seg faults with the Report to Apple error window.
demo-lwjgl3-packr4.0-thread.app is created with PACKR 4.0 and does include -XstartOnMainThread. This app runs fine with no problems.
Conclusion
On Catalina, the only way to get LWJGL2 applications to work with PACKR is to use PACKR 1.2 without -XstartOnMainThread, and to make sure that you are using a JRE with the classic folder structure. No 4.0.0 builds work at all.
On the other hand, LWJGL3 works fine with both PACKR 1.2 and 4.0. PACKR 4.0 has the advantage that it supports a modern version of OpenJDK.
The take-away that I am getting from this experiment is that LWJGL3 should be required for all Mac builds going forward. Therefore, I am making two humble requests of the build team.
- Add the lwjgl3 backend API to the API documentation, so that it is official.
- Add an option to the setup app to enable LWJGL3.
I understand not wanting to make LWJGL3 the default. But it would be nice to have an option to configure the files for you.
As for the offer for helping on LibGDX... I do have A LOT of custom classes I have made over the years. But my primary focus is a C++ engine that I am maintaining on top of SDL. Most of my LibGDX features are things that migrate down from that engine.
Huh, the javadocs are definitely not up-to-date, and not just regarding LWJGL3. In https://libgdx.badlogicgames.com/ci/nightlies/docs/api/com/badlogic/gdx/math/MathUtils.html#sinDeg-float- , there's the old incorrect docstring, but in version 1.10.0, the docs correctly say that it should use degrees between -360 and 360, not radians. The docs are built only intermittently, and I think are hosted somewhere that might not be easily accessible... The LWJGL3 backend does need to be there, and probably other parts can be updated at the same time.
I definitely agree that LWJGL3 should be the default, but there are currently some blocking issues. AWT (the plain-Java 2D API) and LWJGL2 can be used together without issue, and they are in the gdx-tools extension. But, AWT does not currently work very well with LWJGL3, and the tools would become less usable (if usable at all) if they suddenly were forcibly moved to LWJGL3. In gdx-liftoff, I've defaulted to LWJGL3 as the main desktop backend since the start. I also allow LWJGL2 apps to be made for compatibility purposes, or even apps with both backends (such as one just to run gdx-tools). Few of the core developers seem to actually use gdx-setup (though I use gdx-liftoff heavily), so it's kind of stagnated over the years. While I could add gdx-liftoff to the libgdx organization, I don't want the fate of gdx-setup to befall gdx-liftoff, and I have no guarantees that outcome wouldn't happen. Right now, gdx-liftoff has updated within 24 hours of each of the last several libGDX releases, and updated with versioned incremental releases multiple times between libGDX releases. I think gdx-setup wasn't rebuilt at all since 1.10.0 came out, despite being a nightly build, though there may have been a change to that recently.
I'm going to link this conversation to the currently-ongoing one in the main libgdx repo about updating to LWJGL3, since the issues with LWJGL2 on Mac seem to be worsening of late, and hopefully that can accelerate a change.
I would appreciate if either gdx-setup be fixed or gdx-liftoff be the official one available on the web page. I use LibGDX in the introductory course taught in the Spring (it helps to have a course in a full-featured Java engine before I drop them in C++), and anything that we can do to reduce the overhead of project start-up would be appreciated.
Yes it wasn't being built, but it is now. This is due to incomplete migration from build systems. Docs have also been overlooked, a lower priority since most people would be seeing this in the IDE with attached source jars, but we will have to fix that also. Jenkins is going away, so all of those old paths are not being updated.
As for Lwjgl3, im all for it. Use it in production currently, the only concerns are as @tommyettinger says, the guys that want to be using swing stuff. but afaik that is not our priority, and we can move on.
@WalkerWhite Thank you very much for this in-depth analysis. I can confirm the same exact results coming out of my playtester with Catalina.
Hey, I haven't had time to really look into this, here are some quick notes:
Big Sur results:
demo-lwjgl2-packr4.0-plain.app is created with PACKR 4.0 and does not include -XstartOnMainThread. Launching this app seg faults with the Report to Apple error window.
Works on macOS Big Sur 11.3.1 (20E241) which contradicts what happens on Catalina. Did you confirm that it's not a difference in the JVM? I noticed the packr12 test uses Java 9 vs 15 for packr4. It would be really disappointing that compiling on Big Sur would break backwards compatibility. It would also be a bit shocking, because packr is pretty dead simple, it loads libjli.dylib, finds a couple of function pointers and calls them.
demo-lwjgl3-packr4.0-thread.app is created with PACKR 4.0 and does include -XstartOnMainThread. This app runs fine with no problems.
Works on macOS Big Sur 11.3.1
I have a game which uses LWJGL2 but not libGDX. I'm curious about the way libGDX launches the display creation in a new thread:
https://github.com/libgdx/libgdx/blob/3bd95db22ca84cab995082488a88b4c597003a04/backends/gdx-backend-lwjgl/src/com/badlogic/gdx/backends/lwjgl/LwjglApplication.java#L122
My game doesn't do that and I have users on Catalina, but I'm also using jpackage instead of packr.
If anyone has the time or curiosity it would be interesting to see if LWJGL2 can work without creating a background thread to init the display, I also do not use -XstartOnMainThread for my LWJGL2 game.
It's probably not worth the time to research LWJGL2 solutions, that code was last updated in 2015, well before macOS deprecated OpenGL in 2018. LWJGL3 uses GLFW to create windows and that project is actively maintained.
@karlsabo Now that IS interesting. I am even more confused now.
I quickly confirmed that demo-lwjgl2-packr4.0-plain.app is a PACKR issue. If I run it with the Java 15 JRE directly, then it runs fine. It is only when I combine them with PACKR that it fails (with the error message that started this thread). I was planning on upgrading to Big Sur before semester start, so I will try them again then.
I also used packr 4.0 on the Java 9 JRE (the one that I needed to use for 1.2). That has the same results. It crashes with the initial error report.
I will add that I would not be surprised if the reason why it works on Big Sur is if Apple relaxed the "on main thread" requirement in that release, but then refused to change it in Catalina. Since deprecating OpenGL, Apple has several times introduced graphical issues which they only fix on the next OS release.
For example, if you try to Python > 3.6.5 on Mojave then Tkinter will reboot your machine when you ask for a window. Apple fixed this in Catalina, but never fixed it in Mojave. So I have tell students on Mojave to use an older Python.
@WalkerWhite ok thanks for trying it out. I'll see if I can get an older SDK installed and build packr with it as you mentioned in your first post. I won't make any promises because my past experience with Apple is they make it excessively difficult to go backwards.
@karlsabo Follow-up
I just finished my upgrade to Big Sur. And I can confirm that demo-lwjgl2-packr4.0-plain.app does indeed work now. So apparently this is a Catalina only issue.
Thanks for confirming. I haven't had luck getting the command line tools to build with a specific SDK and I didn't want to try and rollback to an older macOS.
While packr 1.2 works well on 10.15 and 11, I recommend using 2.3.0 instead because of issue #10 which was happening on 10.14.1
I'm not aware of whether it works on other versions (10.9 to 10.13) but at least I have found this version to be the most compatible at this moment.
EDIT: We confirmed it to work on 10.10.1 !
I'm having a related problem when launching a newly created LibGDX app (using gdx-setup-tool) on a Catalina setup. I'm not packing it to release (so I guess Im not using PACKR?), just launching the app in AndroidStudio is causing the crash with the same error as on this thread. I know this issue is for PACKR and not overall LibGDX, but any pointers or links on how to fix or circumvent this problem? Thanks
@rotter This is likely a side effect of the fact that Catalina requires that all OpenGL apps run OpenGL on the main thread. Any form of redirection of the render thread will cause this error. As we have confirmed in this thread, this requirement is relaxed in Big Sur (hence the error went away for one of my builds).
Thanks for the quick reply @WalkerWhite ! My main machine is Big Sur and indeed I dont have this problem, but this other machine is an old Mac which cant be updated past Catalina, so I was looking for any possible workaround/hack. For example what does PACKR 1.2 do differently that allows the app to run on Catalina? I can try to do that same thing manually.
@rotter The application wrapper launches the LibGDX with OpenGL thread on the main thread. I cannot say more than that. This is different from the main application thread. That is why specifying XStartOnFirstThread causes a problem.