no aot build support

Question

no aot build support

dark4eg opened this issue 7 years ago · comments

Artem Espolov commented 7 years ago

See this problem on Windows 10 x64

Peter Monks · Answer 1 · Sat Oct 20 2018 14:13:13 GMT+0800 (China Standard Time)

Would you mind providing a bit more detail on the issue you're experiencing, @dark4eg ?

Timo T · Answer 2 · Sun Dec 02 2018 00:39:00 GMT+0800 (China Standard Time)

@pmonks have you ever run a lein compile with {:aot :all} in the profile? If you do that, the compile never finishes.

The reason for that is that AOT compilation executes all (except -main) top-level forms (as every defn generates a class, and it needs to find those, and defns may be generated at runtime), and thus also evaluates all top-level forms of all dependencies.

I tried to understand how fn-fx does its rendering, but it's entirely mysterious for me. The only thing that I see actually doing anything JavaFX related is this line but I don't understand how it would do anything, as a JFXPanel is just something that embeds in Swing. Anyway, sorry for the tangent.

Whatever fn-fx does to start an Application also starts the JavaFX Platform and its ThreadPool(s). This happens on AOT compilation. But now, for the JVM to exit, Platform.exit() has to be called. The problem is that you normally call Platform.exit() in your -main somewhere.

The result is that AOT compilation starts, the JavaFX Platform gets started by fn-fx, but it's never stopped so the compilation never finishes.

So, the (I think, but as I said, I don't understand fn-fx even after looking at the source, so something else could work, too) only solution is to add an additional method (or do it implicitly with another) to fn-fx, which has to be called for the Application/Platform to start, instead of starting the Application/Platform automatically.

This is honestly how it should be, anyway. Simply :requireing a library should not run threadpools and so on and so forth. I would never have expected that by simply :requireing fn-fx, I have to call Platform.exit().

Is this enough info? :D But yeah, the original issue is severely lacking.

Peter Monks · Answer 3 · Sun Dec 02 2018 09:26:45 GMT+0800 (China Standard Time)

@Azzurite that's an awesome explanation, and sounds like it may also explain issue #54.

Agreed re not implicitly starting the JavaFX machinery upon require - that seems "wrong" to me too. I wonder if @halgari has time to chime in and update us on his thinking there?

In parallel, if you (or anyone else!) wants to play around with making initialisation explicit instead of implicit, that would be fantastic!

Timothy Baldridge · Answer 4 · Sun Dec 02 2018 11:47:20 GMT+0800 (China Standard Time)

I'm pretty sure this is all related to: https://github.com/fn-fx/fn-fx/blob/master/src/fn_fx/render_core.clj#L8

It's been several years since I've tried to debug this, but that line was required to get JavaFX to work at all from a Clojure REPL. This seems to be related to how JavaFX initializes, without that line, JavaFX would complain that some part of the rendering core wasn't properly initialized. The reason for that is because JavaFX apps don't normally use a static void main(...) entry point, but instead subclass the JavaFX application class (whatever the name is), so we have to somehow trigger the internals to initialize from inside a REPL.

Thinking about it now, there's no reason we couldn't put all that code in a init! function somewhere, and it should work if we simply call (import 'javax.swing.JFrame) from inside that function. But strange enough, it's the importing of that class that triggered the creation of the GUI taskbar icon in OSX, so I think this is all related.

Timo T · Answer 5 · Sun Dec 02 2018 20:59:39 GMT+0800 (China Standard Time)

The reason for that is because JavaFX apps don't normally use a static void main(...) entry point, but instead subclass the JavaFX application class

Yes, I know JavaFX in Java very well (from work), you essentially do:

class MyApp extends Application {
  public void start(Stage stage) {
    // do whatever with the stage
  }
  public static void main(String[] args) {
    Application.launch(MyApp.class);
  }
}

I just don't understand at all why fn-fx imports Swings JFrame and does (JFXPanel.), and normally when you create a JFXPanel you have to add it to the children of the JFrame, which I don't see fn-fx doing anywhere! Actually, you also have to create a JFrame somewhere, and not just import it, but I don't see that happening either. That's why I'm confused, fn-fx seems to be creating a JFXPanel and then simply discarding the object, and somehow an actual GUI window comes into existence?!

I would have thought it'd just be possibly to subclass Application with reify (or proxy idk I'm new to Clojure), have a (defn init! [] (Application/launch app-proxy)) and then use the returned stage in the start method (like you do it "normally" in Java) to do all the magic fn-fx does currently.

That's why I also don't feel qualified to change anything.

Peter Monks · Answer 6 · Mon Dec 03 2018 05:00:55 GMT+0800 (China Standard Time)

@Azzurite I know Clojure fairly well, but don't know JavaFX at all, so we could make a good team if you'd be interested in hacking away on a branch together! 😉

I'd be more than happy to give you commit access to this repo if that makes collaboration easier - the master branch is protected, so there's no risk of permanently screwing anything up.

Lars Andersen · Answer 7 · Wed Dec 19 2018 19:17:48 GMT+0800 (China Standard Time)

Thank you for your wonderful explanation of the underlying cause of of this problem @Azzurite.

To anyone else encountering this: I managed to get lein uberjar working by ending core.clj with:

(when *compile-files*
  (Platform/exit))

Timo T · Answer 8 · Thu Dec 20 2018 08:29:58 GMT+0800 (China Standard Time)

Ha! Had I known that. I didn't know *compile-files* existed, so I set an environment variable during compilation, which I checked instead.

Thanks @expez, I'll use this instead :D

Anuj Seth · Answer 9 · Thu Dec 27 2018 18:51:17 GMT+0800 (China Standard Time)

Hi,
I think the way to create an application should be something like this

(ns javafx-simple.core
   (:gen-class
   :extends javafx.application.Application))

(defn -start
  [app stage]
;; the UI stuff goes here
)

(defn -main
  [& args]
  (javafx.application.Application/launch javafx_simple.core
                                         (into-array String args)))

That way the user controls the JavaFx application thread and not the library.
Thanks.

Peter Monks · Answer 10 · Sat Jan 19 2019 03:25:38 GMT+0800 (China Standard Time)

I've taken a first swing at explicit initialisation, in the explicit-init branch. Best place to start is probably the new init namespace. Comments / contributions welcome, as I'm a JavaFX n00b!

Couple of things I've learnt from spending a little time on this:

@Azzurite, @anuj-seth - it turns out that using a JFXPanel is indeed a valid way to initialise JavaFX. It may or may not be the ideal way to do so, of course (I don't know enough yet to have formed an opinion), but just wanted to confirm that this is legal. From what I can tell it's primarily intended to support JavaFX panels within Swing apps.
I don't think this is going to fix issue #54, since many of the functions in fn-fx are generated at runtime, and I would expect to see the documentation for those functions to be included. Compounding this is that it doesn't appear to be possible to even load JavaFX classes in order to introspect them until the platform is initialised (though I may have missed some tricky way to do that - suggestions welcome!).

All of this has me wondering if we might consider switch gears on this, perhaps by:

Making the code generation stage explicit i.e. emit actual source files that then, in a second step, get built / documented, etc.
Revisiting Monocle and trying to get it to work more reliably under TravisCI (which will address issue #54).

Thoughts?

Anuj Seth · Answer 11 · Wed Jan 23 2019 14:24:21 GMT+0800 (China Standard Time)

Hi,
I will give the explicit-init branch a spin and let you know how it goes.

Thanks.

Anuj Seth · Answer 12 · Wed Jan 30 2019 21:15:37 GMT+0800 (China Standard Time)

Hi,
After this change I was able to work on the repl after calling fi/init! but uberjar creation failed with the error below.
I am guessing that JavaFx does not like that we are trying to create an instance of javafx.stage.Screen on a non-JavaFX Application thread.

@pmonks, the explicit code generation step that you suggested may solve this.

Notes: fn-fx-simple.core is a file in my sample project. I am not copying the full stack trace below only the relevant parts

Compiling fn-fx-simple.core
java.lang.ExceptionInInitializerError, compiling:(fn_fx/controls.clj:255:34)
Exception in thread "main" java.lang.ExceptionInInitializerError, compiling:(fn_fx/controls.clj:255:34)
at clojure.lang.Compiler.analyze(Compiler.java:6792)
at clojure.lang.Compiler.analyze(Compiler.java:6729)
.
.
.Caused by: java.lang.IllegalStateException: This operation is permitted on the event thread only; currentThread = main
at com.sun.glass.ui.Application.checkEventThread(Application.java:441)
at com.sun.glass.ui.Screen.setEventHandler(Screen.java:369)
at com.sun.javafx.tk.quantum.QuantumToolkit.setScreenConfigurationListener(QuantumToolkit.java:684)
at javafx.stage.Screen.(Screen.java:74)

Răzvan Rotaru · Answer 13 · Mon Mar 18 2019 03:48:24 GMT+0800 (China Standard Time)

I get the same exception as @anuj-seth. It happens while the Screen class is loaded, not an instance of Screen being created. See Screen.java:74. The class has a static initialisation block which adds a ScreenConfigurationListener which (apparently) should only happen on the event thread. This looks weird to me, the loading of a class should not depend on a thread. Maybe it's a bug, and if not there is probably some protection somewhere for it. I asked a question on the OpenJFX mailing list, maybe we can find some advice there. The relevant stacktrace would be this:

java.lang.IllegalStateException: This operation is permitted on the event thread only; currentThread = main
	at com.sun.glass.ui.Application.checkEventThread(Application.java:441)
	at com.sun.glass.ui.Screen.setEventHandler(Screen.java:369)
	at com.sun.javafx.tk.quantum.QuantumToolkit.setScreenConfigurationListener(QuantumToolkit.java:684)
	at javafx.stage.Screen.<clinit>(Screen.java:74)
	at java.base/java.lang.Class.forName0(Native Method)
	at java.base/java.lang.Class.forName(Class.java:398)
	at clojure.lang.RT.classForName(RT.java:2207)
	at clojure.lang.RT.classForName(RT.java:2216)
	at clojure.lang.Compiler.resolveIn(Compiler.java:7394)
	at clojure.lang.Compiler.resolve(Compiler.java:7357)
	at clojure.lang.Compiler.analyzeSymbol(Compiler.java:7318)
	at clojure.lang.Compiler.analyze(Compiler.java:6768)
	at clojure.lang.Compiler.analyze(Compiler.java:6745)
	at clojure.lang.Compiler$InvokeExpr.parse(Compiler.java:3888)
	at clojure.lang.Compiler.analyzeSeq(Compiler.java:7108)
	at clojure.lang.Compiler.analyze(Compiler.java:6789)
	at clojure.lang.Compiler.analyze(Compiler.java:6745)
	at clojure.lang.Compiler$BodyExpr$Parser.parse(Compiler.java:6120)
	at clojure.lang.Compiler$LetExpr$Parser.parse(Compiler.java:6436)
	at clojure.lang.Compiler.analyzeSeq(Compiler.java:7106)
	at clojure.lang.Compiler.analyze(Compiler.java:6789)
	at clojure.lang.Compiler.analyzeSeq(Compiler.java:7094)
	at clojure.lang.Compiler.analyze(Compiler.java:6789)
	at clojure.lang.Compiler.analyze(Compiler.java:6745)
	at clojure.lang.Compiler$BodyExpr$Parser.parse(Compiler.java:6120)
	at clojure.lang.Compiler$FnMethod.parse(Compiler.java:5467)
	at clojure.lang.Compiler$FnExpr.parse(Compiler.java:4029)
	at clojure.lang.Compiler.analyzeSeq(Compiler.java:7104)
	at clojure.lang.Compiler.analyze(Compiler.java:6789)
	at clojure.lang.Compiler.analyzeSeq(Compiler.java:7094)
	at clojure.lang.Compiler.analyze(Compiler.java:6789)
	at clojure.lang.Compiler.access$300(Compiler.java:38)
	at clojure.lang.Compiler$DefExpr$Parser.parse(Compiler.java:596)
	at clojure.lang.Compiler.analyzeSeq(Compiler.java:7106)
	at clojure.lang.Compiler.analyze(Compiler.java:6789)
	at clojure.lang.Compiler.analyze(Compiler.java:6745)
	at clojure.lang.Compiler.compile1(Compiler.java:7725)
	at clojure.lang.Compiler.compile1(Compiler.java:7720)
	at clojure.lang.Compiler.compile(Compiler.java:7797)
	at clojure.lang.RT.compile(RT.java:415)
	at clojure.lang.RT.load(RT.java:461)
	at clojure.lang.RT.load(RT.java:428)
	at clojure.core$load$fn__6824.invoke(core.clj:6126)
	at clojure.core$load.invokeStatic(core.clj:6125)
	at clojure.core$load.doInvoke(core.clj:6109)
	at clojure.lang.RestFn.invoke(RestFn.java:408)
	at clojure.core$load_one.invokeStatic(core.clj:5908)
	at clojure.core$load_one.invoke(core.clj:5903)
	at clojure.core$load_lib$fn__6765.invoke(core.clj:5948)
	at clojure.core$load_lib.invokeStatic(core.clj:5947)
	at clojure.core$load_lib.doInvoke(core.clj:5928)
	at clojure.lang.RestFn.applyTo(RestFn.java:142)
	at clojure.core$apply.invokeStatic(core.clj:667)
	at clojure.core$load_libs.invokeStatic(core.clj:5985)
	at clojure.core$load_libs.doInvoke(core.clj:5969)
	at clojure.lang.RestFn.applyTo(RestFn.java:137)
	at clojure.core$apply.invokeStatic(core.clj:667)
	at clojure.core$require.invokeStatic(core.clj:6007)
	at clojure.core$require.doInvoke(core.clj:6007)
	at clojure.lang.RestFn.invoke(RestFn.java:703)

Regarding JavaFX Toolkit initalisation I think there are only two possiblities:

create an instance of JFXPanel
extend javafx.application.Application

The better approach would probably be the latter. Unfortunately it works only with gen-class. I tried to make it work with proxy, but Application.launch needs a true class in order to work.

This worked for me (with aot):

(ns aottest.core
  (:gen-class
    :extends javafx.application.Application
    :main false))

(defn -start [app stage]
  ;;stage is the main stage, might be worth keeping somewhere in an atom
  (println "JavaFX Toolkit initialized"))

(defn init! []
  (javafx.application.Application/launch aottest.core (make-array String 0)))

init! should then be simply called on the first rendering.

Răzvan Rotaru · Answer 14 · Sun Mar 24 2019 22:12:23 GMT+0800 (China Standard Time)

I found a better way to initialise the JavaFX Toolkit: apparently we can call Platform.startup directly:
https://openjfx.io/javadoc/11/javafx.graphics/javafx/application/Platform.html#startup(java.lang.Runnable)
This removes the need to use JFXPanel (which is actually part of the Swing interop) and extending Application with gen-class.

There is one additional thing to do: JavaFX by default will terminate its thread when the last Stage is closed. This can be changed with Platform.setImplicitExit. Since we are working with the REPL almost always, I think it's best if the default for fn-fx is false.

Răzvan Rotaru · Answer 15 · Fri Mar 29 2019 19:00:35 GMT+0800 (China Standard Time)

Here's the answer I got from the OpenJFX Community: https://mail.openjdk.java.net/pipermail/openjfx-dev/2019-March/023154.html

This means that fn-fx will be able to support aot only if the clojure compiler can afford to load the classes without initializing them (i.e. use the three argument Class.forName method).

Răzvan Rotaru · Answer 16 · Fri Mar 29 2019 23:20:21 GMT+0800 (China Standard Time)

So, apparently there was some work done in this direction:
https://dev.clojure.org/jira/browse/CLJ-1743
https://dev.clojure.org/jira/browse/CLJ-1743

My conclusion is fn-fx will be able to support aot only when (and if) the clojure compiler will load classes without initialising them.

Timo T · Answer 17 · Sat Mar 30 2019 02:19:35 GMT+0800 (China Standard Time)

At which stage does the Screen class initialization happen? Would a workaround be possible, to initialize it on the JavaFX application thread, before Clojure tries to initialize it on whatever other thread?

Răzvan Rotaru · Answer 18 · Sat Apr 20 2019 22:57:52 GMT+0800 (China Standard Time)

At which stage does the Screen class initialization happen? Would a workaround be possible, to initialize it on the JavaFX application thread, before Clojure tries to initialize it on whatever other thread?

When the class is loaded. The only workaround I see is to shutdown the Java FX application thread when compilation is over. This would require a hook in leiningen, and I don't know if one exists. What needs to happen is a call to Platform.exit when the compilation is over.

marcandrefontaine · Answer 19 · Sun Apr 21 2019 00:27:42 GMT+0800 (China Standard Time)

The following hack exits the JavaFX thread when doing aot. Haven't used it with fn-fx but it might work.
https://gist.github.com/marcandrefontaine/9d9d596e0324a2bf0152a40059f1b923

Răzvan Rotaru · Answer 20 · Mon Apr 22 2019 06:59:03 GMT+0800 (China Standard Time)

I'm sometimes amazed at how easy a solution is, but I fail to see it. I pushed a potential fix to explicit-init. I think it could qualify as a solution, rather than workaround. The thing I'm not sure about is whether *compile-files* is part of the public Clojure Compiler API (and therefore guaranteed to be there in the future) and how exactly it is set. In my tests I stumbled upon a case in which it was false during AOT compilation! Likely because of the asynchronous functions we and JavaFX play around with. Anyway, I think it's good enough for us.

The commit contains two changes:

use of Platform.startup to initialize Java FX instead of JFXPanel
make the Java FX Application Thread a Daemon Thread when *compile-files* is true

Two questions remain:

what value should Platform.isImplicitExit have by default in fn-fx? For REPL development you want false, for a production app true.
when developing with a REPL, in a project with has AOT configured in project.clj, *compile-files* is true when starting the REPL, so the Java FX Application Thread will be a daemon Thread. This is unlikely to be a problem, but what if the Java FX Application Thread would always be a daemon thread in fn-fx (to avoid being dependent on *compile-files*)?

@pmonks I took the liberty to revert your explicit-init solution, I hope you don't mind. Please have a look at the changes around fn-fx.util.reflect-utils/all-javafx-types and fn-fx.util.reflect-utils/enum-classes as I didn't completely understood their purpose (I reverted them as well).

Peter Monks · Answer 21 · Mon Apr 22 2019 07:19:04 GMT+0800 (China Standard Time)

@roti I don't mind at all - in fact I'm stoked that you're working on this!

From memory those two methods were added so that the initialisation code they contain wouldn't run at (require) time (as was previously the case).

Timo T · Answer 22 · Tue Apr 23 2019 22:59:40 GMT+0800 (China Standard Time)

I'm sometimes amazed at how easy a solution is, but I fail to see it

It's not only that you didn't see it code-wise, but that it was mentioned when this issue was first brought up :D #25 (comment)

But yeah, I would definitely consider it a workaround, because it works around the bug of Clojure initializing static classes for compilation.

Răzvan Rotaru · Answer 23 · Sun Apr 28 2019 05:01:10 GMT+0800 (China Standard Time)

It's not only that you didn't see it code-wise, but that it was mentioned when this issue was first brought up :D #25 (comment)

I don't think that's the case, though I would mind if it were. This code

(when *compile-files*
  (Platform/exit))

works only if the compilation is fast enough.
What happens is following:

leiningen calls the clojure compiler (as far as I can tell in a new JVM)
the clojure compiler loads some JavaFX code, which starts the JavaFX Application Thread
the clojure compiler finishes, but the JVM remains open because of the Java FX application thread

The only safe way to call Platform/exit is when you are sure that the clojure compiler is finished. A better solution is #25 (comment) , which adds a delay, but of course that only works if compilation time is less than the delay. I tried to find a leiningen hook to use, but that is not possible because the JavaFX Thread is part of the compilation process from leiningen's point of view. Making the Java FX Application Thread a daemon thread makes sure the JVM process finishes when the compilation finishes, regardless of how long it takes.

But yeah, I would definitely consider it a workaround, because it works around the bug of Clojure initializing static classes for compilation.

I'm not sure that it's a bug in the Clojure compiler. The solution suggested by somebody in the JavaFX community was to load the JavaFX classes without initializing them during compilation (i.e. so that the static initialization blocks are not executed), but Clojure might not be able to that, ever, because of macros.

In my oppinion, it's JavaFX that is doing something which is not ok. Starting a Thread in a static block of a class and also relying on it in other static blocks is something I don't consider to be a good design choice. Event if it works for Java, I would not care about the rest of the JVM world. In fact I think we're lucky that fn-fx loads the Java FX classes in the right order, and I suspect it will break when this will change, even with this hack/workaround.

I think another (better) solution would be possible only with a change in either the Clojure compiler or JavaFX.

Răzvan Rotaru · Answer 24 · Sat May 18 2019 02:01:00 GMT+0800 (China Standard Time)

Pull request has been merged. I'll go ahead and close this issue.