PaperMC / paperweight

While trying to find out why hot reloading single classes takes "ages", I found out, that when Gradle is no longer forced to recompile everything (through the log4j2 annotation processor) and instead uses incremental compilation, it fails with lots of compiler errors: https://paste.gg/p/anonymous/6b0dcd7a59b04cf5a5eb3ad653530714

The reason for the compiler errors is that paperweight adds the remapped jar as a dependency:

paperweight/paperweight-lib/src/main/kotlin/io/papermc/paperweight/util/project-util.kt

Line 82 in 3f6c7a5

add(create(parent.files(remappedJar)))

This dependency ends up being the first entry in the javac classpath argument (I don't know much about gradle internals but after some playing around with the code and googling I believe that through gradle's dependency resolution file dependencies always end up at the beginning in front of any non-file dependency).

In incremental compilation, gradle instructs javac to compile only those classes that have changed (and dependants). Any missing class will be loaded from the files defined in the classpath where the remapped jar is the first entry to be searched in.

javac will load any NMS related class from the remapped jar which then causes compiler errors as these classes are the wrong ones.

I wrote a discord message while investigating this morning (it contains a little more in-depth info on what I did): https://discord.com/channels/289587909051416579/555462289851940864/1119107748030844938

Screenshot of the message

Simple Reproduction

Spin up a clean PaperMC project
Remove the log4j annotation processor from Paper-Server
Run ./gradlew compileJava
- This one should run as it has no cache yet and performs a full recompilation
Change something in a class (in my case I added a println to the constructor of EmptyLevelChunk)
Run ./gradlew compileJava
- This run should fail now. If you want, you can add a --info to see whether (and why) gradle compiles
- Adding --debug and searching for compiler arguments should lead you to the javac command args

The log4j annotation processor is currently the only reason why this bug has not been noticed. If Apache marks it as incremental or maybe Gradle further improves the incremental compilation, then this will probably break every non-CI build. A quick workaround would be to disable the incremental compilation feature.

While browsing around in the source of paperweight I've noticed that @Machine-Maker recently added a task that generates a filtered remapped jar. I've tried to use that jar instead of the normal remapped jar. It resolved most of the compiler errors. But I'm getting new errors towards LogUtils.getClassLogger() missing. I'll maybe look into that later today.

After the merge of gradle/gradle#27942 this issue will be fixed once we upgrade to Gradle 8.7

I'm not sure about that, I don't see anything in that change that addresses my comments here and here. So the likely outcome is that #200 is superseded, but incremental compilation will still be effectively broken.

Do you have a reproduction case for incremental compilation being overzealous in its selections? Is it related to jars? I think if a downstream jar changes then it may attempt to recompile anything it thinks depends on it, but we can probably be smarter on indicating changes.

As expected Gradle 8.7-rc-2 shows the same behavior as I described here.

If you need more specific reproduction steps:

Clone Paper master
Update the Gradle wrapper to 8.7-rc-2
Run the applyPatches task
build -x test
Make any change to one class that doesn't require recompilation of any other classes. For example in Commands.java comment out the registration of a command.
build -x test --info
Observe Incremental compilation of 2788 classes completed in 14.377 secs in the log

I can reproduce this behavior in a normal gradle repository too. The compiler seems to recompile everything that depends on a class which makes sense. However, gradle also does this when only changing something in a method body (a single int or a string in a hardcoded println).

If I haven't missed something, then changing anything within the Commands.java or any other class that somehow connects to a ton of other classes (either directly or indirectly), you end up with gradle needlessly recompiling lots of classes.

There is a comment in an issue of 2021 where this behavior is described (although I'm not sure what exactly "any change to the API" means).

If that behavior is indeed intended, then I think that there is not much we can do from a paper-perspective except maybe splitting code into a bunch of modules which would be a nightmare on its own.

I've tried something with my paper fork:

Compile
Create a new class with a method, don't use it anywhere.
Compile (It should end up only compiling the new class)
Change something within the method of the unused class (it should still only recompile that class)
Use the class/function in Main.java & Compile (should now recompile ~20-30 classes)
Use the class/function in MinecraftServer.java constructor & Compile (Compiles ~3k classes)

For smaller projects, the tradeoff for doing ABI-level analysis might not make sense, but I have a feeling for larger ones it probably could... either way it looks like this is indeed intended behavior from Gradle.

(core/patcher): Gradle incremental compilation broken