Giters

gradle / gradle

Adaptable, fast automation for all

Home Page:https://gradle.org

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Parent project classloader scopes should be locked before evaluating any children project

eskatos opened this issue · comments

commented

When using evaluationDependsOn(p) or Configure-On-Demand, it can happen a project is evaluated before its parent.

When this happens, parent classloader scopes aren't locked. This works in most cases with Groovy scripts because Gradle doesn't run with -Dorg.gradle.classloaderscope.strict=true. It currently fails with Kotlin scripts because the Kotlin DSL provider requires parent scopes to be locked in order to deterministically compute script compilation classpaths.

Groovy or Kotlin, letting the classpath vary depending on the order the projects are evaluated lead to hard to debug errors particularly when combined with configure-on-demand:

Kotlin
Why my script fails to compile when I run ./gradlew :foo:bar but compiles cleanly when I run ./gradlew bar?

Groovy
Why my script fails to execute when I run ./gradlew :foo:bar but executes properly when I run ./gradlew bar?

As suggested by @adammurdoch, this should be fixed by introducing some sort of implicit script classpath evaluation dependency between a child project and its parent.

See gradle/kotlin-dsl-samples#782 for some context

commented

Is this relationship something that would be specified in the settings.gradle file or would it simply be some sort of standard that can't be modified?

commented

Gradle class loader scope hierarchy follows the project hierarchy. Note that configure-on-demand (COD) or not, the root project is always configured first.

evaluationDependsOn() and COD don't respect that hierarchy, they would configure parents projects first if they did. It could mean more projects configured.

If you are looking for workarounds see gradle/kotlin-dsl-samples#607 (comment)

commented

The issue can be reproduced without COD with Groovy scripts as follows.

// settings.gradle
include(":a", ":b:c")

// a/build.gradle
evaluationDependsOn(":b:c")

// b/build.gradle
plugins {
    id("org.gradle.hello-world") version "0.2" apply false
}

// b/c/build.gradle
import org.gradle.plugin.HelloWorldTask

https://scans.gradle.com/s/h3p6l5js65fig

If you rename :a to :d, no more failure.

commented

What adds to the difficulty to track that kind of issues with Groovy scripts is that if, in the reproducer above, b/c/build.gradle was using the org.gradle.plugin.HelloWorldTask without making a static reference to the type (the import above), then the Groovy script would compile, but would be relying on Groovy dynamics and reflection at runtime. Add COD to the mix and it also depends on what task you request on the command line.

Another manifestation can be observed by replacing the content of b/c/build.gradle in the reproducer above by:

// b/c/build.gradle
plugins {
    id("org.gradle.hello-world")
}

Then you get a different error for the same issue: Plugin [id: 'org.gradle.hello-world'] was not found
https://scans.gradle.com/s/snxz65x332enm

Again, if you rename :a to :d, no more failure, because then :b gets evaluated first.

commented

Any updates? I just encountered this, kinda unpleasant.

commented

We are running into this in one of our builds that is mixed Groovy/Kotlin build scripts but use neither COD nor evaluationDependsOn. We removed both as we ran into this a few times in converting, but see it now in a medium-sized multi-project build where most of the configuration is done in the root build.gradle.kts

commented

I can't share the full project but I do see it happen where we have something like this:

build.gradle.kts

allprojects {
  // some configuration of plugins.withId, tasks, other stuff
}

tasks {
  val copyBunchOfStuff by creating(Copy::class) {
    from(tasks.findByPath(":code:in:sub:assembleTar")) {
        rename { "renamed-assembly.tar.gz" }
    }
  }
}

Getting rid of the tasks.findByPath seems to get around the classloader scope exception. Hopefully this gives some more insight!

commented

@mkobit I can't reproduce with your snippet, any chance you could narrow it down to a reproducer?

Just a gut feeling, but does from(provider { tasks.findByPath(":code:in:sub:assembleTar") }) { .. } fix it?

commented

This issue is several years old now. We have a somewhat consistent repro, can we help report anything?

commented

We made a workground, hope it will work also for some of you.

We have many projects, some of them use kotlin script, we have problem when run task of kotlin project with CoD(it can be easily reproduced by deleting the gradle cache and stopping the deamon), the root project is using groovy, so we add the following code to the build script of the root project, the idea is, for each kotlin project, evaluate its parent projects first.

StartParameter startParameter = gradle.getStartParameter();
if(startParameter.isConfigureOnDemand()) {
  subprojects.each {s ->
    if(s.buildscript.sourceFile.name.endsWith(".kts")){ //kotlin project
      def current = s
      def depPath = current.path
      while(current.path != rootProject.path && current.parent.path != rootProject.path) {
        depPath += " -> ${current.parent.path}"
        current = current.evaluationDependsOn(current.parent.path)
      }
      logger.info("Adding kotlin project dependency for CoD: ${depPath}")
    }
  }
}
commented

Thanks @andy-maca for your solution. We've sporadically run into this issue for our projects since switching over to Kotlin build scripts. We had been re-ordering dependencies to try to encourage Gradle to resolve the parent projects first, which worked in most cases but was inconsistent and painful. I implemented your changes today and it seems to have fixed things for us so far.

subprojects {
	// Address https://github.com/gradle/gradle/issues/4823: Force parent project evaluation before sub-project evaluation for Kotlin build scripts
	if (gradle.startParameter.isConfigureOnDemand
			&& buildscript.sourceFile?.extension?.toLowerCase() == "kts"
			&& parent != rootProject) {
		generateSequence(parent) { project -> project.parent.takeIf { it != rootProject } }
				.forEach { evaluationDependsOn(it.path) }
	}
}
commented

This issue didn't appear until Gradle 6.8.1, not sure why but either it was always broken and now it's being surfaced or something regressed in 6.8...

commented

I've just got bitten by this issue in 6.8.2 - What triggered it was performing a dependency replacement within my buildscript block, i.e:

buildscript {
    repositories { ... }
    dependencies { ... }
    configurations.all {
        resolutionStrategy.eachDependency {
            if (requested.group == "com.squareup.okhttp" && requested.name == "okhttp") {
                useTarget(Dependencies.OkHttp)
            }
        }
    }
}

☝️ Where Dependencies is defined in buildSrc.
If I swap Dependencies.OkHttp for a string literal of the same value the problem goes away.

commented

@chris-hatton what if you move your dependencies class into root of buildSrc? (As in no package name? I noticed other issues where dependencies wasn't detected when it was under a package)

commented

Hello everyone,
I might have a consistent repro.

Let me know if I should create an extra ticket for it or this is enough.

Issue is IMHO the same
org.gradle.api.internal.initialization.DefaultClassLoaderScope@46de2ed5 must be locked before it can be used to compute a classpath!.

With Gradle 6.7 and older we were getting this issue randomly.
With Gradle 6.8 and up (last tested was 7) it was consistent with configureondemand=true. With configureondemand=false, Windows agents stopped seeing this issue but Mac agents still couldn't get rid of it.

Fix was to remove intermediate (parent) folders of modules.

This works:
root/app
root/lib_device_pairing

This doesn't work:
root/app
root/lib/device_pairing

commented

This works:
root/app
root/lib_device_pairing

This doesn't work:
root/app
root/lib/device_pairing

@MartinRajniak In my case, the latter could be made to work by flattening just the module reference name, but keeping the directory structure:

include(":lib_device_pairing")
project(":lib_device_pairing").projectDir = File("lib/device_pairing")

Still a bit of a nuisance, but seems to work around the issue.

commented

Posting here as this seems to be the only left opened issue about the locking issue in kts.

The issue happens for me when I try to use version catalog inside the buildscript of the main build.gradle.kts.
Everything works fine before in a multi module project. But have the
java.lang.IllegalArgumentException: project.classLoaderScope must be locked before querying the project schema and java.lang.IllegalArgumentException: org.gradle.api.internal.initialization.DefaultClassLoaderScope@1b8d0a50 must be locked before it can be used to compute a classpath! as soon as I use the libs.xxxx in buildscript

I can workaround by using:

    val catalogs = extensions.getByType<VersionCatalogsExtension>()
    val libs = catalogs.named("libs")

    dependencies {
        classpath(libs.findDependency("plugin-r8").get())

But I'm not sure to understand why this happens when reading all the other issues related to this and this one.

commented

At this point this ticket is 3 years old with limited input from Gradle.

I guess the only real solution is to switch all our kts files back to groovy as no one has given us any sort of indication this is being investigated or a timeframe for a fix...

commented

We disabled configure on demand and have been fine since. It's being effectively replaced by configuration caching anyway. We don't see any real performance issue with disabling it either.

commented

How many modules do you have? an invalid config is a good 1m+ for us so anything to help at this point

commented

I have configure on demand to false and it worked nicely until I used the version catalog and tried to use it in main gradle.kts

commented

We're approaching 200 modules but honestly didn't see much performance change in disabling configuration on demand

commented

There is not much change when running the whole app from the root. But there would be a difference when running unit tests of a leaf module. In that case, the whole repo will be configured first before running the tests.

commented

There is not much change when running the whole app from the root. But there would be a difference when running unit tests of a leaf module. In that case, the whole repo will be configured first before running the tests.

That was my understanding too, when working on feature modules you would only configure your direct modules required. In some cases that could be a large difference?

commented

This issue is about using evaluationDependsOn and configure-on-demand. Other use cases should be fixed by #14616 already.

@Tolriq, sorry you are getting into troubles with dependencies catalogs and thanks for the report. The fact that this happens without configure-on-demand enabled make it look like a bug in how dependencies catalogs accessors work. Could you please open a separate issue, ideally with a self contained reproducer?

commented

@Tolriq your issue is via CLI or syncing the IDE

commented

Disabling configureOnDemand helped fixed my issue

commented

In my case, it was happening while creating & adding a new module to the Android project. Turns out, it was happening because Android Studio was adding these 2 lines to project level gradle file:

id("com.android.library") version "7.1.2" apply false
id("org.jetbrains.kotlin.android") version "1.6.10" apply false

Once I removed these it worked. I already had many modules added to the project without any issues, so I looked for what was different for this new one and that's how I found the issue. Would suggest anyone with this issue to do the same.

commented

Another case i found today:

root/settings.gradle.kts:

pluginManagement {
    apply(from = "myplugin.settings.gradle.kts")
}

any exception thrown inside this custom settings script during IDE gradle sync leads to same:

java.lang.IllegalArgumentException: org.gradle.api.internal.initialization.DefaultClassLoaderScope@568c7753 must be locked before it can be used to compute a classpath!
	at org.gradle.kotlin.dsl.provider.KotlinScriptClassPathProvider.exportClassPathFromHierarchyOf(KotlinScriptClassPathProvider.kt:152)
	at org.gradle.kotlin.dsl.provider.KotlinScriptClassPathProvider.computeCompilationClassPath(KotlinScriptClassPathProvider.kt:148)
	at org.gradle.kotlin.dsl.provider.KotlinScriptClassPathProvider.access$computeCompilationClassPath(KotlinScriptClassPathProvider.kt:95)
	at org.gradle.kotlin.dsl.provider.KotlinScriptClassPathProvider$compilationClassPathOf$1.invoke(KotlinScriptClassPathProvider.kt:144)
	at org.gradle.kotlin.dsl.provider.KotlinScriptClassPathProvider$compilationClassPathOf$1.invoke(KotlinScriptClassPathProvider.kt:95)
	at org.gradle.kotlin.dsl.provider.KotlinScriptClassPathProviderKt$sam$java_util_function_Function$0.apply(KotlinScriptClassPathProvider.kt)

running ./gradlew help in terminal shows correct stack trace

commented

I believe I'm running into the same bug with Gradle 8.0.1 (interestingly only on Windows) when inputs / from for a Copy-task depend on Jars produced by tasks in another project, see this:

val processResources = tasks.named<Copy>("processResources").configure {
    val gradleModelProject = project(":plugins:package-managers:gradle-model")
    val gradleModelJarTask = gradleModelProject.tasks.named<Jar>("jar")
    val gradleModelJarFile = gradleModelJarTask.get().outputs.files.singleFile

    val gradlePluginProject = project(":plugins:package-managers:gradle-plugin")
    val gradlePluginJarTask = gradlePluginProject.tasks.named<Jar>("jar")
    val gradlePluginJarFile = gradlePluginJarTask.get().outputs.files.singleFile

    // As the Copy-task simply skips non-existing files, add explicit dependencies on the Jar-tasks.
    dependsOn(gradleModelJarTask, gradlePluginJarTask)

    // Bundle the model and plugins JARs as resources, so the inspector can copy them at runtime to the init script's
    // classpath.
    from(gradleModelJarFile, gradlePluginJarFile)

    // Ensure constant file names without a version suffix.
    rename(gradleModelJarFile.name, "gradle-model.jar")
    rename(gradlePluginJarFile.name, "gradle-plugin.jar")
}

Running

./gradlew :plugins:package-managers:gradle-inspector:processResources --rerun-tasks --no-build-cache -Dorg.gradle.configureondemand=true

fails on Windows with a lot of unresolved referenced and

* What went wrong:
Execution failed for task ':plugins:package-managers:gradle-plugin:compileKotlin'.
> A failure occurred while executing org.jetbrains.kotlin.compilerRunner.GradleCompilerRunnerWithWorkers$GradleKotlinCompilerWorkAction
   > Compilation error. See log for more details

But when running the failed task :plugins:package-managers:gradle-plugin:compileKotlin manually, it succeeds.

Also, running with configure-on-demand disabled works:

./gradlew :plugins:package-managers:gradle-inspector:processResources --rerun-tasks --no-build-cache -Dorg.gradle.configureondemand=false

Wrapping inputs in provider {} as suggested here unfortunately does not help.

commented

Here's a summary of complete reproducers shared in this issue and their outcome as of Today.

As noted in the comment for the second one, they are about using evaluationDependsOn() even without enabling Configure On Demand.
They also all fail in Groovy when they fail in Kotlin and respectively succeed in Groovy when they succeed in Kotlin. The error messages are different though but that's not the most important thing here.

Reproducers:

1/ #4823 (comment)

2/ #4823 (comment)

Results with links to build scans:

Reproducer Last version failing First version fixed
repro-1-kotlin 🔴 7.3 🟢 7.4
repro-1-groovy 🔴 7.3 🟢 7.4
repro-2-kotlin 🔴 7.3 🟢 7.4
repro-2-groovy 🔴 7.3 🟢 7.4

All the actionable reproducers from this issue are fixed since Gradle 7.4.
I could not figure out what PR in 7.4.x fixed those use cases though.
I'm going to close this as fixed in 7.4.

There are other open issues with Configure On Demand, see https://github.com/gradle/gradle/issues?q=is%3Aopen+is%3Aissue+label%3Ain%3Aconfigure-on-demand

If you come across this issue, please look for existing ones and add your 👍 there or create a new issue.

Also note that most of the classloading issues were already fixed in 6.8 by

commented

I see the PR merged but I'm confused by how to use it. Could you please elaborate ? I'm on Gradle 8.5 and my faulty script looks like this:

tasks {
    wrapper {
        gradleVersion = "8.5"
        distributionType = Wrapper.DistributionType.BIN
    }

    create("publishSdkLocal") {
        val publicationTasks = this.project.flattenChildren().filter { sdkProjects.contains(it.name) }.map { p ->
            p.getTasksByName("publishToMavenLocal", false)
        }
        dependsOn(publicationTasks)
    }
}

fun Project.flattenChildren(): List<Project> {

    return if (this.childProjects.isNotEmpty()) {
        childProjects.values.flatMap { it.flattenChildren() } + this
    } else
        listOf(this)
}