Bloop fails to compile project with weird error

Question

Bloop fails to compile project with weird error

ktonga opened this issue a year ago · comments

It happens for only one of the modules (maybe some other after passing that one too) in a multi-module project, there are many modules before that one is attempted which finish compiling without issue. Also if I compile using sbt it doesn't happen.

I wasn't able to isolate the issue in a smaller self-contained project. I can help with any debugging if I'm guided, TBH I don't know where I could start from. To see that error I had to add the --verbose flag.

[E] Unexpected error when compiling merchant-model: 'null'
[T] java.nio.file.ClosedFileSystemException
[T] 	jdk.zipfs/jdk.nio.zipfs.ZipFileSystem.ensureOpen(ZipFileSystem.java:1628)
[T] 	jdk.zipfs/jdk.nio.zipfs.ZipFileSystem.getFileAttributes(ZipFileSystem.java:536)
[T] 	jdk.zipfs/jdk.nio.zipfs.ZipPath.readAttributes(ZipPath.java:767)
[T] 	jdk.zipfs/jdk.nio.zipfs.ZipPath.readAttributes(ZipPath.java:777)
[T] 	jdk.zipfs/jdk.nio.zipfs.ZipFileSystemProvider.readAttributes(ZipFileSystemProvider.java:276)
[T] 	java.base/java.nio.file.Files.readAttributes(Files.java:1851)
[T] 	java.base/java.nio.file.Files.size(Files.java:2468)
[T] 	sbt.internal.inc.HashUtil$.farmHash(HashUtil.scala:31)
[T] 	sbt.internal.inc.FarmHash$.ofPath(Stamp.scala:89)
[T] 	sbt.internal.inc.Stamper$.$anonfun$forHashInRootPaths$3(Stamp.scala:233)
[T] 	sbt.internal.inc.Stamper$.tryStamp(Stamp.scala:188)
[T] 	sbt.internal.inc.Stamper$.$anonfun$forHashInRootPaths$1(Stamp.scala:233)
[T] 	sbt.internal.inc.UncachedStamps.library(Stamp.scala:468)
[T] 	sbt.internal.inc.InitialStamps.$anonfun$library$1(Stamp.scala:422)
[T] 	scala.collection.mutable.HashMap.getOrElseUpdate(HashMap.scala:86)
[T] 	sbt.internal.inc.InitialStamps.library(Stamp.scala:422)
[T] 	sbt.internal.inc.TimeWrapBinaryStamps.$anonfun$library0$1(Stamp.scala:446)
[T] 	sbt.internal.inc.Stamper$.$anonfun$timeWrap$1(Stamp.scala:252)
[T] 	sbt.internal.inc.TimeWrapBinaryStamps.library(Stamp.scala:447)
[T] 	sbt.internal.inc.bloop.internal.BloopAnalysisCallback.$anonfun$addProductsAndDeps$16(BloopAnalysisCallback.scala:400)
[T] 	scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:286)
[T] 	scala.collection.mutable.HashSet.foreach(HashSet.scala:79)
[T] 	scala.collection.TraversableLike.map(TraversableLike.scala:286)
[T] 	scala.collection.TraversableLike.map$(TraversableLike.scala:279)
[T] 	scala.collection.mutable.AbstractSet.scala$collection$SetLike$$super$map(Set.scala:50)
[T] 	scala.collection.SetLike.map(SetLike.scala:105)
[T] 	scala.collection.SetLike.map$(SetLike.scala:105)
[T] 	scala.collection.mutable.AbstractSet.map(Set.scala:50)
[T] 	sbt.internal.inc.bloop.internal.BloopAnalysisCallback.$anonfun$addProductsAndDeps$1(BloopAnalysisCallback.scala:398)
[T] 	scala.collection.TraversableOnce$folder$1.apply(TraversableOnce.scala:196)
[T] 	scala.collection.TraversableOnce$folder$1.apply(TraversableOnce.scala:194)
[T] 	scala.collection.mutable.HashSet.foreach(HashSet.scala:79)
[T] 	scala.collection.TraversableOnce.foldLeft(TraversableOnce.scala:199)
[T] 	scala.collection.TraversableOnce.foldLeft$(TraversableOnce.scala:192)
[T] 	scala.collection.AbstractTraversable.foldLeft(Traversable.scala:108)
[T] 	sbt.internal.inc.bloop.internal.BloopAnalysisCallback.addProductsAndDeps(BloopAnalysisCallback.scala:356)
[T] 	sbt.internal.inc.bloop.internal.BloopAnalysisCallback.get(BloopAnalysisCallback.scala:298)
[T] 	sbt.internal.inc.bloop.internal.BloopIncremental$.$anonfun$compileIncremental$5(BloopIncremental.scala:119)
[T] 	bloop.task.Task.$anonfun$runAsync$7(Task.scala:265)
[T] 	monix.eval.Task$Map.apply(Task.scala:4604)
[T] 	monix.eval.Task$Map.apply(Task.scala:4600)
[T] 	monix.eval.internal.TaskRunLoop$.startFull(TaskRunLoop.scala:170)
[T] 	monix.eval.internal.TaskRestartCallback.syncOnSuccess(TaskRestartCallback.scala:101)
[T] 	monix.eval.internal.TaskRestartCallback.onSuccess(TaskRestartCallback.scala:74)
[T] 	monix.eval.internal.TaskExecuteOn$AsyncRegister$$anon$1.run(TaskExecuteOn.scala:71)
[T] 	java.base/java.util.concurrent.ForkJoinTask$RunnableExecuteAction.exec(ForkJoinTask.java:1395)
[T] 	java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:373)
[T] 	java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1182)
[T] 	java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1655)
[T] 	java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1622)
[T] 	java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:165)
[E] Failed to compile 'merchant-model'

Tomasz Godzik · Answer 1 · Thu May 18 2023 18:34:27 GMT+0800 (China Standard Time)

Thanks for reporting! This looks like an issue with Zinc it or the way we are using it 🤔

Gastón Tonietti · Answer 2 · Sat May 27 2023 20:41:17 GMT+0800 (China Standard Time)

Hey @tgodzik, after testing with the snapshot as you suggested on Twitter, I got the following error logs:

2023.05.27 22:34:42 ERROR Could not calculate hash for /BC/java.base/java/lang/CharSequence.sig because of null
2023.05.27 22:34:42 ERROR Could not calculate hash for /BC/java.xml/javax/xml/parsers/DocumentBuilderFactory.sig because of null
2023.05.27 22:34:42 ERROR Could not calculate hash for /BCDEFG/java.xml/org/w3c/dom/Node.sig because of null
2023.05.27 22:34:42 ERROR Could not calculate hash for /B/java.base/java/io/InputStream.sig because of null
2023.05.27 22:34:42 ERROR Could not calculate hash for /BCDEFG/java.xml/javax/xml/xpath/XPathConstants.sig because of null
2023.05.27 22:34:42 ERROR Could not calculate hash for /BCDEFG/java.xml/org/w3c/dom/NodeList.sig because of null
2023.05.27 22:34:42 ERROR Could not calculate hash for /BCDEFG/java.base/java/io/ByteArrayInputStream.sig because of null
2023.05.27 22:34:42 ERROR Could not calculate hash for /BCDEF/java.xml/javax/xml/xpath/XPathExpression.sig because of null
2023.05.27 22:34:42 ERROR Could not calculate hash for /BCDEF/java.xml/javax/xml/xpath/XPath.sig because of null
2023.05.27 22:34:42 ERROR Could not calculate hash for /BCDEF/java.xml/javax/xml/namespace/QName.sig because of null
2023.05.27 22:34:42 ERROR Could not calculate hash for /BCDEF/java.xml/javax/xml/parsers/DocumentBuilder.sig because of null
2023.05.27 22:34:42 ERROR Could not calculate hash for /BCDEF/java.xml/javax/xml/xpath/XPathFactory.sig because of null
2023.05.27 22:34:42 ERROR Could not calculate hash for /BCDEFG/java.xml/org/w3c/dom/Document.sig because of null
2023.05.27 22:34:42 ERROR Could not calculate hash for /9ABCDEFG/java.base/java/io/File.sig because of null
2023.05.27 22:34:42 ERROR Could not calculate hash for /9ABCDEFG/java.base/java/lang/ClassLoader.sig because of null
2023.05.27 22:34:42 ERROR Could not calculate hash for /BC/java.base/java/nio/file/Path.sig because of null
2023.05.27 22:34:42 ERROR Could not calculate hash for /BCDEFG/java.base/java/nio/file/Paths.sig because of null
2023.05.27 22:34:42 ERROR Could not calculate hash for /9ABCDEF/java.base/java/time/OffsetDateTime.sig because of null
2023.05.27 22:34:42 ERROR Could not calculate hash for /89ABCDEFG/java.base/java/util/HashMap.sig because of null
2023.05.27 22:34:42 ERROR Could not calculate hash for /BCDEF/java.base/java/time/ZoneOffset.sig because of null
2023.05.27 22:34:42 ERROR Could not calculate hash for /9ABCDEF/java.base/java/time/LocalDate.sig because of null
2023.05.27 22:34:42 ERROR Could not calculate hash for /9ABCDEF/java.base/java/time/ZoneId.sig because of null
2023.05.27 22:34:42 ERROR Could not calculate hash for /879ABCDEFG/java.base/java/nio/charset/Charset.sig because of null
2023.05.27 22:34:42 ERROR Could not calculate hash for /BCDEFG/java.base/java/nio/charset/StandardCharsets.sig because of null
2023.05.27 22:34:43 ERROR Could not calculate hash for /BCDEFG/java.base/java/util/concurrent/atomic/AtomicLong.sig because of null
2023.05.27 22:34:43 ERROR Could not calculate hash for /BCDEFG/java.base/java/util/Iterator.sig because of null
2023.05.27 22:34:44 ERROR Could not calculate hash for /BCDEF/java.base/java/util/stream/Stream.sig because of null
2023.05.27 22:34:44 ERROR Could not calculate hash for /BCDEFG/java.base/java/util/stream/BaseStream.sig because of null
2023.05.27 22:34:44 ERROR Could not calculate hash for /9AB/java.base/java/lang/Integer.sig because of null
2023.05.27 22:34:44 ERROR Could not calculate hash for /BCD/java.base/java/lang/Thread.sig because of null
2023.05.27 22:34:44 ERROR Could not calculate hash for /BCDEFG/java.base/java/util/concurrent/Future.sig because of null
2023.05.27 22:34:44 ERROR Could not calculate hash for /B/java.base/java/util/concurrent/CompletableFuture.sig because of null
2023.05.27 22:34:44 ERROR Could not calculate hash for /BCDEFG/java.base/java/util/concurrent/Callable.sig because of null
2023.05.27 22:34:44 ERROR Could not calculate hash for /89ABCDEFG/java.base/java/util/Collections.sig because of null
2023.05.27 22:34:44 ERROR Could not calculate hash for /9ABCDEFG/java.base/java/util/concurrent/Executors.sig because of null
2023.05.27 22:34:44 ERROR Could not calculate hash for /BCDEFG/java.base/java/util/concurrent/ExecutorService.sig because of null
2023.05.27 22:34:44 ERROR Could not calculate hash for /BCDEFG/java.base/java/lang/Iterable.sig because of null
2023.05.27 22:34:44 ERROR Could not calculate hash for /BCDEF/java.base/java/util/function/Consumer.sig because of null
2023.05.27 22:34:44 ERROR Could not calculate hash for /9ABCDEF/java.base/java/time/ZonedDateTime.sig because of null
2023.05.27 22:34:44 ERROR Could not calculate hash for /9ABCDEF/java.base/java/time/LocalTime.sig because of null
2023.05.27 22:34:46 ERROR Could not calculate hash for /9ABCDEFG/java.base/java/util/concurrent/ArrayBlockingQueue.sig because of null
2023.05.27 22:34:46 ERROR Could not calculate hash for /BCDEF/java.base/java/util/function/IntUnaryOperator.sig because of null
2023.05.27 22:34:46 ERROR Could not calculate hash for /BCDEFG/java.base/java/util/concurrent/atomic/AtomicInteger.sig because of null
2023.05.27 22:34:46 ERROR Could not calculate hash for /BCDEFG/java.base/java/util/Collection.sig because of null
2023.05.27 22:34:46 ERROR Could not calculate hash for /BCDEF/java.base/java/util/concurrent/atomic/AtomicReference.sig because of null

Hope it helps, If you need any more help just let me know.
Cheers.

Tomasz Godzik · Answer 3 · Sun May 28 2023 02:11:13 GMT+0800 (China Standard Time)

Och, that's curious. It shouldn't reach into the jar, just calculate the hash of the entire jar.

Tomasz Godzik · Answer 4 · Sun May 28 2023 02:36:17 GMT+0800 (China Standard Time)

Also, any idea where those .sig files are coming from? Should they be part of the JDK itself? I wonder if we should just filter them out and only calculate hashes for .class files.

Gastón Tonietti · Answer 5 · Mon May 29 2023 15:51:04 GMT+0800 (China Standard Time)

Hey @tgodzik

It doesn't look like those files are in the JDK runtime FS.
I played around a bit with it and it seems it only contains .class files:

val fs = java.nio.file.FileSystems.getFileSystem(java.net.URI.create("jrt:/"))
java.nio.file.Files.list(fs.getPath("modules", "java.base", "java/lang")).forEach(println(_))

modules/java.base/java/lang/AbstractMethodError.class
modules/java.base/java/lang/AbstractStringBuilder.class
modules/java.base/java/lang/Appendable.class
modules/java.base/java/lang/ApplicationShutdownHooks$1.class
modules/java.base/java/lang/ApplicationShutdownHooks.class
modules/java.base/java/lang/ArithmeticException.class
modules/java.base/java/lang/ArrayIndexOutOfBoundsException.class
modules/java.base/java/lang/ArrayStoreException.class
modules/java.base/java/lang/AssertionError.class
modules/java.base/java/lang/AssertionStatusDirectives.class
modules/java.base/java/lang/AutoCloseable.class
modules/java.base/java/lang/Boolean.class
modules/java.base/java/lang/BootstrapMethodError.class
modules/java.base/java/lang/Byte$ByteCache.class
modules/java.base/java/lang/Byte.class
modules/java.base/java/lang/CharSequence$1CharIterator.class
modules/java.base/java/lang/CharSequence$1CodePointIterator.class
modules/java.base/java/lang/CharSequence.class
modules/java.base/java/lang/Character$CharacterCache.class
....

 java.nio.file.Files.lines(fs.getPath("modules", "java.base", "java/lang/CharSequence.sig")).forEach(println(_))

java.nio.file.NoSuchFileException: /modules/java.base/java/lang/CharSequence.sig
  at java.base/jdk.internal.jrtfs.JrtFileSystem.checkNode(JrtFileSystem.java:486)
  at java.base/jdk.internal.jrtfs.JrtFileSystem.getFileContent(JrtFileSystem.java:245)
  at java.base/jdk.internal.jrtfs.JrtFileSystem.newInputStream(JrtFileSystem.java:334)
  at java.base/jdk.internal.jrtfs.JrtPath.newInputStream(JrtPath.java:637)
  at java.base/jdk.internal.jrtfs.JrtFileSystemProvider.newInputStream(JrtFileSystemProvider.java:325)
  at java.base/java.nio.file.Files.newInputStream(Files.java:160)
  at java.base/java.nio.file.Files.newBufferedReader(Files.java:2922)
  at java.base/java.nio.file.Files.lines(Files.java:4141)
  at java.base/java.nio.file.Files.lines(Files.java:4227)
  ... 59 elided

Tomasz Godzik · Answer 6 · Wed May 31 2023 02:01:22 GMT+0800 (China Standard Time)

the .sig files are supposed to be pickles needed for pipelining (not only probably) but we don't have any pickles for Java files, especially not inside a jar.

I will dig in a bit more.

Tomasz Godzik · Answer 7 · Wed May 31 2023 02:10:06 GMT+0800 (China Standard Time)

Are you using any JDK specific classes or options? How is the modules defined, would you be able to share that?

Looks like the sig files should not pop up there and they are somehow 🤔

Gastón Tonietti · Answer 8 · Fri Jun 02 2023 08:23:25 GMT+0800 (China Standard Time)

Are you using any JDK specific classes or options? How is the modules defined, would you be able to share that?

Looks like the sig files should not pop up there and they are somehow 🤔

So this is the JDK im using locally:

$ java -version
openjdk version "17.0.7" 2023-04-18
OpenJDK Runtime Environment Homebrew (build 17.0.7+0)
OpenJDK 64-Bit Server VM Homebrew (build 17.0.7+0, mixed mode, sharing)

And this, I think, are all the relevant build definition bits:

val baseSettings =
  Seq(
    scalaVersion := Dependencies.Version.scala,
    organization := "...",
    logLevel     := Level.Info,
    scalacOptions ++= Seq("-release", "11"),
    Compile / scalacOptions ++= Seq(
      "-Wconf:any:warning-verbose",
      "-Wunused:nowarn",
      "-Ymacro-annotations",
      "-Xsource:3"
    ),
    PB.protocVersion := Version.protobufJava, // generate objects at same version as lib
    Compile / scalacOptions ~= {
      options: Seq[String] =>
        options.filterNot(
          Set(
            "-Wunused:imports",
            "-Wunused:params",
            "-Wunused:privates",
            "-Xlint:infer-any"
          )
        )
    },
    javacOptions ++= Seq("-source", "16", "-target", "16"),
    //      scalafmtOnCompile := true,
    testFrameworks ++= Seq(
      TestFramework("zio.test.sbt.ZTestFramework"),
      TestFramework("hedgehog.sbt.Framework")
    ),
    Global / cancelable           := true,
    Global / turbo                := true, // 🤞
    Global / fork                 := true, // https://github.com/sbt/sbt/issues/2274
    Global / onChangedBuildSource := ReloadOnSourceChanges,
    resolvers ++= Seq(
      "Sonatype OSS Snapshots" at "https://oss.sonatype.org/content/repositories/snapshots",
      "Confluent" at "https://packages.confluent.io/maven",
      "Ovotech" at "https://kaluza.jfrog.io/artifactory/maven/"
      //        Resolver.bintrayRepo("ovotech", "maven")
    ),
    // Don't make or publish scaladoc
    Compile / doc / sources                := Nil,
    Compile / packageDoc / publishArtifact := false,
    // parallel and forked tests
    Test / fork               := true,
    Test / testForkedParallel := true,
    Test / parallelExecution  := true,
    // [for the moment] Test / turbo := true,
    Test / concurrentRestrictions := Seq(
      Tags.limit(Tags.CPU, java.lang.Runtime.getRuntime.availableProcessors()),
      Tags.limit(Tags.Test, java.lang.Runtime.getRuntime.availableProcessors())
    ),
    // incremental compilation cache
    pushRemoteCacheTo := Some(MavenCache("sbt-build-cache", file(".sbt-build-cache")))
  )

val compilerPlugins =
  List(
    kindProjector,
    betterMonadicFor
  )

val commonSettings =
  baseSettings ++
    Seq(
      Compile / compile / wartremoverErrors ++= warts,
      libraryDependencies ++= compilerPlugins,
      dependencyOverrides ++= List(
        ...
      )
    )

lazy val `merchant-model` = (project in file("modules/merchant-model"))
  .settings(
    commonSettings,
    libraryDependencies ++=
      List(
        ...
      )
  )
  .dependsOn(
    ...
  )

Tomasz Godzik · Answer 9 · Fri Jun 30 2023 18:32:35 GMT+0800 (China Standard Time)

I finally got back to the PR with a potential fix for your issue @ktonga <- it should be in the newest snapshot version

Gastón Tonietti · Answer 10 · Fri Jun 30 2023 18:37:10 GMT+0800 (China Standard Time)

I finally got back to the PR with a potential fix for your issue @ktonga <- it should be in the newest snapshot version

Thank you so much, I'll give it a go. 🙏

Gastón Tonietti · Answer 11 · Wed Jul 05 2023 09:53:34 GMT+0800 (China Standard Time)

Hey! I was able to update to 1.5.7 and I'm not seeing the error any more, so I think we could close this one as fixed.