Scalafix sbt causing OOM errors
tgodzik opened this issue · comments
This started happening after the most recent release and always breaks on a stack trace:
java.lang.OutOfMemoryError: Java heap space
at org.eclipse.jgit.internal.storage.file.PackIndexV2.<init>(PackIndexV2.java:95)
at org.eclipse.jgit.internal.storage.file.PackIndex.read(PackIndex.java:101)
at org.eclipse.jgit.internal.storage.file.PackIndex.open(PackIndex.java:67)
at org.eclipse.jgit.internal.storage.file.Pack.idx(Pack.java:161)
at org.eclipse.jgit.internal.storage.file.Pack.get(Pack.java:273)
at org.eclipse.jgit.internal.storage.file.PackDirectory.open(PackDirectory.java:216)
at org.eclipse.jgit.internal.storage.file.ObjectDirectory.openPackedObject(ObjectDirectory.java:393)
at org.eclipse.jgit.internal.storage.file.ObjectDirectory.openPackedFromSelfOrAlternate(ObjectDirectory.java:356)
at org.eclipse.jgit.internal.storage.file.ObjectDirectory.openObjectWithoutRestoring(ObjectDirectory.java:346)
at org.eclipse.jgit.internal.storage.file.ObjectDirectory.openObject(ObjectDirectory.java:331)
at org.eclipse.jgit.internal.storage.file.WindowCursor.open(WindowCursor.java:132)
at org.eclipse.jgit.revwalk.RevWalk.getCachedBytes(RevWalk.java:1119)
at org.eclipse.jgit.revwalk.RevCommit.parseHeaders(RevCommit.java:126)
at org.eclipse.jgit.revwalk.RevWalk.markStart(RevWalk.java:308)
at org.eclipse.jgit.api.LogCommand.add(LogCommand.java:343)
at org.eclipse.jgit.api.LogCommand.add(LogCommand.java:180)
at org.eclipse.jgit.api.LogCommand.call(LogCommand.java:133)
at scalafix.internal.sbt.JGitCompletion.$anonfun$x$1$1(JGitCompletions.scala:29)
at scalafix.internal.sbt.JGitCompletion$$Lambda$9375/0x00007ec7[2180](https://github.com/scalameta/metals/actions/runs/8298008580/job/22710386419#step:4:2181)8600.apply(Unknown Source)
at scala.util.Try$.apply(Try.scala:213)
at scalafix.internal.sbt.JGitCompletion.<init>(JGitCompletions.scala:29)
at scalafix.internal.sbt.ScalafixCompletions.gitDiffParser$lzycompute(ScalafixCompletions.scala:119)
at scalafix.internal.sbt.ScalafixCompletions.gitDiffParser(ScalafixCompletions.scala:118)
at scalafix.internal.sbt.ScalafixCompletions.parser(ScalafixCompletions.scala:186)
at scalafix.sbt.ScalafixPlugin$.$anonfun$scalafixInputTask$4(ScalafixPlugin.scala:357)
at scalafix.sbt.ScalafixPlugin$$$Lambda$8642/0x00007ec7215f5600.apply(Unknown Source)
at scala.Function1.$anonfun$compose$1(Function1.scala:49)
at scala.Function1$$Lambda$1172/0x00007ec7204dee78.apply(Unknown Source)
even if no scalafix task is actually run.
Were there any changes recently that could cause it?
Maybe the jgit update broke things?
Or more likely https://github.com/scalacenter/sbt-scalafix/pull/378/files
Or it might be scalameta/scalameta#3650 though it's curious the stack trace show ScalafixCompletions and scalafix is not even being run
I think jgit use too many memory.
here is profile memory result just launch sbt in my private large repository.
just launch sbt
if remove .git directory before launch sbt. for avoid isGitRepository == true
Or more likely https://github.com/scalacenter/sbt-scalafix/pull/378/files
Yes scalacenter/sbt-scalafix#378 remove scalafixCompletions from globalSettings. I think it generate many JGitCompletion instances
also jgit use many memory if repo is too many commits and/or many branch, tag.
I think JGitCompletions.scala should not retain jgit objects for avoid memory leak.
refListisSeq[org.eclipse.jgit.lib.Ref]refsisList[org.eclipse.jgit.revwalk.RevCommit]
see following reproduce repo
also jgit use many memory if repo is too many commits and/or many branch, tag.
I think
JGitCompletions.scalashould not retain jgit objects for avoid memory leak.
Indeed, there is an inherent leak there, which predates sbt-scalafix 0.12.0.
I am on my phone right now so I cannot check, but I am unsure how the upper bound set by setMaxCount(20) causes so much memory consumption after the initial lookup.
In any case, feel free to open a follow-up PR 👍
workaround
-addSbtPlugin("ch.epfl.scala" % "sbt-scalafix" % "0.12.0")
+// TODO remove explicit "scalafix-interfaces" dependency if sbt-scalafix 0.12.1 released
+addSbtPlugin("ch.epfl.scala" % "sbt-scalafix" % "0.11.1")
+libraryDependencies += "ch.epfl.scala" % "scalafix-interfaces" % "0.12.0"Any plans to release new version? 👀
Yes! 2.13.14 is around the corner so I am holding the release a bit (to provide 2.13.14 compatibility in the testkit). Based on the current ETA, it should be last days of April.