Proposal: Add withAssemblyUnzipDirectory to `AssemblyOption`
er1c opened this issue · comments
Description
Today, we currently have a single setter (withAssemblyDirectory
) to configure the working directory for the assembly process. This defaults to a directory within target/
.
I propose adding two one additional setters:
withAssemblyUnzipDirectory
- which specifies the directory location to unzip library dependency jars into.
*withAssemblyOutputDirectory
- which specifies the directory location to cache the fully merged/etc assembly targets into.
Motivation
When assembling an über artifact, that has many library dependencies, the unzip process can be very IO intensive (I've seen just this step take 10+ minutes in our CI). These unzipped directories are very suitable for CI systems to persist in between job runs.
However, on a CI agent, the withCacheOutput
is not always suitable to persist in-between job runs, especially if re-used for different branches with lots of changes. From a practical standpoint, the target/
directory is a perfect location for this scenario.
Additionally, other CI workflows may want to persist the withCacheOutput
in-between runs, or manage pruning the contents of these directories differently than the withCacheUnzip
directory cache.
Example Usages
Global Cache, Two Different Locations
lazy val sbtAssemblyUnzipDirectory = taskKey[File]("Directory to cache sbt assembly library jars.")
~~ lazy val sbtAssemblyOutputDirectory = taskKey[File]("Directory to cache sbt assembly output.")~~
lazy val sbtAssemblyDirectory = taskKey[File]("Directory to cache sbt assembly output.")
assembly / assemblyOption := {
val opt = (assembly / assemblyOption).value
val dir = sbtAssemblyDirectory.value
opt
.withAssemblyUnzipDirectory(sbtAssemblyUnzipDirectory.value)
.withCacheUnzip(true) // Keep dependency .jar files pre-extracted
~~ .withAssemblyOutputDirectory(sbtAssemblyOutputDirectory.value)~~
.withAssemblyDirectory(sbtAssemblyDirectory.value)
.withCacheOutput(true) // Cache pre-merged assembly file .class files to diff changes
}
Global Cache, One Location
lazy val sbtAssemblyDirectory = taskKey[File]("Directory to cache sbt assembly output.")
assembly / assemblyOption := {
val opt = (assembly / assemblyOption).value
val dir = sbtAssemblyDirectory.value
opt
.withAssemblyDirectory(dir)
.withCacheUnzip(true) // Keep dependency .jar files pre-extracted
.withCacheOutput(true) // Cache pre-merged assembly file .class files to diff changes
}
Global Unzip Cache, Local Output Cache
lazy val sbtAssemblyUnzipDirectory = taskKey[File]("Directory to cache sbt assembly library jars.")
assembly / assemblyOption := {
val opt = (assembly / assemblyOption).value
opt
.withAssemblyUnzipDirectory(sbtAssemblyUnzipDirectory.value)
.withCacheUnzip(true) // Keep dependency .jar files pre-extracted
.withCacheOutput(true) // Cache pre-merged assembly file .class files to diff changes
}
I see the motivation for changing assemblyUnzipDirectory
, so that makes sense.
assemblyOutputDirectory
on the other hand sounds like it's pretty much what we have already with withAssemblyDirectory
?
I see the motivation for changing
assemblyUnzipDirectory
, so that makes sense.
assemblyOutputDirectory
on the other hand sounds like it's pretty much what we have already withwithAssemblyDirectory
?
It probably is, I conceptually was thinking of withAssemblyDirectory
as just setting both withAssemblyUnzipDirectory
and withAssemblyOutputDirectory
to the same directory. But perhaps that can just be an internal/implementation nuance, and just add the additional one setter withAssemblyUnzipDirectory
for the public API 🤔
I'm also noodling on adding a separate task assemblyUnzip
that could be used to bootstrap/pre-warm a CI agent, without doing the full assembly
I'm also noodling on adding a separate task
assemblyUnzip
that could be used to bootstrap/pre-warm a CI agent, without doing the fullassembly
This might be close enough to the existing assemblyPackageDependency
or assemblyCacheUnzipDependency
@eed3si9n what do you think about this name? Was also thinking assemblyCacheUnzipDependencies