Scala Kryo Serialization provides a convenient way of using Kryo with Scala and is the base for Pekko Kryo Serialization providing the same functionality to pekko.
=====================================================================
This library provides custom Kryo-based serializers for Scala. See Pekko Kryo Serialization for serialization in Pekko.
It can also be used for a general purpose and very efficient Kryo-based serialization of such Scala types like Option, Tuple, Enumeration and most of Scala's collection types.
- It is more efficient than Java serialization - both in size and speed
- Does not require any additional build steps like compiling proto files, when using protobuf serialization
- Almost any Scala and Java class can be serialized using it without any additional configuration or code changes
- Efficient serialization of such Scala types like Option, Tuple, Enumeration, most of Scala's collection types
- Supports transparent AES encryption and different modes of compression
- Apache 2.0 license
Note that this serializer is mainly intended to be used for remoting and not for (long term) persisted data. The underlying kryo serializer does not guarantee compatibility between major versions.
To use this serializer, you need to do two things:
-
Include a dependency on this library into your project:
libraryDependencies += "io.altoo" %% "scala-kryo-serialization" % "? not yet released"
-
Register and configure the serializer in your Typesafe Config configuration file, e.g.
application.conf
.
We provide several versions of the library:
Version | Kryo Compatibility | Available Scala Versions | Tested with |
---|---|---|---|
v1.2.x | Kryo-5.6 | 2.12,2.13,3 | JDK: OpenJdk11,OpenJdk17,OpenJdk21 Scala: 2.12.18,2.13.11,3.3.1 |
v1.1.x | Kryo-5.5 | 2.12,2.13,3 | JDK: OpenJdk11,OpenJdk17,OpenJdk21 Scala: 2.12.18,2.13.11,3.3.1 |
v1.0.x | Kryo-5.4 | 2.12,2.13,3 | JDK: OpenJdk11,OpenJdk17 Scala: 2.12.18,2.13.11,3.3.1 |
Note that we use semantic versioning - see semver.org.
To use the latest stable release of scala-kryo-serialization in sbt projects you just need to add this dependency:
libraryDependencies += "io.altoo" %% "scala-kryo-serialization" % "1.2.0"
To use the official release of scala-kryo-serialization in Maven projects, please use the following snippet in your pom.xml
<repository>
<snapshots>
<enabled>false</enabled>
</snapshots>
<id>central</id>
<name>Maven Central Repository</name>
<url>https://repo1.maven.org/maven2</url>
</repository>
<dependency>
<groupId>io.altoo</groupId>
<artifactId>scala-kryo-serialization_2.13</artifactId>
<version>1.2.0</version>
</dependency>
For snapshots see Snapshots.md
The following options are available for configuring this serializer:
-
You can add a new
scala-kryo-serialization
section to the configuration to customize the serializer. Consult the supplied reference.conf for a detailed explanation of all the options available. -
Then you can create an instance of
ScalaKryoSerializer
and use it to serialize data. The serializer implements pooling to perform serialization across multiple threads.
One of the easiest ways to understand which classes you need to register in those sections is to leave both sections first empty and then set
implicit-registration-logging = true
As a result, you'll eventually see log messages about implicit registration of some classes. By default, they will receive some random default ids. Once you see the names of implicitly registered classes, you can copy them into your mappings or classes sections and assign an id of your choice to each of those classes.
You may need to repeat the process several times until you see no further log messages about implicitly registered classes.
Another useful trick is to provide your own custom initializer for Kryo (see below) and inside it, you registerclasses of a few objects that are typically used by your application, for example:
kryo.register(myObj1.getClass)
kryo.register(myObj2.getClass)
Obviously, you can also explicitly assign IDs to your classes in the initializer, if you wish:
kryo.register(myObj3.getClass, 123)
If you use this library as an alternative serialization method when sending messages between actors, it is extremely important that the order of class registration and the assigned class IDs are the same for senders and for receivers!
To further customize kryo you can extend the io.altoo.serialization.kryo.scala.DefaultKryoInitializer
and
configure the FQCN under scala-kryo-serialization.kryo-initializer
.
In preInit
a different default serializer can be configured
as it will be picked up by serializers added afterward.
By default, the com.esotericsoftware.kryo.serializers.FieldSerializer
will be used.
The available options are:
-
com.esotericsoftware.kryo.serializers.FieldSerializer
Serializes objects using direct field assignment. FieldSerializer is generic and can serialize most classes without any configuration. It is efficient and writes only the field data, without any extra information. It does not support adding, removing, or changing the type of fields without invalidating previously serialized bytes. This can be acceptable in many situations, such as when sending data over a network, but may not be a good choice for long term data storage because the Java classes cannot evolve. -
com.esotericsoftware.kryo.serializers.CompatibleFieldSerializer
Serializes objects using direct field assignment, providing both forward and backward compatibility. This means fields can be added or removed without invalidating previously serialized bytes. Changing the type of a field is not supported. The forward and backward compatibility comes at a cost: the first time the class is encountered in the serialized bytes, a simple schema is written containing the field name strings. -
com.esotericsoftware.kryo.serializers.VersionFieldSerializer
Serializes objects using direct field assignment, with versioning backward compatibility. Allows fields to have a @Since(int) annotation to indicate the version they were added. For a particular field, the value in @Since should never change once created. This is less flexible than FieldSerializer, which can handle most classes without needing annotations, but it provides backward compatibility. This means that new fields can be added, but removing, renaming or changing the type of any field will invalidate previous serialized bytes. VersionFieldSerializer has very little overhead (a single additional varint) compared to FieldSerializer. Forward compatibility is not supported. -
com.esotericsoftware.kryo.serializers.TaggedFieldSerializer
Serializes objects using direct field assignment for fields that have a @Tag(int) annotation. This provides backward compatibility so new fields can be added. TaggedFieldSerializer has two advantages over VersionFieldSerializer:- fields can be renamed
- fields marked with the @Deprecated annotation will be ignored when reading old bytes and won't be written to new bytes.
Deprecation effectively removes the field from serialization, though the field and @Tag annotation must remain in the class. The downside is that it has a small amount of additional overhead compared to VersionFieldSerializer (additional per field variant). Forward compatibility is not supported.
Create a custom initializer
class XyzKryoInitializer extends DefaultKryoInitializer {
def preInit(kryo: ScalaKryo): Unit = {
kryo.setDefaultSerializer(classOf[com.esotericsoftware.kryo.serializers.TaggedFieldSerializer[_]])
}
}
And register the custom initializer in your application.conf
by overriding
scala-kryo-serialization.kryo-initializer = "com.example.XyzKryoInitializer"
To configure the field serializer a serializer factory can be used as described here: https://github.com/EsotericSoftware/kryo#serializer-factories
Using the DefaultKeyProvider
an encryption key can statically be set by defining encryption.aes.password
and encryption.aes.salt
.
Refere to the reference.conf for an example configuration.
Sometimes you need to pass a custom aes key, depending on the context you are in, instead of having a static key. For example, you might have the key in a data store, or provided by some other application. In such instances, you might want to provide the key dynamically to kryo serializer.
You can override the
encryption.aes.key-provider = "CustomKeyProviderFQCN"
Where CustomKeyProviderFQCN
is a fully qualified class name of your custom aes key
provider class. The key provider must extend the DefaultKeyProvider
and can override the aesKey
method.
An example of such a custom aes-key supplier class could be something like this:
class CustomKeyProvider extends DefaultKeyProvider {
override def aesKey(config: Config): String = "ThisIsASecretKey"
}
The encryption transformer (selected for aes
in post serialization transformations) only
supports GCM modes (currently recommended default mode is AES/GCM/NoPadding
).
Important: The old encryption transformer only supported CBC modes without manual authentication which is
deemed problematic. It is currently available for backwards compatibility by specifying aesLegacy
in
post serialization transformations instead of aes
. Its usage is deprecated and will be removed in future versions.
If you are using id-strategy="explicit"
, you may find that some of the standard Scala types are a bit hard to register properly.
This is because these types are exposed in the API as simple traits or abstract classes, but they are actually implemented as many
specialized subclasses that are used as necessary. Examples include:
- scala.collection.immutable.Map
- scala.collection.immutable.Set
The problem is that Kryo thinks in terms of the exact class being serialized, but you are
rarely working with the actual implementation class -- the application code only cares about
the more abstract trait. The implementation class often isn't obvious, and is sometimes
private to the library it comes from. This isn't an issue for idstrategies that add registrations
when needed, or which use the class name, but in explicit
you must register every class to be
serialized, and that may turn out to be more than you expect.
For cases like these, you can use the SubclassResolver
. This is a variant of the standard
Kryo ClassResolver, which is able to deal with subclasses of the registered types. You turn it
on by setting
resolve-subclasses = true
With that turned on, unregistered subclasses of a registered supertype are serialized as that
supertype. So for example, if you have registered immutable.Set
, and the object being serialized
is actually an immutable.Set.Set3
(the subclass used for Sets of 3 elements), it will serialize and
deserialize that as an immutable.Set
.
If you register immutable.Map
, you should use the ScalaImmutableAbstractMapSerializer
with it.
If you register immutable.Set
, you should use the ScalaImmutableAbstractSetSerializer
. These
serializers are specifically designed to work with those traits.
The SubclassResolver
approach should only be used in cases where the implementation types are completely
opaque, chosen by the implementation library, and not used explicitly in application code. If you have
subclasses that have their own distinct semantics, such as immutable.ListMap
, you should register
those separately. You can register both a higher-level class like immutable.Map
and a subclass
like immutable.ListMap
-- the resolver will choose the more-specific one when appropriate.
SubclassResolver
should be used with care -- even when it is turned on, you should define and
register most of your classes explicitly, as usual. But it is a helpful way to tame the complexity
of some class hierarchies, when that complexity can be treated as an implementation detail and all
the subclasses can be serialized and deserialized identically.
There may be the need to use different configurations for different use cases.
To support this the KryoSerializer
can be extended to use a different configuration path.
Define a custom configuration:
scala-kryo-serialization-xyz = ${scala-kryo-serialization} {
# configuration overrides like...
# id-strategy = "explicit"
}
Create new serializer subclass overriding the config key to the matching config section.
package xyz
class XyzKryoSerializer(config: Config, classLoader: ClassLoader) extends ScalaKryoSerializer(config, classLoader) {
override def configKey: String = "scala-kryo-serialization-xyz"
}
Serialization of Java and Scala 3 enums is done by name (and not by index) to avoid having reordering of enum values breaking serialization.
Kryo needs modules to be opened for reflection when serializing basic JDK classes. Those options have to be passed to the JVM, for example in sbt:
javaOptions ++= Seq("--add-opens", "java.base/java.util=ALL-UNNAMED", "--add-opens", "java.base/java.util.concurrent=ALL-UNNAMED", "--add-opens", "java.base/java.lang=ALL-UNNAMED", "--add-opens", "java.base/java.lang.invoke=ALL-UNNAMED", "--add-opens", "java.base/java.math=ALL-UNNAMED")
To use unsafe transformations, the following access must be granted:
javaOptions ++= Seq("--add-opens", "java.base/java.nio=ALL-UNNAMED", "--add-opens", "java.base/sun.nio.ch=ALL-UNNAMED")
If you wish to build the library on your own, you need to check out the project from GitHub and do
sbt compile publishM2