study: serialization
oldratlee opened this issue · comments
李鼎 commented
🍺 serialization framework
Kryo
- repo: https://github.com/EsotericSoftware/kryo
- EsotericSoftware/kryo: Java binary serialization and cloning: fast, efficient, automatic.
- Kryo is a fast and efficient binary object graph serialization framework for Java.
- Additional serializers(for some jdk types and some external libs like e.g. joda time) are available in a separate project on github, kryo-serializers.
JBoss Marshalling
- repo: https://github.com/jboss-remoting/jboss-marshalling
- an alternative serialization API that
- fixes many of the problems found in the JDK serialization API while remaining fully compatible with
java.io.Serializable
and its relatives, - and adds several new tunable parameters and additional features, all of which are pluggable via factory configuration (externalizers, class/instance lookup tables, class resolution, and object replacement, to name a few).
- fixes many of the problems found in the JDK serialization API while remaining fully compatible with
- this framework was inspired by the need for certain features unavailable with the standard
Object*Stream
classes- Pluggable class resolvers, making it easy to customize classloader policy, by implementing a small interface (rather than having to subclass the
Object*Stream
classes) - Pluggable object replacement (also without subclassing)
- Pluggable predefined class tables, which can dramatically decrease stream size and serialization time for stream types which frequently use a common set of classes
- Pluggable predefined instance tables, which make it easy to handle remote references
- Pluggable externalizers which may be used to serialize classes which are not Serializable, or for which an alternate strategy is needed
- Customizable stream headers
- Each marshaller instance is highly configurable and tunable to maximize performance based on expected usage patterns
- A generalized API which can support many different protocol implementations, including protocols which do not necessarily provide all the above features
- Inexpensive instance creation, beneficial to applications where many short-lived streams are used
- Support for separate class and instance caches, if the protocol permits; useful for sending multiple messages or requests with a single stream, with separate object graphs but retaining the class cache
- Pluggable class resolvers, making it easy to customize classloader policy, by implementing a small interface (rather than having to subclass the
- home page: http://jbossmarshalling.jboss.org/
- maintenance is not active...
- although personally I like the product of jboss and it's high quality 😩
🍺 serialization specification
Protocol Buffers
- repo: https://github.com/protocolbuffers/protobuf
- most widely used, especially when need support multi-languages.
- Protocol Buffers (a.k.a., protobuf) are Google's language-neutral, platform-neutral, extensible mechanism for serializing structured data.
- home page: https://developers.google.com/protocol-buffers/
protostuff
improvement/enhancement of java; independent implementation of java.
- Java serialization library, proto compiler, code generator
- repo: https://github.com/protostuff/protostuff
- official docs: https://protostuff.github.io/docs/
李鼎 commented
Kryo
分析
- Kryo为什么比Hessian快 https://x-rip.iteye.com/blog/1555344
有Kryo生成的 二进制字节 分析 - Kryo 使用指南 https://www.cnblogs.com/hntyzgn/p/7122709.html
提到使用注意的问题,如- 缺省的设置 不支持 增加或删除 Bean 中的字段,使用的是
FieldSerializer
- 注册机制生成ClassId
- Kryo 对象不是线程安全的
- 缺省的设置 不支持 增加或删除 Bean 中的字段,使用的是
- Kryo 序列化说明 https://my.oschina.net/u/725800/blog/823467
- Strategy for registering classes with kryo https://stackoverflow.com/questions/7780707
- Kryo Serialization Type Detection https://stackoverflow.com/questions/22079054
- Kryo Class Registration https://stackoverflow.com/questions/31576169
- Since Kryo use Minlog library for logging, you can enable the trace level log by adding
com.esotericsoftware.minlog.Log.TRACE();
to the source code.
Enabling TRACE level log will print the details about all the classes which are registered with Kryo.
- Since Kryo use Minlog library for logging, you can enable the trace level log by adding
- Kryo read and write generically https://stackoverflow.com/questions/30184019
- Dealing with an incompatible version change of a serialization framework
https://stackoverflow.com/questions/16083263 - Spark 2.0.2, double[], 使用Kyro序列化加速,和手动注册类名
https://facaiy.com/misc/2017/01/21/spark-kyro.html - Kryo官方文档翻译
注意的问题
- 字段 + 循环引用下有的一个问题:
CompatibleFieldSerializer fails with IndexOutOfBoundsException on field removal
EsotericSoftware/kryo#286 (comment) - 使用kryo做序列化会遇到的几个坑 https://blog.csdn.net/paoma_1008/article/details/79827803
com.esotericsoftware.kryo.KryoException: Class cannot be created (missing no-arg constructor): java.util.Collections$SynchronizedRandomAccessList
com.esotericsoftware.kryo.KryoException: Encountered unregistered class ID: XXXX