Benchmarks for Haskell serialization libraries.
For every library, we measure the serialization time, deserialization time and size of the encoded output for a set of test datasets.
Summary Results
The summary tables list, for every test, all libraries that are within 30% of the best result (with best library listed first).
Transfer Time
To merge all measures in a concrete use case, we estimate the total transfer time at different transmission speeds, that's to say the the time that it takes to serialize, move across a network (with no compression and no protocol overheads) and deserialize a given dataset.
| Dataset\Measure | transfer [10 MBits] | transfer [100 MBits] | transfer [1000 MBits] |
|---|---|---|---|
| BinTree Direction | flat | flat | flat,persist,store |
| BinTree Int | flat | flat | flat,store,persist |
| Cars | flat | flat | flat,store,persist |
| Iris | flat,store,packman,persist,cereal,serialise | store,flat,persist | store,flat |
| [Direction] | flat | flat,persist | flat,persist,store |
When the network speed is low, transfer times are dominated by the size of the encoded dataset. At higher speeds, by the (de)serialisation times.
Tests are executed on an Intel Xeon W3520 @ 2.67GHz. On a slower machine, de(serialization) times would matter more, on a faster machine size would be more significant.
(De)serialization Times and Size
| Dataset\Measure | deserialization | serialization | size |
|---|---|---|---|
| BinTree Direction | flat,persist,store | persist,store | flat |
| BinTree Int | store,persist,flat,cereal | flat,persist,store | flat |
| Cars | persist,flat,store | store | flat |
| Iris | store,persist,flat | store | packman,flat,cereal,persist,store,serialise |
| [Direction] | flat,persist,cereal,store | persist,store,flat | flat |
Full Results
Tested Libraries
Performance is not the only relevant property, depending on your needs you should also consider other features like laziness and compatibility.
| Package | Laziness | Compatibility |
|---|---|---|
| store-0.5.0.1 | Strict | Haskell-Same-Architecture |
| packman-0.5.0 | Lazy | Haskell-Same-Architecture |
| cereal-0.5.7.0 | Strict | Haskell |
| binary-0.8.5.1 | Lazy | Haskell |
| serialise-0.2.1.0 | Lazy | Multi-Language |
| flat-0.3.4 | Strict | Multi-Language |
| persist-0.1 | Strict | Haskell |
Compatibility Levels (lowest to highest):
- Haskell-Same-Architecture
- Compatible across Haskell systems sharing the same CPU and endianness
- Haskell
- Compatible across Haskell systems
- Multi-Language
- Compatible across different programming languages
Test Data
| Test | Description |
|---|---|
| BinTree Direction | Binary Tree of a simple enumeration data type |
| BinTree Int | Binary Tree of Ints |
| [Direction] | A List of a simple enumeration data type |
| Cars | A dataset of Cars descriptions (mostly lists, enumerations and Ints) |
| Iris | A dataset of Iris descriptions (mostly lists and floats) |
Shout if you would like other tests to be added!
Running the Benchmarks
Run the benchmarks with:
stack bench :all
If you get this error:
...<stdout>: commitBuffer: invalid argument (invalid character)
Try:
export LC_ALL=C.UTF-8
The executable will write the report.md, report.html and report.json files containing the full test results and will print out the two summary results tables in markdown format.
Tests will be compiled with GHC 8.4.4.