performance issues

[pavlexander]

I would like to use this library to save/append the candlestick trading data (OHLCV) into files.

After looking at advertised performance I have tried using this library, but it's just not as fast as simply saving the data to a binary file.. I would like to know if I am misusing the library or it's simply not meant for the given use-case? The test data set consists of 3_020_871 records.

the test code looks like this

using MemoryStream memoryStream = new MemoryStream();
foreach (var data in dataStructLong)
{
    var bytes = BinaryConverter.Serialize(data);
    await memoryStream.WriteAsync(bytes);
}

File.WriteAllBytes("customHyper.hyper", memoryStream.ToArray());

for performance comparison, I am also using the teafiles library (which basically is a wrapper for brute-force binary serialization):

    using (var tf = TeaFile<CandlestickLongStruct>.Create("teaFile.tea"))
    {
        foreach (var item in dataStructLong)
        {
            tf.Write(item);
        }
    }

the results are:

Tea: 528 ms, File size: 184.37969970703125 mb
BinaryPack: 886 ms, File size: 184.37933349609375 mb

it does seems like the BinaryPack produces almost exact size output file, but the performance is worse.

The aim of this post of course, not to compare this lib to others. I genuinely want to replace the teafiles library and looking for a better solution. I would appreciate to hear a feedback on the performance issue..

for the sake of completeness, here's the serialized data type

public readonly struct CandlestickLongStruct
{
    public long Id { get; init; }
    public Time OpenTime { get; init; }
    public long Open { get; init; }
    public long High { get; init; }
    public long Low { get; init; }
    public long Close { get; init; }
    public long Volume { get; init; }
    public Time CloseTime { get; init; }
}

.net 7, BinaryPack 1.0.3, Rubble.TeaFiles.Net 2.0.0

additionally I would like to know how would you solve the problem of appending the data to an existing file and also getting/upating a total number of records in file.