1. To start a new data manager, need parameters
data_dir: the dir to store the data and metadata
format_version: the version of index and data file
size_per_index: size in byte per index
index_num_per_file: number of index per index file
max_size_data_file: maximum size per data file
data_blk_size: block size for data file
data_dir +
+ metadata [bin file] #generated by data manager
+ transid [dir] #transid for every backend
+ 0...n [dir]
+ index.0
+ data.0
metadata structrue:
[format_version, size_per_index, index_num_per_file, index_file_num_per_dir,max_size_data_file, data_blk_size, need_compress],
size: 2+4+4+4+8+4+2 = 28 bytes
index structrue:
[transid, type_no, dir_no, file_no, data_blk_no, data_len, checksum],
size: 8+4+4+4+8+4+4 = 36
data structure:
[transid, checksum, raw_datapack]
format_version: 16-bits unsigned integer (S)
size_per_index: 32-bits unsigned integer (L)
index_num_per_file: 32-bits unsigned integer (L)
index_file_num_per_dir: 32-bits unsigned integer (L)
max_size_data_file: 64-bits unsigned integer (Q)
data_blk_size: 32-bits unsigned integer (L)
need_compress: 16-bits unsigned integer (S)
transid: 64-bits unsigned integer (Q)
type_no: 32-bits unsigned integer (L)
dir_no: 32-bits unsigned integer (L)
file_no: 32-bits unsigned integer (L)
data_blk_no: 64-bits unsigned integer (Q)
data_len: 32-bits unsigned integer(L)
check_sum: 32-bits unsigned integer(L) CRC check sum
NOTICE:
1. transid start from 1, 0 stands for nil