Easy to hack and dead simple RecordIO-like data storing. Inspired by MXNet's RecordIO and Protobuf.
- Binary-based data format with index file
- Easy custom serialization schemes
- Common serialization schemes (more TBA)
- Documentation
- Append, update, delete data sample
- Pack/unpack form
- Schema serialization
See the overview section in the documentations.
Also, check out this notebook for a quick example, this notebook has not been updated in a while though.
- Open an empty file, this is the data file.
- Take whatever data you got, serialize it in a way (for example, write to PNG buffer)
- Write the serialized data to the said file buffer, store the offset and size of the data
- Repeat until all data are written
- Write all the offsets and sizes to another file (index file)
- Load the data file and the index file.
- When given an index, use the index file to get the offset and size of the data.
- Seek the file pointer to the offset, read the exact size and deserialize the data.