Feature: Storage persistence
opened this issue · comments
Would it be possible to use something like flatbuffers or cap'n'proto as backing for the tree which would allow it to be persisted into disk, hence it would be loadable from disk to memory without the need to rebuild it?
Its fairly easy to do this yourself; Bitcask persists the ART to disk for later retrieval on startup: https://github.com/prologic/bitcask/blob/master/internal/index/codec_index.go
that is exactly what I wanted to avoid. you are working with single items at a time instead of the whole blob representing the entire tree.
I must be missing something here because I don't understand how you can serialize a tree structure like this (let alone a simple Go struct) to disk without some kind of traversal of the structure and some kind of encoding. I may be wrong but at the end of the day you will end up encoding items/pairs of some kind with some way to identify them as a header + data when you read them back as raw bytes. Its unclear to me how flatbuffers could even help here?
I think, @nonpcnpc is talking about serializing a tree structure as a set of structures and then load it back without a tree.Insert method to avoid resizing and shrinking nodes.
@nonpcnpc I'm not sure if it is in the scope of the project but it is quite simple to implement. I don't have time right now but I will think about it.
I think, @nonpcnpc is talking about serializing a tree structure as a set of structures and then load it back without a
tree.Insertmethod to avoid resizing and shrinking nodes.
Kind of curious how you'd do this in the first place? I had a quick poke around Google and didn't turn up anything obvious. Maybe I'll go figure out how flatbuffers work. Is there some magic here you know of? :)