Support persisting search index in FoundationDB
ovaistariq opened this issue · comments
We plan to have FoundationDB as the source of truth for the indexed data. This improves the reliability story of TypeSense. As the first step towards that, we need to do the following work:
- FDB has a limit on the value size. Because of that, we need to break up the document_to_index into multiple KV pairs
- Since the write doesn't need to be transactional and to avoid running into the 10MB transaction size limit, we will break down a batch of documents_to_index before writing it to FDB
We will remove the RocksDB storage backend in TypeSense at a later stage.
For future reference
The rocksdb write code in typesense:
https://github.com/typesense/typesense/blob/main/src/collection.cpp#L357
General rocksdb usage:
https://github.com/search?q=repo%3Atypesense%2Ftypesense%20rocksdb&type=code