Infinite storage
purplefox opened this issue · comments
We should allow for older data (cold-set) to be offloaded to S3 buckets (when running on AWS), this would allow Prana dataset to scale to virtually unlimited sizes. Blocks of data would be read and asynchronously pushed while keeping a hot-set local. This would also allow for smaller local storage to be used, reducing costs.
Both Kafka (Confluent) and Red Panda do something similar.