Record Structure and Stream Batching
khkiley opened this issue · comments
Hello @kriszyp,
I was reading through #36 , and had a question about record structures and streams.
The intent is to stream records from a database, pipe them through msgpackr, buffer them, and once the buffer reaches a certain size store the batch in S3, then reset the buffer and start over.
I want to 1. ensure the batch that is created on the fly is correctly formed and can be decompressed downstream, and 2 I can store the record structure inside each batch.
Any guidelines or suggestions on how to safely do this ?
Thank you for your work on this library!
Yes, this should work. I believe you will want to create a new PackrStream instance for each entry (when you reset and start over), to ensure that the structures get written in each new buffer/entry. And when read the entry, you will probably want to use unpackMultiple to read all the data items so the references to previous structures within the entry work properly.
Let me know if you run into any issues with that.
Excellent, thank you for the quick response. I will let you know how it turns out.
Kurt