Supporting out-of-order writing
jptmoore opened this issue · comments
The current backend log-based storage is suited to in-order time-series writes. Do we need to support out-of-order time-series writes?
For complex values like tweets we have the requirement to be able to redact them, which is a limited kind of out of order (over-)write, but might be implemented some other way.
At the moment I could implement the drivers I am doing without writing out of order.
Although I guess it would happen if someone changed the window of data being cached from an external source (e.g. twitter) to make it further in the past. Although again a work-around might be to drop all the data associated with that datasource and rebuild it (could be done by driver if there is an API method for that)
Uploading google take out data would require the user to upload the files in the correct order if we do not support out-of-order writing
Also for drivers that import large amounts of data (e.g email driver) may wish to do this in reverse chronological order which requires out-of-order write support.