storage: bootstrap manifest replay
yyin-dev opened this issue · comments
When bootstrapping secondary storage, the manifest log is replayed to restore the internal state of VersionManager. This is done via the commit_changes call.
risinglight/src/storage/secondary/storage.rs
Line 164 in 365f285
The problem with this is that any effective AddRowset/AddDV are written to the manifest file again at bootstrap. This means that, if the database keeps crashing and coming back, the manifest will keep growing (due to the bootstrap), even if no write happens.
Under the textbook WAL setting, redo&undo phase append more logs (and those logs provide useful information). But I don't think those logs are necessary/useful for RisingLight, because it's using a much simpler logging scheme.
Yes! My idea is to periodically take a snapshot of the database and replace some of the manifest entries with snapshot.