[BUG]: Value missing after power failure
SeienK opened this issue · comments
What version of Badger are you using?
github.com/dgraph-io/badger/v3 v3.2103.4
What version of Go are you using?
go version go1.18.10 linux/amd64
Have you tried reproducing the issue with the latest release?
None
What is the hardware spec (RAM, CPU, OS)?
Centos 7
What steps will reproduce the bug?
- Put some data in transaction
- Power failure occurs
- Try to read some data
Expected behavior and actual result.
I expect either all the data in the transaction are well preserved or the whole transaction fail.
The actual result is one of the key value pair in the transaction crashed.
Additional information
Badger options:
myOptions := badger.DefaultOptions(path).WithValueThreshold(256)
db, err := badger.Open(myOptions)
Restart after power failure, one of the key/value pair crashes(key exists and value can not be found).
Some info:
Other key/value pairs in the same transaction can all be found correctly.
The crashed value is not the last write in that transaction.
The crashed value is larger than 256 bytes and other values in the same transaction are less than 256 bytes(ValueThreshold = 256).
Here is the badger logs:
badger 2023/08/29 17:55:01 INFO: All 1 tables opened in 1ms
badger 2023/08/29 17:55:01 INFO: Discard stats nextEmptySlot: 0
badger 2023/08/29 17:55:01 INFO: Set nextTxnTs to 88411
badger 2023/08/29 17:55:01 INFO: Deleting empty file: D:\9\baddata\data\chain\000002.vlog
badger 2023/08/29 17:55:01 ERROR: Unable to read: Key: [0 3 0 2 0 0 117 5 169 109 251 135 105 196 241 91 226 21 139 80 254 38 84 155 99 127 183 103 193 12 181 13 3 96 157 135 225 202], Version : 88411, meta: 66, userMeta: 0 Error: ErrEOF: End of file
badger 2023/08/29 17:55:01 ERROR: Key: [0 3 0 2 0 0 117 5 169 109 251 135 105 196 241 91 226 21 139 80 254 38 84 155 99 127 183 103 193 12 181 13 3 96 157 135 225 202], Version : 88411, meta: 66, userMeta: 0 valuePointer: {Fid:1 Len:523 Offset:9400101}
Thanks for filling the bug @SeienK. Is it possible for you to share the badger data directory? You could send it over email as well.
The data does not have the WAL file (.mem file), I was hoping to look at the WAL file to figure out what happened. If you have the data right after the crash and before restarting (reopening) badger, that'd be helpful.
In any case, this looks related to #1954. We are planning to work on it. You should keep SyncWrite to true to avoid this issue altogether. If you decide to do that, keep in mind that it'd slow down badger a lot.
Thanks for sharing the data and for filling a detailed issue.
The data at that moment was not saved.
Thank you for your advice and I'd like to know if this issue exists in v1 and v2 as well, if SyncWrite is set to false.
This issue is not fixed yet, I do plan to work on it. Closing this issue as it seems that the root cause could be #1954. Feel free to reopen it if needed.