Major data loss
Liz-chan opened this issue · comments
I've been archiving a site for the past few days (the pages of this specific site are incredibly long and full of photos), and when I went to start Diskernet today, it seems that it wiped my archive and there's nothing there. This is a huge bug as I've spent hours archiving the site using this program, and now it's all gone. The data still seems to be inside the library folder, but the index.json file is empty and I don't know of any way to recover it unless I had a backup for it (and I've only been using it for a few days and didn't think it'd just wipe it, so I didn't think to back it up). Also, even though the index.json file is like brand new, now the program is crashing when I try to switch from save to serve mode. I think it'd be a good idea to look into this to prevent any data loss for other people in the future.
Here's the error when it crashes, for reference:
Crash Error
<--- Last few GCs --->
[1040:000001E179DDFED0] 78177 ms: Mark-sweep (reduce) 4091.5 (4136.9) -> 4080.3 (4125.1) MB, 3507.9 / 0.0 ms (+ 24.9 ms in 6 steps since start of marking, biggest step 6.2 ms, walltime since start of marking 3725 ms) (average mu = 0.333, current mu = [1040:000001E179DDFED0] 82573 ms: Mark-sweep (reduce) 4104.3 (4149.1) -> 4092.3 (4136.3) MB, 4204.8 / 0.0 ms (average mu = 0.201, current mu = 0.043) allocation failure scavenge might not succeed
<--- JS stacktrace --->
FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory
1: 00007FF79C36AFFF v8::internal::CodeObjectRegistry::~CodeObjectRegistry+112463
2: 00007FF79C2FA0B6 v8::internal::wasm::WasmCode::safepoint_table_offset+65510
3: 00007FF79C2FAF6D node::OnFatalError+301
4: 00007FF79CC2F03E v8::Isolate::ReportExternalAllocationLimitReached+94
5: 00007FF79CC1943D v8::SharedArrayBuffer::Externalize+781
6: 00007FF79CABCB1C v8::internal::Heap::EphemeronKeyWriteBarrierFromCode+1468
7: 00007FF79CAB9C54 v8::internal::Heap::CollectGarbage+4244
8: 00007FF79CAB75D0 v8::internal::Heap::AllocateExternalBackingStore+2000
9: 00007FF79CAD5240 v8::internal::FreeListManyCached::Reset+1408
10: 00007FF79CAD58F5 v8::internal::Factory::AllocateRaw+37
11: 00007FF79CAE75EE v8::internal::FactoryBasev8::internal::Factory::AllocateRawArray+46
12: 00007FF79CAEA24A v8::internal::FactoryBasev8::internal::Factory::NewFixedArrayWithFiller+74
13: 00007FF79CAEA4A3 v8::internal::FactoryBasev8::internal::Factory::NewFixedArrayWithMap+35
14: 00007FF79C8F1136 v8::internal::HashTablev8::internal::NameDictionary,v8::internal::NameDictionaryShape::EnsureCapacityv8::internal::Isolate+246
15: 00007FF79C8EEEDA v8::internal::Dictionaryv8::internal::NameDictionary,v8::internal::NameDictionaryShape::Addv8::internal::Isolate+58
16: 00007FF79C8F7166 v8::internal::BaseNameDictionaryv8::internal::NameDictionary,v8::internal::NameDictionaryShape::Add+118
17: 00007FF79C923AB6 v8::internal::LookupIterator::ApplyTransitionToDataProperty+982
18: 00007FF79C8F8685 v8::internal::StringSet::Add+1941
19: 00007FF79C93AF7C v8::internal::JSObject::DefineAccessor+1644
20: 00007FF79C9F2B0C v8::base::TimeDelta::operator!=+26092
21: 00007FF79C9F5F4F v8::base::TimeDelta::operator!=+39471
22: 00007FF79C9F3C4A v8::base::TimeDelta::operator!=+30506
23: 00007FF79CBC8CBE v8::internal::Builtins::code_handle+38542
24: 00007FF79CBC8D90 v8::internal::Builtins::code_handle+38752
25: 00007FF79CCBCB31 v8::internal::SetupIsolateDelegate::SetupHeap+494673
26: 00007FF79CC4EFDE v8::internal::SetupIsolateDelegate::SetupHeap+45310
27: 00007FF79CC4EFDE v8::internal::SetupIsolateDelegate::SetupHeap+45310
28: 00007FF79CCCFC43 v8::internal::SetupIsolateDelegate::SetupHeap+572771
29: 00007FF79CC4EFDE v8::internal::SetupIsolateDelegate::SetupHeap+45310
30: 00007FF79CC4EFDE v8::internal::SetupIsolateDelegate::SetupHeap+45310
31: 00007FF79CC4EFDE v8::internal::SetupIsolateDelegate::SetupHeap+45310
32: 00007FF79CC4EFDE v8::internal::SetupIsolateDelegate::SetupHeap+45310
33: 000001E1000CCF4B
Poor you. This must totally suck. I feel a little bit sad for you that you had this happen you, and lost all you work. It's a big loss. You must feel so upset. I totally agree this bug is major. Many people have experienced it.
I've investigated for a long time but not sure of the cause, but I have a feeling that a major version I've been working on for some time does not have this bug anymore, so, while these words won't help in your case, I think there might be hope on the horizon coming soon.
The reason I feel this way is that, in this major version I've been working on, I have not personally experienced this bug for a while. Maybe you can use that in future.
You won't be helped by this, but for future I recommend simply setting up a git repository in the root folder of your archive and setting a regular Cron job or other periodic task to backup in other words, git add and git commit, any changes to that repository of your archive so that you'll have a complete history of your archive. That way in future if you do lose this through a bug or some other means then you'll be able to recover.
A fix that might help for you is because all of the content is still saved it may be actually possible to rebuild the index.json file. I remember there was a customer in the past who experienced the same bug and he told me that he rebuilt the index.json file and I was very impressed to hear this. You may be interested to search through the issues and discover his statement of the same. You may be able to contact this person and find out how he was able to do that.
You responded with an unhappy confused face 😕 ?
I'm sorry if you're unhappy that I spent time replying you for free but you expected me to do more. I guess that's your problem.