init.prune=full stuck at traversing trie database

Question

init.prune=full stuck at traversing trie database

icculp opened this issue 4 months ago · comments

Describe the bug
Running an arbitrum-nova node with --init.prune=full results get stuck at traversing trie database when the eta gets to low seconds, then just hangs with no output. Viewing htop and docker stats, the container is still utilizing resources, but no logs indicate anything is happening or continuing with prune. I tried it initially on a full node which synced from scratch long ago, and then again with a node which synced from pruned snapshot to recent tip, same issue, has been hung for several days at
traversing trie database key=ff902b..6f73af elapsed=1h54m39.89785619s eta=11.778250562s

To Reproduce
Steps to reproduce the behavior:
Running version nitro-node:v2.3.3-6a1c1a7
add --init.prune-full to startup command
Wait until it's nearly finished traversing trie database
get stuck

Expected behavior
Pruning continues and then becomes available for syncing to head.
Screenshots

Ian Culp · Answer 1 · Tue Apr 16 2024 05:29:19 GMT+0800 (China Standard Time)

LOL! Just a few minutes after I post this issue it finally moves forward with building bloom filter and iterating state snapshot. If you look at the timestamp it was roughly 42 hours of no output before finally moving forward...

INFO [04-14|03:39:03.562] traversing trie database                 key=f5f228..29bb1c elapsed=1h50m9.836712822s  eta=4m30.221696138s
INFO [04-14|03:39:33.568] traversing trie database                 key=f6f612..755d69 elapsed=1h50m39.842709109s eta=4m3.026407819s
INFO [04-14|03:40:03.569] traversing trie database                 key=f80809..8a266c elapsed=1h51m9.844159196s  eta=3m34.288491812s
INFO [04-14|03:40:33.572] traversing trie database                 key=f90db6..041d51 elapsed=1h51m39.847104443s eta=3m6.944735301s
INFO [04-14|03:41:03.573] traversing trie database                 key=fa4433..c89e51 elapsed=1h52m9.848273142s  eta=2m34.201540362s
INFO [04-14|03:41:33.575] traversing trie database                 key=fb7563..512d24 elapsed=1h52m39.849335595s eta=2m2.126918869s
INFO [04-14|03:42:03.575] traversing trie database                 key=fc82e7..c8d1a3 elapsed=1h53m9.849513856s  eta=1m33.904076928s
INFO [04-14|03:42:33.587] traversing trie database                 key=fd9eed..cfd26c elapsed=1h53m39.861496849s eta=1m4.075070447s
INFO [04-14|03:43:03.622] traversing trie database                 key=fe9d56..5d33c8 elapsed=1h54m9.897232593s  eta=37.306697519s
INFO [04-14|03:43:33.623] traversing trie database                 key=ff902b..6f73af elapsed=1h54m39.89785619s  eta=11.778250562s
INFO [04-15|21:23:41.643] Building bloom filter for pruning        root=a08915..1e43d5
INFO [04-15|21:23:41.654] Iterating state snapshot                 accounts=0 slots=0 elapsed="138.392µs"
INFO [04-15|21:23:49.657] Iterating state snapshot                 accounts=287,012 slots=4,712,541 elapsed=8.002s             eta=57.418s
INFO [04-15|21:23:57.658] Iterating state snapshot                 accounts=602,115 slots=11,896,014 elapsed=16.003s            eta=5m41.203s
INFO [04-15|21:24:05.658] Iterating state snapshot                 accounts=864,543 slots=18,227,080 elapsed=24.003s            eta=12m34.962s
INFO [04-15|21:24:13.658] Iterating state snapshot                 accounts=1,016,524 slots=24,471,753 elapsed=32.004s            eta=16m51.752s
INFO [04-15|21:24:21.658] Iterating state snapshot                 accounts=1,224,551 slots=31,147,391 elapsed=40.004s            eta=18m0.639s
INFO [04-15|21:24:29.658] Iterating state snapshot                 accounts=1,376,086 slots=38,270,575 elapsed=48.004s            eta=18m21.009s
INFO [04-15|21:24:37.658] Iterating state snapshot                 accounts=1,561,664 slots=47,024,947 elapsed=56.004s            eta=19m52.995s
INFO [04-15|21:24:45.659] Iterating state snapshot                 accounts=1,718,920 slots=55,405,834 elapsed=1m4.004s           eta=20m0.571s
INFO [04-15|21:24:53.659] Iterating state snapshot                 accounts=1,823,150 slots=63,631,466 elapsed=1m12.004s          eta=1h55m27.426s
INFO [04-15|21:25:01.659] Iterating state snapshot                 accounts=1,973,258 slots=71,819,089 elapsed=1m20.004s          eta=3h49m49.158s
INFO [04-15|21:25:09.659] Iterating state snapshot                 accounts=2,188,920 slots=80,153,309 elapsed=1m28.004s          eta=4h44m14.131s
INFO [04-15|21:25:17.661] Iterating state snapshot                 accounts=2,346,862 slots=88,436,396 elapsed=1m36.006s          eta=4h36m7.947s
INFO [04-15|21:25:25.661] Iterating state snapshot                 accounts=2,346,862 slots=96,953,273 elapsed=1m44.006s          eta=4h0m43.856s
INFO [04-15|21:25:33.661] Iterating state snapshot                 accounts=2,346,862 slots=105,073,007 elapsed=1m52.007s          eta=3h55m46.537s
INFO [04-15|21:25:41.661] Iterating state snapshot                 accounts=2,346,862 slots=113,540,414 elapsed=2m0.007s           eta=3h25m9.288s
INFO [04-15|21:25:49.661] Iterating state snapshot                 accounts=2,346,862 slots=122,125,045 elapsed=2m8.007s           eta=2h59m56.295s
INFO [04-15|21:25:57.661] Iterating state snapshot                 accounts=2,346,862 slots=130,460,158 elapsed=2m16.007s          eta=2h57m10.155s
INFO [04-15|21:26:05.662] Iterating state snapshot                 accounts=2,346,862 slots=137,816,609 elapsed=2m24.007s          eta=2h43m12.293s
INFO [04-15|21:26:13.662] Iterating state snapshot                 accounts=2,346,862 slots=146,413,816 elapsed=2m32.007s          eta=2h28m52.358s
INFO [04-15|21:26:21.662] Iterating state snapshot                 accounts=2,346,862 slots=153,250,666 elapsed=2m40.007s          eta=2h23m27.875s