Erigon is stuck in a loop at "2/15 headers" with a repeated error

Question

Erigon is stuck in a loop at "2/15 headers" with a repeated error

subtletech opened this issue 7 months ago · comments

subtletech commented 7 months ago

System information

Erigon version: 1.1.9-dev-f29b6c9c

OS & Version: Ubuntu 22.04.3 LTS

Commit hash: f29b6c9

Erigon Command (with flags/config):

./build/bin/erigon --datadir ./data --networkid=56 --chain=bsc --p2p.protocol=66 --nat=any --db.pagesize=16k --http --ws --http.api=web3,net,eth,debug,trace,txpool --http.addr=127.0.0.1 --snapshots --torrent.download.rate=1000mb --maxpeers=500 --prune hrtc --bootnodes=enode://0637d1e62026e0c8685b1db0ca1c767c78c95c3fab64abc468d1a64b12ca4b530b46b8f80c915aec96f74f7ffc5999e8ad6d1484476f420f0c10e3d42361914b@52.199.214.252:30311,enode://330d768f6de90e7825f0ea6fe59611ce9d50712e73547306846a9304663f9912bf1611037f7f90f21606242ded7fb476c7285cb7cd792836b8c0c5ef0365855c@18.181.52.189:30311,enode://df1e8eb59e42cad3c4551b2a53e31a7e55a2fdde1287babd1e94b0836550b489ba16c40932e4dacb16cba346bd442c432265a299c4aca63ee7bb0f832b9f45eb@52.51.80.128:30311,enode://0bd566a7fd136ecd19414a601bfdc530d5de161e3014033951dd603e72b1a8959eb5b70b06c87a5a75cbf45e4055c387d2a842bd6b1bd8b5041b3a61bab615cf@34.242.33.165:30311,enode://ecd664250ca19b1074dcfbfb48576a487cc18d052064222a363adacd2650f8e08fb3db9de7a7aecb48afa410eaeb3285e92e516ead01fb62598553aed91ee15e@3.209.122.123:30311,enode://665cf77ca26a8421cfe61a52ac312958308d4912e78ce8e0f61d6902e4494d4cc38f9b0dd1b23a427a7a5734e27e5d9729231426b06bb9c73b56a142f83f6b68@52.72.123.113:30311

Expected behaviour

Normal operation.

Actual behaviour

I'm trying to sync a node with "hrtc" pruning. My erigon is stuck in a loop at stage "2/15 headers". It is going with "inserting headers" up to a block around 32949184 (always a bit different number), then prints an error and restarts from block 31300000. It has happened several times during the last day and I'm guessing it will continue like that. Not sure what to do about it.

The error is:

Staged Sync                              err="runtime error: index out of range [2] with length 2, trace: [stageloop.go:128 panic.go:890 panic.go:113 btree_generic.go:522 btree_generic.go:779 header_algos.go:393 stage_headers.go:844 stage_headers.go:154 default_stages.go:36 sync.go:358 sync.go:260 stageloop.go:177 stageloop.go:92 asm_amd64.s:1598]"

A chunk of my log:

INFO[11-18|09:45:59.638] [2/15 Headers] Inserting headers         progress=32782034 queue=1
INFO[11-18|09:46:19.635] [2/15 Headers] Inserting headers         progress=32789213 queue=1
INFO[11-18|09:46:32.315] [p2p] GoodPeers                          eth66=500
INFO[11-18|09:46:39.636] [2/15 Headers] Inserting headers         progress=32796442 queue=1
INFO[11-18|09:46:49.598] [parlia] snapshots build, gather headers block=32800000
INFO[11-18|09:46:49.598] [parlia] snapshots build, recover from headers block=32800000
INFO[11-18|09:46:59.637] [2/15 Headers] Inserting headers         progress=32803568 queue=1
INFO[11-18|09:47:19.635] [2/15 Headers] Inserting headers         progress=32810680 queue=1
INFO[11-18|09:47:39.637] [2/15 Headers] Inserting headers         progress=32817839 queue=1
INFO[11-18|09:47:59.635] [2/15 Headers] Inserting headers         progress=32824943 queue=1
INFO[11-18|09:48:19.637] [2/15 Headers] Inserting headers         progress=32831700 queue=1
INFO[11-18|09:48:32.315] [p2p] GoodPeers                          eth66=500
INFO[11-18|09:48:39.638] [2/15 Headers] Inserting headers         progress=32838822 queue=1
INFO[11-18|09:48:59.635] [2/15 Headers] Inserting headers         progress=32845864 queue=1
INFO[11-18|09:49:19.637] [2/15 Headers] Inserting headers         progress=32853027 queue=1                                                                                                             INFO[11-18|09:49:39.637] [2/15 Headers] Inserting headers         progress=32860507 queue=1
INFO[11-18|09:49:59.635] [2/15 Headers] Inserting headers         progress=32867884 queue=1
INFO[11-18|09:50:19.636] [2/15 Headers] Inserting headers         progress=32875017 queue=1
INFO[11-18|09:50:32.314] [p2p] GoodPeers                          eth66=500
INFO[11-18|09:50:39.637] [2/15 Headers] Inserting headers         progress=32882206 queue=1
INFO[11-18|09:50:59.638] [2/15 Headers] Inserting headers         progress=32889483 queue=1
INFO[11-18|09:51:19.638] [2/15 Headers] Inserting headers         progress=32896671 queue=1
INFO[11-18|09:51:29.001] [parlia] snapshots build, gather headers block=32900000
INFO[11-18|09:51:29.001] [parlia] snapshots build, recover from headers block=32900000
INFO[11-18|09:51:39.635] [2/15 Headers] Inserting headers         progress=32903797 queue=1
INFO[11-18|09:51:59.636] [2/15 Headers] Wrote block headers       number=32909797 blk/second=7012.500 alloc=5.3GB sys=10.0GB
INFO[11-18|09:52:19.635] [2/15 Headers] No block headers to write in this log period block number=32909797
INFO[11-18|09:52:32.315] [p2p] GoodPeers                          eth66=500
INFO[11-18|09:52:39.635] [2/15 Headers] No block headers to write in this log period block number=32909797
EROR[11-18|09:52:50.769] Staged Sync                              err="runtime error: index out of range [0] with length 0, trace: [stageloop.go:128 panic.go:890 panic.go:113 btree_generic.go:522 btree_generic.go:779 header_algos.go:393 stage_headers.go:831 stage_headers.go:154 default_stages.go:36 sync.go:358 sync.go:260 stageloop.go:177 stageloop.go:92 asm_amd64.s:1598]"
INFO[11-18|09:52:51.670] [2/15 Headers] Waiting for headers...    from=31306138
INFO[11-18|09:52:59.970] [parlia] snapshots build, gather headers block=31300000
INFO[11-18|09:53:08.079] [parlia] snapshots build, recover from headers block=31300000
INFO[11-18|09:53:11.672] [2/15 Headers] Inserting headers         progress=31307235 queue=1
INFO[11-18|09:53:31.673] [2/15 Headers] Inserting headers         progress=31314421 queue=1
INFO[11-18|09:53:51.673] [2/15 Headers] Inserting headers         progress=31319143 queue=1
INFO[11-18|09:54:11.672] [2/15 Headers] Inserting headers         progress=31326313 queue=1
INFO[11-18|09:54:31.672] [2/15 Headers] Inserting headers         progress=31331126 queue=1
INFO[11-18|09:54:32.315] [p2p] GoodPeers                          eth66=499
INFO[11-18|09:54:51.671] [2/15 Headers] Inserting headers         progress=31335692 queue=1
INFO[11-18|09:55:11.672] [2/15 Headers] Inserting headers         progress=31341381 queue=1
INFO[11-18|09:55:31.674] [2/15 Headers] Inserting headers         progress=31348610 queue=1
INFO[11-18|09:55:51.674] [2/15 Headers] Inserting headers         progress=31355842 queue=1
INFO[11-18|09:56:11.672] [2/15 Headers] Inserting headers         progress=31363039 queue=1
INFO[11-18|09:56:31.671] [2/15 Headers] Inserting headers         progress=31370228 queue=1
INFO[11-18|09:56:32.315] [p2p] GoodPeers                          eth66=500
INFO[11-18|09:56:51.672] [2/15 Headers] Inserting headers         progress=31375522 queue=1
INFO[11-18|09:57:11.671] [2/15 Headers] Inserting headers         progress=31382706 queue=1
INFO[11-18|09:57:31.671] [2/15 Headers] Inserting headers         progress=31389900 queue=1

Steps to reproduce the behaviour

Not sure.

subtletech · Answer 1 · Sun Nov 19 2023 05:55:05 GMT+0800 (China Standard Time)

I deleted the data/snapshots/db directory and added the --downloader.verify flag. Seems it helped. Transitioned to "5/15 bodies" finally.