Erigon drop peers and stop synchronization
ivan-genev opened this issue · comments
System information
Erigon version: v1.1.12
OS & Version: Linux/Docker
Commit hash: git_branch=HEAD git_tag=v1.1.12-dirty git_commit=f46bb114004c20f63e0d252c9d9c650b6da66c25
Erigon Command (with flags/config):
erigon --chain=bsc --private.api.addr=0.0.0.0:9090 --http=false --metrics --metrics.addr=0.0.0.0 --metrics.port=6060 --torrent.download.rate=70mb --datadir=/data/erigon --p2p.protocol=66 --bodies.cache=214748364800 --batchSize=4096M --db.pagesize=16k --sentry.drop-useless-peers
Chain/Network: bsc
Expected behavior
To reconnect to peers
Actual behavior
At some point erigon start drooping GoodPeers and synchronization stops
Steps to reproduce the behavior
Backtrace
[INFO] [01-22|14:00:49.080] [p2p] GoodPeers eth66=33
[INFO] [01-22|14:02:49.080] [p2p] GoodPeers eth66=32
[INFO] [01-22|14:04:49.080] [p2p] GoodPeers eth66=23
[INFO] [01-22|14:06:49.080] [p2p] GoodPeers eth66=11
[INFO] [01-22|14:08:49.080] [p2p] GoodPeers eth66=6
[INFO] [01-22|14:10:49.080] [p2p] GoodPeers eth66=3
[INFO] [01-22|14:12:49.080] [p2p] GoodPeers eth66=4
[INFO] [01-22|14:14:49.080] [p2p] GoodPeers eth66=8
[INFO] [01-22|14:16:49.080] [p2p] GoodPeers eth66=5
[INFO] [01-22|14:18:49.080] [p2p] GoodPeers eth66=3
[INFO] [01-22|14:20:49.080] [p2p] GoodPeers eth66=4
[INFO] [01-22|14:22:49.080] [p2p] GoodPeers eth66=4
[INFO] [01-22|14:24:49.080] [p2p] GoodPeers eth66=4
[INFO] [01-22|14:26:49.080] [p2p] GoodPeers eth66=1
even with one peer it will continue to synchronize for some time but at some point it will fail without errors and will print
[INFO] [01-23|09:17:05.773] [2/15 Headers] No block headers to write in this log period block number=35490443
[INFO] [01-23|09:17:05.773] Req/resp stats req=0 reqMin=0 reqMax=0 skel=18 skelMin=35490442 skelMax=35490634 resp=18 respMin=35490442 respMax=35490443 dups=36
[INFO] [01-23|09:17:25.773] [2/15 Headers] No block headers to write in this log period block number=35490443
[INFO] [01-23|09:17:25.774] Req/resp stats req=0 reqMin=0 reqMax=0 skel=18 skelMin=35490442 skelMax=35490634 resp=18 respMin=35490442 respMax=35490443 dups=36
restart fix the problem.
Hi there, Try removing this --sentry.drop-useless-peers
as it seems to work fine after that. JFYR: #249 (comment).
I try that but synchronization didn't start at all even when it say that GoodPeers was 32.
Now I test one of out nodes without --p2p.protocol=66
and our second node is with only one GoodPeer for last 24H and still working.
Try out these:
integration state_stages --unwind=100 --chain=bsc --datadir yourdatadir
(JFYR: #258 (comment)),- Stop and run : integration stage_headers --reset, as it will load the missing header. (JFYR: #258 (comment))
- Try not specify the body cache and batch size, these will bring heavy to RAM caches :--bodies.cache=214748364800 --batchSize=4096M (JFYR: #39 (comment))
Do share more logs if it still doesn't work.
Use the latest commit in devel branch.
I started it on our backup node, for now everything looks good.
ps: It was still working with this 1 peer :) my hero
Reopen if still have problem