Constant missing block headers, even after resetting stage_headers

[tpalaz]

System information

Erigon version: ./erigon --version
erigon version 1.1.15-dev-9a42b90a

OS & Version: Windows/Linux/OSX
Ubuntu 18

Commit hash:
9a42b90

Erigon Command (with flags/config):
/home/bsc/bsc-erigon/build/bin/erigon --chain=bsc --datadir=/home/bsc/data --metrics --metrics.port=6060 --http=false --pprof --pprof.port=6061 --torrent.download.rate=100000mb --db.pagesize=16k --db.read.concurrency=1000 --sentry.drop-useless-peers=true --p2p.protocol=66 --downloader.verify

Chain/Network:
BSC

Actual behaviour

My FULLY SYNCED ACRHIVE node for the past month or so has been constantly getting the "no block headers to write" error. Having synced from scratch, this problem did not appear until v.1.1.14 and since then has been getting seemingly worse. It is very sporadic when it happens, and restarting the node often fixes it but sometimes it will only fix it for ~5 minutes (and sometimes, for several hours). I have a script to automatically monitor the output but having to restart the node automatically sometimes every several minutes is not ideal. This was never a problem for the longest time, and I was hoping this bug was gone completely. But alas, it seems to be back.

Steps to reproduce the behaviour

Iv'e already reset the stage_headers, unwinded, and attempted to automatically restart HOWEVER, After running stage_headers --reset, and starting the node with --downloader.verify, I get the "no block headers" when trying to catchup to the chain-tip again, and my node WILL NOT RESYNC

Backtrace

INFO[03-15|22:11:33.147] [2/15 Headers] Waiting for headers...    from=37000807
INFO[03-15|22:11:35.304] [2/15 Headers] Processed                 highest inserted=37000808 age=0
INFO[03-15|22:11:35.392] [7/15 Execution] Completed on            block=37000808
INFO[03-15|22:11:35.685] RPC Daemon notified of new headers       from=37000807 to=37000808 hash=0xdd55f59fc258e9082389c8d06b6ac0334cccbc970e9ea3f5505dfc0d91c103f2 header sending=25.892_s log sending=451ns
INFO[03-15|22:11:35.685] Commit cycle                             in=78.493058ms                                    INFO[03-15|22:11:35.685] Timings (slower than 50ms)               Headers=2.157s Execution=72ms IntermediateHashes=72ms
INFO[03-15|22:11:35.685] Tables                                   PlainState=298.6GB AccountChangeSet=682.3GB StorageChangeSet=1.5TB BlockTransaction=7.9GB TransactionLog=2.8TB FreeList=156.8MB ReclaimableSpace=627.2GB              INFO[03-15|22:11:35.741] [2/15 Headers] Waiting for headers...    from=37000808
INFO[03-15|22:11:48.970] [2/15 Headers] Processed                 highest inserted=37000809 age=10s
INFO[03-15|22:12:08.972] [5/15 Bodies] No block bodies to write in this log period block number=37000808
INFO[03-15|22:12:28.972] [5/15 Bodies] No block bodies to write in this log period block number=37000808
INFO[03-15|22:12:42.235] [p2p] GoodPeers
INFO[03-15|22:12:48.974] [5/15 Bodies] No block bodies to write in this log period block number=37000808
INFO[03-15|22:13:08.972] [5/15 Bodies] No block bodies to write in this log period block number=37000808
INFO[03-15|22:13:28.972] [5/15 Bodies] No block bodies to write in this log period block number=37000808
INFO[03-15|22:13:28.972] [5/15 Bodies] DONE                       in=1m40.001545834s
INFO[03-15|22:13:28.974] RPC Daemon notified of new headers       from=37000808 to=37000809 hash=0xfcc216be9c16ebef2d6bacb514db91d40b3bcfa3ad0125a87a8e62643b2c1652 header sending=22.856_s log sending=491ns
INFO[03-15|22:13:28.974] Commit cycle                             in=703.701_s
INFO[03-15|22:13:28.974] Timings (slower than 50ms)               Headers=13.228s Bodies=1m40.001s
INFO[03-15|22:13:28.974] Tables                                   PlainState=298.6GB AccountChangeSet=682.3GB StorageChangeSet=1.5TB BlockTransaction=7.9GB TransactionLog=2.8TB FreeList=156.8MB ReclaimableSpace=627.2GB
INFO[03-15|22:13:28.974] [2/15 Headers] Waiting for headers...    from=37000809
INFO[03-15|22:13:48.974] [2/15 Headers] No block headers to write in this log period block number=37000809
INFO[03-15|22:14:08.974] [2/15 Headers] No block headers to write in this log period block number=37000809
INFO[03-15|22:14:28.975] [2/15 Headers] No block headers to write in this log period block number=37000809
INFO[03-15|22:14:42.235] [p2p] GoodPeers
INFO[03-15|22:14:48.975] [2/15 Headers] No block headers to write in this log period block number=37000809
INFO[03-15|22:15:08.974] [2/15 Headers] No block headers to write in this log period block number=37000809
INFO[03-15|22:15:08.974] Req/resp stats                           req=0 reqMin=0 reqMax=0 skel=0 skelMin=0 skelMax=0 resp=0 respMin=0 respMax=0 dups=0
INFO[03-15|22:15:28.974] [2/15 Headers] No block headers to write in this log period block number=37000809
INFO[03-15|22:15:28.974] Req/resp stats                           req=0 reqMin=0 reqMax=0 skel=0 skelMin=0 skelMax=0 resp=0 respMin=0 respMax=0 dups=0
INFO[03-15|22:15:48.974] [2/15 Headers] No block headers to write in this log period block number=37000809
INFO[03-15|22:15:48.974] Req/resp stats                           req=0 reqMin=0 reqMax=0 skel=0 skelMin=0 skelMax=0 resp=0 respMin=0 respMax=0 dups=0
INFO[03-15|22:16:08.974] [2/15 Headers] No block headers to write in this log period block number=37000809
INFO[03-15|22:16:08.974] Req/resp stats                           req=0 reqMin=0 reqMax=0 skel=0 skelMin=0 skelMax=0 resp=0 respMin=0 respMax=0 dups=0
INFO[03-15|22:16:28.974] [2/15 Headers] No block headers to write in this log period block number=37000809
INFO[03-15|22:16:28.974] Req/resp stats                           req=0 reqMin=0 reqMax=0 skel=0 skelMin=0 skelMax=0 resp=0 respMin=0 respMax=0 dups=0
INFO[03-15|22:16:42.236] [p2p] GoodPeers
INFO[03-15|22:16:48.974] [2/15 Headers] No block headers to write in this log period block number=37000809````

[Lil Crypto]

Same

[blxdyx]

remove those flags: --sentry.drop-useless-peers=true --p2p.protocol=66