[Bug] Potential temporary sync stall from weak-link or malicious peers
Meshiest opened this issue · comments
🐛 Bug Report
A circumstance where block requests are forced to timeout can be reached when a peer with a pending request does not respond with a block response. This is similar to the temporary stall outlined in #3321.
Steps to Reproduce
- prevent a block response from being created from the validator or the client's router
- observe syncing nodes using the modified nodes as peers get stuck for ~10 minutes when reaching a block with an "incomplete request"
We encountered a 10 minute stall during a sync of a node on high performance hardware after merging the latest client sync fixes and deduced it may be related to a swarm of low-quality clients as connected peers (from another test). We were able to resume block sync immediately by forcibly disconnecting the weak links.
The 34.16.96.117 IP is one of 20 16core servers under extreme stress from 10 clients syncing blocks with dozens of transactions per block and as a result is unreliably able to respond to block requests, preventing a block request from being marked as complete.
Expected Behavior
Blocks to sync without stalls when a peer unable to respond.
Potential Fixes
- timeout for block requests on a peer-node basis
- disconnect from peers that are unreliable
- allow for a certain # of peers to be unresponsive when marking a request as complete in
block_sync.rs - shorter total block request timeouts
