Sync Error - Attempt to start an already started synchronizer.

[michaels0184]

besu-config.txt

Description

2 boot nodes, 4 validator nodes, 3 member nodes, 1 rpc node.

Steps to Reproduce (Bug)

1. Running tests against the BESU network over 3-4 hours

Expected behavior:
All validators continue to mine at the same rate

Actual behavior:
One validator stops mining and hangs indefinitely.

Frequency:
Happens when running tests after a 3-4 hours (12 hours max). This happens every-time we run tests against the network.

Logs (if a bug)

Besu has identified a problem with its worldstate database.

│ # Your node will fetch the correct data from peers to repair the problem.
│ # Starting the sync pipeline...
#################################################################################################### │
│ validator-4-besu Throwable summary: java.lang.IllegalStateException: Attempt to start an already started synchronizer. │
│ validator-4-besu at: org.hyperledger.besu.ethereum.eth.sync.DefaultSynchronizer.start(DefaultSynchronizer.java:218) │
│ validator-4-besu 2024-05-28 23:58:01.418+00:00 | EthScheduler-Services-7 (importBlock) | ERROR | PipelineChainDownloade │
│ validator-4-besu java.util.concurrent.CompletionException: java.lang.IllegalStateException: Attempt to start an already started synchronizer.

Versions

• Software version: [24.5.2]
• Java version: [openjdk-java-21]
• OS Name & Version: [AKSUbuntu-2204gen2containerd-202405.03.0]
• Kernel Version: [5.15.0-1061-azure]
• Kubernetes Version: [1.28.5]
• Cloud VM, type, size: [Standard_D4as_v4]

Additional Information (Add any of the following or anything else that may be relevant)

• Besu setup info - genesis file, config options
• `genesis.json:

---

{
"nonce": "0x0",
"timestamp": "0x58ee40ba",
"extraData": "0xf87aa00000000000000000000000000000000000000000000000000000000000000000f8549452dfcfb57df309cb83a2a5873ab9e02b5286adc294c78292173664af0153c943b67d6b7371c5bbbbcf94130ee989acf35492f16ac796c2672a7ae085bf7f940f96dea61aa4303a35f5f0c29938973fc0d0bf52c080c0",
"gasLimit": "0x989680",
"gasUsed": "0x0",
"number": "0x0",
"difficulty": "0x1",
"coinbase": "0x0000000000000000000000000000000000000000",
"mixHash": "0x63746963616c2062797a616e74696e65206661756c7420746f6c6572616e6365",
"parentHash": "0x0000000000000000000000000000000000000000000000000000000000000000",
"config": {
"chainId": 1337,
"homesteadBlock": 0,
"eip150Block": 0,
"eip150Hash": "0x0000000000000000000000000000000000000000000000000000000000000000",
"eip155Block": 0,
"eip158Block": 0,
"byzantiumBlock": 0,
"constantinopleBlock": 0,
"petersburgBlock": 0,
"istanbulBlock": 0,
"muirglacierblock": 0,
"berlinBlock": 0,
"londonBlock": 0,
"zeroBaseFee": true,
"qbft": {
"blockperiodseconds": 2,
"epochlength": 30000,
"requesttimeoutseconds": 20
}
},
"baseFeePerGas": "0x0",
"nonce": "0x0",
"alloc": {
"0xa1c0d3e39e5524dd0284f7fe768f9f5aebfbae8a": {
"balance": "1000000000000000000000000000"
},
"0x8d5903e624be9cddd0a23ea5d1dc3efc077ea438": {
"balance": "1000000000000000000000000000"
},
"0xceee8609e8950c22edbf0adca5a36b52a350dba5": {
"balance": "1000000000000000000000000000"
},
"0x29f9c20171de55c323e95070c4eea1ccaec2f5ec": {
"balance": "1000000000000000000000000000"
}
}
}`

• System info - memory, CPU
  Total memory: 15.62 GB
  CPU cores: 4

[Matt Nelson]

Hi there - Bonsai and Tessera can have issues together - have you tried FOREST mode for your data storage format?

https://besu.hyperledger.org/development/public-networks/concepts/data-storage-formats

[GGos3]

In my case, I encountered the same issue, but changing the node's option from data-storage-format="BONSAI" to "FOREST" resolved the error.