Testnet nodes crashing after hard-fork
nmiceli-simtlix opened this issue · comments
And once again, a hard-fork comes, and our nodes stop working.
I am on v0.10.15.
I[2023-07-11|12:22:37.816] Relaying block for order book module=dex height=41649999
I[2023-07-11|12:22:37.816] replayed all tx. Starting match module=dex height=41649999
I[2023-07-11|12:22:37.816] No order comes in for the block module=dexkeeper
I[2023-07-11|12:22:37.816] Relaying block for order book module=dex height=41650000
I[2023-07-11|12:22:37.816] replayed all tx. Starting match module=dex height=41650000
I[2023-07-11|12:22:37.816] No order comes in for the block module=dexkeeper
I[2023-07-11|12:22:38.144] Located day to load breathe block height module=dexkeeper epochDay="]L\u0000\u0000\u0000\u0000\u0000\u0000"
I[2023-07-11|12:22:38.144] Loaded breathe block height module=dexkeeper height=41641219
I[2023-07-11|12:22:38.144] get last breathe block height module=main height=41641219
I[2023-07-11|12:22:38.144] reload latest snapshot module=statesync height=41641219
I[2023-07-11|12:22:38.211] ABCI Handshake App Info module=consensus height=41650000 hash=E72E007C0DE98D8557538AD3C12CA543DD93C9AEAA06D481FAD295BF795EB7A5 software-version= protocol-version=0
I[2023-07-11|12:22:38.211] ABCI Replay Blocks module=consensus appHeight=41650000 storeHeight=41650001 stateHeight=41650000
I[2023-07-11|12:22:38.212] Replay last block using real app module=consensus
ERROR: error during handshake: error on replay: Wrong Block.Header.AppHash. Expected E72E007C0DE98D8557538AD3C12CA543DD93C9AEAA06D481FAD295BF795EB7A5, got 465A524C0D00723A8C0F40840149B355FAED5909592E8CD47E3840F47A4457D9
Hi there, Can you provide some more logs please?
Also, Is this happened after sometime post you re-started user node? Can you stop your node and re-start it after a while (not immediately) and let it run for a while and see if it works.
Also, Can you confirm if you're using latest (v0.10.15) app, config and genesis files from here: https://github.com/bnb-chain/node/releases/tag/v0.10.15.
Can you also share the output of: curl localhost:27147/status.
Just a side note, I tried it on my end and it seems working for me.
Hi @deepcrazy
I am using bnb v0.10.15 (commit hash 3195e6b)
The config files I am using are this ones --> https://github.com/bnb-chain/node/releases/download/v0.10.15/testnet_config.zip
I[2023-07-11|19:12:43.604] Starting ABCI with Tendermint module=main
I[2023-07-11|19:12:45.734] Located day to load breathe block height module=dexkeeper epochDay="]L\u0000\u0000\u0000\u0000\u0000\u0000"
I[2023-07-11|19:12:45.734] Loaded breathe block height module=dexkeeper height=41641219
I[2023-07-11|19:12:45.734] Loading order book snapshot from last breathe block module=main blockHeight=41641219
I[2023-07-11|19:12:45.810] Successfully Loaded order snapshot module=main pair="{BaseAssetSymbol:175-0B3M QuoteAssetSymbol:BNB ListPrice:1.00000000 TickSize:0.00001000 LotSize:0.00100000}"
I[2023-07-11|19:12:45.810] Successfully Loaded order snapshot module=main pair="{BaseAssetSymbol:3LAU-544 QuoteAssetSymbol:BNB ListPrice:1.00000000 TickSize:0.00001000 LotSize:0.00100000}"
I[2023-07-11|19:12:45.811] Successfully Loaded order snapshot module=main pair="{BaseAssetSymbol:ABC-001 QuoteAssetSymbol:BUSD-BAF ListPrice:1.00000000 TickSize:0.00001000 LotSize:0.00100000}"
I[2023-07-11|19:12:45.811] Successfully Loaded order snapshot module=main pair="{BaseAssetSymbol:ABC-59C QuoteAssetSymbol:BNB ListPrice:1.00000000 TickSize:0.00001000 LotSize:0.00100000}"
I[2023-07-11|19:12:45.811] Successfully Loaded order snapshot module=main pair="{BaseAssetSymbol:ABCD-AA5M QuoteAssetSymbol:BNB ListPrice:10.00000000 TickSize:0.00010000 LotSize:0.00010000}"
I[2023-07-11|19:12:45.811] Successfully Loaded order snapshot module=main pair="{BaseAssetSymbol:ALELE-979M QuoteAssetSymbol:BUSD-BAF ListPrice:1.00000000 TickSize:0.00001000 LotSize:0.00100000}"
I[2023-07-11|19:12:45.811] Successfully Loaded order snapshot module=main pair="{BaseAssetSymbol:AT10-AF2 QuoteAssetSymbol:BNB ListPrice:2.00000000 TickSize:0.00000001 LotSize:100.00000000}"
I[2023-07-11|19:12:45.812] Successfully Loaded order snapshot module=main pair="{BaseAssetSymbol:AT25-C88 QuoteAssetSymbol:BNB ListPrice:2.00000000 TickSize:0.00000001 LotSize:10.00000000}"
I[2023-07-11|19:12:45.812] Successfully Loaded order snapshot module=main pair="{BaseAssetSymbol:AT9-4F1 QuoteAssetSymbol:BNB ListPrice:2.00000000 TickSize:0.00001000 LotSize:0.00100000}"
I[2023-07-11|19:12:45.812] Successfully Loaded order snapshot module=main pair="{BaseAssetSymbol:AT9-693 QuoteAssetSymbol:BNB ListPrice:2.00000000 TickSize:0.00000001 LotSize:100.00000000}"
I[2023-07-11|19:12:45.812] Successfully Loaded order snapshot module=main pair="{BaseAssetSymbol:AVA-524 QuoteAssetSymbol:BNB ListPrice:1.00000000 TickSize:0.00001000 LotSize:0.00100000}"
I[2023-07-11|19:12:45.813] Successfully Loaded order snapshot module=main pair="{BaseAssetSymbol:BBEX-6BAM QuoteAssetSymbol:BNB ListPrice:1.00000000 TickSize:0.00001000 LotSize:0.00100000}"
I[2023-07-11|19:12:45.813] Successfully Loaded order snapshot module=main pair="{BaseAssetSymbol:BDX-129M QuoteAssetSymbol:BNB ListPrice:1.00000000 TickSize:0.00001000 LotSize:0.00100000}"
I[2023-07-11|19:12:45.813] Successfully Loaded order snapshot module=main pair="{BaseAssetSymbol:BDX-129M QuoteAssetSymbol:BUSD-BAF ListPrice:1.00000000 TickSize:0.00001000 LotSize:0.00100000}"
I[2023-07-11|19:12:45.813] Successfully Loaded order snapshot module=main pair="{BaseAssetSymbol:BEP-5DDM QuoteAssetSymbol:BNB ListPrice:1000000.00000000 TickSize:10.00000000 LotSize:0.00000001}"
I[2023-07-11|19:12:45.813] Successfully Loaded order snapshot module=main pair="{BaseAssetSymbol:BHT-17FM QuoteAssetSymbol:BUSD-BAF ListPrice:1.00000000 TickSize:0.00001000 LotSize:0.00100000}"
I[2023-07-11|19:12:45.814] Successfully Loaded order snapshot module=main pair="{BaseAssetSymbol:BNBM-742M QuoteAssetSymbol:BNB ListPrice:10.00000000 TickSize:0.00010000 LotSize:0.00010000}"
I[2023-07-11|19:12:45.814] Successfully Loaded order snapshot module=main pair="{BaseAssetSymbol:BNBM-742M QuoteAssetSymbol:BUSD-BAF ListPrice:1.00000000 TickSize:0.00001000 LotSize:0.00010000}"
I[2023-07-11|19:12:45.814] Successfully Loaded order snapshot module=main pair="{BaseAssetSymbol:BNB QuoteAssetSymbol:BTCB-101 ListPrice:0.02000000 TickSize:0.00000010 LotSize:0.00100000}"
I[2023-07-11|19:12:45.814] Successfully Loaded order snapshot module=main pair="{BaseAssetSymbol:BNB QuoteAssetSymbol:BUSD-BAF ListPrice:18.00000000 TickSize:0.00000100 LotSize:0.00100000}"
I[2023-07-11|19:12:45.814] Successfully Loaded order snapshot module=main pair="{BaseAssetSymbol:BNB QuoteAssetSymbol:ETH-D5B ListPrice:0.07000000 TickSize:0.00000100 LotSize:0.00100000}"
I[2023-07-11|19:12:45.815] Successfully Loaded order snapshot module=main pair="{BaseAssetSymbol:BNB QuoteAssetSymbol:TUSD-5D7 ListPrice:17.00000000 TickSize:0.00010000 LotSize:0.00100000}"
I[2023-07-11|19:12:45.815] Successfully Loaded order snapshot module=main pair="{BaseAssetSymbol:BNB QuoteAssetSymbol:USDS-5BD ListPrice:17.00000000 TickSize:0.00010000 LotSize:0.00100000}"
I[2023-07-11|19:12:45.815] Successfully Loaded order snapshot module=main pair="{BaseAssetSymbol:BOKM-8D7M QuoteAssetSymbol:BNB ListPrice:10.00000000 TickSize:0.00010000 LotSize:0.00010000}"
I[2023-07-11|19:12:45.815] Successfully Loaded order snapshot module=main pair="{BaseAssetSymbol:BTCB-101 QuoteAssetSymbol:BUSD-BAF ListPrice:9000.00000000 TickSize:0.01000000 LotSize:0.00001000}"
I[2023-07-11|19:12:45.816] Successfully Loaded order snapshot module=main pair="{BaseAssetSymbol:BTCKFE-353M QuoteAssetSymbol:BNB ListPrice:0.00160000 TickSize:0.00000001 LotSize:1.00000000}"
I[2023-07-11|19:12:45.816] Successfully Loaded order snapshot module=main pair="{BaseAssetSymbol:BTCKFE-353M QuoteAssetSymbol:BUSD-BAF ListPrice:0.10000000 TickSize:0.00000100 LotSize:1.00000000}"
I[2023-07-11|19:12:45.816] Successfully Loaded order snapshot module=main pair="{BaseAssetSymbol:BUSD-227 QuoteAssetSymbol:BNB ListPrice:1.00000000 TickSize:0.00010000 LotSize:0.00010000}"
I[2023-07-11|19:12:45.817] Successfully Loaded order snapshot module=main pair="{BaseAssetSymbol:CBIX-7A1 QuoteAssetSymbol:BNB ListPrice:1.00000000 TickSize:0.00000001 LotSize:100.00000000}"
I[2023-07-11|19:12:45.817] Successfully Loaded order snapshot module=main pair="{BaseAssetSymbol:CRAD-D5DM QuoteAssetSymbol:BNB ListPrice:0.10000000 TickSize:0.00000100 LotSize:0.01000000}"
I[2023-07-11|19:12:45.818] Successfully Loaded order snapshot module=main pair="{BaseAssetSymbol:CUB-EE5 QuoteAssetSymbol:BNB ListPrice:1.00000000 TickSize:0.01000000 LotSize:0.00000100}"
I[2023-07-11|19:12:45.818] Successfully Loaded order snapshot module=main pair="{BaseAssetSymbol:DEEZ-BE4M QuoteAssetSymbol:BNB ListPrice:0.00130000 TickSize:0.00000001 LotSize:1.00000000}"
I[2023-07-11|19:12:45.818] Successfully Loaded order snapshot module=main pair="{BaseAssetSymbol:DWU-B6BM QuoteAssetSymbol:BUSD-BAF ListPrice:0.01000000 TickSize:0.00000010 LotSize:0.10000000}"
I[2023-07-11|19:12:45.818] Successfully Loaded order snapshot module=main pair="{BaseAssetSymbol:ELON-72BM QuoteAssetSymbol:BNB ListPrice:0.01000000 TickSize:0.00000010 LotSize:0.10000000}"
I[2023-07-11|19:12:45.819] Successfully Loaded order snapshot module=main pair="{BaseAssetSymbol:ELON-72BM QuoteAssetSymbol:BUSD-BAF ListPrice:100000.00000000 TickSize:1.00000000 LotSize:0.10000000}"
I[2023-07-11|19:12:45.819] Successfully Loaded order snapshot module=main pair="{BaseAssetSymbol:ETH-D5B QuoteAssetSymbol:BUSD-BAF ListPrice:200.00000000 TickSize:0.00100000 LotSize:0.00100000}"
I[2023-07-11|19:12:45.819] Successfully Loaded order snapshot module=main pair="{BaseAssetSymbol:EWE-31DM QuoteAssetSymbol:BUSD-BAF ListPrice:1.00000000 TickSize:0.00001000 LotSize:0.00100000}"
I[2023-07-11|19:12:45.820] Successfully Loaded order snapshot module=main pair="{BaseAssetSymbol:FABT-A4DM QuoteAssetSymbol:BNB ListPrice:0.00200000 TickSize:0.00000001 LotSize:1.00000000}"
I[2023-07-11|19:12:45.820] Successfully Loaded order snapshot module=main pair="{BaseAssetSymbol:FEHU-639M QuoteAssetSymbol:BNB ListPrice:0.00050000 TickSize:0.00000001 LotSize:10.00000000}"
I[2023-07-11|19:12:45.820] Successfully Loaded order snapshot module=main pair="{BaseAssetSymbol:GAEA-E4AM QuoteAssetSymbol:BNB ListPrice:0.66000000 TickSize:0.00000100 LotSize:0.01000000}"
I[2023-07-11|19:12:45.820] Successfully Loaded order snapshot module=main pair="{BaseAssetSymbol:GBC-F29M QuoteAssetSymbol:BUSD-BAF ListPrice:10.00000000 TickSize:0.00010000 LotSize:0.00010000}"
I[2023-07-11|19:12:45.820] Successfully Loaded order snapshot module=main pair="{BaseAssetSymbol:GLT-2E8M QuoteAssetSymbol:BNB ListPrice:0.00150000 TickSize:0.00000001 LotSize:1.00000000}"
I[2023-07-11|19:12:45.821] Successfully Loaded order snapshot module=main pair="{BaseAssetSymbol:GLT-2E8M QuoteAssetSymbol:BUSD-BAF ListPrice:0.75000000 TickSize:0.00000100 LotSize:1.00000000}"
I[2023-07-11|19:12:45.821] Successfully Loaded order snapshot module=main pair="{BaseAssetSymbol:HEM-452 QuoteAssetSymbol:BNB ListPrice:1.00000000 TickSize:0.00000100 LotSize:0.01000000}"
I[2023-07-11|19:12:45.821] Successfully Loaded order snapshot module=main pair="{BaseAssetSymbol:HEM-452 QuoteAssetSymbol:BUSD-227 ListPrice:1.00000000 TickSize:0.00001000 LotSize:0.01000000}"
I[2023-07-11|19:12:45.821] Successfully Loaded order snapshot module=main pair="{BaseAssetSymbol:HMT-7FEM QuoteAssetSymbol:BNB ListPrice:1.00000000 TickSize:0.00001000 LotSize:0.00100000}"
I[2023-07-11|19:12:45.821] Successfully Loaded order snapshot module=main pair="{BaseAssetSymbol:HMT-7FEM QuoteAssetSymbol:BUSD-BAF ListPrice:40.00000000 TickSize:0.00010000 LotSize:0.00100000}"
I[2023-07-11|19:12:45.822] Successfully Loaded order snapshot module=main pair="{BaseAssetSymbol:ICC-C6BM QuoteAssetSymbol:BNB ListPrice:0.10000000 TickSize:0.00000100 LotSize:0.01000000}"
I[2023-07-11|19:12:45.822] Successfully Loaded order snapshot module=main pair="{BaseAssetSymbol:IMC-CFCM QuoteAssetSymbol:BUSD-BAF ListPrice:2.50000000 TickSize:0.00001000 LotSize:0.00100000}"
I[2023-07-11|19:12:45.822] Successfully Loaded order snapshot module=main pair="{BaseAssetSymbol:INNBC-1A1M QuoteAssetSymbol:BNB ListPrice:0.03000000 TickSize:0.00000010 LotSize:0.10000000}"
I[2023-07-11|19:12:45.822] Successfully Loaded order snapshot module=main pair="{BaseAssetSymbol:INNBC-1A1M QuoteAssetSymbol:BUSD-BAF ListPrice:1.00000000 TickSize:0.00001000 LotSize:0.10000000}"
I[2023-07-11|19:12:45.822] Successfully Loaded order snapshot module=main pair="{BaseAssetSymbol:IRMO-C9D QuoteAssetSymbol:BNB ListPrice:1.00000000 TickSize:0.00100000 LotSize:0.00001000}"
I[2023-07-11|19:12:45.823] Successfully Loaded order snapshot module=main pair="{BaseAssetSymbol:ITTT-730M QuoteAssetSymbol:BNB ListPrice:0.00010000 TickSize:0.00000001 LotSize:100.00000000}"
I[2023-07-11|19:12:45.823] Successfully Loaded order snapshot module=main pair="{BaseAssetSymbol:KAI-04E QuoteAssetSymbol:BNB ListPrice:1.00000000 TickSize:0.00001000 LotSize:0.00100000}"
I[2023-07-11|19:12:45.823] Successfully Loaded order snapshot module=main pair="{BaseAssetSymbol:LPHP-88F QuoteAssetSymbol:BNB ListPrice:1.00000000 TickSize:0.00001000 LotSize:0.00100000}"
I[2023-07-11|19:12:45.823] Successfully Loaded order snapshot module=main pair="{BaseAssetSymbol:MAE-082M QuoteAssetSymbol:BNB ListPrice:0.10000000 TickSize:0.00000100 LotSize:0.01000000}"
I[2023-07-11|19:12:45.823] Successfully Loaded order snapshot module=main pair="{BaseAssetSymbol:MATIC-A7D QuoteAssetSymbol:BNB ListPrice:0.01000000 TickSize:0.00000010 LotSize:0.10000000}"
I[2023-07-11|19:12:45.824] Successfully Loaded order snapshot module=main pair="{BaseAssetSymbol:MILAD-ECEM QuoteAssetSymbol:BNB ListPrice:0.00100000 TickSize:0.00000010 LotSize:0.10000000}"
I[2023-07-11|19:12:45.824] Successfully Loaded order snapshot module=main pair="{BaseAssetSymbol:MILAD-ECEM QuoteAssetSymbol:BUSD-BAF ListPrice:1.00000000 TickSize:0.00000010 LotSize:0.10000000}"
I[2023-07-11|19:12:45.824] Successfully Loaded order snapshot module=main pair="{BaseAssetSymbol:MIMI-02FM QuoteAssetSymbol:BNB ListPrice:0.01000000 TickSize:0.00000010 LotSize:0.10000000}"
I[2023-07-11|19:12:45.824] Successfully Loaded order snapshot module=main pair="{BaseAssetSymbol:MINIA-7A2M QuoteAssetSymbol:BNB ListPrice:1.00000000 TickSize:0.00001000 LotSize:0.00100000}"
I[2023-07-11|19:12:45.824] Successfully Loaded order snapshot module=main pair="{BaseAssetSymbol:MINIA-7A2M QuoteAssetSymbol:BUSD-BAF ListPrice:1.00000000 TickSize:0.00001000 LotSize:0.00100000}"
I[2023-07-11|19:12:45.824] Successfully Loaded order snapshot module=main pair="{BaseAssetSymbol:MJBEP8-409M QuoteAssetSymbol:BNB ListPrice:0.50000000 TickSize:0.00000100 LotSize:0.01000000}"
I[2023-07-11|19:12:45.825] Successfully Loaded order snapshot module=main pair="{BaseAssetSymbol:MOCKT-094 QuoteAssetSymbol:BNB ListPrice:10.00000000 TickSize:0.00010000 LotSize:0.00010000}"
I[2023-07-11|19:12:45.825] Successfully Loaded order snapshot module=main pair="{BaseAssetSymbol:MOCKT-1D0 QuoteAssetSymbol:BNB ListPrice:10.00000000 TickSize:0.00010000 LotSize:0.00010000}"
I[2023-07-11|19:12:45.825] Successfully Loaded order snapshot module=main pair="{BaseAssetSymbol:MOCKT-2D6 QuoteAssetSymbol:BNB ListPrice:10.00000000 TickSize:0.00010000 LotSize:0.00010000}"
I[2023-07-11|19:12:45.825] Successfully Loaded order snapshot module=main pair="{BaseAssetSymbol:MOCKT-477 QuoteAssetSymbol:BNB ListPrice:10.00000000 TickSize:0.00010000 LotSize:0.00010000}"
I[2023-07-11|19:12:45.825] Successfully Loaded order snapshot module=main pair="{BaseAssetSymbol:MOCKT-4FF QuoteAssetSymbol:BNB ListPrice:10.00000000 TickSize:0.00010000 LotSize:0.00010000}"
I[2023-07-11|19:12:45.826] Successfully Loaded order snapshot module=main pair="{BaseAssetSymbol:MOCKT-68A QuoteAssetSymbol:BNB ListPrice:10.00000000 TickSize:0.00010000 LotSize:0.00010000}"
I[2023-07-11|19:12:45.826] Successfully Loaded order snapshot module=main pair="{BaseAssetSymbol:MOCKT-D1B QuoteAssetSymbol:BNB ListPrice:10.00000000 TickSize:0.00010000 LotSize:0.00010000}"
I[2023-07-11|19:12:45.826] Successfully Loaded order snapshot module=main pair="{BaseAssetSymbol:MOCKT-E62 QuoteAssetSymbol:BNB ListPrice:10.00000000 TickSize:0.00010000 LotSize:0.00010000}"
I[2023-07-11|19:12:45.826] Successfully Loaded order snapshot module=main pair="{BaseAssetSymbol:MSDK-0A3M QuoteAssetSymbol:BNB ListPrice:10.00000000 TickSize:0.00010000 LotSize:0.00010000}"
I[2023-07-11|19:12:45.830] Successfully Loaded order snapshot module=main pair="{BaseAssetSymbol:MSDK-132M QuoteAssetSymbol:BNB ListPrice:10.00000000 TickSize:0.00010000 LotSize:0.00010000}"
I[2023-07-11|19:12:45.830] Successfully Loaded order snapshot module=main pair="{BaseAssetSymbol:MSDK-1F6M QuoteAssetSymbol:BNB ListPrice:10.00000000 TickSize:0.00010000 LotSize:0.00010000}"
I[2023-07-11|19:12:45.831] Successfully Loaded order snapshot module=main pair="{BaseAssetSymbol:MSDK-229M QuoteAssetSymbol:BNB ListPrice:10.00000000 TickSize:0.00010000 LotSize:0.00010000}"
I[2023-07-11|19:12:45.831] Successfully Loaded order snapshot module=main pair="{BaseAssetSymbol:MSDK-25CM QuoteAssetSymbol:BNB ListPrice:10.00000000 TickSize:0.00010000 LotSize:0.00010000}"
I[2023-07-11|19:12:45.832] Successfully Loaded order snapshot module=main pair="{BaseAssetSymbol:MSDK-327M QuoteAssetSymbol:BNB ListPrice:10.00000000 TickSize:0.00010000 LotSize:0.00010000}"
I[2023-07-11|19:12:45.832] Successfully Loaded order snapshot module=main pair="{BaseAssetSymbol:MSDK-3B4M QuoteAssetSymbol:BNB ListPrice:10.00000000 TickSize:0.00010000 LotSize:0.00010000}"
I[2023-07-11|19:12:45.833] Successfully Loaded order snapshot module=main pair="{BaseAssetSymbol:MSDK-3F7M QuoteAssetSymbol:BNB ListPrice:10.00000000 TickSize:0.00010000 LotSize:0.00010000}"
I[2023-07-11|19:12:45.833] Successfully Loaded order snapshot module=main pair="{BaseAssetSymbol:MSDK-41CM QuoteAssetSymbol:BNB ListPrice:10.00000000 TickSize:0.00010000 LotSize:0.00010000}"
I[2023-07-11|19:12:45.834] Successfully Loaded order snapshot module=main pair="{BaseAssetSymbol:MSDK-5C9M QuoteAssetSymbol:BNB ListPrice:10.00000000 TickSize:0.00010000 LotSize:0.00010000}"
I[2023-07-11|19:12:45.834] Successfully Loaded order snapshot module=main pair="{BaseAssetSymbol:MSDK-C59M QuoteAssetSymbol:BNB ListPrice:10.00000000 TickSize:0.00010000 LotSize:0.00010000}"
I[2023-07-11|19:12:45.834] Successfully Loaded order snapshot module=main pair="{BaseAssetSymbol:MSDK-D1FM QuoteAssetSymbol:BNB ListPrice:10.00000000 TickSize:0.00010000 LotSize:0.00010000}"
I[2023-07-11|19:12:45.835] Successfully Loaded order snapshot module=main pair="{BaseAssetSymbol:MSDK-DA2M QuoteAssetSymbol:BNB ListPrice:10.00000000 TickSize:0.00010000 LotSize:0.00010000}"
I[2023-07-11|19:12:45.835] Successfully Loaded order snapshot module=main pair="{BaseAssetSymbol:MSDK-DC1M QuoteAssetSymbol:BNB ListPrice:10.00000000 TickSize:0.00010000 LotSize:0.00010000}"
I[2023-07-11|19:12:45.835] Successfully Loaded order snapshot module=main pair="{BaseAssetSymbol:MSDK-E1DM QuoteAssetSymbol:BNB ListPrice:10.00000000 TickSize:0.00010000 LotSize:0.00010000}"
I[2023-07-11|19:12:45.836] Successfully Loaded order snapshot module=main pair="{BaseAssetSymbol:MSDK-E53M QuoteAssetSymbol:BNB ListPrice:10.00000000 TickSize:0.00010000 LotSize:0.00010000}"
I[2023-07-11|19:12:45.836] Successfully Loaded order snapshot module=main pair="{BaseAssetSymbol:MSFA-3D2M QuoteAssetSymbol:BNB ListPrice:0.15000000 TickSize:0.00000100 LotSize:0.01000000}"
I[2023-07-11|19:12:45.836] Successfully Loaded order snapshot module=main pair="{BaseAssetSymbol:MST-EB4M QuoteAssetSymbol:BNB ListPrice:0.10000000 TickSize:0.00000100 LotSize:0.01000000}"
I[2023-07-11|19:12:45.836] Successfully Loaded order snapshot module=main pair="{BaseAssetSymbol:NWCT-61FM QuoteAssetSymbol:BNB ListPrice:0.30000000 TickSize:0.00000100 LotSize:0.01000000}"
I[2023-07-11|19:12:45.837] Successfully Loaded order snapshot module=main pair="{BaseAssetSymbol:PRV1-E9BM QuoteAssetSymbol:BNB ListPrice:0.00000500 TickSize:0.00000001 LotSize:1000.00000000}"
I[2023-07-11|19:12:45.837] Successfully Loaded order snapshot module=main pair="{BaseAssetSymbol:QWE-684M QuoteAssetSymbol:BNB ListPrice:0.01000000 TickSize:0.00000010 LotSize:0.10000000}"
I[2023-07-11|19:12:45.837] Successfully Loaded order snapshot module=main pair="{BaseAssetSymbol:RCO-30EM QuoteAssetSymbol:BNB ListPrice:0.04500000 TickSize:0.00000100 LotSize:0.01000000}"
I[2023-07-11|19:12:45.837] Successfully Loaded order snapshot module=main pair="{BaseAssetSymbol:RCO-30EM QuoteAssetSymbol:BUSD-BAF ListPrice:1.00000000 TickSize:0.00001000 LotSize:0.01000000}"
I[2023-07-11|19:12:45.838] Successfully Loaded order snapshot module=main pair="{BaseAssetSymbol:RUBY-366M QuoteAssetSymbol:BNB ListPrice:1.00000000 TickSize:0.00001000 LotSize:0.00100000}"
I[2023-07-11|19:12:45.838] Successfully Loaded order snapshot module=main pair="{BaseAssetSymbol:RUBY-366M QuoteAssetSymbol:BUSD-BAF ListPrice:1.00000000 TickSize:0.00001000 LotSize:0.00100000}"
I[2023-07-11|19:12:45.838] Successfully Loaded order snapshot module=main pair="{BaseAssetSymbol:SAT-30B QuoteAssetSymbol:BNB ListPrice:1.00000000 TickSize:0.00000100 LotSize:0.01000000}"
I[2023-07-11|19:12:45.838] Successfully Loaded order snapshot module=main pair="{BaseAssetSymbol:SDDS-A84M QuoteAssetSymbol:BUSD-BAF ListPrice:0.10000000 TickSize:0.00000100 LotSize:0.01000000}"
I[2023-07-11|19:12:45.839] Successfully Loaded order snapshot module=main pair="{BaseAssetSymbol:SEFA-A0AM QuoteAssetSymbol:BNB ListPrice:0.00000001 TickSize:0.00000001 LotSize:100000.00000000}"
I[2023-07-11|19:12:45.839] Successfully Loaded order snapshot module=main pair="{BaseAssetSymbol:SLV-C4EM QuoteAssetSymbol:BNB ListPrice:1.00000000 TickSize:0.00001000 LotSize:0.00100000}"
I[2023-07-11|19:12:45.839] Successfully Loaded order snapshot module=main pair="{BaseAssetSymbol:SLV-C4EM QuoteAssetSymbol:BUSD-BAF ListPrice:26.00000000 TickSize:0.00010000 LotSize:0.00100000}"
I[2023-07-11|19:12:45.839] Successfully Loaded order snapshot module=main pair="{BaseAssetSymbol:START-561M QuoteAssetSymbol:BNB ListPrice:1.00000000 TickSize:0.00001000 LotSize:0.00100000}"
I[2023-07-11|19:12:45.839] Successfully Loaded order snapshot module=main pair="{BaseAssetSymbol:START-BA0M QuoteAssetSymbol:BNB ListPrice:0.10000000 TickSize:0.00000100 LotSize:0.01000000}"
I[2023-07-11|19:12:45.840] Successfully Loaded order snapshot module=main pair="{BaseAssetSymbol:STEEP-423M QuoteAssetSymbol:BNB ListPrice:0.00010000 TickSize:0.00000001 LotSize:10.00000000}"
I[2023-07-11|19:12:45.840] Successfully Loaded order snapshot module=main pair="{BaseAssetSymbol:STL-7C6M QuoteAssetSymbol:BNB ListPrice:1.00000000 TickSize:0.00001000 LotSize:0.00100000}"
I[2023-07-11|19:12:45.840] Successfully Loaded order snapshot module=main pair="{BaseAssetSymbol:STLM-DD7M QuoteAssetSymbol:BNB ListPrice:1.00000000 TickSize:0.00001000 LotSize:0.00100000}"
I[2023-07-11|19:12:45.841] Successfully Loaded order snapshot module=main pair="{BaseAssetSymbol:TDZR-1DCM QuoteAssetSymbol:BNB ListPrice:0.00100000 TickSize:0.00000001 LotSize:1.00000000}"
I[2023-07-11|19:12:45.841] Successfully Loaded order snapshot module=main pair="{BaseAssetSymbol:TES-BCCM QuoteAssetSymbol:BNB ListPrice:0.00100000 TickSize:0.00000001 LotSize:1.00000000}"
I[2023-07-11|19:12:45.841] Successfully Loaded order snapshot module=main pair="{BaseAssetSymbol:TTT-29DM QuoteAssetSymbol:BNB ListPrice:0.10000000 TickSize:0.00000100 LotSize:0.01000000}"
I[2023-07-11|19:12:45.841] Successfully Loaded order snapshot module=main pair="{BaseAssetSymbol:TTT-3FAM QuoteAssetSymbol:BNB ListPrice:2.00000000 TickSize:0.00000010 LotSize:0.10000000}"
I[2023-07-11|19:12:45.842] Successfully Loaded order snapshot module=main pair="{BaseAssetSymbol:U2U-DF5M QuoteAssetSymbol:BNB ListPrice:1.00000000 TickSize:0.00001000 LotSize:0.00100000}"
I[2023-07-11|19:12:45.842] Successfully Loaded order snapshot module=main pair="{BaseAssetSymbol:VNB-708M QuoteAssetSymbol:BNB ListPrice:0.10000000 TickSize:0.00000100 LotSize:0.01000000}"
I[2023-07-11|19:12:45.842] Successfully Loaded order snapshot module=main pair="{BaseAssetSymbol:VTBC-CB0M QuoteAssetSymbol:BNB ListPrice:10.00000000 TickSize:0.00000100 LotSize:0.01000000}"
I[2023-07-11|19:12:45.842] Successfully Loaded order snapshot module=main pair="{BaseAssetSymbol:WOOPT-FADM QuoteAssetSymbol:BNB ListPrice:1.00000000 TickSize:0.00000001 LotSize:100.00000000}"
I[2023-07-11|19:12:45.842] Successfully Loaded order snapshot module=main pair="{BaseAssetSymbol:WTR-ED7M QuoteAssetSymbol:BNB ListPrice:1.00000000 TickSize:0.00001000 LotSize:0.00100000}"
I[2023-07-11|19:12:45.843] Successfully Loaded order snapshot module=main pair="{BaseAssetSymbol:WWWC-12CM QuoteAssetSymbol:BUSD-BAF ListPrice:0.20000000 TickSize:0.00000100 LotSize:0.01000000}"
I[2023-07-11|19:12:45.843] Successfully Loaded order snapshot module=main pair="{BaseAssetSymbol:XIN-F25 QuoteAssetSymbol:BNB ListPrice:1.00000000 TickSize:0.00001000 LotSize:0.00100000}"
I[2023-07-11|19:12:45.843] Successfully Loaded order snapshot module=main pair="{BaseAssetSymbol:XRP-FB8 QuoteAssetSymbol:BNB ListPrice:0.01000000 TickSize:0.00000001 LotSize:1.00000000}"
I[2023-07-11|19:12:45.843] Successfully Loaded order snapshot module=main pair="{BaseAssetSymbol:YGEM-B61M QuoteAssetSymbol:BNB ListPrice:0.00100000 TickSize:0.00010000 LotSize:0.00010000}"
I[2023-07-11|19:12:45.844] Successfully Loaded order snapshot module=main pair="{BaseAssetSymbol:YGEM-B61M QuoteAssetSymbol:BUSD-BAF ListPrice:0.00100000 TickSize:0.00000001 LotSize:0.00010000}"
I[2023-07-11|19:12:45.844] Successfully Loaded order snapshot module=main pair="{BaseAssetSymbol:YNT2-C51M QuoteAssetSymbol:BNB ListPrice:10.00000000 TickSize:0.00001000 LotSize:0.00100000}"
I[2023-07-11|19:12:45.844] Recovered active orders. Snapshot is fully loaded module=main
I[2023-07-11|19:12:45.844] Initialized Block Store for replay module=dex fromHeight=41641219 toHeight=41650000
I[2023-07-11|19:12:45.845] Relaying block for order book module=dex height=41641220
I[2023-07-11|19:12:45.845] replayed all tx. Starting match module=dex height=41641220
I[2023-07-11|19:12:45.845] No order comes in for the block module=dexkeeper
I[2023-07-11|19:12:45.845] Relaying block for order book module=dex height=41641221
I[2023-07-11|19:12:45.845] replayed all tx. Starting match module=dex height=41641221
I[2023-07-11|19:12:45.845] No order comes in for the block module=dexkeeper
REMOVED PART OF THE LOG, SO IT'S NOT THAT BIG
I[2023-07-11|19:13:37.075] Relaying block for order book module=dex height=41649999
I[2023-07-11|19:13:37.075] replayed all tx. Starting match module=dex height=41649999
I[2023-07-11|19:13:37.075] No order comes in for the block module=dexkeeper
I[2023-07-11|19:13:37.075] Relaying block for order book module=dex height=41650000
I[2023-07-11|19:13:37.075] replayed all tx. Starting match module=dex height=41650000
I[2023-07-11|19:13:37.075] No order comes in for the block module=dexkeeper
I[2023-07-11|19:13:37.368] Located day to load breathe block height module=dexkeeper epochDay="]L\u0000\u0000\u0000\u0000\u0000\u0000"
I[2023-07-11|19:13:37.368] Loaded breathe block height module=dexkeeper height=41641219
I[2023-07-11|19:13:37.368] get last breathe block height module=main height=41641219
I[2023-07-11|19:13:37.368] reload latest snapshot module=statesync height=41641219
I[2023-07-11|19:13:37.450] ABCI Handshake App Info module=consensus height=41650000 hash=E72E007C0DE98D8557538AD3C12CA543DD93C9AEAA06D481FAD295BF795EB7A5 software-version= protocol-version=0
I[2023-07-11|19:13:37.450] ABCI Replay Blocks module=consensus appHeight=41650000 storeHeight=41650001 stateHeight=41650000
I[2023-07-11|19:13:37.450] Replay last block using real app module=consensus
ERROR: error during handshake: error on replay: Wrong Block.Header.AppHash. Expected E72E007C0DE98D8557538AD3C12CA543DD93C9AEAA06D481FAD295BF795EB7A5, got 465A524C0D00723A8C0F40840149B355FAED5909592E8CD47E3840F47A4457D9
Hi there, Thanks for more info and logs.
As mentioned, Can you try to stop your node and re-start it after a while (not immediately) and let it run for a while and see if it works.
Also, Have you seen similar following log lines in your logs?:
I[2023-07-11|19:55:39.023] FeeCalculation distributeFeeBEP159 module=main height=41670629 prevBlockFee=
I[2023-07-11|19:55:39.025] Executed block module=state height=41670629 validTxs=0 invalidTxs=0
I[2023-07-11|19:55:39.027] Committed state module=state height=41670629 txs=0 appHash=3EBBC6C01AA914EF6B51FE73BAA41B4CE269971E4D355D3C192691593D5278E3
I[2023-07-11|19:55:39.033] FeeCalculation distributeFeeBEP159 module=main height=41670630 prevBlockFee=
I[2023-07-11|19:55:39.036] Executed block module=state height=41670630 validTxs=0 invalidTxs=0
I[2023-07-11|19:55:39.037] Committed state module=state height=41670630 txs=0 appHash=3EBBC6C01AA914EF6B51FE73BAA41B4CE269971E4D355D3C192691593D5278E3
I[2023-07-11|19:55:39.044] FeeCalculation distributeFeeBEP159 module=main height=41670631 prevBlockFee=
I[2023-07-11|19:55:39.046] Executed block module=state height=41670631 validTxs=0 invalidTxs=0
I[2023-07-11|19:55:39.047] Committed state module=state height=41670631 txs=0 appHash=3EBBC6C01AA914EF6B51FE73BAA41B4CE269971E4D355D3C192691593D5278E3
I stopped one of my nodes and waited for hour. The outcome is the same. What difference will it make if I am not doing anything different?
I am not seeing those logs since my nodes are panicking with this error:
I[2023-07-11|21:06:45.260] Relaying block for order book module=dex height=41649999
I[2023-07-11|21:06:45.260] replayed all tx. Starting match module=dex height=41649999
I[2023-07-11|21:06:45.260] No order comes in for the block module=dexkeeper
I[2023-07-11|21:06:45.260] Relaying block for order book module=dex height=41650000
I[2023-07-11|21:06:45.260] replayed all tx. Starting match module=dex height=41650000
I[2023-07-11|21:06:45.260] No order comes in for the block module=dexkeeper
I[2023-07-11|21:06:45.687] Located day to load breathe block height module=dexkeeper epochDay="]L\u0000\u0000\u0000\u0000\u0000\u0000"
I[2023-07-11|21:06:45.687] Loaded breathe block height module=dexkeeper height=41641219
I[2023-07-11|21:06:45.687] get last breathe block height module=main height=41641219
I[2023-07-11|21:06:45.688] reload latest snapshot module=statesync height=41641219
I[2023-07-11|21:06:45.770] ABCI Handshake App Info module=consensus height=41650000 hash=E72E007C0DE98D8557538AD3C12CA543DD93C9AEAA06D481FAD295BF795EB7A5 software-version= protocol-version=0
I[2023-07-11|21:06:45.770] ABCI Replay Blocks module=consensus appHeight=41650000 storeHeight=41650001 stateHeight=41650000
I[2023-07-11|21:06:45.770] Replay last block using real app module=consensus
ERROR: error during handshake: error on replay: Wrong Block.Header.AppHash. Expected E72E007C0DE98D8557538AD3C12CA543DD93C9AEAA06D481FAD295BF795EB7A5, got 465A524C0D00723A8C0F40840149B355FAED5909592E8CD47E3840F47A4457D9
I wiped the data I had in one of my nodes and restarted. The node runs fine, but that means I loose the block history I had in that node.
There is definitely something wrong with the code here. Every time there's a hard fork, the same issue happens.
Same thing happened with our mainnet nodes a month ago (#960) and no solution was given.
Let me check this internally with the team and we or someone from team will get back to you.
@nmiceli-simtlix For any hardfork, you should 1) stop your node, 2) update app.toml, 3) then start with the new binary. And these steps must be done before the hark fork height. Meanwhile, make sure the config file is updated before binary update, otherwise it will not pick up the new configs.
To fix the current issue, you can try following steps:
-
Download the tool state-recover from release binary for fixing the upgrade issue https://github.com/bnb-chain/node/releases
-
Backup your home directory
-
Since the height of the upgrade is set to be 41,650,000, you can run the following command to recover the state to just one block before the upgrade, which is 41,649,999:
./state_recover 41649999 <your_home_path> -
Restart with the version v0.10.15 or v0.10.16 of bnbchaind
bnbchaind start --home