Hard fork failed to apply using v1.1.10
frangrech opened this issue · comments
System information
Erigon version: erigon version 1.1.10
OS & Version: Linux
Commit hash: 4e041df
Erigon Command (with flags/config): erigon --p2p.protocol=66 --txpool.disable --log.console.verbosity=info --snapshots=true --chain=bsc --http --http.addr 0.0.0.0 --http.port "8545" --http.corsdomain "*" --http.api="eth,net,web3,debug,trace,erigon" --ws --metrics --metrics.addr 0.0.0.0 --maxpeers 50 --rpc.gascap=0 --datadir /data/ --db.pagesize=16k --bodies.cache=2294967296
Expected behaviour
Node processes new blocks post-fork normally
Actual behaviour
Node is not able to proceed syncing after fork.
Steps to reproduce the behaviour
Running v1.1.10. Also tried upgrading to v1.1.12 but to no avail
Backtrace
At hard fork:
[INFO] [01-23|08:00:10.704] [2/15 Headers] Waiting for headers... from=35490443
[INFO] [01-23|08:00:10.718] [2/15 Headers] Processed highest inserted=35490447 age=1s
[INFO] [01-23|08:00:10.785] Apply upgrade kepler at height 35490444 system-contract-upgrade=default
[INFO] [01-23|08:00:10.787] Upgrade contract 0x0000000000000000000000000000000000001000 to commit https://github.com/bnb-chain/bsc-genesis-contract/commit/ff75c61ecd457c1370327e136b38c7a0d63ca821 system-contract-upgrade=default
[INFO] [01-23|08:00:10.788] Upgrade contract 0x0000000000000000000000000000000000001001 to commit https://github.com/bnb-chain/bsc-genesis-contract/commit/ff75c61ecd457c1370327e136b38c7a0d63ca821 system-contract-upgrade=default
[INFO] [01-23|08:00:10.790] Upgrade contract 0x0000000000000000000000000000000000001002 to commit https://github.com/bnb-chain/bsc-genesis-contract/commit/ff75c61ecd457c1370327e136b38c7a0d63ca821 system-contract-upgrade=default
[INFO] [01-23|08:00:12.214] [7/15 Execution] Completed on block=35490447
[EROR] [01-23|08:00:15.792] [9/15 IntermediateHashes] Wrong trie root of block 35490447: ac30a906bba2266dbb300f3747f20a8a26e93ed0c59bd4183025e4c955aa5ab1, expected (from header): c1a5c0adcf543394217b1d88458bffcc8599180ae6ea95c611d78ca4e164e5c8. Block hash: 62abed0636f07ea3a122d686756c079508c8e00f9a6404aba1cae4f2ddc6e437
[WARN] [01-23|08:00:15.792] Unwinding due to incorrect root hash to=35490445
[INFO] [01-23|08:00:15.792] UnwindTo block=35490445 bad_block_hash=0x62abed0636f07ea3a122d686756c079508c8e00f9a6404aba1cae4f2ddc6e437
[INFO] [01-23|08:00:15.792] [8/15 HashState] Unwinding started from=35490447 to=35490445 storage=false codes=true
[INFO] [01-23|08:00:15.796] [8/15 HashState] Unwinding started from=35490447 to=35490445 storage=false codes=false
[INFO] [01-23|08:00:15.800] [8/15 HashState] Unwinding started from=35490447 to=35490445 storage=true codes=false
[INFO] [01-23|08:00:15.812] [7/15 Execution] Unwind Execution from=35490447 to=35490445
[INFO] [01-23|08:00:16.011] RPC Daemon notified of new headers from=35490443 to=35490447 hash=0x0000000000000000000000000000000000000000000000000000000000000000 header sending=25.762µs log sending=436ns
[INFO] [01-23|08:00:16.011] Commit cycle in=172.432447ms
[INFO] [01-23|08:00:16.011] Timings (slower than 50ms) Execution=1.43s HashState=1.528s IntermediateHashes=2.049s
[INFO] [01-23|08:00:16.011] Tables PlainState=287.2GB AccountChangeSet=658.9GB StorageChangeSet=1.5TB BlockTransaction=5.5GB TransactionLog=2.7TB FreeList=134.5MB ReclaimableSpace=537.8GB
[INFO] [01-23|08:00:16.219] [2/15 Headers] Waiting for headers... from=35490445
[WARN] [01-23|08:00:16.248] [downloader] Rejected header marked as bad hash=0x62abed0636f07ea3a122d686756c079508c8e00f9a6404aba1cae4f2ddc6e437 height=35490447
[WARN] [01-23|08:00:17.444] [downloader] Rejected header marked as bad hash=0x62abed0636f07ea3a122d686756c079508c8e00f9a6404aba1cae4f2ddc6e437 height=35490447
[WARN] [01-23|08:00:18.677] [downloader] Rejected header marked as bad hash=0x62abed0636f07ea3a122d686756c079508c8e00f9a6404aba1cae4f2ddc6e437 height=35490447
[WARN] [01-23|08:00:19.091] [downloader] Rejected header marked as bad hash=0x62abed0636f07ea3a122d686756c079508c8e00f9a6404aba1cae4f2ddc6e437 height=35490447
[WARN] [01-23|08:00:20.212] [downloader] Rejected header marked as bad hash=0x62abed0636f07ea3a122d686756c079508c8e00f9a6404aba1cae4f2ddc6e437 height=35490447
Tried unwinding, but to no avail:
./build/bin/integration state_stages --unwind 1500 --chain=bsc --datadir=/data
INFO[01-23|09:11:55.213] logging to file system log dir=/data/logs file prefix=erigon log level=info json=false
INFO[01-23|09:11:59.527] [snapshots] Blocks Stat blocks=35400k indices=35400k alloc=2.4GB sys=2.6GB
INFO[01-23|09:11:59.960] [9/15 IntermediateHashes] Generating intermediate hashes from=35490443 to=35490709
EROR[01-23|09:12:07.052] [9/15 IntermediateHashes] Wrong trie root of block 35490709: 7411a51cef7bb11141dca511e59506f80f17aa78a38b33197bd115077910a610, expected (from header): 0da332e8767a898b9c875615c355912eb6062e3d87490313219237778ec548fc. Block hash: cbff3ffdd9f120a797b8cfd430ebb1140655552c61c3b10edf81f6df5b5f576c
WARN[01-23|09:12:07.052] Unwinding due to incorrect root hash to=35490576
INFO[01-23|09:12:07.052] UnwindTo block=35490576 bad_block_hash=0xcbff3ffdd9f120a797b8cfd430ebb1140655552c61c3b10edf81f6df5b5f576c
INFO[01-23|09:12:07.053] [8/15 HashState] Unwinding started from=35490709 to=35490576 storage=false codes=true
INFO[01-23|09:12:07.220] [8/15 HashState] Unwinding started from=35490709 to=35490576 storage=false codes=false
INFO[01-23|09:12:07.565] [8/15 HashState] Unwinding started from=35490709 to=35490576 storage=true codes=false
INFO[01-23|09:12:08.441] [7/15 Execution] Unwind Execution from=35490709 to=35490576
INFO[01-23|09:12:13.474] UnwindTo block=35489076 bad_block_hash=0x0000000000000000000000000000000000000000000000000000000000000000
INFO[01-23|09:14:31.064] [12/15 StorageHistoryIndex] Unwind done in=1m8.906898684s
INFO[01-23|09:15:04.936] [10/15 CallTraces] Unwind from=35490443 to=35489076
INFO[01-23|09:16:02.386] [8/15 HashState] Unwinding started from=35490576 to=35489076 storage=false codes=true
INFO[01-23|09:16:04.088] [8/15 HashState] Unwinding started from=35490576 to=35489076 storage=false codes=false
INFO[01-23|09:16:27.455] [8/15 HashState] Unwinding started from=35490576 to=35489076 storage=true codes=false
INFO[01-23|09:17:01.815] [8/15 HashState] ETL [2/2] Loading into=HashedStorage progress=92
INFO[01-23|09:17:30.389] [8/15 HashState] ETL [2/2] Loading into=HashedStorage progress=99
INFO[01-23|09:17:31.602] [8/15 HashState] Unwind done in=1m29.216725108s
INFO[01-23|09:17:31.602] [9/15 IntermediateHashes] Unwinding from=35490443 to=35489076 csbucket=AccountChangeSet
INFO[01-23|09:17:44.967] [9/15 IntermediateHashes] Unwinding from=35490443 to=35489076 csbucket=StorageChangeSet
INFO[01-23|09:18:38.494] [9/15 IntermediateHashes] Calculating Merkle root current key=9f531056
INFO[01-23|09:18:59.247] [9/15 IntermediateHashes] Flushed buffer file name=/data/temp/erigon-sortable-buf-597632521
INFO[01-23|09:19:12.790] [9/15 IntermediateHashes] Calculating Merkle root current key=e9dae3d7
EROR[01-23|09:19:16.205] [9/15 IntermediateHashes] wrong trie root: 04d23447abd157df0bf641e78cbcdf0f26c5f9d318a65114c968847c26a5cdce, expected (from header): f1a785a5ff0b707fede08e23572ac2425796e257ea06612bfa0da535a7756f95
Is there a proper way to unwind and get it to re-apply the hard fork using a newer version? I thought the hard fork was supported by v1.1.10. Thank you
Hi there, Try out following:
- Try: integration stage_headers —reset. JFYR: #289 (comment)
- Upgrade to v1.1.12
- Lastly, you can also refer this solution: #288 (comment) and underneath conversation.
There are some bugs about upgrade fixed in v1.1.12.
try: ./build/bin/integration stage_hash_state --datadir= --reset
./build/bin/integration stage_trie --datadir= --reset
I've tried all of the above (and much more) but to no avail - still getting:
[INFO] [01-24|10:13:42.043] [6/15 Senders] DONE in=4m21.667983656s
[INFO] [01-24|10:13:42.043] [7/15 Execution] Blocks execution from=35486576 to=35521047
[WARN] [01-24|10:13:42.186] [7/15 Execution] Execution failed block=35486577 hash=0x96cc040971b068bd23e993d7fea65240743cc5882a92f8d69c52f9ccc39ce47b err="mismatched receipt headers for block 35486577 (0xd30e2808d1fb477d428ed784dfdf72b871985fe88d6e09d1af8b40ca6acc49e1 != 0x154724b5ecb9e7dcf3becf3b37345781701fbe799f2d2d91ff2128a363608692)"
[INFO] [01-24|10:13:42.186] UnwindTo block=35486576 bad_block_hash=0x96cc040971b068bd23e993d7fea65240743cc5882a92f8d69c52f9ccc39ce47b
[INFO] [01-24|10:13:42.187] [7/15 Execution] Completed on block=35486576
[INFO] [01-24|10:13:47.159] [2/15 Headers] Waiting for headers... from=35486576
[WARN] [01-24|10:13:48.449] [downloader] Rejected header marked as bad hash=0x96cc040971b068bd23e993d7fea65240743cc5882a92f8d69c52f9ccc39ce47b height=35486577
[WARN] [01-24|10:13:49.500] [downloader] Rejected header marked as bad hash=0x96cc040971b068bd23e993d7fea65240743cc5882a92f8d69c52f9ccc39ce47b height=35486577
[WARN] [01-24|10:13:51.132] [downloader] Rejected header marked as bad hash=0x96cc040971b068bd23e993d7fea65240743cc5882a92f8d69c52f9ccc39ce47b height=35486577
[WARN] [01-24|10:13:52.716] [downloader] Rejected header marked as bad hash=0x96cc040971b068bd23e993d7fea65240743cc5882a92f8d69c52f9ccc39ce47b height=35486577
...
[INFO] [01-24|10:14:07.160] [2/15 Headers] No block headers to write in this log period block number=35486576
I tried unwinding stage_exec but whichever block it has to resume from, I still get the same error. Will be downloading the newest snapshot and syncing from that.