Syncing from 0 errors at different blocks
roks0n opened this issue · comments
II can't say with certainty but I think it happens at different heights, I believe that 1st time when I synced from 0 it errored at a different height.
Example log:
blockchain_1 | 2019-03-29T14:31:04.126688190Z Started processing block 291310634791671175
blockchain_1 | 2019-03-29T14:31:04.126775964Z Last block height: 90461
blockchain_1 | 2019-03-29T14:31:04.127196485Z Delegate genesis_43 (021b0f58eca7f123428a8647ffe0644a9454c510f066d3864c27d8c7ad8f5a8aa4) allowed to forge block 90462
blockchain_1 | 2019-03-29T14:31:04.127313039Z 11811961852261780043 == 11811961852261780043
blockchain_1 | 2019-03-29T14:31:04.127568406Z follows_previous True
blockchain_1 | 2019-03-29T14:31:04.127649010Z is_plus_one True
blockchain_1 | 2019-03-29T14:31:04.127888575Z is_after_previous_slot True
blockchain_1 | 2019-03-29T14:31:04.127990451Z ====== Handle apply block ======
blockchain_1 | 2019-03-29T14:31:04.128219268Z Apply round next height: 90463
blockchain_1 | 2019-03-29T14:31:04.128315730Z Current round 1774
blockchain_1 | 2019-03-29T14:31:04.128444313Z Saving block 291310634791671175
blockchain_1 | 2019-03-29T14:31:04.130990310Z Block 291310634791671175 was accepted
blockchain_1 | 2019-03-29T14:31:04.131012374Z
blockchain_1 | 2019-03-29T14:31:04.131062479Z
blockchain_1 | 2019-03-29T14:31:04.131141447Z Started processing block 9004138333315911259
blockchain_1 | 2019-03-29T14:31:04.131242983Z Last block height: 90462
blockchain_1 | 2019-03-29T14:31:04.131685963Z Delegate dulhunty (0315130b2fae02070ff6fb867f189456a31caddbc7b52ceac0147601ac88adf9cb) allowed to forge block 90463
blockchain_1 | 2019-03-29T14:31:04.131791169Z 291310634791671175 == 291310634791671175
blockchain_1 | 2019-03-29T14:31:04.131940622Z follows_previous True
blockchain_1 | 2019-03-29T14:31:04.132141323Z is_plus_one True
blockchain_1 | 2019-03-29T14:31:04.132294986Z is_after_previous_slot True
blockchain_1 | 2019-03-29T14:31:04.132380591Z ====== Handle apply block ======
blockchain_1 | 2019-03-29T14:31:04.132612844Z Apply round next height: 90464
blockchain_1 | 2019-03-29T14:31:04.132716496Z Current round 1774
blockchain_1 | 2019-03-29T14:31:04.132822060Z Saving block 9004138333315911259
blockchain_1 | 2019-03-29T14:31:04.134809821Z Block 9004138333315911259 was accepted
blockchain_1 | 2019-03-29T14:31:04.134901344Z
blockchain_1 | 2019-03-29T14:31:04.134914577Z
blockchain_1 | 2019-03-29T14:31:04.135008266Z Started processing block 10185777960505886816
blockchain_1 | 2019-03-29T14:31:04.135110225Z Last block height: 90463
blockchain_1 | 2019-03-29T14:31:04.135539331Z ['Invalid number of transactions', 'Invalid total amount', 'Invalid total fee', 'Invalid payload hash']
blockchain_1 | 2019-03-29T14:31:04.135556637Z Block 10185777960505886816 was rejected
blockchain_1 | 2019-03-29T14:31:04.135711930Z {'id': '10185777960505886816', 'idHex': '8d5b2716a8f4c860', 'timestamp': 45759274, 'version': 0, 'height': 90464, 'previousBlockHex': '7cf5201bb1c5ce5b', 'previousBlock': '9004138333315911259', 'numberOfTransactions': 5, 'totalAmount': '199998498', 'totalFee': '1500', 'reward': '200000000', 'payloadLength': 160, 'payloadHash': '741f25d46667bd22536516df3aa0778914039a9d9a7024b7d38dae8b40f0e77d', 'generatorPublicKey': '037997a6553ea8073eb199e9f5ff23b8f0892e79433ef35e13966e0a12849d02e3', 'blockSignature': '304402206de84ae8d279a40a85d4c5ed813e888b79b1d17ff7314c3ef17b760d7b1b216a022023578f4e288605f131d6bcc2d22ef953ccceadb506c2aca0308fd1c4be537943', 'transactions': []}
blockchain_1 | 2019-03-29T14:31:04.135817112Z Block 10185777960505886816 was rejected. Skipping all other blocks in this batch
blockchain_1 | 2019-03-29T14:31:04.135962420Z Traceback (most recent call last):
blockchain_1 | 2019-03-29T14:31:04.135994954Z File "chain/blockchain/start.py", line 6, in <module>
blockchain_1 | 2019-03-29T14:31:04.136283312Z bc.start()
blockchain_1 | 2019-03-29T14:31:04.136355956Z File "/home/app/chain/blockchain/blockchain.py", line 150, in start
blockchain_1 | 2019-03-29T14:31:04.136679083Z raise e # TODO:
blockchain_1 | 2019-03-29T14:31:04.136736917Z File "/home/app/chain/blockchain/blockchain.py", line 136, in start
blockchain_1 | 2019-03-29T14:31:04.137021717Z self.start_syncing()
blockchain_1 | 2019-03-29T14:31:04.137057048Z File "/home/app/chain/blockchain/blockchain.py", line 161, in start_syncing
blockchain_1 | 2019-03-29T14:31:04.137343860Z self.sync_blocks(last_block)
blockchain_1 | 2019-03-29T14:31:04.137378742Z File "/home/app/chain/blockchain/blockchain.py", line 204, in sync_blocks
blockchain_1 | 2019-03-29T14:31:04.137632945Z raise Exception(msg)
blockchain_1 | 2019-03-29T14:31:04.137783410Z Exception: Block 10185777960505886816 was rejected. Skipping all other blocks in this batch
Just to give an example of another block that caused the same issue:
blockchain_1 | 2019-03-30T12:24:01.172787855Z ====== Handle apply block ======
blockchain_1 | 2019-03-30T12:24:01.172856617Z Apply round next height: 145240
blockchain_1 | 2019-03-30T12:24:01.172862481Z Current round 2848
blockchain_1 | 2019-03-30T12:24:01.172866404Z Saving block 11781820942017196620
blockchain_1 | 2019-03-30T12:24:01.174137156Z Block 11781820942017196620 was accepted
blockchain_1 | 2019-03-30T12:24:01.174149054Z
blockchain_1 | 2019-03-30T12:24:01.174153143Z
blockchain_1 | 2019-03-30T12:24:01.174157027Z Started processing block 567452547527180521
blockchain_1 | 2019-03-30T12:24:01.174161075Z Last block height: 145239
blockchain_1 | 2019-03-30T12:24:01.174418715Z Delegate arkmoon (029e4e8188fd4f0dd9a73407fe995cb9bab86405da1246b3cef46065e301f1f9eb) allowed to forge block 145240
blockchain_1 | 2019-03-30T12:24:01.174427454Z 11781820942017196620 == 11781820942017196620
blockchain_1 | 2019-03-30T12:24:01.174457004Z follows_previous True
blockchain_1 | 2019-03-30T12:24:01.174461893Z is_plus_one True
blockchain_1 | 2019-03-30T12:24:01.174465919Z is_after_previous_slot True
blockchain_1 | 2019-03-30T12:24:01.174469740Z ====== Handle apply block ======
blockchain_1 | 2019-03-30T12:24:01.174548952Z Apply round next height: 145241
blockchain_1 | 2019-03-30T12:24:01.174580613Z Current round 2848
blockchain_1 | 2019-03-30T12:24:01.174584952Z Saving block 567452547527180521
blockchain_1 | 2019-03-30T12:24:01.175825713Z Block 567452547527180521 was accepted
blockchain_1 | 2019-03-30T12:24:01.175837290Z
blockchain_1 | 2019-03-30T12:24:01.175841487Z
blockchain_1 | 2019-03-30T12:24:01.175874914Z Started processing block 6126200342167803987
blockchain_1 | 2019-03-30T12:24:01.175879965Z Last block height: 145240
blockchain_1 | 2019-03-30T12:24:01.176203622Z ['Invalid number of transactions', 'Invalid total amount', 'Invalid total fee', 'Invalid payload hash']
blockchain_1 | 2019-03-30T12:24:01.176212764Z Block 6126200342167803987 was rejected
blockchain_1 | 2019-03-30T12:24:01.176261528Z {'id': '6126200342167803987', 'idHex': '5504a2c0d1c9d853', 'timestamp': 46207250, 'version': 0, 'height': 145241, 'previousBlockHex': '07dfff15e23f80e9', 'previousBlock': '567452547527180521', 'numberOfTransactions': 2, 'totalAmount': '1000000000', 'totalFee': '2', 'reward': '200000000', 'payloadLength': 64, 'payloadHash': 'e385f67469acc02b337ac875824fcd0e3f43d15ce0c2bef71d7511906b8dd89f', 'generatorPublicKey': '03bbfb43ecb5a54a1e227bb37b5812b5321213838d376e2b455b6af78442621dec', 'blockSignature': '304402207820b36a106cf5315b97bad634468900375264528f9e4f71f1269a367ab85357022040bdfc7776eeb92b9a90b6e2e02a0399b92be1af4d24a74b7d47c74a972d45b4', 'transactions': []}
blockchain_1 | 2019-03-30T12:24:01.176288442Z Block 6126200342167803987 was rejected. Skipping all other blocks in this batch
blockchain_1 | 2019-03-30T12:24:01.176334187Z Traceback (most recent call last):
blockchain_1 | 2019-03-30T12:24:01.176340238Z File "chain/blockchain/start.py", line 6, in <module>
blockchain_1 | 2019-03-30T12:24:01.176483922Z bc.start()
blockchain_1 | 2019-03-30T12:24:01.176490174Z File "/home/app/chain/blockchain/blockchain.py", line 150, in start
blockchain_1 | 2019-03-30T12:24:01.176641844Z raise e # TODO:
blockchain_1 | 2019-03-30T12:24:01.176647883Z File "/home/app/chain/blockchain/blockchain.py", line 136, in start
blockchain_1 | 2019-03-30T12:24:01.176789079Z self.start_syncing()
blockchain_1 | 2019-03-30T12:24:01.176795290Z File "/home/app/chain/blockchain/blockchain.py", line 161, in start_syncing
blockchain_1 | 2019-03-30T12:24:01.176939497Z self.sync_blocks(last_block)
blockchain_1 | 2019-03-30T12:24:01.176945497Z File "/home/app/chain/blockchain/blockchain.py", line 204, in sync_blocks
blockchain_1 | 2019-03-30T12:24:01.177096775Z raise Exception(msg)
blockchain_1 | 2019-03-30T12:24:01.177127725Z Exception: Block 6126200342167803987 was rejected. Skipping all other blocks in this batch
Restart seems to fix the issue until another block is rejected - always a different one, as I was able to sync from 0 once with only restarting the sync process once.
blockchain_1 | 2019-03-30T17:30:53.444493929Z ====== Handle apply block ======
blockchain_1 | 2019-03-30T17:30:53.444678886Z Apply round next height: 229701
blockchain_1 | 2019-03-30T17:30:53.444757967Z Current round 4504
blockchain_1 | 2019-03-30T17:30:53.444856549Z Saving block 115042344007529380
blockchain_1 | 2019-03-30T17:30:53.446486975Z Block 115042344007529380 was accepted
blockchain_1 | 2019-03-30T17:30:53.446501974Z
blockchain_1 | 2019-03-30T17:30:53.446506277Z
blockchain_1 | 2019-03-30T17:30:53.446535812Z Started processing block 16320702779932281256
blockchain_1 | 2019-03-30T17:30:53.446658870Z Last block height: 229700
blockchain_1 | 2019-03-30T17:30:53.447025614Z Delegate arklabs (0394685435d7331d178effe91d6b101ce7c4a6e03d2a96cfd5be1fffb0ae156e58) allowed to forge block 229701
blockchain_1 | 2019-03-30T17:30:53.447037289Z 115042344007529380 == 115042344007529380
blockchain_1 | 2019-03-30T17:30:53.447219185Z follows_previous True
blockchain_1 | 2019-03-30T17:30:53.447363579Z is_plus_one True
blockchain_1 | 2019-03-30T17:30:53.447489312Z is_after_previous_slot True
blockchain_1 | 2019-03-30T17:30:53.447558444Z ====== Handle apply block ======
blockchain_1 | 2019-03-30T17:30:53.447765952Z Apply round next height: 229702
blockchain_1 | 2019-03-30T17:30:53.447858247Z Current round 4504
blockchain_1 | 2019-03-30T17:30:53.447950305Z Saving block 16320702779932281256
blockchain_1 | 2019-03-30T17:30:53.449736012Z Block 16320702779932281256 was accepted
blockchain_1 | 2019-03-30T17:30:53.449831893Z
blockchain_1 | 2019-03-30T17:30:53.449842213Z
blockchain_1 | 2019-03-30T17:30:53.449845158Z Started processing block 2349589560048512919
blockchain_1 | 2019-03-30T17:30:53.449898484Z Last block height: 229701
blockchain_1 | 2019-03-30T17:30:53.450315685Z ['Invalid number of transactions', 'Invalid total amount', 'Invalid total fee', 'Invalid payload hash']
blockchain_1 | 2019-03-30T17:30:53.450414876Z Block 2349589560048512919 was rejected
blockchain_1 | 2019-03-30T17:30:53.450532490Z {'id': '2349589560048512919', 'idHex': '209b6b4196a86f97', 'timestamp': 46906528, 'version': 0, 'height': 229702, 'previousBlockHex': 'e27ec8b5229809a8', 'previousBlock': '16320702779932281256', 'numberOfTransactions': 1, 'totalAmount': '77870000000', 'totalFee': '10000000', 'reward': '200000000', 'payloadLength': 32, 'payloadHash': '40eb06a513f384f4ab4ef65d400e1d3acb76a7e55d47b0f43394b00b55476ec6', 'generatorPublicKey': '0349e7e2afb470994a8323e9623a6dab227c69d5f09f1a59991fd92880123ffe75', 'blockSignature': '3044022020440ba379505e7a75a1038beac6f527941ce3609581514d3ed10e33cd079c8702203779069a74b84ebcced0d86bd5add88f33b0e91137193b2379e1d16856b491f6', 'transactions': []}
blockchain_1 | 2019-03-30T17:30:53.450603576Z Block 2349589560048512919 was rejected. Skipping all other blocks in this batch
blockchain_1 | 2019-03-30T17:30:53.450702684Z Traceback (most recent call last):
blockchain_1 | 2019-03-30T17:30:53.450761282Z File "chain/blockchain/start.py", line 6, in <module>
blockchain_1 | 2019-03-30T17:30:53.451027071Z bc.start()
blockchain_1 | 2019-03-30T17:30:53.451030785Z File "/home/app/chain/blockchain/blockchain.py", line 150, in start
blockchain_1 | 2019-03-30T17:30:53.451244576Z raise e # TODO:
blockchain_1 | 2019-03-30T17:30:53.451300994Z File "/home/app/chain/blockchain/blockchain.py", line 136, in start
blockchain_1 | 2019-03-30T17:30:53.451534828Z self.start_syncing()
blockchain_1 | 2019-03-30T17:30:53.451579778Z File "/home/app/chain/blockchain/blockchain.py", line 161, in start_syncing
blockchain_1 | 2019-03-30T17:30:53.451767798Z self.sync_blocks(last_block)
blockchain_1 | 2019-03-30T17:30:53.451835397Z File "/home/app/chain/blockchain/blockchain.py", line 204, in sync_blocks
blockchain_1 | 2019-03-30T17:30:53.452024358Z raise Exception(msg)
blockchain_1 | 2019-03-30T17:30:53.452194782Z Exception: Block 2349589560048512919 was rejected. Skipping all other blocks in this batch
I'll stop reporting these now.
This should solve it: 5040147
After applying the latest fixes I trashed the db, tried again and got exception again. I assume the node is not handling scenarios where there's something wrong with the block yet?
blockchain_1 | 2019-04-02T22:20:36.123876282Z ====== Handle apply block ======
blockchain_1 | 2019-04-02T22:20:36.123996895Z Apply round next height: 930910
blockchain_1 | 2019-04-02T22:20:36.124001330Z Current round 18254
blockchain_1 | 2019-04-02T22:20:36.124015944Z Saving block 9619715565620886705
blockchain_1 | 2019-04-02T22:20:36.131534301Z Block 9619715565620886705 was accepted
blockchain_1 | 2019-04-02T22:20:36.131550536Z
blockchain_1 | 2019-04-02T22:20:36.131553489Z
blockchain_1 | 2019-04-02T22:20:36.131556050Z Started processing block 7832333697578835274
blockchain_1 | 2019-04-02T22:20:36.131558889Z Last block height: 930909
blockchain_1 | 2019-04-02T22:20:36.131817787Z Delegate arkmoon (029e4e8188fd4f0dd9a73407fe995cb9bab86405da1246b3cef46065e301f1f9eb) allowed to forge block 930910
blockchain_1 | 2019-04-02T22:20:36.131822869Z 9619715565620886705 == 9619715565620886705
blockchain_1 | 2019-04-02T22:20:36.131838945Z follows_previous True
blockchain_1 | 2019-04-02T22:20:36.131842228Z is_plus_one True
blockchain_1 | 2019-04-02T22:20:36.131845074Z is_after_previous_slot True
blockchain_1 | 2019-04-02T22:20:36.131847644Z ====== Handle apply block ======
blockchain_1 | 2019-04-02T22:20:36.131947031Z Apply round next height: 930911
blockchain_1 | 2019-04-02T22:20:36.131951175Z Current round 18254
blockchain_1 | 2019-04-02T22:20:36.131953614Z Saving block 7832333697578835274
blockchain_1 | 2019-04-02T22:20:36.134030585Z Block 7832333697578835274 was accepted
blockchain_1 | 2019-04-02T22:20:36.134303572Z Time taken 1:30:38.412947
blockchain_1 | 2019-04-02T22:20:36.135663075Z
blockchain_1 | 2019-04-02T22:20:36.135672450Z
blockchain_1 | 2019-04-02T22:20:36.135675179Z Fetching blocks from height 930910
blockchain_1 | 2019-04-02T22:20:36.137095013Z Got 84 peers from redis
blockchain_1 | 2019-04-02T22:20:36.138176153Z Downloading blocks from height 930910 via 178.128.32.56
blockchain_1 | 2019-04-02T22:20:36.138186023Z http://178.128.32.56:4002/peer/blocks
blockchain_1 | 2019-04-02T22:20:36.138189102Z {'lastBlockHeight': 930910}
blockchain_1 | 2019-04-02T22:20:36.355948830Z 18175717730746410780 == 7832333697578835274
blockchain_1 | 2019-04-02T22:20:36.356164067Z follows_previous False
blockchain_1 | 2019-04-02T22:20:36.356275014Z is_plus_one True
blockchain_1 | 2019-04-02T22:20:36.356401160Z is_after_previous_slot False
blockchain_1 | 2019-04-02T22:20:36.356480530Z chained False
blockchain_1 | 2019-04-02T22:20:36.356675835Z exception False
blockchain_1 | 2019-04-02T22:20:36.356717468Z 18175717730746410780 == 7832333697578835274
blockchain_1 | 2019-04-02T22:20:36.356845657Z follows_previous False
blockchain_1 | 2019-04-02T22:20:36.356951482Z is_plus_one True
blockchain_1 | 2019-04-02T22:20:36.357066477Z is_after_previous_slot False
blockchain_1 | 2019-04-02T22:20:36.357154726Z Last downloaded block: 7832333697578835274
blockchain_1 | 2019-04-02T22:20:36.357232971Z WTF: 930910
blockchain_1 | 2019-04-02T22:20:36.364619192Z Traceback (most recent call last):
blockchain_1 | 2019-04-02T22:20:36.364638099Z File "chain/blockchain/start.py", line 6, in <module>
blockchain_1 | 2019-04-02T22:20:36.365721615Z bc.start()
blockchain_1 | 2019-04-02T22:20:36.365736211Z File "/home/app/chain/blockchain/blockchain.py", line 164, in start
blockchain_1 | 2019-04-02T22:20:36.365889646Z raise e # TODO:
blockchain_1 | 2019-04-02T22:20:36.365895883Z File "/home/app/chain/blockchain/blockchain.py", line 150, in start
blockchain_1 | 2019-04-02T22:20:36.366022149Z self.start_syncing()
blockchain_1 | 2019-04-02T22:20:36.366027788Z File "/home/app/chain/blockchain/blockchain.py", line 175, in start_syncing
blockchain_1 | 2019-04-02T22:20:36.366158801Z self.sync_blocks(last_block)
blockchain_1 | 2019-04-02T22:20:36.366164193Z File "/home/app/chain/blockchain/blockchain.py", line 228, in sync_blocks
blockchain_1 | 2019-04-02T22:20:36.366329920Z "Downloaded block not accepted: {}".format(blocks[0].id)
blockchain_1 | 2019-04-02T22:20:36.366335744Z Exception: Downloaded block not accepted: 11726543974491624121
This should fix this issue: 990da6e
Bear in mind that this might cause other issues, like getting in an infinite loop, downloading the same batch of blocks again and again. If that happens, please raise an issue as I thing we'll need to ban the peer we've downloaded these broken blocks from, but would rather test it first than do something extra at the moment :)