can not replay block 90913098, get appHash mismatch

[zoro]

refer to issue bnb-chain/node-binary#204.

For now this issue affect one of our Uganda node, and two project teams.

version: Binance Chain Release: 0.6.3-hf.1;Binance Chain Commit: a5abb0f; Cosmos Release: v0.25.0-binance.21; Tendermint Release: v0.32.3-binance.1;

we have excluded following condition:

1. Exceed open file limit. Because the end block result and tx result is exactly the same, the tx result is ok.
2. Wrong binary version or config file. Exactly the same.

More:

1. We try to reset disaster node to height 90910000, who have already passed 90913098, and start it without any other change, the node failed at the height 90913098.
2. We try to do it in another witness, the same result. And no matter reset how many times, still failed.
3. Compare the rootMulti store of height 90913098 and 90913097, only atomic_swap and acc changed.

{"Version":90913098,"StoreInfos":[{"Name":"time_lock","Core":{"CommitID":{"Version":90913098,"Hash":"9DgSAgIxRXyqNqbit2rJQjOhtiIcivXzku3RhYLlab8="}}},{"Name":"main","Core":{"CommitID":{"Version":90913098,"Hash":null}}},{"Name":"dex","Core":{"CommitID":{"Version":90913098,"Hash":"bzv4xwAKTEqDq8ogcgCOtSeJahNcOJsfmcJhiP6yzro="}}},{"Name":"atomic_swap","Core":{"CommitID":{"Version":90913098,"Hash":"NQUnS6VHU/Z9Bj/bTKoX0NeDeH4+Q/w1GUAOUZx8KTU="}}},{"Name":"params","Core":{"CommitID":{"Version":90913098,"Hash":"PPMWYUY9nEpWkXSjvEhC4LScFeyYLRgzcB6Ix5wVlzg="}}},{"Name":"gov","Core":{"CommitID":{"Version":90913098,"Hash":"MsYyDHk4FrGqXinikS5LusWrMCVy/oI11QbF3jCmN2w="}}},{"Name":"stake","Core":{"CommitID":{"Version":90913098,"Hash":"dpOEm2eR9cJv+G3kT/rEartOTRXh11TL1ocK5kbF0hE="}}},{"Name":"val","Core":{"CommitID":{"Version":90913098,"Hash":null}}},{"Name":"acc","Core":{"CommitID":{"Version":90913098,"Hash":"UQA6OlByTaFZHx73pEXXh/nfiITMQjICwBSIbxPY4ss="}}},{"Name":"tokens","Core":{"CommitID":{"Version":90913098,"Hash":"wz28H34fJ5GHlRFTCLFLwuzUTR7aqMewmKJblp9k/Sg="}}},{"Name":"pairs","Core":{"CommitID":{"Version":90913098,"Hash":"qAgXWjg2yKWnwV4wrU3xuqjw0E2FTp0iP86nfWz0/MY="}}}]}

{"Version":90913097,"StoreInfos":[{"Name":"atomic_swap","Core":{"CommitID":{"Version":90913097,"Hash":"qN703D3AFLy5Zpt5oqslwd8zEV1E7pYpEZ9Xq140Iik="}}},{"Name":"params","Core":{"CommitID":{"Version":90913097,"Hash":"PPMWYUY9nEpWkXSjvEhC4LScFeyYLRgzcB6Ix5wVlzg="}}},{"Name":"gov","Core":{"CommitID":{"Version":90913097,"Hash":"MsYyDHk4FrGqXinikS5LusWrMCVy/oI11QbF3jCmN2w="}}},{"Name":"time_lock","Core":{"CommitID":{"Version":90913097,"Hash":"9DgSAgIxRXyqNqbit2rJQjOhtiIcivXzku3RhYLlab8="}}},{"Name":"main","Core":{"CommitID":{"Version":90913097,"Hash":null}}},{"Name":"dex","Core":{"CommitID":{"Version":90913097,"Hash":"bzv4xwAKTEqDq8ogcgCOtSeJahNcOJsfmcJhiP6yzro="}}},{"Name":"acc","Core":{"CommitID":{"Version":90913097,"Hash":"NjRARXyNPyfXcLqadX3a5Stu6PdV7K0roCQQYEPMWWU="}}},{"Name":"tokens","Core":{"CommitID":{"Version":90913097,"Hash":"wz28H34fJ5GHlRFTCLFLwuzUTR7aqMewmKJblp9k/Sg="}}},{"Name":"pairs","Core":{"CommitID":{"Version":90913097,"Hash":"qAgXWjg2yKWnwV4wrU3xuqjw0E2FTp0iP86nfWz0/MY="}}},{"Name":"stake","Core":{"CommitID":{"Version":90913097,"Hash":"dpOEm2eR9cJv+G3kT/rEartOTRXh11TL1ocK5kbF0hE="}}},{"Name":"val","Core":{"CommitID":{"Version":90913097,"Hash":null}}}]}

4. The all key-value in atomic_swap store looks fine.

key 0197a56b8db06ade0fd9a17e5260386401a9d500dfe0961220ff904068e9e9875f, value 0a14c219ec0efa1e0b5e53137b6cc96bc98c3ae826f312142fee2c3082d3ed905d4ee93782a058d66600464b1a090a03424e4210a08d062a0a3130303030303a424e42322b6b6176613167307179776b78366d74356a6d7665667636687337633768333333716173356b7336336136743a20d22ff4cd448ee400fd73a62757f03d0550c29bcb7a3af0a615c8ef4d74c35fe0488287c7f605500158dfc0ad2b60037001
key 02c219ec0efa1e0b5e53137b6cc96bc98c3ae826f30000000000000003, value 97a56b8db06ade0fd9a17e5260386401a9d500dfe0961220ff904068e9e9875f
key 032fee2c3082d3ed905d4ee93782a058d66600464b0000000000000003, value 97a56b8db06ade0fd9a17e5260386401a9d500dfe0961220ff904068e9e9875f
key 05, value 0000000000000004

5. We use state_sync to sync from 90891900(breath block), and failed at 90913098. Try use accout_view tool to check the relative account: account number 2(the proposer) , the msg sender and 293906(the system atomic swap addr), every thing looks fine.

The embarrassing thing is that we get no chance to compare the correct commit id with the wrong one, once use reset tool and sync again, it will fail too. But we do not think it is an issue of reset tool, if the root cause is the reset tool, the apphash mismatch should happen at the next block of the reseted height, but it did not.

Please post here if you know any suspicious code.