node suddenly stopped working, restart not helping
mindstyle85 opened this issue · comments
here is the log:
release/harmony/node/node.go:276","time":"2019-09-23T09:22:39.392095567Z","message":"Got more transactions"}
{"level":"info","port":"9000","ip":"209.97.180.175","numPeersNow":0,"caller":"/mnt/jenkins/workspace/harmony-release/harmony/node/node_handler.go:580","time":"2019-09-23T09:22:41.369054045Z","message":"No peers, continue"}
{"level":"warn","port":"9000","ip":"209.97.180.175","error":"rpc error: code = Unavailable desc = all SubConns are in TransientFailure, latest connection error: connection error: desc = "transport: Error while dialing dial tcp 34.228.143.91:6000: connect: connection refused"","peerIP":"34.228.143.91","peerPort":"6000","caller":"/mnt/jenkins/workspace/harmony-release/harmony/api/service/syncing/syncing.go:709","time":"2019-09-23T09:22:43.692263741Z","message":"[Sync]GetBlockChainHeight failed"}
{"level":"debug","port":"9000","ip":"209.97.180.175","OtherHeight":15177,"MyHeight":15177,"IsOutOfSync":false,"caller":"/mnt/jenkins/workspace/harmony-release/harmony/api/service/syncing/syncing.go:739","time":"2019-09-23T09:22:43.923313548Z","message":"[SYNC] Checking sync status"}
{"level":"info","port":"9000","ip":"209.97.180.175","numPeersNow":0,"caller":"/mnt/jenkins/workspace/harmony-release/harmony/node/node_handler.go:580","time":"2019-09-23T09:22:46.368978629Z","message":"No peers, continue"}
{"level":"info","port":"9000","ip":"209.97.180.175","numPeersNow":0,"caller":"/mnt/jenkins/workspace/harmony-release/harmony/node/node_handler.go:580","time":"2019-09-23T09:22:51.369081986Z","message":"No peers, continue"}
{"level":"debug","port":"9000","ip":"209.97.180.175","caller":"/mnt/jenkins/workspace/harmony-release/harmony/node/node_handler.go:152","time":"2019-09-23T09:22:51.555345883Z","message":"NET: received message: Node/Transaction"}
{"level":"info","port":"9000","ip":"209.97.180.175","length of newTxs":1,"totalPending":15,"caller":"/mnt/jenkins/workspace/harmony-release/harmony/node/node.go:276","time":"2019-09-23T09:22:51
{"level":"info","port":"9000","ip":"165.22.73.42","publicKey":"e2a2ec8d5f95444203b8ad80f9cd50775c90cfa9b1b0a5fdb95f1c10336aa364200cbce0828b236ef0dd9494d7b21094","caller":"/mnt/jenkins/workspace/harmony-release/harmony/consensus/consensus.go:276","time":"2019-09-23T10:32:09.226959637Z","message":"My Public Key"}
{"level":"warn","port":"9000","ip":"165.22.73.42","hash":"0x00675e82fbe5e7457d3662da1298d21e70db49d4d87ac455952648a95518a227","caller":"/mnt/jenkins/workspace/harmony-release/harmony/core/blockchain.go:284","time":"2019-09-23T10:32:09.286411459Z","message":"Head block missing, resetting chain"}
{"level":"warn","port":"9000","ip":"165.22.73.42","target":0,"caller":"/mnt/jenkins/workspace/harmony-release/harmony/core/blockchain.go:352","time":"2019-09-23T10:32:09.286447272Z","message":"Rewinding blockchain"}
{"level":"info","port":"9000","ip":"165.22.73.42","port":"8000","caller":"/mnt/jenkins/workspace/harmony-release/harmony/internal/memprofiling/lib.go:52","time":"2019-09-23T10:32:22.708523773Z","message":"running mem profiling"}
{"level":"debug","port":"9000","ip":"165.22.73.42","hostID":"QmZJGt7vEbdyZhNkex3rEvbMzwt5JPnfkCEsi4juBGHc7F","port":"9000","id":"QmZJGt7vEbdyZhNkex3rEvbMzwt5JPnfkCEsi4juBGHc7F","addr":"/ip4/0.0.0.0/tcp/9000","PubKey":"e2a2ec8d5f95444203b8ad80f9cd50775c90cfa9b1b0a5fdb95f1c10336aa364200cbce0828b236ef0dd9494d7b21094","caller":"/mnt/jenkins/workspace/harmony-release/harmony/p2p/host/hostv2/hostv2.go:193","time":"2019-09-23T10:32:22.716065263Z","message":"HostV2 is up!"}
{"level":"info","port":"9000","ip":"165.22.73.42","self":"165.22.73.42:9000","PeerID":"QmZJGt7vEbdyZhNkex3rEvbMzwt5JPnfkCEsi4juBGHc7F","PubKey":"e2a2ec8d5f95444203b8ad80f9cd50775c90cfa9b1b0a5fdb95f1c10336aa364200cbce0828b236ef0dd9494d7b21094","caller":"/mnt/jenkins/workspace/harmony-release/harmony/p2p/p2pimpl/p2pimpl.go:23","time":"2019-09-23T10:32:22.716705756Z","message":"NewHost"}
{"level":"info","port":"9000","ip":"165.22.73.42","publicKey":"e2a2ec8d5f95444203b8ad80f9cd50775c90cfa9b1b0a5fdb95f1c10336aa364200cbce0828b236ef0dd9494d7b21094","caller":"/mnt/jenkins/workspace/harmony-release/harmony/consensus/consensus.go:276","time":"2019-09-23T10:32:22.717582296Z","message":"My Public Key"}
{"level":"warn","port":"9000","ip":"165.22.73.42","hash":"0x00675e82fbe5e7457d3662da1298d21e70db49d4d87ac455952648a95518a227","caller":"/mnt/jenkins/workspace/harmony-release/harmony/core/blockchain.go:284","time":"2019-09-23T10:32:22.78108322Z","message":"Head block missing, resetting chain"}
{"level":"warn","port":"9000","ip":"165.22.73.42","target":0,"caller":"/mnt/jenkins/workspace/harmony-release/harmony/core/blockchain.go:352","time":"2019-09-23T10:32:22.781143573Z","message":"Rewinding blockchain"}
I encountered a similar issue, node block height stopped at 13768. Restarts seem futile.
The first copy of the log seems a network connection issue, which should be fixable by restart. The second copy of the log "Head block missing, resetting chain" is a database corruption issue. The solution is to clean the database and do sync'ing again. Similar to Ethereum's issue: https://ethereum.stackexchange.com/questions/41882/fatal-error-starting-protocol-stack-missing-block-number-for-head-header-hash
@rlan35 agree with the first one, however this happened on a machine that ran all pangaea and mainnet versions successfully till now - i had to change dns to google's in the network settings to make it work
for the head block missing - ok, in this case it would be great if we have tarballs for all shards ready, since this happened to a mainnet node, which would take a long time to re-sync
For mainnet, people can follow this to fast sync: https://nodes.harmony.one/foundational-node-playbook/fast-state-syncing-using-db-snapshot.
Clean install , node block height stopped at 13768 (Again).
Latest / *.Log
https://www.dropbox.com/s/xxcmu7y9ei4a7xn/v4662-pangaea-20190920.0-0-gc2324c41.tar.gz?dl=0
This issue can be related to the fact that currently shard 0,1,2 are down