Peer connections hang temporarily or fail permanently

Question

Peer connections hang temporarily or fail permanently

oxarbitrage opened this issue 4 years ago · comments

Analysis

Peer connection failures can result in a slow sync, temporary sync hangs, or permanent sync hangs.

These failures can happen after network interruptions, or due to normal connection churn. They might also happen because Zebra's protocol state machine gets in an invalid or unrecoverable state.

Next Steps

Here are some things we could try:

Run Zebra with a single local peer, and make it panic as soon as that peer disappears. It's ok for the peer to go from ready to unready and back, and maybe disconnect, but it should never disappear.

Version

zebrad 3.0.0-alpha.0

Current main branch
Edit: as of 2020-12-02

Platform
Linux oxarbitrage 4.15.0-20-generic #21-Ubuntu SMP Tue Apr 24 06:16:15 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

Description

During sync on the mainnet the sync component stopped responding. The program didn't crashed or hanged totally as the inbound requests where still being responded however no more block downloads were made. I have a log of this at https://gist.github.com/oxarbitrage/2f067fed9588c3d942e499d1252fc777 where the last msg from the sync component is at https://gist.github.com/oxarbitrage/2f067fed9588c3d942e499d1252fc777#file-gistfile1-txt-L2843 and no further msg after 30 minutes.

I tried stooping the program and start again. The sync resumed.

--
When trying to sync the zcash blockchain using zebra i will expect to download all the blocks from start to end without intervention. Instead, i had to stop(ctrl-c) the program and restart to keep going.

teor · Answer 1 · Thu Dec 03 2020 05:24:00 GMT+0800 (China Standard Time)

Your logs say that you're using commit bfbc737.

This looks like issue #1409, which is fixed by PR #1425 in a later commit b449fe9.

Please pull the latest main branch, recompile, and relaunch Zebra. If the issue happens again, please re-open this ticket.

Alfredo Garcia · Answer 2 · Thu Dec 03 2020 05:31:35 GMT+0800 (China Standard Time)

Yea, the PR was not merged when i started that test earlier today. Thanks, i will reopen if i see it again.

Alfredo Garcia · Answer 3 · Thu Dec 03 2020 07:23:18 GMT+0800 (China Standard Time)

I am seeing the same issue with an updated branch that includes #1425

No response from the sync component after 30 minutes.

A restart fixes it.

Click triangle to reveal logs

Dec 02 19:46:32.007  INFO {zebrad="7d1dc174"}:peer{addr=39.97.178.10:8233}:msg_as_req{msg=getheaders}:inbound:state: zebra_state::service: responding to peer GetBlocks or GetHeaders final_height=Height(14491) response_len=158 chain_tip_height=Height(419181) stop_height=None intersection_height=Some(Height(14333))
Dec 02 19:46:35.684  INFO {zebrad="7d1dc174"}:sync: zebrad::components::sync: waiting for pending blocks tips.len=1 in_flight=2372 lookahead_limit=2000
Dec 02 19:46:35.685  INFO {zebrad="7d1dc174"}:sync: zebrad::components::sync: extending tips tips.len=1 in_flight=2000 lookahead_limit=2000
Dec 02 19:46:36.668  INFO {zebrad="7d1dc174"}:sync:extend_tips:checkpoint: zebra_consensus::checkpoint: verified checkpoint range block_count=400 current_range=(Excluded(Height(419181)), Included(Height(419581)))
Dec 02 19:46:45.185  INFO {zebrad="7d1dc174"}:sync: zebrad::components::sync: waiting for pending blocks tips.len=1 in_flight=2372 lookahead_limit=2000
Dec 02 19:46:45.185  INFO {zebrad="7d1dc174"}:sync: zebrad::components::sync: extending tips tips.len=1 in_flight=2000 lookahead_limit=2000
Dec 02 19:46:49.016  INFO {zebrad="7d1dc174"}:sync: zebrad::components::sync: waiting for pending blocks tips.len=1 in_flight=2372 lookahead_limit=2000
Dec 02 19:46:49.017  INFO {zebrad="7d1dc174"}:sync: zebrad::components::sync: extending tips tips.len=1 in_flight=2000 lookahead_limit=2000
Dec 02 19:46:52.900  INFO {zebrad="7d1dc174"}:sync: zebrad::components::sync: waiting for pending blocks tips.len=1 in_flight=2374 lookahead_limit=2000
Dec 02 19:46:52.901  INFO {zebrad="7d1dc174"}:sync: zebrad::components::sync: extending tips tips.len=1 in_flight=2000 lookahead_limit=2000
Dec 02 19:46:57.378  INFO {zebrad="7d1dc174"}:sync: zebrad::components::sync: waiting for pending blocks tips.len=1 in_flight=2489 lookahead_limit=2000
Dec 02 19:47:52.769  INFO {zebrad="7d1dc174"}:peer{addr=39.97.178.10:8233}:msg_as_req{msg=inv}:inbound:download_and_verify{hash=000000000036349c3579797cff490c80a6d337c83ab20839b7b1c831caf5cedf}: zebra_state::service: pruned utxo requests old_len=5 new_len=0 prune_count=5 tip=Some((Height(419581), block::Hash("000000000003de545eedfd47b6ae10327fa3b317a6d1eb66a29c3bd6029a243d")))
Dec 02 19:51:31.471  INFO {zebrad="7d1dc174"}:peer{addr=47.254.69.198:8233}:msg_as_req{msg=getheaders}:inbound: zebra_state::service: pruned utxo requests old_len=104 new_len=103 prune_count=1 tip=Some((Height(419581), block::Hash("000000000003de545eedfd47b6ae10327fa3b317a6d1eb66a29c3bd6029a243d")))
Dec 02 19:51:31.521  INFO {zebrad="7d1dc174"}:peer{addr=47.254.69.198:8233}:msg_as_req{msg=getheaders}:inbound:state: zebra_state::service: responding to peer GetBlocks or GetHeaders final_height=Height(14496) response_len=158 chain_tip_height=Height(419581) stop_height=None intersection_height=Some(Height(14338))
Dec 02 19:56:13.322  INFO {zebrad="7d1dc174"}:peer{addr=59.110.243.62:8233}:msg_as_req{msg=inv}:inbound:download_and_verify{hash=00000000010eaac9ff05c2db4c17a64ea5ec024efb300f517c6fec60236817e8}: zebra_state::service: pruned utxo requests old_len=203 new_len=0 prune_count=203 tip=Some((Height(419581), block::Hash("000000000003de545eedfd47b6ae10327fa3b317a6d1eb66a29c3bd6029a243d")))
Dec 02 20:01:13.848  INFO {zebrad="7d1dc174"}:peer{addr=47.254.69.198:8233}:msg_as_req{msg=getheaders}:inbound: zebra_state::service: pruned utxo requests old_len=96 new_len=91 prune_count=5 tip=Some((Height(419581), block::Hash("000000000003de545eedfd47b6ae10327fa3b317a6d1eb66a29c3bd6029a243d")))
Dec 02 20:01:13.867  INFO {zebrad="7d1dc174"}:peer{addr=47.254.69.198:8233}:msg_as_req{msg=getheaders}:inbound:state: zebra_state::service: responding to peer GetBlocks or GetHeaders final_height=Height(14507) response_len=158 chain_tip_height=Height(419581) stop_height=None intersection_height=Some(Height(14349))
Dec 02 20:04:41.853  INFO {zebrad="7d1dc174"}:peer{addr=120.79.50.67:8233}:msg_as_req{msg=inv}:inbound:download_and_verify{hash=000000000010e5d01e28c86a99c18906fd6cf6c975e565908fc471c5fbe42e83}: zebra_state::service: pruned utxo requests old_len=109 new_len=0 prune_count=109 tip=Some((Height(419581), block::Hash("000000000003de545eedfd47b6ae10327fa3b317a6d1eb66a29c3bd6029a243d")))
Dec 02 20:04:57.586  INFO {zebrad="7d1dc174"}:peer{addr=66.70.180.144:8233}:msg_as_req{msg=getheaders}:inbound:state: zebra_state::service: responding to peer GetBlocks or GetHeaders final_height=Height(14510) response_len=158 chain_tip_height=Height(419581) stop_height=None intersection_height=Some(Height(14352))
Dec 02 20:08:51.840  INFO {zebrad="7d1dc174"}:peer{addr=139.99.123.157:8233}:msg_as_req{msg=getheaders}:inbound: zebra_state::service: pruned utxo requests old_len=275 new_len=9 prune_count=266 tip=Some((Height(419581), block::Hash("000000000003de545eedfd47b6ae10327fa3b317a6d1eb66a29c3bd6029a243d")))
Dec 02 20:08:51.856  INFO {zebrad="7d1dc174"}:peer{addr=139.99.123.157:8233}:msg_as_req{msg=getheaders}:inbound:state: zebra_state::service: responding to peer GetBlocks or GetHeaders final_height=Height(14513) response_len=158 chain_tip_height=Height(419581) stop_height=None intersection_height=Some(Height(14355))
Dec 02 20:12:51.891  INFO {zebrad="7d1dc174"}:peer{addr=120.79.50.67:8233}:msg_as_req{msg=inv}:inbound:download_and_verify{hash=000000000060fd9d327ae2680d946af863c191a96b1b8981574a6310f1830e20}: zebra_state::service: pruned utxo requests old_len=21 new_len=0 prune_count=21 tip=Some((Height(419581), block::Hash("000000000003de545eedfd47b6ae10327fa3b317a6d1eb66a29c3bd6029a243d")))
Dec 02 20:14:14.204  INFO {zebrad="7d1dc174"}:peer{addr=39.99.44.175:8233}:msg_as_req{msg=getheaders}:inbound:state: zebra_state::service: responding to peer GetBlocks or GetHeaders final_height=Height(14516) response_len=158 chain_tip_height=Height(419581) stop_height=None intersection_height=Some(Height(14358))
Dec 02 20:14:18.081  INFO {zebrad="7d1dc174"}:peer{addr=176.74.154.45:8233}:msg_as_req{msg=getheaders}:inbound:state: zebra_state::service: responding to peer GetBlocks or GetHeaders final_height=Height(14516) response_len=158 chain_tip_height=Height(419581) stop_height=None intersection_height=Some(Height(14358))
Dec 02 20:16:27.824  INFO {zebrad="7d1dc174"}:peer{addr=47.254.69.198:8233}:msg_as_req{msg=inv}:inbound:download_and_verify{hash=000000000152e1077b370d87f446e04c10c7c586c0645266b0ba8c3c94d24451}: zebra_state::service: pruned utxo requests old_len=74 new_len=2 prune_count=72 tip=Some((Height(419581), block::Hash("000000000003de545eedfd47b6ae10327fa3b317a6d1eb66a29c3bd6029a243d")))
Dec 02 20:18:41.734  INFO {zebrad="7d1dc174"}:peer{addr=39.97.178.10:8233}:msg_as_req{msg=inv}:inbound:download_and_verify{hash=0000000000b8bf272dcc55cfc71b478c59116ccdf713a73dab85882c975fe3da}: zebra_state::service: pruned utxo requests old_len=2 new_len=0 prune_count=2 tip=Some((Height(419581), block::Hash("000000000003de545eedfd47b6ae10327fa3b317a6d1eb66a29c3bd6029a243d")))

teor · Answer 4 · Thu Dec 03 2020 11:22:35 GMT+0800 (China Standard Time)

@oxarbitrage if you have a 10s of gigabytes of spare disk space, you can help us diagnose the issue using a trace log:

would it be possible to run a sync with filter = 'trace' and save a complete execution trace? It will likely be quite large, but we'd have the ability to trace exactly what data is causing the timeout.
#1406 (comment)

If you don't have lots of disk space, using filter = 'debug' might help us diagnose the issue - the logs would be a lot smaller than trace.

Alfredo Garcia · Answer 5 · Fri Dec 04 2020 05:30:53 GMT+0800 (China Standard Time)

I was able to find the problem in debug mode so i have some logs https://gist.github.com/oxarbitrage/d2369fd4e5abb24e683e5a6301fcc6ff#file-sync_debug_logs_mainnet-txt

I didn't researched much yet but one possibility is i am losing connection locally(my adsl change IP every 12 hours or so) and by the amount of marking candidate as failed candidate.addr= msgs in that log all candidates are blacklisted when the connection returns back.

I will confirm this, can post bigger logs if needed.

Alfredo Garcia · Answer 6 · Fri Dec 04 2020 05:51:29 GMT+0800 (China Standard Time)

Yea, it seems that is what is happening. I resumed and blocks are downloaded fine, then i closed my connection in purpose, while zebra is still running. Waited like 30 seconds and connected back, zebra was never able to download more blocks long after my connection was active again.

I have some logs for this too:

https://gist.github.com/oxarbitrage/916ed4d49db39116917ad6e938555e8e#file-sync_disconnect_mainnet-txt

Alfredo Garcia · Answer 7 · Fri Dec 04 2020 06:00:28 GMT+0800 (China Standard Time)

Is the cut is short enough(tested a cut of 3-5 seconds) and there is no time to mark all the peers as failing then the sync will continue

Alfredo Garcia · Answer 8 · Fri Dec 04 2020 06:59:10 GMT+0800 (China Standard Time)

The cut haves to be long(of at least 1 minute here) to actually be totally unrecoverable in the same session.

In a 1>minute cut sometimes i get this warnings long after the connection is restored:

Dec 03 19:49:46.245  WARN crawl_and_dial: zebra_network::peer_set::set: network request with no peer connections. Hint: check your network connection

They at least indicate something is going on with the network with default info level. In some other runs i am not seeing this warns, i suppose this can depend on the amount of time the connection was off.

Henry de Valence · Answer 9 · Fri Dec 04 2020 07:15:33 GMT+0800 (China Standard Time)

Hmm, I don't think that the failure of an existing connection causes a peer address to be put in the "failed candidates" list, that's a list of peers where the connection attempt failed. (It's not a permanent rejection, it's just the lowest-preference for reconnection attempts). The peer set will attempt to reconnect to peers it disconnected from, by waiting until they could no longer be connected (ie. waiting beyond a timeout interval), then treating them as high-priority candidates for new connections.

The peer set is supposed to send a demand signal when it does not have enough peers, causing attempts to connect to new peers. It's unclear why that mechanism isn't working.

teor · Answer 10 · Fri Dec 04 2020 08:42:54 GMT+0800 (China Standard Time)

I'm also seeing this issue, it looks like my machine or router stopped allowing connections for a few hours this morning. When it came back up, Zebra reconnected to 1 peer, but has failed to reconnect to any other peers.

Edit: this is on testnet, so I should be seeing 5-20 active peers

teor · Answer 11 · Fri Dec 04 2020 11:03:23 GMT+0800 (China Standard Time)

I've just seen a Testnet Zebra rapidly lose all its peers, and never reconnect:

Click triangle to reveal logs

Dec 04 12:52:33.915  INFO {zebrad="af01a78e"}:sync:extend_tips: zebra_network::peer_set::set: network request with no ready peers: finding more peers, waiting for 15 peers to answer requests
Dec 04 12:52:49.767  INFO {zebrad="af01a78e"}:sync: zebrad::components::sync: extending tips tips.len=1 in_flight=1606 lookahead_limit=2000
Dec 04 12:53:06.779  INFO {zebrad="af01a78e"}:sync: zebrad::components::sync: extending tips tips.len=1 in_flight=1977 lookahead_limit=2000
Dec 04 12:53:32.834  INFO {zebrad="af01a78e"}:sync:extend_tips: zebra_state::service: pruned utxo requests old_len=1365 new_len=40 prune_count=1325 tip=Some((Height(282491), block::Hash("0013da35d7196632f70cfa9fcc6e22906db20d21430ec08cfd2ba231bfd594b4")))
Dec 04 12:53:34.275  INFO {zebrad="af01a78e"}:sync:extend_tips: zebra_network::peer_set::set: network request with no ready peers: finding more peers, waiting for 15 peers to answer requests
Dec 04 12:54:17.672  INFO {zebrad="af01a78e"}:sync:extend_tips: zebra_state::service: pruned utxo requests old_len=729 new_len=45 prune_count=684 tip=Some((Height(282491), block::Hash("0013da35d7196632f70cfa9fcc6e22906db20d21430ec08cfd2ba231bfd594b4")))
Dec 04 12:54:35.569  INFO {zebrad="af01a78e"}:sync:extend_tips: zebra_network::peer_set::set: network request with no ready peers: finding more peers, waiting for 5 peers to answer requests
Dec 04 12:55:19.382  INFO {zebrad="af01a78e"}:peer{addr=[2600:3c00::f03c:92ff:fe48:bd98]:18233}:msg_as_req{msg=getheaders}:inbound: zebra_state::service: pruned utxo requests old_len=423 new_len=49 prune_count=374 tip=Some((Height(282491), block::Hash("0013da35d7196632f70cfa9fcc6e22906db20d21430ec08cfd2ba231bfd594b4")))
Dec 04 12:55:19.384  INFO {zebrad="af01a78e"}:peer{addr=[2600:3c00::f03c:92ff:fe48:bd98]:18233}:msg_as_req{msg=getheaders}:inbound:state: zebra_state::service: responding to peer GetBlocks or GetHeaders final_height=Height(141750) response_len=158 chain_tip_height=Height(282491) stop_height=None intersection_height=Some(Height(141592))
Dec 04 12:55:40.353  INFO {zebrad="af01a78e"}:sync:extend_tips: zebra_network::peer_set::set: network request with no ready peers: finding more peers, waiting for 1 peers to answer requests
Dec 04 12:55:50.508  INFO {zebrad="af01a78e"}:sync:extend_tips: zebra_state::service: pruned utxo requests old_len=4801 new_len=62 prune_count=4739 tip=Some((Height(282491), block::Hash("0013da35d7196632f70cfa9fcc6e22906db20d21430ec08cfd2ba231bfd594b4")))
Dec 04 12:56:45.185  WARN {zebrad="af01a78e"}:sync:extend_tips: zebra_network::peer_set::set: network request with no peer connections. Hint: check your network connection
Dec 04 12:57:47.290  WARN {zebrad="af01a78e"}:sync:extend_tips: zebra_network::peer_set::set: network request with no peer connections. Hint: check your network connection
Dec 04 12:59:02.290  WARN {zebrad="af01a78e"}:sync:extend_tips: zebra_network::peer_set::set: network request with no peer connections. Hint: check your network connection
Dec 04 13:00:17.291  WARN {zebrad="af01a78e"}:sync:extend_tips: zebra_network::peer_set::set: network request with no peer connections. Hint: check your network connection
Dec 04 13:01:32.291  WARN {zebrad="af01a78e"}:sync:extend_tips: zebra_network::peer_set::set: network request with no peer connections. Hint: check your network connection
Dec 04 13:02:47.291  WARN crawl_and_dial: zebra_network::peer_set::set: network request with no peer connections. Hint: check your network connection

teor · Answer 12 · Fri Dec 04 2020 13:21:08 GMT+0800 (China Standard Time)

I've taken the triage off this bug, because we already triaged some of its duplicates into the first alpha

Alfredo Garcia · Answer 13 · Mon Dec 07 2020 05:51:17 GMT+0800 (China Standard Time)

Between each warn here is what i can get in a trace level:

Click triangle to reveal logs

...
Dec 06 18:42:32.966 TRACE {zebrad="5b687f87"}:sync:extend_tips:download_and_verify{hash=0000000000176a3344d93598c018fddef80f2edd59b194a7c0a0b7a59aa52c4a}: zebra_network::peer_set::set: ready_peers=0 unready_peers=0
Dec 06 18:42:32.966  WARN {zebrad="5b687f87"}:sync:extend_tips:download_and_verify{hash=0000000000176a3344d93598c018fddef80f2edd59b194a7c0a0b7a59aa52c4a}: zebra_network::peer_set::set: network request with no peer connections. Hint: check your network connection
Dec 06 18:42:32.967 TRACE {zebrad="5b687f87"}:sync:extend_tips:download_and_verify{hash=0000000000176a3344d93598c018fddef80f2edd59b194a7c0a0b7a59aa52c4a}: zebra_network::peer_set::set: preselected service was not ready, reselecting
Dec 06 18:42:32.967 TRACE {zebrad="5b687f87"}:sync:extend_tips:download_and_verify{hash=0000000000176a3344d93598c018fddef80f2edd59b194a7c0a0b7a59aa52c4a}: zebra_network::peer_set::set: no ready services, sending demand signal
Dec 06 18:42:32.967 TRACE {zebrad="5b687f87"}:sync:extend_tips:download_and_verify{hash=0000000000176a3344d93598c018fddef80f2edd59b194a7c0a0b7a59aa52c4a}: tower::buffer::worker: service.ready=false delay
Dec 06 18:42:52.968 DEBUG {zebrad="5b687f87"}:sync:extend_tips:download_and_verify{hash=0000000000176a3344d93598c018fddef80f2edd59b194a7c0a0b7a59aa52c4a}: zebra_network::policies: retrying req=BlocksByHash({block::Hash("0000000000176a3344d93598c018fddef80f2edd59b194a7c0a0b7a59aa52c4a")}) e=Elapsed(()) remaining_tries=1
Dec 06 18:42:52.968 DEBUG {zebrad="5b687f87"}:sync:extend_tips:download_and_verify{hash=0000000000176a3344d93598c018fddef80f2edd59b194a7c0a0b7a59aa52c4a}: zebra_network::policies: retrying req=BlocksByHash({block::Hash("0000000000176a3344d93598c018fddef80f2edd59b194a7c0a0b7a59aa52c4a")}) e=Elapsed(()) remaining_tries=2
Dec 06 18:43:47.966 TRACE {zebrad="5b687f87"}:task{kind=task spawn.location=/home/oxarbitrage/.cargo/git/checkouts/tower-b098c32cf5a1bcca/d4d1c67/tower/src/buffer/service.rs:67:9}: tower::buffer::worker: worker polling for next message
Dec 06 18:43:47.966 TRACE {zebrad="5b687f87"}:task{kind=task spawn.location=/home/oxarbitrage/.cargo/git/checkouts/tower-b098c32cf5a1bcca/d4d1c67/tower/src/buffer/service.rs:67:9}: tower::buffer::worker: dropping cancelled buffered request
Dec 06 18:43:47.966 TRACE {zebrad="5b687f87"}:task{kind=task spawn.location=/home/oxarbitrage/.cargo/git/checkouts/tower-b098c32cf5a1bcca/d4d1c67/tower/src/buffer/service.rs:67:9}: tower::buffer::worker: dropping cancelled request
Dec 06 18:43:47.966 TRACE {zebrad="5b687f87"}:task{kind=task spawn.location=/home/oxarbitrage/zebra/zebra_sync/zebra/zebra-network/src/peer_set/initialize.rs:162:23}:crawl_and_dial: tower::buffer::service: sending request to buffer worker
Dec 06 18:43:47.966 TRACE {zebrad="5b687f87"}:task{kind=task spawn.location=/home/oxarbitrage/.cargo/git/checkouts/tower-b098c32cf5a1bcca/d4d1c67/tower/src/buffer/service.rs:67:9}: tower::buffer::worker: worker polling for next message
Dec 06 18:43:47.966 TRACE {zebrad="5b687f87"}:task{kind=task spawn.location=/home/oxarbitrage/.cargo/git/checkouts/tower-b098c32cf5a1bcca/d4d1c67/tower/src/buffer/service.rs:67:9}: tower::buffer::worker: processing new request
Dec 06 18:43:47.966 TRACE {zebrad="5b687f87"}:task{kind=task spawn.location=/home/oxarbitrage/zebra/zebra_sync/zebra/zebra-network/src/peer_set/initialize.rs:162:23}:crawl_and_dial: tower::buffer::worker: resumed=false worker received request; waiting for service readiness
Dec 06 18:43:47.966 TRACE {zebrad="5b687f87"}:sync:extend_tips:download_and_verify{hash=000000000100c07024462b6221c73daca37afc18047cc0a5fcf2e4520573c821}: tower::hedge::rotating_histogram: Time since last rotation is 189.693851337s.  clearing!
Dec 06 18:43:47.967 TRACE {zebrad="5b687f87"}:task{kind=task spawn.location=/home/oxarbitrage/zebra/zebra_sync/zebra/zebra-network/src/peer_set/initialize.rs:162:23}:crawl_and_dial: zebra_network::peer_set::set: ready_peers=0 unready_peers=0
Dec 06 18:43:47.967 TRACE {zebrad="5b687f87"}:sync:extend_tips:download_and_verify{hash=000000000100c07024462b6221c73daca37afc18047cc0a5fcf2e4520573c821}: tower::buffer::service: sending request to buffer worker
Dec 06 18:43:47.967  WARN {zebrad="5b687f87"}:task{kind=task spawn.location=/home/oxarbitrage/zebra/zebra_sync/zebra/zebra-network/src/peer_set/initialize.rs:162:23}:crawl_and_dial: zebra_network::peer_set::set: network request with no peer connections. Hint: check your network connection
Dec 06 18:43:47.967 DEBUG {zebrad="5b687f87"}:sync:extend_tips:download_and_verify{hash=000000000100c07024462b6221c73daca37afc18047cc0a5fcf2e4520573c821}: zebrad::components::sync::downloads: requested block
Dec 06 18:43:47.967 TRACE {zebrad="5b687f87"}:task{kind=task spawn.location=/home/oxarbitrage/zebra/zebra_sync/zebra/zebra-network/src/peer_set/initialize.rs:162:23}:crawl_and_dial: zebra_network::peer_set::set: preselected service was not ready, reselecting
Dec 06 18:43:47.967 TRACE {zebrad="5b687f87"}:task{kind=task spawn.location=/home/oxarbitrage/zebra/zebra_sync/zebra/zebra-network/src/peer_set/initialize.rs:162:23}:crawl_and_dial: zebra_network::peer_set::set: no ready services, sending demand signal
Dec 06 18:43:47.967 TRACE {zebrad="5b687f87"}:task{kind=task spawn.location=/home/oxarbitrage/zebra/zebra_sync/zebra/zebra-network/src/peer_set/initialize.rs:162:23}:crawl_and_dial: tower::buffer::worker: service.ready=false delay
Dec 06 18:43:47.967 DEBUG {zebrad="5b687f87"}:sync:extend_tips:download_and_verify{hash=000000000254e1d8cf90681e84f877d56fb4f7d4965b3eb09c175ca56820b635}: zebrad::components::sync::downloads: waiting to request block
Dec 06 18:44:07.968 DEBUG {zebrad="5b687f87"}:sync:extend_tips:download_and_verify{hash=000000000100c07024462b6221c73daca37afc18047cc0a5fcf2e4520573c821}: zebra_network::policies: retrying req=BlocksByHash({block::Hash("000000000100c07024462b6221c73daca37afc18047cc0a5fcf2e4520573c821")}) e=Elapsed(()) remaining_tries=2
Dec 06 18:45:03.017 TRACE {zebrad="5b687f87"}:task{kind=task spawn.location=/home/oxarbitrage/.cargo/git/checkouts/tower-b098c32cf5a1bcca/d4d1c67/tower/src/buffer/service.rs:67:9}: tower::buffer::worker: worker polling for next message
Dec 06 18:45:03.017 TRACE {zebrad="5b687f87"}:task{kind=task spawn.location=/home/oxarbitrage/.cargo/git/checkouts/tower-b098c32cf5a1bcca/d4d1c67/tower/src/buffer/service.rs:67:9}: tower::buffer::worker: resuming buffered request
Dec 06 18:45:03.017 TRACE {zebrad="5b687f87"}:task{kind=task spawn.location=/home/oxarbitrage/zebra/zebra_sync/zebra/zebra-network/src/peer_set/initialize.rs:162:23}:crawl_and_dial: tower::buffer::worker: resumed=true worker received request; waiting for service readiness
Dec 06 18:45:03.017 TRACE {zebrad="5b687f87"}:task{kind=task spawn.location=/home/oxarbitrage/zebra/zebra_sync/zebra/zebra-network/src/peer_set/initialize.rs:162:23}:crawl_and_dial: zebra_network::peer_set::set: ready_peers=0 unready_peers=0
Dec 06 18:45:03.017  WARN {zebrad="5b687f87"}:task{kind=task spawn.location=/home/oxarbitrage/zebra/zebra_sync/zebra/zebra-network/src/peer_set/initialize.rs:162:23}:crawl_and_dial: zebra_network::peer_set::set: network request with no peer connections. Hint: check your network connection
Dec 06 18:45:03.017 TRACE {zebrad="5b687f87"}:task{kind=task spawn.location=/home/oxarbitrage/zebra/zebra_sync/zebra/zebra-network/src/peer_set/initialize.rs:162:23}:crawl_and_dial: zebra_network::peer_set::set: preselected service was not ready, reselecting
Dec 06 18:45:03.017 TRACE {zebrad="5b687f87"}:task{kind=task spawn.location=/home/oxarbitrage/zebra/zebra_sync/zebra/zebra-network/src/peer_set/initialize.rs:162:23}:crawl_and_dial: zebra_network::peer_set::set: no ready services, sending demand signal
Dec 06 18:45:03.017 TRACE {zebrad="5b687f87"}:task{kind=task spawn.location=/home/oxarbitrage/zebra/zebra_sync/zebra/zebra-network/src/peer_set/initialize.rs:162:23}:crawl_and_dial: tower::buffer::worker: service.ready=false delay
...

At around every 1 minute(connected or not connected) we get this one: https://github.com/ZcashFoundation/zebra/blob/main/zebra-network/src/peer_set/set.rs#L452 so we have a working loop but it seems that the demand signal at the following line is not working.

teor · Answer 14 · Mon Dec 14 2020 09:54:51 GMT+0800 (China Standard Time)

We just merged #1468, which should provide better diagnostics for this issue, and panics if our assumptions about the peer state machine don't hold.

teor · Answer 15 · Mon Dec 14 2020 16:49:57 GMT+0800 (China Standard Time)

A similar bug was reported in #1516, with very slow syncing and only 2 peers.

Here is the debug log

Dimitris Apostolou · Answer 16 · Tue Dec 15 2020 01:14:39 GMT+0800 (China Standard Time)

Another case.

Click triangle to reveal logs

Dec 14 17:45:34.315  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: starting sync, obtaining new tips
Dec 14 17:45:34.316  INFO {zebrad="b8646fa7"}:sync:obtain_tips:state: zebra_state::util: created block locator tip_height=Height(580963) min_locator_height=580864 locators=[Height(580963), Height(580962), Height(580961), Height(580959), Height(580955), Height(580947), Height(580931), Height(580899), Height(580864)]
Dec 14 17:45:40.452  INFO {zebrad="b8646fa7"}:sync:obtain_tips: zebra_network::peer_set::set: network request with no ready peers: finding more peers, waiting for 11 peers to answer requests
Dec 14 17:45:46.068  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: extending tips tips.len=1 in_flight=499 lookahead_limit=2000
Dec 14 17:45:47.392  INFO {zebrad="b8646fa7"}:sync:obtain_tips: zebra_state::service: pruned utxo requests old_len=9865 new_len=9797 prune_count=68 tip=Some((Height(580971), block::Hash("00000000014627f59cbb1f203038c80e17fcb7f64ceaa1fb48e0838c8f3c395e")))
Dec 14 17:45:57.600  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: extending tips tips.len=1 in_flight=918 lookahead_limit=2000
Dec 14 17:46:03.202  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: extending tips tips.len=1 in_flight=1415 lookahead_limit=2000
Dec 14 17:46:04.249  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: extending tips tips.len=1 in_flight=1498 lookahead_limit=2000
Dec 14 17:46:09.769  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: extending tips tips.len=1 in_flight=1996 lookahead_limit=2000
Dec 14 17:46:15.597  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: waiting for pending blocks tips.len=1 in_flight=2494 lookahead_limit=2000
Dec 14 17:46:17.532  INFO {zebrad="b8646fa7"}:sync:extend_tips: zebra_state::service: pruned utxo requests old_len=38517 new_len=38003 prune_count=514 tip=Some((Height(581096), block::Hash("0000000000374f4efd3efc760bf6a934ff0fce3643483acd88356ddd783e4f12")))
Dec 14 17:46:47.565  INFO {zebrad="b8646fa7"}:sync:obtain_tips: zebra_state::service: pruned utxo requests old_len=56230 new_len=56206 prune_count=24 tip=Some((Height(581445), block::Hash("000000000137afe054e777a0559e232801546e4cdf0f88ec8143aace5b812611")))
Dec 14 17:46:55.016  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: extending tips tips.len=1 in_flight=2000 lookahead_limit=2000
Dec 14 17:47:01.023  INFO {zebrad="b8646fa7"}:sync:extend_tips: zebra_network::peer_set::set: network request with no ready peers: finding more peers, waiting for 11 peers to answer requests
Dec 14 17:47:06.627  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: waiting for pending blocks tips.len=1 in_flight=2371 lookahead_limit=2000
Dec 14 17:47:17.572  INFO {zebrad="b8646fa7"}:sync:extend_tips: zebra_state::service: pruned utxo requests old_len=58811 new_len=58769 prune_count=42 tip=Some((Height(581800), block::Hash("0000000000c452643c46a6069a59ef084092569564307a84e3143ac205586875")))
Dec 14 17:47:30.753  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: extending tips tips.len=1 in_flight=2000 lookahead_limit=2000
Dec 14 17:47:42.784  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: waiting for pending blocks tips.len=1 in_flight=2419 lookahead_limit=2000
Dec 14 17:47:47.614  INFO {zebrad="b8646fa7"}:sync:extend_tips: zebra_state::service: pruned utxo requests old_len=59189 new_len=59158 prune_count=31 tip=Some((Height(582199), block::Hash("000000000117739fd3e750eed4cad81f93fa505d8269212d84064e33690d41cb")))
Dec 14 17:48:06.180  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: extending tips tips.len=1 in_flight=2000 lookahead_limit=2000
Dec 14 17:48:07.096  INFO {zebrad="b8646fa7"}:sync:extend_tips: zebra_network::peer_set::set: network request with no ready peers: finding more peers, waiting for 11 peers to answer requests
Dec 14 17:48:12.932  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: waiting for pending blocks tips.len=1 in_flight=2479 lookahead_limit=2000
Dec 14 17:48:17.693  INFO {zebrad="b8646fa7"}:sync:extend_tips: zebra_state::service: pruned utxo requests old_len=62786 new_len=62744 prune_count=42 tip=Some((Height(582609), block::Hash("000000000057b415481fef161227879323f240ce4874cfb3fced74689d7826c6")))
Dec 14 17:48:47.699  INFO {zebrad="b8646fa7"}:sync:extend_tips: zebra_state::service: pruned utxo requests old_len=59908 new_len=59900 prune_count=8 tip=Some((Height(582853), block::Hash("00000000017a047cfc28bf8fb6a324bdf8ea71bfc9cc9b7bfa1dd2a949bea725")))
Dec 14 17:48:58.876  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: extending tips tips.len=1 in_flight=2000 lookahead_limit=2000
Dec 14 17:49:07.119  INFO {zebrad="b8646fa7"}:sync:extend_tips: zebra_network::peer_set::set: network request with no ready peers: finding more peers, waiting for 11 peers to answer requests
Dec 14 17:49:12.362  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: waiting for pending blocks tips.len=1 in_flight=2371 lookahead_limit=2000
Dec 14 17:49:14.520  WARN {zebrad="b8646fa7"}:sync: zebrad::components::sync: error downloading and verifying block e=Block(Transaction(InternalDowncastError("downcast to redjubjub::Error failed, original error: Elapsed(())")))
Dec 14 17:49:14.521  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: waiting to restart sync timeout=45s
Dec 14 17:49:59.526  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: starting sync, obtaining new tips
Dec 14 17:49:59.679  INFO {zebrad="b8646fa7"}:sync:obtain_tips: zebra_state::service: pruned utxo requests old_len=59818 new_len=2 prune_count=59816 tip=Some((Height(583206), block::Hash("00000000000475ba6023e31e1a13c046bb3bebd4887efb13e045dc2abdc83bdc")))
Dec 14 17:49:59.679  INFO {zebrad="b8646fa7"}:sync:obtain_tips:state: zebra_state::util: created block locator tip_height=Height(583206) min_locator_height=583107 locators=[Height(583206), Height(583205), Height(583204), Height(583202), Height(583198), Height(583190), Height(583174), Height(583142), Height(583107)]
Dec 14 17:50:06.521  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: extending tips tips.len=1 in_flight=499 lookahead_limit=2000
Dec 14 17:50:07.125  INFO {zebrad="b8646fa7"}:sync:extend_tips: zebra_network::peer_set::set: network request with no ready peers: finding more peers, waiting for 11 peers to answer requests
Dec 14 17:50:13.039  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: extending tips tips.len=1 in_flight=997 lookahead_limit=2000
Dec 14 17:50:13.503  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: exhausted prospective tip set
Dec 14 17:50:13.503  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: waiting to restart sync timeout=45s
Dec 14 17:50:35.371  INFO {zebrad="b8646fa7"}:peer{addr=40.118.90.57:55002}:msg_as_req{msg=inv}:inbound:download_and_verify{hash=0000000000a4cda3732a231f2e7881b092e5864d9c5b34a60f114bc78fbefda9}: zebra_state::service: pruned utxo requests old_len=15135 new_len=8 prune_count=15127 tip=Some((Height(583254), block::Hash("00000000004c58d2b26794d6f8ffc9f759f17ce106d59a7b23cfd4cba375b7bc")))
Dec 14 17:50:58.527  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: starting sync, obtaining new tips
Dec 14 17:50:58.527  INFO {zebrad="b8646fa7"}:sync:obtain_tips:state: zebra_state::util: created block locator tip_height=Height(583254) min_locator_height=583155 locators=[Height(583254), Height(583253), Height(583252), Height(583250), Height(583246), Height(583238), Height(583222), Height(583190), Height(583155)]
Dec 14 17:51:05.365  INFO {zebrad="b8646fa7"}:sync:obtain_tips: zebra_state::service: pruned utxo requests old_len=1052 new_len=1047 prune_count=5 tip=Some((Height(583255), block::Hash("000000000111bb6f5da6e28b04e1fc5346bde4a50f56e55fe8eb51dab5e96325")))
Dec 14 17:51:07.172  INFO {zebrad="b8646fa7"}:sync:obtain_tips: zebra_network::peer_set::set: network request with no ready peers: finding more peers, waiting for 11 peers to answer requests
Dec 14 17:51:09.900  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: extending tips tips.len=1 in_flight=451 lookahead_limit=2000
Dec 14 17:51:13.821  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: extending tips tips.len=1 in_flight=793 lookahead_limit=2000
Dec 14 17:51:14.000  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: exhausted prospective tip set
Dec 14 17:51:14.001  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: waiting to restart sync timeout=45s
Dec 14 17:51:59.010  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: starting sync, obtaining new tips
Dec 14 17:51:59.031  INFO {zebrad="b8646fa7"}:sync:obtain_tips: zebra_state::service: pruned utxo requests old_len=11761 new_len=217 prune_count=11544 tip=Some((Height(583280), block::Hash("0000000000d7dcaf50a498e6abf1dd6caf7a3cb598959d4404f10e16ecd4b20c")))
Dec 14 17:51:59.031  INFO {zebrad="b8646fa7"}:sync:obtain_tips:state: zebra_state::util: created block locator tip_height=Height(583280) min_locator_height=583181 locators=[Height(583280), Height(583279), Height(583278), Height(583276), Height(583272), Height(583264), Height(583248), Height(583216), Height(583181)]
Dec 14 17:52:07.183  INFO {zebrad="b8646fa7"}:sync:obtain_tips: zebra_network::peer_set::set: network request with no ready peers: finding more peers, waiting for 11 peers to answer requests
Dec 14 17:52:11.430  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: extending tips tips.len=2 in_flight=496 lookahead_limit=2000
Dec 14 17:52:17.745  WARN {zebrad="b8646fa7"}:sync: zebrad::components::sync: error extending tips e=
   0: duplicate hash queued for download

Location:
   zebrad/src/components/sync/downloads.rs:116

  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ SPANTRACE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  
   0: zebrad::components::sync::extend_tips
      at zebrad/src/components/sync.rs:381
   1: zebrad::components::sync::sync
      at zebrad/src/components/sync.rs:150
   2: zebrad::commands:: with zebrad="b8646fa7"
      at zebrad/src/commands.rs:56

  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ BACKTRACE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
   1: backtrace::backtrace::libunwind::trace::h56b5b89d46ea18ee
      at /Users/rex/.cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.54/src/backtrace/libunwind.rs:90
        88 │ #[inline(always)]
        89 │ pub unsafe fn trace(mut cb: &mut dyn FnMut(&super::Frame) -> bool) {
        90 >     uw::_Unwind_Backtrace(trace_fn, &mut cb as *mut _ as *mut _);
        91 │ 
        92 │     extern "C" fn trace_fn(
   2: backtrace::backtrace::trace_unsynchronized::h456d388ebbb97b58
      at /Users/rex/.cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.54/src/backtrace/mod.rs:66
        64 │ /// See information on `trace` for caveats on `cb` panicking.
        65 │ pub unsafe fn trace_unsynchronized<F: FnMut(&Frame) -> bool>(mut cb: F) {
        66 >     trace_imp(&mut cb)
        67 │ }
        68 │ 
   3: backtrace::backtrace::trace::h0c2ee1e6653540fd
      at /Users/rex/.cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.54/src/backtrace/mod.rs:53
        51 │ pub fn trace<F: FnMut(&Frame) -> bool>(cb: F) {
        52 │     let _guard = crate::lock::lock();
        53 >     unsafe { trace_unsynchronized(cb) }
        54 │ }
        55 │ 
   4: backtrace::capture::Backtrace::create::h8d054d598eb8a93e
      at /Users/rex/.cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.54/src/capture.rs:176
       174 │         let mut frames = Vec::new();
       175 │         let mut actual_start_index = None;
       176 >         trace(|frame| {
       177 │             frames.push(BacktraceFrame {
       178 │                 frame: Frame::Raw(frame.clone()),
   5: backtrace::capture::Backtrace::new::h89e80605e5dd4af5
      at /Users/rex/.cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.54/src/capture.rs:140
       138 │     #[inline(never)] // want to make sure there's a frame here to remove
       139 │     pub fn new() -> Backtrace {
       140 >         let mut bt = Self::create(Self::new as usize);
       141 │         bt.resolve();
       142 │         bt
   6: color_eyre::config::EyreHook::default::h2b86268b3a262bdc
      at /Users/rex/.cargo/registry/src/github.com-1ecc6299db9ec823/color-eyre-0.5.10/src/config.rs:1016
      1014 │     pub(crate) fn default(&self, error: &(dyn std::error::Error + 'static)) -> crate::Handler {
      1015 │         let backtrace = if lib_verbosity() != Verbosity::Minimal {
      1016 >             Some(backtrace::Backtrace::new())
      1017 │         } else {
      1018 │             None
   7: color_eyre::config::EyreHook::into_eyre_hook::{{closure}}::h0fba5a5bb754400f
      at /Users/rex/.cargo/registry/src/github.com-1ecc6299db9ec823/color-eyre-0.5.10/src/config.rs:1070
      1068 │             + 'static,
      1069 │     > {
      1070 >         Box::new(move |e| Box::new(self.default(e)))
      1071 │     }
      1072 │ }

Run with COLORBT_SHOW_HIDDEN=1 environment variable to disable frame filtering.

Consider reporting this error using this URL: https://github.com/ZcashFoundation/zebra/issues/new?title=%3Cautogenerated-issue%3E&body=%23%23+Error%0A%60%60%60%0AError%3A+%0A+++0%3A+duplicate+hash+queued+for+download%0A%60%60%60%0A%0A%23%23+Metadata%0A%7Ckey%7Cvalue%7C%0A%7C--%7C--%7C%0A%7C**version**%7C3.0.0-alpha.0%7C%0A%7C**git+commit**%7Cb8646fa7%7C%0A%0A%0A%23%23+SpanTrace%0A%0A%3Cdetails%3E%0A%0A%60%60%60%0ASpanTrace%3A%0A+++0%3A+zebrad%3A%3Acomponents%3A%3Async%3A%3Aextend_tips%0A+++++++++++++at+zebrad%2Fsrc%2Fcomponents%2Fsync.rs%3A381%0A+++1%3A+zebrad%3A%3Acomponents%3A%3Async%3A%3Async%0A+++++++++++++at+zebrad%2Fsrc%2Fcomponents%2Fsync.rs%3A150%0A+++2%3A+zebrad%3A%3Acommands%3A%3A%0A+++++++++++with+zebrad%3D%22b8646fa7%22%0A+++++++++++++at+zebrad%2Fsrc%2Fcommands.rs%3A56%0A%60%60%60%0A%3C%2Fdetails%3E%0A%0A%23%23+Backtrace%0A%0A%3Cdetails%3E%0A%0A%60%60%60%0ABacktrace%3A%0A+++0%3A+backtrace%3A%3Abacktrace%3A%3Alibunwind%3A%3Atrace%0A+++++++++++++at+%2FUsers%2Frex%2F.cargo%2Fregistry%2Fsrc%2Fgithub.com-1ecc6299db9ec823%2Fbacktrace-0.3.54%2Fsrc%2Fbacktrace%2Flibunwind.rs%3A90%3A5%0A++++++backtrace%3A%3Abacktrace%3A%3Atrace_unsynchronized%0A+++++++++++++at+%2FUsers%2Frex%2F.cargo%2Fregistry%2Fsrc%2Fgithub.com-1ecc6299db9ec823%2Fbacktrace-0.3.54%2Fsrc%2Fbacktrace%2Fmod.rs%3A66%3A5%0A+++1%3A+backtrace%3A%3Abacktrace%3A%3Atrace%0A+++++++++++++at+%2FUsers%2Frex%2F.cargo%2Fregistry%2Fsrc%2Fgithub.com-1ecc6299db9ec823%2Fbacktrace-0.3.54%2Fsrc%2Fbacktrace%2Fmod.rs%3A53%3A14%0A+++2%3A+backtrace%3A%3Acapture%3A%3ABacktrace%3A%3Acreate%0A+++++++++++++at+%2FUsers%2Frex%2F.cargo%2Fregistry%2Fsrc%2Fgithub.com-1ecc6299db9ec823%2Fbacktrace-0.3.54%2Fsrc%2Fcapture.rs%3A176%3A9%0A+++3%3A+backtrace%3A%3Acapture%3A%3ABacktrace%3A%3Anew%0A+++++++++++++at+%2FUsers%2Frex%2F.cargo%2Fregistry%2Fsrc%2Fgithub.com-1ecc6299db9ec823%2Fbacktrace-0.3.54%2Fsrc%2Fcapture.rs%3A140%3A22%0A+++4%3A+color_eyre%3A%3Aconfig%3A%3AEyreHook%3A%3Adefault%0A+++++++++++++at+%2FUsers%2Frex%2F.cargo%2Fregistry%2Fsrc%2Fgithub.com-1ecc6299db9ec823%2Fcolor-eyre-0.5.10%2Fsrc%2Fconfig.rs%3A1016%3A18%0A+++5%3A+color_eyre%3A%3Aconfig%3A%3AEyreHook%3A%3Ainto_eyre_hook%3A%3A%7B%7Bclosure%7D%7D%0A+++++++++++++at+%2FUsers%2Frex%2F.cargo%2Fregistry%2Fsrc%2Fgithub.com-1ecc6299db9ec823%2Fcolor-eyre-0.5.10%2Fsrc%2Fconfig.rs%3A1070%3A36%0A%0A%60%60%60%0A%3C%2Fdetails%3E
Dec 14 17:52:17.755  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: waiting to restart sync timeout=45s
Dec 14 17:53:02.758  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: starting sync, obtaining new tips
Dec 14 17:53:02.773  INFO {zebrad="b8646fa7"}:sync:obtain_tips: zebra_state::service: pruned utxo requests old_len=7417 new_len=217 prune_count=7200 tip=Some((Height(583411), block::Hash("00000000000b2b6879b29873fbbc77590220ed1dad556f002ab399bd98d5327c")))
Dec 14 17:53:02.773  INFO {zebrad="b8646fa7"}:sync:obtain_tips:state: zebra_state::util: created block locator tip_height=Height(583411) min_locator_height=583312 locators=[Height(583411), Height(583410), Height(583409), Height(583407), Height(583403), Height(583395), Height(583379), Height(583347), Height(583312)]
Dec 14 17:53:08.807  INFO {zebrad="b8646fa7"}:sync:obtain_tips: zebra_network::peer_set::set: network request with no ready peers: finding more peers, waiting for 10 peers to answer requests
Dec 14 17:53:09.520  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: extending tips tips.len=1 in_flight=79 lookahead_limit=2000
Dec 14 17:53:20.935  WARN {zebrad="b8646fa7"}:sync: zebrad::components::sync: error downloading and verifying block e=Block(Block { source: AlreadyInChain(block::Hash("000000000173b3fd7200c7c2837ce3fe496fdcd1e0f0d22388ebcac1a6c67ac3"), 3) })
Dec 14 17:53:20.935  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: waiting to restart sync timeout=45s
Dec 14 17:54:05.940  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: starting sync, obtaining new tips
Dec 14 17:54:05.963  INFO {zebrad="b8646fa7"}:sync:obtain_tips: zebra_state::service: pruned utxo requests old_len=12155 new_len=211 prune_count=11944 tip=Some((Height(583457), block::Hash("000000000089342a2001e491f306f03cbd6e7d702179a066093c58ee69449e28")))
Dec 14 17:54:05.963  INFO {zebrad="b8646fa7"}:sync:obtain_tips:state: zebra_state::util: created block locator tip_height=Height(583457) min_locator_height=583358 locators=[Height(583457), Height(583456), Height(583455), Height(583453), Height(583449), Height(583441), Height(583425), Height(583393), Height(583358)]
Dec 14 17:54:12.044  INFO {zebrad="b8646fa7"}:sync:obtain_tips: zebra_network::peer_set::set: network request with no ready peers: finding more peers, waiting for 10 peers to answer requests
Dec 14 17:54:15.181  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: extending tips tips.len=1 in_flight=279 lookahead_limit=2000
Dec 14 17:54:21.232  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: exhausted prospective tip set
Dec 14 17:54:21.232  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: waiting to restart sync timeout=45s
Dec 14 17:55:06.240  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: starting sync, obtaining new tips
Dec 14 17:55:06.253  INFO {zebrad="b8646fa7"}:sync:obtain_tips: zebra_state::service: pruned utxo requests old_len=6296 new_len=0 prune_count=6296 tip=Some((Height(583457), block::Hash("000000000089342a2001e491f306f03cbd6e7d702179a066093c58ee69449e28")))
Dec 14 17:55:06.253  INFO {zebrad="b8646fa7"}:sync:obtain_tips:state: zebra_state::util: created block locator tip_height=Height(583457) min_locator_height=583358 locators=[Height(583457), Height(583456), Height(583455), Height(583453), Height(583449), Height(583441), Height(583425), Height(583393), Height(583358)]
Dec 14 17:55:12.333  INFO {zebrad="b8646fa7"}:sync:obtain_tips: zebra_network::peer_set::set: network request with no ready peers: finding more peers, waiting for 10 peers to answer requests
Dec 14 17:55:15.976  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: extending tips tips.len=1 in_flight=268 lookahead_limit=2000
Dec 14 17:55:22.039  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: extending tips tips.len=1 in_flight=165 lookahead_limit=2000
Dec 14 17:55:29.452  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: extending tips tips.len=1 in_flight=264 lookahead_limit=2000
Dec 14 17:55:36.243  INFO {zebrad="b8646fa7"}:sync:extend_tips: zebra_state::service: pruned utxo requests old_len=4439 new_len=4386 prune_count=53 tip=Some((Height(583724), block::Hash("0000000000557b090283ffc133f9dad981bdae3f651883623d73c25d4db93775")))
Dec 14 17:55:37.685  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: extending tips tips.len=1 in_flight=470 lookahead_limit=2000
Dec 14 17:55:43.159  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: extending tips tips.len=1 in_flight=841 lookahead_limit=2000
Dec 14 17:55:58.261  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: extending tips tips.len=1 in_flight=1339 lookahead_limit=2000
Dec 14 17:56:06.252  INFO {zebrad="b8646fa7"}:sync:extend_tips: zebra_state::service: pruned utxo requests old_len=35794 new_len=35693 prune_count=101 tip=Some((Height(583854), block::Hash("000000000060263aa11a9d919cb29fe50374f8a1f8b63e5469e88fce41abac75")))
Dec 14 17:56:12.343  INFO {zebrad="b8646fa7"}:sync:extend_tips: zebra_network::peer_set::set: network request with no ready peers: finding more peers, waiting for 10 peers to answer requests
Dec 14 17:56:13.227  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: extending tips tips.len=1 in_flight=1835 lookahead_limit=2000
Dec 14 17:56:19.276  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: exhausted prospective tip set
Dec 14 17:56:19.276  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: waiting to restart sync timeout=45s
Dec 14 17:56:40.232  INFO {zebrad="b8646fa7"}:peer{addr=40.118.90.57:55002}:msg_as_req{msg=inv}:inbound:download_and_verify{hash=0000000000f226d172ef09b5c4825a292452e5d8ad03227ff7ef7061e748b1f2}: zebra_state::service: pruned utxo requests old_len=47029 new_len=211 prune_count=46818 tip=Some((Height(583854), block::Hash("000000000060263aa11a9d919cb29fe50374f8a1f8b63e5469e88fce41abac75")))
Dec 14 17:57:04.284  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: starting sync, obtaining new tips
Dec 14 17:57:04.284  INFO {zebrad="b8646fa7"}:sync:obtain_tips:state: zebra_state::util: created block locator tip_height=Height(583854) min_locator_height=583755 locators=[Height(583854), Height(583853), Height(583852), Height(583850), Height(583846), Height(583838), Height(583822), Height(583790), Height(583755)]
Dec 14 17:57:12.347  INFO {zebrad="b8646fa7"}:sync:obtain_tips: zebra_network::peer_set::set: network request with no ready peers: finding more peers, waiting for 10 peers to answer requests
Dec 14 17:57:16.682  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: extending tips tips.len=1 in_flight=499 lookahead_limit=2000
Dec 14 17:57:27.490  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: extending tips tips.len=1 in_flight=969 lookahead_limit=2000
Dec 14 17:57:27.819  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: exhausted prospective tip set
Dec 14 17:57:27.819  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: waiting to restart sync timeout=45s
Dec 14 17:58:12.823  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: starting sync, obtaining new tips
Dec 14 17:58:12.867  INFO {zebrad="b8646fa7"}:sync:obtain_tips: zebra_state::service: pruned utxo requests old_len=23077 new_len=253 prune_count=22824 tip=Some((Height(583910), block::Hash("00000000001acbd31f2ee7401ae510c609a223e7a345beea7e915df789a02c2c")))
Dec 14 17:58:12.867  INFO {zebrad="b8646fa7"}:sync:obtain_tips:state: zebra_state::util: created block locator tip_height=Height(583910) min_locator_height=583811 locators=[Height(583910), Height(583909), Height(583908), Height(583906), Height(583902), Height(583894), Height(583878), Height(583846), Height(583811)]
Dec 14 17:58:13.127  INFO {zebrad="b8646fa7"}:sync:obtain_tips: zebra_network::peer_set::set: network request with no ready peers: finding more peers, waiting for 10 peers to answer requests
Dec 14 17:58:18.502  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: extending tips tips.len=2 in_flight=496 lookahead_limit=2000
Dec 14 17:58:24.804  WARN {zebrad="b8646fa7"}:sync: zebrad::components::sync: error extending tips e=
   0: duplicate hash queued for download

Location:
   zebrad/src/components/sync/downloads.rs:116

  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ SPANTRACE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  
   0: zebrad::components::sync::extend_tips
      at zebrad/src/components/sync.rs:381
   1: zebrad::components::sync::sync
      at zebrad/src/components/sync.rs:150
   2: zebrad::commands:: with zebrad="b8646fa7"
      at zebrad/src/commands.rs:56

  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ BACKTRACE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
   1: backtrace::backtrace::libunwind::trace::h56b5b89d46ea18ee
      at /Users/rex/.cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.54/src/backtrace/libunwind.rs:90
        88 │ #[inline(always)]
        89 │ pub unsafe fn trace(mut cb: &mut dyn FnMut(&super::Frame) -> bool) {
        90 >     uw::_Unwind_Backtrace(trace_fn, &mut cb as *mut _ as *mut _);
        91 │ 
        92 │     extern "C" fn trace_fn(
   2: backtrace::backtrace::trace_unsynchronized::h456d388ebbb97b58
      at /Users/rex/.cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.54/src/backtrace/mod.rs:66
        64 │ /// See information on `trace` for caveats on `cb` panicking.
        65 │ pub unsafe fn trace_unsynchronized<F: FnMut(&Frame) -> bool>(mut cb: F) {
        66 >     trace_imp(&mut cb)
        67 │ }
        68 │ 
   3: backtrace::backtrace::trace::h0c2ee1e6653540fd
      at /Users/rex/.cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.54/src/backtrace/mod.rs:53
        51 │ pub fn trace<F: FnMut(&Frame) -> bool>(cb: F) {
        52 │     let _guard = crate::lock::lock();
        53 >     unsafe { trace_unsynchronized(cb) }
        54 │ }
        55 │ 
   4: backtrace::capture::Backtrace::create::h8d054d598eb8a93e
      at /Users/rex/.cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.54/src/capture.rs:176
       174 │         let mut frames = Vec::new();
       175 │         let mut actual_start_index = None;
       176 >         trace(|frame| {
       177 │             frames.push(BacktraceFrame {
       178 │                 frame: Frame::Raw(frame.clone()),
   5: backtrace::capture::Backtrace::new::h89e80605e5dd4af5
      at /Users/rex/.cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.54/src/capture.rs:140
       138 │     #[inline(never)] // want to make sure there's a frame here to remove
       139 │     pub fn new() -> Backtrace {
       140 >         let mut bt = Self::create(Self::new as usize);
       141 │         bt.resolve();
       142 │         bt
   6: color_eyre::config::EyreHook::default::h2b86268b3a262bdc
      at /Users/rex/.cargo/registry/src/github.com-1ecc6299db9ec823/color-eyre-0.5.10/src/config.rs:1016
      1014 │     pub(crate) fn default(&self, error: &(dyn std::error::Error + 'static)) -> crate::Handler {
      1015 │         let backtrace = if lib_verbosity() != Verbosity::Minimal {
      1016 >             Some(backtrace::Backtrace::new())
      1017 │         } else {
      1018 │             None
   7: color_eyre::config::EyreHook::into_eyre_hook::{{closure}}::h0fba5a5bb754400f
      at /Users/rex/.cargo/registry/src/github.com-1ecc6299db9ec823/color-eyre-0.5.10/src/config.rs:1070
      1068 │             + 'static,
      1069 │     > {
      1070 >         Box::new(move |e| Box::new(self.default(e)))
      1071 │     }
      1072 │ }

Run with COLORBT_SHOW_HIDDEN=1 environment variable to disable frame filtering.

Consider reporting this error using this URL: https://github.com/ZcashFoundation/zebra/issues/new?title=%3Cautogenerated-issue%3E&body=%23%23+Error%0A%60%60%60%0AError%3A+%0A+++0%3A+duplicate+hash+queued+for+download%0A%60%60%60%0A%0A%23%23+Metadata%0A%7Ckey%7Cvalue%7C%0A%7C--%7C--%7C%0A%7C**version**%7C3.0.0-alpha.0%7C%0A%7C**git+commit**%7Cb8646fa7%7C%0A%0A%0A%23%23+SpanTrace%0A%0A%3Cdetails%3E%0A%0A%60%60%60%0ASpanTrace%3A%0A+++0%3A+zebrad%3A%3Acomponents%3A%3Async%3A%3Aextend_tips%0A+++++++++++++at+zebrad%2Fsrc%2Fcomponents%2Fsync.rs%3A381%0A+++1%3A+zebrad%3A%3Acomponents%3A%3Async%3A%3Async%0A+++++++++++++at+zebrad%2Fsrc%2Fcomponents%2Fsync.rs%3A150%0A+++2%3A+zebrad%3A%3Acommands%3A%3A%0A+++++++++++with+zebrad%3D%22b8646fa7%22%0A+++++++++++++at+zebrad%2Fsrc%2Fcommands.rs%3A56%0A%60%60%60%0A%3C%2Fdetails%3E%0A%0A%23%23+Backtrace%0A%0A%3Cdetails%3E%0A%0A%60%60%60%0ABacktrace%3A%0A+++0%3A+backtrace%3A%3Abacktrace%3A%3Alibunwind%3A%3Atrace%0A+++++++++++++at+%2FUsers%2Frex%2F.cargo%2Fregistry%2Fsrc%2Fgithub.com-1ecc6299db9ec823%2Fbacktrace-0.3.54%2Fsrc%2Fbacktrace%2Flibunwind.rs%3A90%3A5%0A++++++backtrace%3A%3Abacktrace%3A%3Atrace_unsynchronized%0A+++++++++++++at+%2FUsers%2Frex%2F.cargo%2Fregistry%2Fsrc%2Fgithub.com-1ecc6299db9ec823%2Fbacktrace-0.3.54%2Fsrc%2Fbacktrace%2Fmod.rs%3A66%3A5%0A+++1%3A+backtrace%3A%3Abacktrace%3A%3Atrace%0A+++++++++++++at+%2FUsers%2Frex%2F.cargo%2Fregistry%2Fsrc%2Fgithub.com-1ecc6299db9ec823%2Fbacktrace-0.3.54%2Fsrc%2Fbacktrace%2Fmod.rs%3A53%3A14%0A+++2%3A+backtrace%3A%3Acapture%3A%3ABacktrace%3A%3Acreate%0A+++++++++++++at+%2FUsers%2Frex%2F.cargo%2Fregistry%2Fsrc%2Fgithub.com-1ecc6299db9ec823%2Fbacktrace-0.3.54%2Fsrc%2Fcapture.rs%3A176%3A9%0A+++3%3A+backtrace%3A%3Acapture%3A%3ABacktrace%3A%3Anew%0A+++++++++++++at+%2FUsers%2Frex%2F.cargo%2Fregistry%2Fsrc%2Fgithub.com-1ecc6299db9ec823%2Fbacktrace-0.3.54%2Fsrc%2Fcapture.rs%3A140%3A22%0A+++4%3A+color_eyre%3A%3Aconfig%3A%3AEyreHook%3A%3Adefault%0A+++++++++++++at+%2FUsers%2Frex%2F.cargo%2Fregistry%2Fsrc%2Fgithub.com-1ecc6299db9ec823%2Fcolor-eyre-0.5.10%2Fsrc%2Fconfig.rs%3A1016%3A18%0A+++5%3A+color_eyre%3A%3Aconfig%3A%3AEyreHook%3A%3Ainto_eyre_hook%3A%3A%7B%7Bclosure%7D%7D%0A+++++++++++++at+%2FUsers%2Frex%2F.cargo%2Fregistry%2Fsrc%2Fgithub.com-1ecc6299db9ec823%2Fcolor-eyre-0.5.10%2Fsrc%2Fconfig.rs%3A1070%3A36%0A%0A%60%60%60%0A%3C%2Fdetails%3E
Dec 14 17:58:24.810  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: waiting to restart sync timeout=45s
Dec 14 17:59:09.821  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: starting sync, obtaining new tips
Dec 14 17:59:09.839  INFO {zebrad="b8646fa7"}:sync:obtain_tips: zebra_state::service: pruned utxo requests old_len=8687 new_len=42 prune_count=8645 tip=Some((Height(584005), block::Hash("000000000146297bedcefcc9f47123b007e62946f569fdd62d16828659f1f2fb")))
Dec 14 17:59:09.840  INFO {zebrad="b8646fa7"}:sync:obtain_tips:state: zebra_state::util: created block locator tip_height=Height(584005) min_locator_height=583906 locators=[Height(584005), Height(584004), Height(584003), Height(584001), Height(583997), Height(583989), Height(583973), Height(583941), Height(583906)]
Dec 14 17:59:15.866  INFO {zebrad="b8646fa7"}:sync:obtain_tips: zebra_network::peer_set::set: network request with no ready peers: finding more peers, waiting for 10 peers to answer requests
Dec 14 17:59:16.364  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: extending tips tips.len=1 in_flight=55 lookahead_limit=2000
Dec 14 17:59:22.372  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: exhausted prospective tip set
Dec 14 17:59:22.372  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: waiting to restart sync timeout=45s
Dec 14 18:00:07.374  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: starting sync, obtaining new tips
Dec 14 18:00:07.379  INFO {zebrad="b8646fa7"}:sync:obtain_tips: zebra_state::service: pruned utxo requests old_len=1045 new_len=0 prune_count=1045 tip=Some((Height(584005), block::Hash("000000000146297bedcefcc9f47123b007e62946f569fdd62d16828659f1f2fb")))
Dec 14 18:00:07.379  INFO {zebrad="b8646fa7"}:sync:obtain_tips:state: zebra_state::util: created block locator tip_height=Height(584005) min_locator_height=583906 locators=[Height(584005), Height(584004), Height(584003), Height(584001), Height(583997), Height(583989), Height(583973), Height(583941), Height(583906)]
Dec 14 18:00:15.884  INFO {zebrad="b8646fa7"}:sync:obtain_tips: zebra_network::peer_set::set: network request with no ready peers: finding more peers, waiting for 10 peers to answer requests
Dec 14 18:00:15.951  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: extending tips tips.len=1 in_flight=227 lookahead_limit=2000
Dec 14 18:00:22.014  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: extending tips tips.len=1 in_flight=164 lookahead_limit=2000
Dec 14 18:00:28.023  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: exhausted prospective tip set
Dec 14 18:00:28.023  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: waiting to restart sync timeout=45s
Dec 14 18:01:13.028  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: starting sync, obtaining new tips
Dec 14 18:01:13.037  INFO {zebrad="b8646fa7"}:sync:obtain_tips: zebra_state::service: pruned utxo requests old_len=3399 new_len=0 prune_count=3399 tip=Some((Height(584098), block::Hash("0000000001245515c78a627c3be896b4eb6964f5ae90ee5e0f343bced536eac6")))
Dec 14 18:01:13.037  INFO {zebrad="b8646fa7"}:sync:obtain_tips:state: zebra_state::util: created block locator tip_height=Height(584098) min_locator_height=583999 locators=[Height(584098), Height(584097), Height(584096), Height(584094), Height(584090), Height(584082), Height(584066), Height(584034), Height(583999)]
Dec 14 18:01:19.117  INFO {zebrad="b8646fa7"}:sync:obtain_tips: zebra_network::peer_set::set: network request with no ready peers: finding more peers, waiting for 10 peers to answer requests
Dec 14 18:01:22.010  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: extending tips tips.len=2 in_flight=262 lookahead_limit=2000
Dec 14 18:01:34.034  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: exhausted prospective tip set
Dec 14 18:01:34.034  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: waiting to restart sync timeout=45s
Dec 14 18:02:19.039  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: starting sync, obtaining new tips
Dec 14 18:02:19.044  INFO {zebrad="b8646fa7"}:sync:obtain_tips: zebra_state::service: pruned utxo requests old_len=1785 new_len=0 prune_count=1785 tip=Some((Height(584244), block::Hash("0000000000a06adc72e1a2747e865fde2af0a1bcbd5cf3bbeab91a8b25df3603")))
Dec 14 18:02:19.044  INFO {zebrad="b8646fa7"}:sync:obtain_tips:state: zebra_state::util: created block locator tip_height=Height(584244) min_locator_height=584145 locators=[Height(584244), Height(584243), Height(584242), Height(584240), Height(584236), Height(584228), Height(584212), Height(584180), Height(584145)]
Dec 14 18:02:25.049  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: exhausted prospective tip set
Dec 14 18:02:25.049  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: waiting to restart sync timeout=45s
Dec 14 18:03:10.055  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: starting sync, obtaining new tips
Dec 14 18:03:10.056  INFO {zebrad="b8646fa7"}:sync:obtain_tips:state: zebra_state::util: created block locator tip_height=Height(584244) min_locator_height=584145 locators=[Height(584244), Height(584243), Height(584242), Height(584240), Height(584236), Height(584228), Height(584212), Height(584180), Height(584145)]
Dec 14 18:03:16.063  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: exhausted prospective tip set
Dec 14 18:03:16.063  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: waiting to restart sync timeout=45s
Dec 14 18:04:01.066  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: starting sync, obtaining new tips
Dec 14 18:04:01.067  INFO {zebrad="b8646fa7"}:sync:obtain_tips:state: zebra_state::util: created block locator tip_height=Height(584244) min_locator_height=584145 locators=[Height(584244), Height(584243), Height(584242), Height(584240), Height(584236), Height(584228), Height(584212), Height(584180), Height(584145)]
Dec 14 18:04:07.073  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: exhausted prospective tip set
Dec 14 18:04:07.073  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: waiting to restart sync timeout=45s
Dec 14 18:04:52.076  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: starting sync, obtaining new tips
Dec 14 18:04:52.076  INFO {zebrad="b8646fa7"}:sync:obtain_tips:state: zebra_state::util: created block locator tip_height=Height(584244) min_locator_height=584145 locators=[Height(584244), Height(584243), Height(584242), Height(584240), Height(584236), Height(584228), Height(584212), Height(584180), Height(584145)]
Dec 14 18:04:58.079  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: exhausted prospective tip set
Dec 14 18:04:58.079  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: waiting to restart sync timeout=45s
Dec 14 18:05:43.081  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: starting sync, obtaining new tips
Dec 14 18:05:43.082  INFO {zebrad="b8646fa7"}:sync:obtain_tips:state: zebra_state::util: created block locator tip_height=Height(584244) min_locator_height=584145 locators=[Height(584244), Height(584243), Height(584242), Height(584240), Height(584236), Height(584228), Height(584212), Height(584180), Height(584145)]
Dec 14 18:05:49.088  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: exhausted prospective tip set
Dec 14 18:05:49.088  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: waiting to restart sync timeout=45s
Dec 14 18:06:34.092  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: starting sync, obtaining new tips
Dec 14 18:06:34.093  INFO {zebrad="b8646fa7"}:sync:obtain_tips:state: zebra_state::util: created block locator tip_height=Height(584244) min_locator_height=584145 locators=[Height(584244), Height(584243), Height(584242), Height(584240), Height(584236), Height(584228), Height(584212), Height(584180), Height(584145)]
Dec 14 18:06:40.096  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: exhausted prospective tip set
Dec 14 18:06:40.097  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: waiting to restart sync timeout=45s
Dec 14 18:07:25.102  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: starting sync, obtaining new tips
Dec 14 18:07:25.103  INFO {zebrad="b8646fa7"}:sync:obtain_tips:state: zebra_state::util: created block locator tip_height=Height(584244) min_locator_height=584145 locators=[Height(584244), Height(584243), Height(584242), Height(584240), Height(584236), Height(584228), Height(584212), Height(584180), Height(584145)]
Dec 14 18:07:31.107  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: exhausted prospective tip set
Dec 14 18:07:31.107  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: waiting to restart sync timeout=45s
Dec 14 18:08:16.110  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: starting sync, obtaining new tips
Dec 14 18:08:16.112  INFO {zebrad="b8646fa7"}:sync:obtain_tips: zebra_state::service: pruned utxo requests old_len=28 new_len=4 prune_count=24 tip=Some((Height(584244), block::Hash("0000000000a06adc72e1a2747e865fde2af0a1bcbd5cf3bbeab91a8b25df3603")))
Dec 14 18:08:16.112  INFO {zebrad="b8646fa7"}:sync:obtain_tips:state: zebra_state::util: created block locator tip_height=Height(584244) min_locator_height=584145 locators=[Height(584244), Height(584243), Height(584242), Height(584240), Height(584236), Height(584228), Height(584212), Height(584180), Height(584145)]
Dec 14 18:08:22.115  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: exhausted prospective tip set
Dec 14 18:08:22.115  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: waiting to restart sync timeout=45s
Dec 14 18:09:07.118  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: starting sync, obtaining new tips
Dec 14 18:09:07.118  INFO {zebrad="b8646fa7"}:sync:obtain_tips:state: zebra_state::util: created block locator tip_height=Height(584244) min_locator_height=584145 locators=[Height(584244), Height(584243), Height(584242), Height(584240), Height(584236), Height(584228), Height(584212), Height(584180), Height(584145)]
Dec 14 18:09:13.122  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: exhausted prospective tip set
Dec 14 18:09:13.122  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: waiting to restart sync timeout=45s
Dec 14 18:09:58.127  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: starting sync, obtaining new tips
Dec 14 18:09:58.129  INFO {zebrad="b8646fa7"}:sync:obtain_tips:state: zebra_state::util: created block locator tip_height=Height(584244) min_locator_height=584145 locators=[Height(584244), Height(584243), Height(584242), Height(584240), Height(584236), Height(584228), Height(584212), Height(584180), Height(584145)]
Dec 14 18:10:04.132  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: exhausted prospective tip set
Dec 14 18:10:04.132  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: waiting to restart sync timeout=45s
Dec 14 18:10:47.789  INFO {zebrad="b8646fa7"}:peer{addr=40.118.90.57:55002}:msg_as_req{msg=inv}:inbound:download_and_verify{hash=0000000000dc3328e239295061bb780b554020c0d8babae9c42e10e23afd79ef}: zebra_state::service: pruned utxo requests old_len=5 new_len=1 prune_count=4 tip=Some((Height(584244), block::Hash("0000000000a06adc72e1a2747e865fde2af0a1bcbd5cf3bbeab91a8b25df3603")))
Dec 14 18:10:49.135  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: starting sync, obtaining new tips
Dec 14 18:10:49.136  INFO {zebrad="b8646fa7"}:sync:obtain_tips:state: zebra_state::util: created block locator tip_height=Height(584244) min_locator_height=584145 locators=[Height(584244), Height(584243), Height(584242), Height(584240), Height(584236), Height(584228), Height(584212), Height(584180), Height(584145)]
Dec 14 18:10:55.139  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: exhausted prospective tip set
Dec 14 18:10:55.139  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: waiting to restart sync timeout=45s
Dec 14 18:11:40.141  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: starting sync, obtaining new tips
Dec 14 18:11:40.141  INFO {zebrad="b8646fa7"}:sync:obtain_tips:state: zebra_state::util: created block locator tip_height=Height(584244) min_locator_height=584145 locators=[Height(584244), Height(584243), Height(584242), Height(584240), Height(584236), Height(584228), Height(584212), Height(584180), Height(584145)]
Dec 14 18:11:46.147  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: exhausted prospective tip set
Dec 14 18:11:46.147  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: waiting to restart sync timeout=45s
Dec 14 18:12:31.154  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: starting sync, obtaining new tips
Dec 14 18:12:31.156  INFO {zebrad="b8646fa7"}:sync:obtain_tips: zebra_state::service: pruned utxo requests old_len=21 new_len=20 prune_count=1 tip=Some((Height(584244), block::Hash("0000000000a06adc72e1a2747e865fde2af0a1bcbd5cf3bbeab91a8b25df3603")))
Dec 14 18:12:31.156  INFO {zebrad="b8646fa7"}:sync:obtain_tips:state: zebra_state::util: created block locator tip_height=Height(584244) min_locator_height=584145 locators=[Height(584244), Height(584243), Height(584242), Height(584240), Height(584236), Height(584228), Height(584212), Height(584180), Height(584145)]
Dec 14 18:12:37.160  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: exhausted prospective tip set
Dec 14 18:12:37.160  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: waiting to restart sync timeout=45s
Dec 14 18:13:22.164  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: starting sync, obtaining new tips
Dec 14 18:13:22.165  INFO {zebrad="b8646fa7"}:sync:obtain_tips:state: zebra_state::util: created block locator tip_height=Height(584244) min_locator_height=584145 locators=[Height(584244), Height(584243), Height(584242), Height(584240), Height(584236), Height(584228), Height(584212), Height(584180), Height(584145)]
Dec 14 18:13:28.171  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: exhausted prospective tip set
Dec 14 18:13:28.171  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: waiting to restart sync timeout=45s
Dec 14 18:14:13.175  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: starting sync, obtaining new tips
Dec 14 18:14:13.176  INFO {zebrad="b8646fa7"}:sync:obtain_tips: zebra_state::service: pruned utxo requests old_len=25 new_len=7 prune_count=18 tip=Some((Height(584244), block::Hash("0000000000a06adc72e1a2747e865fde2af0a1bcbd5cf3bbeab91a8b25df3603")))
Dec 14 18:14:13.176  INFO {zebrad="b8646fa7"}:sync:obtain_tips:state: zebra_state::util: created block locator tip_height=Height(584244) min_locator_height=584145 locators=[Height(584244), Height(584243), Height(584242), Height(584240), Height(584236), Height(584228), Height(584212), Height(584180), Height(584145)]
Dec 14 18:14:19.179  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: exhausted prospective tip set
Dec 14 18:14:19.179  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: waiting to restart sync timeout=45s
Dec 14 18:15:04.183  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: starting sync, obtaining new tips
Dec 14 18:15:04.184  INFO {zebrad="b8646fa7"}:sync:obtain_tips: zebra_state::service: pruned utxo requests old_len=7 new_len=5 prune_count=2 tip=Some((Height(584244), block::Hash("0000000000a06adc72e1a2747e865fde2af0a1bcbd5cf3bbeab91a8b25df3603")))
Dec 14 18:15:04.184  INFO {zebrad="b8646fa7"}:sync:obtain_tips:state: zebra_state::util: created block locator tip_height=Height(584244) min_locator_height=584145 locators=[Height(584244), Height(584243), Height(584242), Height(584240), Height(584236), Height(584228), Height(584212), Height(584180), Height(584145)]
Dec 14 18:15:10.188  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: exhausted prospective tip set
Dec 14 18:15:10.188  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: waiting to restart sync timeout=45s
Dec 14 18:15:55.193  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: starting sync, obtaining new tips
Dec 14 18:15:55.195  INFO {zebrad="b8646fa7"}:sync:obtain_tips:state: zebra_state::util: created block locator tip_height=Height(584244) min_locator_height=584145 locators=[Height(584244), Height(584243), Height(584242), Height(584240), Height(584236), Height(584228), Height(584212), Height(584180), Height(584145)]
Dec 14 18:16:01.198  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: exhausted prospective tip set
Dec 14 18:16:01.198  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: waiting to restart sync timeout=45s
Dec 14 18:16:44.079  INFO {zebrad="b8646fa7"}:peer{addr=40.118.90.57:55002}:msg_as_req{msg=inv}:inbound:download_and_verify{hash=0000000001d1ee94d496ea39574b5f885ad2f2d1a0f2084d1951e94d73cebefa}: zebra_state::service: pruned utxo requests old_len=5 new_len=0 prune_count=5 tip=Some((Height(584244), block::Hash("0000000000a06adc72e1a2747e865fde2af0a1bcbd5cf3bbeab91a8b25df3603")))
Dec 14 18:16:46.205  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: starting sync, obtaining new tips
Dec 14 18:16:46.205  INFO {zebrad="b8646fa7"}:sync:obtain_tips:state: zebra_state::util: created block locator tip_height=Height(584244) min_locator_height=584145 locators=[Height(584244), Height(584243), Height(584242), Height(584240), Height(584236), Height(584228), Height(584212), Height(584180), Height(584145)]
Dec 14 18:16:52.210  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: exhausted prospective tip set
Dec 14 18:16:52.210  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: waiting to restart sync timeout=45s
Dec 14 18:17:37.213  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: starting sync, obtaining new tips
Dec 14 18:17:37.214  INFO {zebrad="b8646fa7"}:sync:obtain_tips:state: zebra_state::util: created block locator tip_height=Height(584244) min_locator_height=584145 locators=[Height(584244), Height(584243), Height(584242), Height(584240), Height(584236), Height(584228), Height(584212), Height(584180), Height(584145)]
Dec 14 18:17:43.220  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: exhausted prospective tip set
Dec 14 18:17:43.220  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: waiting to restart sync timeout=45s
Dec 14 18:18:28.222  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: starting sync, obtaining new tips
Dec 14 18:18:28.223  INFO {zebrad="b8646fa7"}:sync:obtain_tips:state: zebra_state::util: created block locator tip_height=Height(584244) min_locator_height=584145 locators=[Height(584244), Height(584243), Height(584242), Height(584240), Height(584236), Height(584228), Height(584212), Height(584180), Height(584145)]
Dec 14 18:18:34.228  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: exhausted prospective tip set
Dec 14 18:18:34.228  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: waiting to restart sync timeout=45s
Dec 14 18:19:19.235  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: starting sync, obtaining new tips
Dec 14 18:19:19.236  INFO {zebrad="b8646fa7"}:sync:obtain_tips:state: zebra_state::util: created block locator tip_height=Height(584244) min_locator_height=584145 locators=[Height(584244), Height(584243), Height(584242), Height(584240), Height(584236), Height(584228), Height(584212), Height(584180), Height(584145)]
Dec 14 18:19:25.240  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: exhausted prospective tip set
Dec 14 18:19:25.240  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: waiting to restart sync timeout=45s
Dec 14 18:20:10.244  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: starting sync, obtaining new tips
Dec 14 18:20:10.245  INFO {zebrad="b8646fa7"}:sync:obtain_tips: zebra_state::service: pruned utxo requests old_len=59 new_len=48 prune_count=11 tip=Some((Height(584244), block::Hash("0000000000a06adc72e1a2747e865fde2af0a1bcbd5cf3bbeab91a8b25df3603")))
Dec 14 18:20:10.245  INFO {zebrad="b8646fa7"}:sync:obtain_tips:state: zebra_state::util: created block locator tip_height=Height(584244) min_locator_height=584145 locators=[Height(584244), Height(584243), Height(584242), Height(584240), Height(584236), Height(584228), Height(584212), Height(584180), Height(584145)]
Dec 14 18:20:16.249  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: exhausted prospective tip set
Dec 14 18:20:16.249  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: waiting to restart sync timeout=45s
Dec 14 18:21:01.253  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: starting sync, obtaining new tips
Dec 14 18:21:01.255  INFO {zebrad="b8646fa7"}:sync:obtain_tips:state: zebra_state::util: created block locator tip_height=Height(584244) min_locator_height=584145 locators=[Height(584244), Height(584243), Height(584242), Height(584240), Height(584236), Height(584228), Height(584212), Height(584180), Height(584145)]
Dec 14 18:21:07.260  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: exhausted prospective tip set
Dec 14 18:21:07.260  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: waiting to restart sync timeout=45s
Dec 14 18:21:52.265  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: starting sync, obtaining new tips
Dec 14 18:21:52.266  INFO {zebrad="b8646fa7"}:sync:obtain_tips:state: zebra_state::util: created block locator tip_height=Height(584244) min_locator_height=584145 locators=[Height(584244), Height(584243), Height(584242), Height(584240), Height(584236), Height(584228), Height(584212), Height(584180), Height(584145)]
Dec 14 18:21:58.276  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: exhausted prospective tip set
Dec 14 18:21:58.276  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: waiting to restart sync timeout=45s
Dec 14 18:22:43.279  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: starting sync, obtaining new tips
Dec 14 18:22:43.281  INFO {zebrad="b8646fa7"}:sync:obtain_tips: zebra_state::service: pruned utxo requests old_len=48 new_len=0 prune_count=48 tip=Some((Height(584244), block::Hash("0000000000a06adc72e1a2747e865fde2af0a1bcbd5cf3bbeab91a8b25df3603")))
Dec 14 18:22:43.281  INFO {zebrad="b8646fa7"}:sync:obtain_tips:state: zebra_state::util: created block locator tip_height=Height(584244) min_locator_height=584145 locators=[Height(584244), Height(584243), Height(584242), Height(584240), Height(584236), Height(584228), Height(584212), Height(584180), Height(584145)]
Dec 14 18:22:49.284  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: exhausted prospective tip set
Dec 14 18:22:49.284  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: waiting to restart sync timeout=45s
Dec 14 18:23:34.292  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: starting sync, obtaining new tips
Dec 14 18:23:34.293  INFO {zebrad="b8646fa7"}:sync:obtain_tips:state: zebra_state::util: created block locator tip_height=Height(584244) min_locator_height=584145 locators=[Height(584244), Height(584243), Height(584242), Height(584240), Height(584236), Height(584228), Height(584212), Height(584180), Height(584145)]
Dec 14 18:23:40.297  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: exhausted prospective tip set
Dec 14 18:23:40.298  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: waiting to restart sync timeout=45s
Dec 14 18:24:25.302  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: starting sync, obtaining new tips
Dec 14 18:24:25.304  INFO {zebrad="b8646fa7"}:sync:obtain_tips:state: zebra_state::util: created block locator tip_height=Height(584244) min_locator_height=584145 locators=[Height(584244), Height(584243), Height(584242), Height(584240), Height(584236), Height(584228), Height(584212), Height(584180), Height(584145)]
Dec 14 18:24:31.309  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: exhausted prospective tip set
Dec 14 18:24:31.309  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: waiting to restart sync timeout=45s
Dec 14 18:25:16.311  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: starting sync, obtaining new tips
Dec 14 18:25:16.312  INFO {zebrad="b8646fa7"}:sync:obtain_tips:state: zebra_state::util: created block locator tip_height=Height(584244) min_locator_height=584145 locators=[Height(584244), Height(584243), Height(584242), Height(584240), Height(584236), Height(584228), Height(584212), Height(584180), Height(584145)]
Dec 14 18:25:22.316  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: exhausted prospective tip set
Dec 14 18:25:22.316  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: waiting to restart sync timeout=45s
Dec 14 18:26:07.320  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: starting sync, obtaining new tips
Dec 14 18:26:07.321  INFO {zebrad="b8646fa7"}:sync:obtain_tips:state: zebra_state::util: created block locator tip_height=Height(584244) min_locator_height=584145 locators=[Height(584244), Height(584243), Height(584242), Height(584240), Height(584236), Height(584228), Height(584212), Height(584180), Height(584145)]
Dec 14 18:26:13.327  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: exhausted prospective tip set
Dec 14 18:26:13.327  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: waiting to restart sync timeout=45s
Dec 14 18:26:58.331  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: starting sync, obtaining new tips
Dec 14 18:26:58.332  INFO {zebrad="b8646fa7"}:sync:obtain_tips:state: zebra_state::util: created block locator tip_height=Height(584244) min_locator_height=584145 locators=[Height(584244), Height(584243), Height(584242), Height(584240), Height(584236), Height(584228), Height(584212), Height(584180), Height(584145)]
Dec 14 18:27:04.336  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: exhausted prospective tip set
Dec 14 18:27:04.336  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: waiting to restart sync timeout=45s
Dec 14 18:27:49.341  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: starting sync, obtaining new tips
Dec 14 18:27:49.343  INFO {zebrad="b8646fa7"}:sync:obtain_tips: zebra_state::service: pruned utxo requests old_len=19 new_len=0 prune_count=19 tip=Some((Height(584244), block::Hash("0000000000a06adc72e1a2747e865fde2af0a1bcbd5cf3bbeab91a8b25df3603")))
Dec 14 18:27:49.343  INFO {zebrad="b8646fa7"}:sync:obtain_tips:state: zebra_state::util: created block locator tip_height=Height(584244) min_locator_height=584145 locators=[Height(584244), Height(584243), Height(584242), Height(584240), Height(584236), Height(584228), Height(584212), Height(584180), Height(584145)]
Dec 14 18:27:55.347  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: exhausted prospective tip set
Dec 14 18:27:55.347  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: waiting to restart sync timeout=45s
Dec 14 18:28:40.349  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: starting sync, obtaining new tips
Dec 14 18:28:40.351  INFO {zebrad="b8646fa7"}:sync:obtain_tips:state: zebra_state::util: created block locator tip_height=Height(584244) min_locator_height=584145 locators=[Height(584244), Height(584243), Height(584242), Height(584240), Height(584236), Height(584228), Height(584212), Height(584180), Height(584145)]
Dec 14 18:28:46.361  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: exhausted prospective tip set
Dec 14 18:28:46.361  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: waiting to restart sync timeout=45s
Dec 14 18:29:31.365  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: starting sync, obtaining new tips
Dec 14 18:29:31.366  INFO {zebrad="b8646fa7"}:sync:obtain_tips:state: zebra_state::util: created block locator tip_height=Height(584244) min_locator_height=584145 locators=[Height(584244), Height(584243), Height(584242), Height(584240), Height(584236), Height(584228), Height(584212), Height(584180), Height(584145)]
Dec 14 18:29:37.373  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: exhausted prospective tip set
Dec 14 18:29:37.373  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: waiting to restart sync timeout=45s
Dec 14 18:30:22.375  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: starting sync, obtaining new tips
Dec 14 18:30:22.376  INFO {zebrad="b8646fa7"}:sync:obtain_tips:state: zebra_state::util: created block locator tip_height=Height(584244) min_locator_height=584145 locators=[Height(584244), Height(584243), Height(584242), Height(584240), Height(584236), Height(584228), Height(584212), Height(584180), Height(584145)]
Dec 14 18:30:28.380  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: exhausted prospective tip set
Dec 14 18:30:28.380  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: waiting to restart sync timeout=45s
Dec 14 18:31:13.384  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: starting sync, obtaining new tips
Dec 14 18:31:13.385  INFO {zebrad="b8646fa7"}:sync:obtain_tips: zebra_state::service: pruned utxo requests old_len=17 new_len=15 prune_count=2 tip=Some((Height(584244), block::Hash("0000000000a06adc72e1a2747e865fde2af0a1bcbd5cf3bbeab91a8b25df3603")))
Dec 14 18:31:13.385  INFO {zebrad="b8646fa7"}:sync:obtain_tips:state: zebra_state::util: created block locator tip_height=Height(584244) min_locator_height=584145 locators=[Height(584244), Height(584243), Height(584242), Height(584240), Height(584236), Height(584228), Height(584212), Height(584180), Height(584145)]
Dec 14 18:31:19.392  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: exhausted prospective tip set
Dec 14 18:31:19.392  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: waiting to restart sync timeout=45s
Dec 14 18:32:04.394  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: starting sync, obtaining new tips
Dec 14 18:32:04.395  INFO {zebrad="b8646fa7"}:sync:obtain_tips:state: zebra_state::util: created block locator tip_height=Height(584244) min_locator_height=584145 locators=[Height(584244), Height(584243), Height(584242), Height(584240), Height(584236), Height(584228), Height(584212), Height(584180), Height(584145)]
Dec 14 18:32:10.401  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: exhausted prospective tip set
Dec 14 18:32:10.401  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: waiting to restart sync timeout=45s
Dec 14 18:32:35.858  INFO {zebrad="b8646fa7"}:peer{addr=40.118.90.57:55002}:msg_as_req{msg=inv}:inbound:download_and_verify{hash=0000000000e3f44784ac1da3f38e6b266a3aa79df08d90e8ae11f21ce91be7f1}: zebra_state::service: pruned utxo requests old_len=15 new_len=4 prune_count=11 tip=Some((Height(584244), block::Hash("0000000000a06adc72e1a2747e865fde2af0a1bcbd5cf3bbeab91a8b25df3603")))
Dec 14 18:32:55.402  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: starting sync, obtaining new tips
Dec 14 18:32:55.403  INFO {zebrad="b8646fa7"}:sync:obtain_tips:state: zebra_state::util: created block locator tip_height=Height(584244) min_locator_height=584145 locators=[Height(584244), Height(584243), Height(584242), Height(584240), Height(584236), Height(584228), Height(584212), Height(584180), Height(584145)]
Dec 14 18:33:01.410  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: exhausted prospective tip set
Dec 14 18:33:01.410  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: waiting to restart sync timeout=45s
Dec 14 18:33:46.412  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: starting sync, obtaining new tips
Dec 14 18:33:46.413  INFO {zebrad="b8646fa7"}:sync:obtain_tips:state: zebra_state::util: created block locator tip_height=Height(584244) min_locator_height=584145 locators=[Height(584244), Height(584243), Height(584242), Height(584240), Height(584236), Height(584228), Height(584212), Height(584180), Height(584145)]
Dec 14 18:33:52.422  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: exhausted prospective tip set
Dec 14 18:33:52.423  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: waiting to restart sync timeout=45s
Dec 14 18:34:18.983  INFO {zebrad="b8646fa7"}:peer{addr=40.118.90.57:55002}:msg_as_req{msg=inv}:inbound:download_and_verify{hash=00000000006effea5bcc0dada4805acb104e78282090290dfe3b68be86f619a9}: zebra_state::service: pruned utxo requests old_len=9 new_len=5 prune_count=4 tip=Some((Height(584244), block::Hash("0000000000a06adc72e1a2747e865fde2af0a1bcbd5cf3bbeab91a8b25df3603")))
Dec 14 18:34:37.426  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: starting sync, obtaining new tips
Dec 14 18:34:37.427  INFO {zebrad="b8646fa7"}:sync:obtain_tips:state: zebra_state::util: created block locator tip_height=Height(584244) min_locator_height=584145 locators=[Height(584244), Height(584243), Height(584242), Height(584240), Height(584236), Height(584228), Height(584212), Height(584180), Height(584145)]
Dec 14 18:34:43.431  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: exhausted prospective tip set
Dec 14 18:34:43.431  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: waiting to restart sync timeout=45s
Dec 14 18:35:28.433  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: starting sync, obtaining new tips
Dec 14 18:35:28.435  INFO {zebrad="b8646fa7"}:sync:obtain_tips:state: zebra_state::util: created block locator tip_height=Height(584244) min_locator_height=584145 locators=[Height(584244), Height(584243), Height(584242), Height(584240), Height(584236), Height(584228), Height(584212), Height(584180), Height(584145)]
Dec 14 18:35:34.442  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: exhausted prospective tip set
Dec 14 18:35:34.442  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: waiting to restart sync timeout=45s
Dec 14 18:36:19.445  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: starting sync, obtaining new tips
Dec 14 18:36:19.446  INFO {zebrad="b8646fa7"}:sync:obtain_tips: zebra_state::service: pruned utxo requests old_len=6 new_len=1 prune_count=5 tip=Some((Height(584244), block::Hash("0000000000a06adc72e1a2747e865fde2af0a1bcbd5cf3bbeab91a8b25df3603")))
Dec 14 18:36:19.447  INFO {zebrad="b8646fa7"}:sync:obtain_tips:state: zebra_state::util: created block locator tip_height=Height(584244) min_locator_height=584145 locators=[Height(584244), Height(584243), Height(584242), Height(584240), Height(584236), Height(584228), Height(584212), Height(584180), Height(584145)]
Dec 14 18:36:25.450  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: exhausted prospective tip set
Dec 14 18:36:25.451  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: waiting to restart sync timeout=45s
Dec 14 18:37:10.456  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: starting sync, obtaining new tips
Dec 14 18:37:10.458  INFO {zebrad="b8646fa7"}:sync:obtain_tips:state: zebra_state::util: created block locator tip_height=Height(584244) min_locator_height=584145 locators=[Height(584244), Height(584243), Height(584242), Height(584240), Height(584236), Height(584228), Height(584212), Height(584180), Height(584145)]
Dec 14 18:37:16.467  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: exhausted prospective tip set
Dec 14 18:37:16.467  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: waiting to restart sync timeout=45s
Dec 14 18:37:44.903  INFO {zebrad="b8646fa7"}:peer{addr=142.4.209.60:8233}:msg_as_req{msg=inv}:inbound:download_and_verify{hash=0000000000fce875ab4588b29d5d38c0e35e64dbb5d941c64f06a6f1e3e8c56c}: zebra_state::service: pruned utxo requests old_len=1 new_len=0 prune_count=1 tip=Some((Height(584244), block::Hash("0000000000a06adc72e1a2747e865fde2af0a1bcbd5cf3bbeab91a8b25df3603")))
Dec 14 18:38:01.469  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: starting sync, obtaining new tips
Dec 14 18:38:01.470  INFO {zebrad="b8646fa7"}:sync:obtain_tips:state: zebra_state::util: created block locator tip_height=Height(584244) min_locator_height=584145 locators=[Height(584244), Height(584243), Height(584242), Height(584240), Height(584236), Height(584228), Height(584212), Height(584180), Height(584145)]
Dec 14 18:38:07.474  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: exhausted prospective tip set
Dec 14 18:38:07.474  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: waiting to restart sync timeout=45s
Dec 14 18:38:52.478  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: starting sync, obtaining new tips
Dec 14 18:38:52.480  INFO {zebrad="b8646fa7"}:sync:obtain_tips:state: zebra_state::util: created block locator tip_height=Height(584244) min_locator_height=584145 locators=[Height(584244), Height(584243), Height(584242), Height(584240), Height(584236), Height(584228), Height(584212), Height(584180), Height(584145)]
Dec 14 18:38:58.484  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: exhausted prospective tip set
Dec 14 18:38:58.484  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: waiting to restart sync timeout=45s
Dec 14 18:39:43.489  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: starting sync, obtaining new tips
Dec 14 18:39:43.490  INFO {zebrad="b8646fa7"}:sync:obtain_tips:state: zebra_state::util: created block locator tip_height=Height(584244) min_locator_height=584145 locators=[Height(584244), Height(584243), Height(584242), Height(584240), Height(584236), Height(584228), Height(584212), Height(584180), Height(584145)]
Dec 14 18:39:49.496  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: exhausted prospective tip set
Dec 14 18:39:49.496  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: waiting to restart sync timeout=45s
Dec 14 18:40:34.503  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: starting sync, obtaining new tips
Dec 14 18:40:34.504  INFO {zebrad="b8646fa7"}:sync:obtain_tips:state: zebra_state::util: created block locator tip_height=Height(584244) min_locator_height=584145 locators=[Height(584244), Height(584243), Height(584242), Height(584240), Height(584236), Height(584228), Height(584212), Height(584180), Height(584145)]
Dec 14 18:40:40.511  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: exhausted prospective tip set
Dec 14 18:40:40.511  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: waiting to restart sync timeout=45s
Dec 14 18:41:25.514  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: starting sync, obtaining new tips
Dec 14 18:41:25.515  INFO {zebrad="b8646fa7"}:sync:obtain_tips: zebra_state::service: pruned utxo requests old_len=17 new_len=0 prune_count=17 tip=Some((Height(584244), block::Hash("0000000000a06adc72e1a2747e865fde2af0a1bcbd5cf3bbeab91a8b25df3603")))
Dec 14 18:41:25.516  INFO {zebrad="b8646fa7"}:sync:obtain_tips:state: zebra_state::util: created block locator tip_height=Height(584244) min_locator_height=584145 locators=[Height(584244), Height(584243), Height(584242), Height(584240), Height(584236), Height(584228), Height(584212), Height(584180), Height(584145)]
Dec 14 18:41:31.523  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: exhausted prospective tip set
Dec 14 18:41:31.523  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: waiting to restart sync timeout=45s
Dec 14 18:42:16.532  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: starting sync, obtaining new tips
Dec 14 18:42:16.535  INFO {zebrad="b8646fa7"}:sync:obtain_tips:state: zebra_state::util: created block locator tip_height=Height(584244) min_locator_height=584145 locators=[Height(584244), Height(584243), Height(584242), Height(584240), Height(584236), Height(584228), Height(584212), Height(584180), Height(584145)]
Dec 14 18:42:22.547  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: exhausted prospective tip set
Dec 14 18:42:22.547  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: waiting to restart sync timeout=45s
Dec 14 18:43:07.550  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: starting sync, obtaining new tips
Dec 14 18:43:07.551  INFO {zebrad="b8646fa7"}:sync:obtain_tips:state: zebra_state::util: created block locator tip_height=Height(584244) min_locator_height=584145 locators=[Height(584244), Height(584243), Height(584242), Height(584240), Height(584236), Height(584228), Height(584212), Height(584180), Height(584145)]
Dec 14 18:43:13.556  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: exhausted prospective tip set
Dec 14 18:43:13.556  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: waiting to restart sync timeout=45s
Dec 14 18:43:58.559  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: starting sync, obtaining new tips
Dec 14 18:43:58.559  INFO {zebrad="b8646fa7"}:sync:obtain_tips:state: zebra_state::util: created block locator tip_height=Height(584244) min_locator_height=584145 locators=[Height(584244), Height(584243), Height(584242), Height(584240), Height(584236), Height(584228), Height(584212), Height(584180), Height(584145)]
Dec 14 18:44:04.567  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: exhausted prospective tip set
Dec 14 18:44:04.567  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: waiting to restart sync timeout=45s
Dec 14 18:44:49.569  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: starting sync, obtaining new tips
Dec 14 18:44:49.570  INFO {zebrad="b8646fa7"}:sync:obtain_tips:state: zebra_state::util: created block locator tip_height=Height(584244) min_locator_height=584145 locators=[Height(584244), Height(584243), Height(584242), Height(584240), Height(584236), Height(584228), Height(584212), Height(584180), Height(584145)]
Dec 14 18:44:55.574  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: exhausted prospective tip set
Dec 14 18:44:55.574  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: waiting to restart sync timeout=45s
Dec 14 18:45:40.577  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: starting sync, obtaining new tips
Dec 14 18:45:40.578  INFO {zebrad="b8646fa7"}:sync:obtain_tips: zebra_state::service: pruned utxo requests old_len=3 new_len=1 prune_count=2 tip=Some((Height(584244), block::Hash("0000000000a06adc72e1a2747e865fde2af0a1bcbd5cf3bbeab91a8b25df3603")))
Dec 14 18:45:40.578  INFO {zebrad="b8646fa7"}:sync:obtain_tips:state: zebra_state::util: created block locator tip_height=Height(584244) min_locator_height=584145 locators=[Height(584244), Height(584243), Height(584242), Height(584240), Height(584236), Height(584228), Height(584212), Height(584180), Height(584145)]
Dec 14 18:45:46.586  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: exhausted prospective tip set
Dec 14 18:45:46.586  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: waiting to restart sync timeout=45s
Dec 14 18:46:31.589  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: starting sync, obtaining new tips
Dec 14 18:46:31.591  INFO {zebrad="b8646fa7"}:sync:obtain_tips: zebra_state::service: pruned utxo requests old_len=7 new_len=6 prune_count=1 tip=Some((Height(584244), block::Hash("0000000000a06adc72e1a2747e865fde2af0a1bcbd5cf3bbeab91a8b25df3603")))
Dec 14 18:46:31.591  INFO {zebrad="b8646fa7"}:sync:obtain_tips:state: zebra_state::util: created block locator tip_height=Height(584244) min_locator_height=584145 locators=[Height(584244), Height(584243), Height(584242), Height(584240), Height(584236), Height(584228), Height(584212), Height(584180), Height(584145)]
Dec 14 18:46:37.604  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: exhausted prospective tip set
Dec 14 18:46:37.604  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: waiting to restart sync timeout=45s
Dec 14 18:47:22.606  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: starting sync, obtaining new tips
Dec 14 18:47:22.608  INFO {zebrad="b8646fa7"}:sync:obtain_tips:state: zebra_state::util: created block locator tip_height=Height(584244) min_locator_height=584145 locators=[Height(584244), Height(584243), Height(584242), Height(584240), Height(584236), Height(584228), Height(584212), Height(584180), Height(584145)]
Dec 14 18:47:28.615  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: exhausted prospective tip set
Dec 14 18:47:28.615  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: waiting to restart sync timeout=45s
Dec 14 18:48:13.618  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: starting sync, obtaining new tips
Dec 14 18:48:13.620  INFO {zebrad="b8646fa7"}:sync:obtain_tips:state: zebra_state::util: created block locator tip_height=Height(584244) min_locator_height=584145 locators=[Height(584244), Height(584243), Height(584242), Height(584240), Height(584236), Height(584228), Height(584212), Height(584180), Height(584145)]
Dec 14 18:48:19.624  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: exhausted prospective tip set
Dec 14 18:48:19.625  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: waiting to restart sync timeout=45s
Dec 14 18:49:04.635  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: starting sync, obtaining new tips
Dec 14 18:49:04.636  INFO {zebrad="b8646fa7"}:sync:obtain_tips: zebra_state::service: pruned utxo requests old_len=6 new_len=0 prune_count=6 tip=Some((Height(584244), block::Hash("0000000000a06adc72e1a2747e865fde2af0a1bcbd5cf3bbeab91a8b25df3603")))
Dec 14 18:49:04.637  INFO {zebrad="b8646fa7"}:sync:obtain_tips:state: zebra_state::util: created block locator tip_height=Height(584244) min_locator_height=584145 locators=[Height(584244), Height(584243), Height(584242), Height(584240), Height(584236), Height(584228), Height(584212), Height(584180), Height(584145)]
Dec 14 18:49:10.644  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: exhausted prospective tip set
Dec 14 18:49:10.644  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: waiting to restart sync timeout=45s
Dec 14 18:49:55.646  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: starting sync, obtaining new tips
Dec 14 18:49:55.648  INFO {zebrad="b8646fa7"}:sync:obtain_tips:state: zebra_state::util: created block locator tip_height=Height(584244) min_locator_height=584145 locators=[Height(584244), Height(584243), Height(584242), Height(584240), Height(584236), Height(584228), Height(584212), Height(584180), Height(584145)]
Dec 14 18:50:01.653  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: exhausted prospective tip set
Dec 14 18:50:01.653  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: waiting to restart sync timeout=45s
Dec 14 18:50:46.655  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: starting sync, obtaining new tips
Dec 14 18:50:46.656  INFO {zebrad="b8646fa7"}:sync:obtain_tips:state: zebra_state::util: created block locator tip_height=Height(584244) min_locator_height=584145 locators=[Height(584244), Height(584243), Height(584242), Height(584240), Height(584236), Height(584228), Height(584212), Height(584180), Height(584145)]
Dec 14 18:50:52.660  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: exhausted prospective tip set
Dec 14 18:50:52.660  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: waiting to restart sync timeout=45s
Dec 14 18:51:37.663  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: starting sync, obtaining new tips
Dec 14 18:51:37.664  INFO {zebrad="b8646fa7"}:sync:obtain_tips:state: zebra_state::util: created block locator tip_height=Height(584244) min_locator_height=584145 locators=[Height(584244), Height(584243), Height(584242), Height(584240), Height(584236), Height(584228), Height(584212), Height(584180), Height(584145)]
Dec 14 18:51:43.668  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: exhausted prospective tip set
Dec 14 18:51:43.668  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: waiting to restart sync timeout=45s
Dec 14 18:52:28.672  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: starting sync, obtaining new tips
Dec 14 18:52:28.673  INFO {zebrad="b8646fa7"}:sync:obtain_tips:state: zebra_state::util: created block locator tip_height=Height(584244) min_locator_height=584145 locators=[Height(584244), Height(584243), Height(584242), Height(584240), Height(584236), Height(584228), Height(584212), Height(584180), Height(584145)]
Dec 14 18:52:34.684  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: exhausted prospective tip set
Dec 14 18:52:34.684  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: waiting to restart sync timeout=45s
Dec 14 18:53:19.688  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: starting sync, obtaining new tips
Dec 14 18:53:19.689  INFO {zebrad="b8646fa7"}:sync:obtain_tips:state: zebra_state::util: created block locator tip_height=Height(584244) min_locator_height=584145 locators=[Height(584244), Height(584243), Height(584242), Height(584240), Height(584236), Height(584228), Height(584212), Height(584180), Height(584145)]
Dec 14 18:53:25.694  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: exhausted prospective tip set
Dec 14 18:53:25.694  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: waiting to restart sync timeout=45s
Dec 14 18:54:10.697  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: starting sync, obtaining new tips
Dec 14 18:54:10.698  INFO {zebrad="b8646fa7"}:sync:obtain_tips:state: zebra_state::util: created block locator tip_height=Height(584244) min_locator_height=584145 locators=[Height(584244), Height(584243), Height(584242), Height(584240), Height(584236), Height(584228), Height(584212), Height(584180), Height(584145)]
Dec 14 18:54:16.701  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: exhausted prospective tip set
Dec 14 18:54:16.702  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: waiting to restart sync timeout=45s
Dec 14 18:55:01.704  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: starting sync, obtaining new tips
Dec 14 18:55:01.706  INFO {zebrad="b8646fa7"}:sync:obtain_tips:state: zebra_state::util: created block locator tip_height=Height(584244) min_locator_height=584145 locators=[Height(584244), Height(584243), Height(584242), Height(584240), Height(584236), Height(584228), Height(584212), Height(584180), Height(584145)]
Dec 14 18:55:07.711  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: exhausted prospective tip set
Dec 14 18:55:07.711  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: waiting to restart sync timeout=45s
Dec 14 18:55:52.720  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: starting sync, obtaining new tips
Dec 14 18:55:52.722  INFO {zebrad="b8646fa7"}:sync:obtain_tips: zebra_state::service: pruned utxo requests old_len=116 new_len=2 prune_count=114 tip=Some((Height(584244), block::Hash("0000000000a06adc72e1a2747e865fde2af0a1bcbd5cf3bbeab91a8b25df3603")))
Dec 14 18:55:52.722  INFO {zebrad="b8646fa7"}:sync:obtain_tips:state: zebra_state::util: created block locator tip_height=Height(584244) min_locator_height=584145 locators=[Height(584244), Height(584243), Height(584242), Height(584240), Height(584236), Height(584228), Height(584212), Height(584180), Height(584145)]
Dec 14 18:55:58.731  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: exhausted prospective tip set
Dec 14 18:55:58.731  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: waiting to restart sync timeout=45s
Dec 14 18:56:43.735  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: starting sync, obtaining new tips
Dec 14 18:56:43.736  INFO {zebrad="b8646fa7"}:sync:obtain_tips:state: zebra_state::util: created block locator tip_height=Height(584244) min_locator_height=584145 locators=[Height(584244), Height(584243), Height(584242), Height(584240), Height(584236), Height(584228), Height(584212), Height(584180), Height(584145)]
Dec 14 18:56:49.742  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: exhausted prospective tip set
Dec 14 18:56:49.742  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: waiting to restart sync timeout=45s
Dec 14 18:57:08.342  INFO {zebrad="b8646fa7"}:peer{addr=40.118.90.57:55002}:msg_as_req{msg=inv}:inbound:download_and_verify{hash=00000000012e210ad5c90beb86c3ce9d581abd15310008c1f1d908ce79648394}: zebra_state::service: pruned utxo requests old_len=8 new_len=6 prune_count=2 tip=Some((Height(584244), block::Hash("0000000000a06adc72e1a2747e865fde2af0a1bcbd5cf3bbeab91a8b25df3603")))
Dec 14 18:57:34.749  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: starting sync, obtaining new tips
Dec 14 18:57:34.750  INFO {zebrad="b8646fa7"}:sync:obtain_tips:state: zebra_state::util: created block locator tip_height=Height(584244) min_locator_height=584145 locators=[Height(584244), Height(584243), Height(584242), Height(584240), Height(584236), Height(584228), Height(584212), Height(584180), Height(584145)]
Dec 14 18:57:40.757  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: exhausted prospective tip set
Dec 14 18:57:40.757  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: waiting to restart sync timeout=45s
Dec 14 18:58:25.762  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: starting sync, obtaining new tips
Dec 14 18:58:25.762  INFO {zebrad="b8646fa7"}:sync:obtain_tips:state: zebra_state::util: created block locator tip_height=Height(584244) min_locator_height=584145 locators=[Height(584244), Height(584243), Height(584242), Height(584240), Height(584236), Height(584228), Height(584212), Height(584180), Height(584145)]
Dec 14 18:58:31.767  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: exhausted prospective tip set
Dec 14 18:58:31.767  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: waiting to restart sync timeout=45s
Dec 14 18:59:16.769  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: starting sync, obtaining new tips
Dec 14 18:59:16.771  INFO {zebrad="b8646fa7"}:sync:obtain_tips:state: zebra_state::util: created block locator tip_height=Height(584244) min_locator_height=584145 locators=[Height(584244), Height(584243), Height(584242), Height(584240), Height(584236), Height(584228), Height(584212), Height(584180), Height(584145)]
Dec 14 18:59:22.776  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: exhausted prospective tip set
Dec 14 18:59:22.777  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: waiting to restart sync timeout=45s
Dec 14 19:00:07.780  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: starting sync, obtaining new tips
Dec 14 19:00:07.781  INFO {zebrad="b8646fa7"}:sync:obtain_tips: zebra_state::service: pruned utxo requests old_len=17 new_len=11 prune_count=6 tip=Some((Height(584244), block::Hash("0000000000a06adc72e1a2747e865fde2af0a1bcbd5cf3bbeab91a8b25df3603")))
Dec 14 19:00:07.781  INFO {zebrad="b8646fa7"}:sync:obtain_tips:state: zebra_state::util: created block locator tip_height=Height(584244) min_locator_height=584145 locators=[Height(584244), Height(584243), Height(584242), Height(584240), Height(584236), Height(584228), Height(584212), Height(584180), Height(584145)]
Dec 14 19:00:13.792  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: exhausted prospective tip set
Dec 14 19:00:13.793  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: waiting to restart sync timeout=45s
Dec 14 19:00:58.801  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: starting sync, obtaining new tips
Dec 14 19:00:58.803  INFO {zebrad="b8646fa7"}:sync:obtain_tips:state: zebra_state::util: created block locator tip_height=Height(584244) min_locator_height=584145 locators=[Height(584244), Height(584243), Height(584242), Height(584240), Height(584236), Height(584228), Height(584212), Height(584180), Height(584145)]
Dec 14 19:01:04.810  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: exhausted prospective tip set
Dec 14 19:01:04.810  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: waiting to restart sync timeout=45s
Dec 14 19:01:39.499  INFO {zebrad="b8646fa7"}:peer{addr=51.75.146.76:8233}:msg_as_req{msg=inv}:inbound:download_and_verify{hash=0000000001628b89fb7496b91df3393964b7cb36a5c02cac0e6f4341e484eb67}: zebra_state::service: pruned utxo requests old_len=13 new_len=2 prune_count=11 tip=Some((Height(584244), block::Hash("0000000000a06adc72e1a2747e865fde2af0a1bcbd5cf3bbeab91a8b25df3603")))
Dec 14 19:01:49.821  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: starting sync, obtaining new tips
Dec 14 19:01:49.822  INFO {zebrad="b8646fa7"}:sync:obtain_tips:state: zebra_state::util: created block locator tip_height=Height(584244) min_locator_height=584145 locators=[Height(584244), Height(584243), Height(584242), Height(584240), Height(584236), Height(584228), Height(584212), Height(584180), Height(584145)]
Dec 14 19:01:55.826  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: exhausted prospective tip set
Dec 14 19:01:55.826  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: waiting to restart sync timeout=45s
Dec 14 19:02:40.829  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: starting sync, obtaining new tips
Dec 14 19:02:40.829  INFO {zebrad="b8646fa7"}:sync:obtain_tips:state: zebra_state::util: created block locator tip_height=Height(584244) min_locator_height=584145 locators=[Height(584244), Height(584243), Height(584242), Height(584240), Height(584236), Height(584228), Height(584212), Height(584180), Height(584145)]
Dec 14 19:02:46.834  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: exhausted prospective tip set
Dec 14 19:02:46.834  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: waiting to restart sync timeout=45s
Dec 14 19:03:31.837  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: starting sync, obtaining new tips
Dec 14 19:03:31.838  INFO {zebrad="b8646fa7"}:sync:obtain_tips:state: zebra_state::util: created block locator tip_height=Height(584244) min_locator_height=584145 locators=[Height(584244), Height(584243), Height(584242), Height(584240), Height(584236), Height(584228), Height(584212), Height(584180), Height(584145)]
Dec 14 19:03:37.843  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: exhausted prospective tip set
Dec 14 19:03:37.843  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: waiting to restart sync timeout=45s
Dec 14 19:04:22.849  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: starting sync, obtaining new tips
Dec 14 19:04:22.849  INFO {zebrad="b8646fa7"}:sync:obtain_tips:state: zebra_state::util: created block locator tip_height=Height(584244) min_locator_height=584145 locators=[Height(584244), Height(584243), Height(584242), Height(584240), Height(584236), Height(584228), Height(584212), Height(584180), Height(584145)]
Dec 14 19:04:28.855  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: exhausted prospective tip set
Dec 14 19:04:28.856  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: waiting to restart sync timeout=45s
Dec 14 19:05:13.860  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: starting sync, obtaining new tips
Dec 14 19:05:13.862  INFO {zebrad="b8646fa7"}:sync:obtain_tips: zebra_state::service: pruned utxo requests old_len=185 new_len=24 prune_count=161 tip=Some((Height(584244), block::Hash("0000000000a06adc72e1a2747e865fde2af0a1bcbd5cf3bbeab91a8b25df3603")))
Dec 14 19:05:13.863  INFO {zebrad="b8646fa7"}:sync:obtain_tips:state: zebra_state::util: created block locator tip_height=Height(584244) min_locator_height=584145 locators=[Height(584244), Height(584243), Height(584242), Height(584240), Height(584236), Height(584228), Height(584212), Height(584180), Height(584145)]
Dec 14 19:05:19.869  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: exhausted prospective tip set
Dec 14 19:05:19.869  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: waiting to restart sync timeout=45s
Dec 14 19:06:04.871  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: starting sync, obtaining new tips
Dec 14 19:06:04.873  INFO {zebrad="b8646fa7"}:sync:obtain_tips: zebra_state::service: pruned utxo requests old_len=30 new_len=25 prune_count=5 tip=Some((Height(584244), block::Hash("0000000000a06adc72e1a2747e865fde2af0a1bcbd5cf3bbeab91a8b25df3603")))
Dec 14 19:06:04.873  INFO {zebrad="b8646fa7"}:sync:obtain_tips:state: zebra_state::util: created block locator tip_height=Height(584244) min_locator_height=584145 locators=[Height(584244), Height(584243), Height(584242), Height(584240), Height(584236), Height(584228), Height(584212), Height(584180), Height(584145)]
Dec 14 19:06:10.883  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: exhausted prospective tip set
Dec 14 19:06:10.884  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: waiting to restart sync timeout=45s
Dec 14 19:06:34.912  INFO {zebrad="b8646fa7"}:peer{addr=40.118.90.57:55002}:msg_as_req{msg=inv}:inbound:download_and_verify{hash=000000000208b098f28f778bf34666f067f33f5c2bf513f356a99b7b5742014d}: zebra_state::service: pruned utxo requests old_len=25 new_len=23 prune_count=2 tip=Some((Height(584244), block::Hash("0000000000a06adc72e1a2747e865fde2af0a1bcbd5cf3bbeab91a8b25df3603")))
Dec 14 19:06:55.887  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: starting sync, obtaining new tips
Dec 14 19:06:55.888  INFO {zebrad="b8646fa7"}:sync:obtain_tips:state: zebra_state::util: created block locator tip_height=Height(584244) min_locator_height=584145 locators=[Height(584244), Height(584243), Height(584242), Height(584240), Height(584236), Height(584228), Height(584212), Height(584180), Height(584145)]
Dec 14 19:07:01.892  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: exhausted prospective tip set
Dec 14 19:07:01.892  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: waiting to restart sync timeout=45s
Dec 14 19:07:46.894  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: starting sync, obtaining new tips
Dec 14 19:07:46.895  INFO {zebrad="b8646fa7"}:sync:obtain_tips: zebra_state::service: pruned utxo requests old_len=26 new_len=9 prune_count=17 tip=Some((Height(584244), block::Hash("0000000000a06adc72e1a2747e865fde2af0a1bcbd5cf3bbeab91a8b25df3603")))
Dec 14 19:07:46.896  INFO {zebrad="b8646fa7"}:sync:obtain_tips:state: zebra_state::util: created block locator tip_height=Height(584244) min_locator_height=584145 locators=[Height(584244), Height(584243), Height(584242), Height(584240), Height(584236), Height(584228), Height(584212), Height(584180), Height(584145)]
Dec 14 19:07:52.900  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: exhausted prospective tip set
Dec 14 19:07:52.900  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: waiting to restart sync timeout=45s
Dec 14 19:08:37.901  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: starting sync, obtaining new tips
Dec 14 19:08:37.903  INFO {zebrad="b8646fa7"}:sync:obtain_tips: zebra_state::service: pruned utxo requests old_len=14 new_len=8 prune_count=6 tip=Some((Height(584244), block::Hash("0000000000a06adc72e1a2747e865fde2af0a1bcbd5cf3bbeab91a8b25df3603")))
Dec 14 19:08:37.903  INFO {zebrad="b8646fa7"}:sync:obtain_tips:state: zebra_state::util: created block locator tip_height=Height(584244) min_locator_height=584145 locators=[Height(584244), Height(584243), Height(584242), Height(584240), Height(584236), Height(584228), Height(584212), Height(584180), Height(584145)]
Dec 14 19:08:43.914  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: exhausted prospective tip set
Dec 14 19:08:43.915  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: waiting to restart sync timeout=45s
Dec 14 19:09:28.917  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: starting sync, obtaining new tips
Dec 14 19:09:28.918  INFO {zebrad="b8646fa7"}:sync:obtain_tips:state: zebra_state::util: created block locator tip_height=Height(584244) min_locator_height=584145 locators=[Height(584244), Height(584243), Height(584242), Height(584240), Height(584236), Height(584228), Height(584212), Height(584180), Height(584145)]
Dec 14 19:09:34.922  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: exhausted prospective tip set
Dec 14 19:09:34.922  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: waiting to restart sync timeout=45s
Dec 14 19:09:44.187  INFO {zebrad="b8646fa7"}:peer{addr=40.118.90.57:55002}:msg_as_req{msg=inv}:inbound:download_and_verify{hash=0000000001557f2169826a4ec1a96abae17b894f3bf39cb7c0695f3389c40252}: zebra_state::service: pruned utxo requests old_len=13 new_len=10 prune_count=3 tip=Some((Height(584244), block::Hash("0000000000a06adc72e1a2747e865fde2af0a1bcbd5cf3bbeab91a8b25df3603")))
Dec 14 19:10:19.927  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: starting sync, obtaining new tips
Dec 14 19:10:19.929  INFO {zebrad="b8646fa7"}:sync:obtain_tips:state: zebra_state::util: created block locator tip_height=Height(584244) min_locator_height=584145 locators=[Height(584244), Height(584243), Height(584242), Height(584240), Height(584236), Height(584228), Height(584212), Height(584180), Height(584145)]
Dec 14 19:10:25.933  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: exhausted prospective tip set
Dec 14 19:10:25.933  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: waiting to restart sync timeout=45s
Dec 14 19:11:10.939  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: starting sync, obtaining new tips
Dec 14 19:11:10.940  INFO {zebrad="b8646fa7"}:sync:obtain_tips:state: zebra_state::util: created block locator tip_height=Height(584244) min_locator_height=584145 locators=[Height(584244), Height(584243), Height(584242), Height(584240), Height(584236), Height(584228), Height(584212), Height(584180), Height(584145)]
Dec 14 19:11:16.943  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: exhausted prospective tip set
Dec 14 19:11:16.943  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: waiting to restart sync timeout=45s
Dec 14 19:12:01.948  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: starting sync, obtaining new tips
Dec 14 19:12:01.950  INFO {zebrad="b8646fa7"}:sync:obtain_tips: zebra_state::service: pruned utxo requests old_len=17 new_len=12 prune_count=5 tip=Some((Height(584244), block::Hash("0000000000a06adc72e1a2747e865fde2af0a1bcbd5cf3bbeab91a8b25df3603")))
Dec 14 19:12:01.950  INFO {zebrad="b8646fa7"}:sync:obtain_tips:state: zebra_state::util: created block locator tip_height=Height(584244) min_locator_height=584145 locators=[Height(584244), Height(584243), Height(584242), Height(584240), Height(584236), Height(584228), Height(584212), Height(584180), Height(584145)]
Dec 14 19:12:07.956  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: exhausted prospective tip set
Dec 14 19:12:07.957  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: waiting to restart sync timeout=45s
Dec 14 19:12:52.959  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: starting sync, obtaining new tips
Dec 14 19:12:52.960  INFO {zebrad="b8646fa7"}:sync:obtain_tips: zebra_state::service: pruned utxo requests old_len=16 new_len=11 prune_count=5 tip=Some((Height(584244), block::Hash("0000000000a06adc72e1a2747e865fde2af0a1bcbd5cf3bbeab91a8b25df3603")))
Dec 14 19:12:52.960  INFO {zebrad="b8646fa7"}:sync:obtain_tips:state: zebra_state::util: created block locator tip_height=Height(584244) min_locator_height=584145 locators=[Height(584244), Height(584243), Height(584242), Height(584240), Height(584236), Height(584228), Height(584212), Height(584180), Height(584145)]
Dec 14 19:12:58.963  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: exhausted prospective tip set
Dec 14 19:12:58.963  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: waiting to restart sync timeout=45s
Dec 14 19:13:43.967  INFO {zebrad="b8646fa7"}:sync: zebrad::components::sync: starting sync, obtaining new tips
Dec 14 19:13:43.969  INFO {zebrad="b8646fa7"}:sync:obtain_tips: zebra_state::service: pruned utxo requests old_len=11 new_len=6 prune_count=5 tip=Some((Height(584244), block::Hash("0000000000a06adc72e1a2747e865fde2af0a1bcbd5cf3bbeab91a8b25df3603")))
Dec 14 19:13:43.969  INFO {zebrad="b8646fa7"}:sync:obtain_tips:state: zebra_state::util: created block locator tip_height=Height(584244) min_locator_height=584145 locators=[Height(584244), Height(584243), Height(584242), Height(584240), Height(584236), Height(584228), Height(584212), Height(584180), Height(584145)]

teor · Answer 17 · Tue Jan 05 2021 11:16:34 GMT+0800 (China Standard Time)

We think #1531 will fix most of these issues - once it's merged we'll close this ticket.

Any new hangs should have separate tickets - ideally based on the part of Zebra that's hanging.

teor · Answer 18 · Wed Jan 06 2021 09:13:07 GMT+0800 (China Standard Time)

Testing shows that #1531 doesn't fix this issue

teor · Answer 19 · Sat Jan 16 2021 08:13:07 GMT+0800 (China Standard Time)

This issue persists after #1586, but it seems less frequent, and has a lower impact (until there are no peers at all, which still results in permanent failure.)

teor · Answer 20 · Mon Jan 25 2021 11:02:44 GMT+0800 (China Standard Time)

When zebrad is stuck in a "network request with no peer connections" loop, starting a peer that connects to zebrad's listener port gets the syncer running again.

So there appear to be two issues here:

the syncer hangs when there are no peers, even though it has a timeout on peer requests - #1633
the peer set is dropping peers, hanging them, or otherwise leaving them in a bad state
- for example, it uses 100% CPU, a sign of a busy-waiting loop - #1634

teor · Answer 21 · Fri Jan 29 2021 10:39:45 GMT+0800 (China Standard Time)

@mpguerra #1620 only fixes one cause of this issue, there could me more instances of #1593, and other causes of peer hangs, like #1633.