Panic on "Transport endpoint is not connected"
zakum1 opened this issue · comments
Hello, When running v0.10.1, in a test environment with two nodes, I get a panic on "Transport endpoint is not connected". This typically happens after a few days of running, so seems to be a result of a very intermittent network glitch rather than my config. I think that this scenario shouldn't cause the node to panic and exit. Although I am using v0.10.1, I have been through the same code on v0.10.3 and it doesn't appear to handle this error differently.
Below are the stack trace and my config file:
Stack trace
thread '<unnamed>' panicked at 'Remote peer address resolve failed: Os { code: 107, kind: NotConnected, message: "Transport endpoint is not connected" }', src/libcore/result.rs:997:5
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
stack backtrace:
0: std::sys::unix::backtrace::tracing::imp::unwind_backtrace
at src/libstd/sys/unix/backtrace/tracing/gcc_s.rs:39
1: std::sys_common::backtrace::_print
at src/libstd/sys_common/backtrace.rs:70
2: std::panicking::default_hook::{{closure}}
at src/libstd/sys_common/backtrace.rs:58
at src/libstd/panicking.rs:200
3: std::panicking::default_hook
at src/libstd/panicking.rs:215
4: std::panicking::rust_panic_with_hook
at src/libstd/panicking.rs:478
5: std::panicking::continue_panic_fmt
at src/libstd/panicking.rs:385
6: rust_begin_unwind
at src/libstd/panicking.rs:312
7: core::panicking::panic_fmt
at src/libcore/panicking.rs:85
8: core::result::unwrap_failed
at /rustc/2aa4c46cfdd726e97360c2734835aa3515e8c858/src/libcore/macros.rs:16
9: <core::result::Result<T, E>>::expect
at /rustc/2aa4c46cfdd726e97360c2734835aa3515e8c858/src/libcore/result.rs:825
10: exonum::events::network::NetworkHandler::listener::{{closure}}
at /home/gitlab-runner/.cargo/registry/src/github.com-1ecc6299db9ec823/exonum-0.10.3/src/events/network.rs:289
11: <futures::stream::for_each::ForEach<S, F, U> as futures::future::Future>::poll
at /home/gitlab-runner/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.25/src/stream/for_each.rs:46
12: <futures::future::join::MaybeDone<A>>::poll
at /home/gitlab-runner/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.25/src/future/join.rs:153
13: <futures::future::join::Join<A, B> as futures::future::Future>::poll
at /home/gitlab-runner/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.25/src/future/join.rs:67
14: <futures::future::map::Map<A, F> as futures::future::Future>::poll
at /home/gitlab-runner/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.25/src/future/map.rs:30
15: <futures::future::select::Select<A, B> as futures::future::Future>::poll
at /home/gitlab-runner/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.25/src/future/select.rs:48
16: <futures::future::map_err::MapErr<A, F> as futures::future::Future>::poll
at /home/gitlab-runner/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.25/src/future/map_err.rs:30
17: <futures::future::map::Map<A, F> as futures::future::Future>::poll
at /home/gitlab-runner/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.25/src/future/map.rs:30
18: <futures::task_impl::Spawn<T>>::poll_future_notify::{{closure}}
at /home/gitlab-runner/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.25/src/task_impl/mod.rs:326
19: <futures::task_impl::Spawn<T>>::enter::{{closure}}
at /home/gitlab-runner/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.25/src/task_impl/mod.rs:396
20: futures::task_impl::std::set
at /home/gitlab-runner/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.25/src/task_impl/std/mod.rs:78
21: <futures::task_impl::Spawn<T>>::enter
at /home/gitlab-runner/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.25/src/task_impl/mod.rs:396
22: <futures::task_impl::Spawn<T>>::poll_fn_notify
at /home/gitlab-runner/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.25/src/task_impl/mod.rs:288
23: <futures::task_impl::Spawn<T>>::poll_future_notify
at /home/gitlab-runner/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.25/src/task_impl/mod.rs:326
24: tokio_core::reactor::Core::run::{{closure}}::{{closure}}::{{closure}}::{{closure}}::{{closure}}
at /home/gitlab-runner/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-core-0.1.17/src/reactor/mod.rs:236
25: <futures::future::lazy::Lazy<F, R>>::get
at /home/gitlab-runner/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.25/src/future/lazy.rs:64
26: <futures::future::lazy::Lazy<F, R> as futures::future::Future>::poll
at /home/gitlab-runner/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.25/src/future/lazy.rs:82
27: <futures::task_impl::Spawn<T>>::poll_future_notify::{{closure}}
at /home/gitlab-runner/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.25/src/task_impl/mod.rs:326
28: <futures::task_impl::Spawn<T>>::enter::{{closure}}
at /home/gitlab-runner/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.25/src/task_impl/mod.rs:396
29: futures::task_impl::std::set
at /home/gitlab-runner/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.25/src/task_impl/std/mod.rs:78
30: <futures::task_impl::Spawn<T>>::enter
at /home/gitlab-runner/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.25/src/task_impl/mod.rs:396
31: <futures::task_impl::Spawn<T>>::poll_fn_notify
at /home/gitlab-runner/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.25/src/task_impl/mod.rs:288
32: <futures::task_impl::Spawn<T>>::poll_future_notify
at /home/gitlab-runner/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.25/src/task_impl/mod.rs:326
33: <tokio_current_thread::Entered<'a, P>>::block_on::{{closure}}
at /home/gitlab-runner/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-current-thread-0.1.4/src/lib.rs:479
34: <tokio_current_thread::Borrow<'a, U>>::enter::{{closure}}::{{closure}}
at /home/gitlab-runner/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-current-thread-0.1.4/src/lib.rs:779
35: tokio_current_thread::CurrentRunner::set_spawn
at /home/gitlab-runner/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-current-thread-0.1.4/src/lib.rs:816
36: <tokio_current_thread::Borrow<'a, U>>::enter::{{closure}}
at /home/gitlab-runner/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-current-thread-0.1.4/src/lib.rs:778
37: <std::thread::local::LocalKey<T>>::try_with
at /rustc/2aa4c46cfdd726e97360c2734835aa3515e8c858/src/libstd/thread/local.rs:299
38: <std::thread::local::LocalKey<T>>::with
at /rustc/2aa4c46cfdd726e97360c2734835aa3515e8c858/src/libstd/thread/local.rs:245
39: <tokio_current_thread::Borrow<'a, U>>::enter
at /home/gitlab-runner/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-current-thread-0.1.4/src/lib.rs:776
40: <tokio_current_thread::Entered<'a, P>>::block_on
at /home/gitlab-runner/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-current-thread-0.1.4/src/lib.rs:478
41: tokio_core::reactor::Core::run::{{closure}}::{{closure}}::{{closure}}::{{closure}}
at /home/gitlab-runner/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-core-0.1.17/src/reactor/mod.rs:234
42: tokio_timer::timer::handle::with_default::{{closure}}
at /home/gitlab-runner/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-timer-0.2.10/src/timer/handle.rs:94
43: <std::thread::local::LocalKey<T>>::try_with
at /rustc/2aa4c46cfdd726e97360c2734835aa3515e8c858/src/libstd/thread/local.rs:299
44: <std::thread::local::LocalKey<T>>::with
at /rustc/2aa4c46cfdd726e97360c2734835aa3515e8c858/src/libstd/thread/local.rs:245
45: tokio_timer::timer::handle::with_default
at /home/gitlab-runner/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-timer-0.2.10/src/timer/handle.rs:81
46: tokio_core::reactor::Core::run::{{closure}}::{{closure}}::{{closure}}
at /home/gitlab-runner/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-core-0.1.17/src/reactor/mod.rs:233
47: tokio_executor::global::with_default::{{closure}}
at /home/gitlab-runner/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-executor-0.1.6/src/global.rs:192
48: <std::thread::local::LocalKey<T>>::try_with
at /rustc/2aa4c46cfdd726e97360c2734835aa3515e8c858/src/libstd/thread/local.rs:299
49: <std::thread::local::LocalKey<T>>::with
at /rustc/2aa4c46cfdd726e97360c2734835aa3515e8c858/src/libstd/thread/local.rs:245
50: tokio_executor::global::with_default
at /home/gitlab-runner/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-executor-0.1.6/src/global.rs:162
51: tokio_core::reactor::Core::run::{{closure}}::{{closure}}
at /home/gitlab-runner/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-core-0.1.17/src/reactor/mod.rs:232
52: tokio_reactor::with_default::{{closure}}
at /home/gitlab-runner/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-reactor-0.1.8/src/lib.rs:229
53: <std::thread::local::LocalKey<T>>::try_with
at /rustc/2aa4c46cfdd726e97360c2734835aa3515e8c858/src/libstd/thread/local.rs:299
54: <std::thread::local::LocalKey<T>>::with
at /rustc/2aa4c46cfdd726e97360c2734835aa3515e8c858/src/libstd/thread/local.rs:245
55: tokio_reactor::with_default
at /home/gitlab-runner/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-reactor-0.1.8/src/lib.rs:212
56: tokio_core::reactor::Core::run::{{closure}}
at /home/gitlab-runner/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-core-0.1.17/src/reactor/mod.rs:231
57: <scoped_tls::ScopedKey<T>>::set
at /home/gitlab-runner/.cargo/registry/src/github.com-1ecc6299db9ec823/scoped-tls-0.1.2/src/lib.rs:155
58: tokio_core::reactor::Core::run
at /home/gitlab-runner/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-core-0.1.17/src/reactor/mod.rs:230
59: exonum::node::Node::run_handler::{{closure}}
at /home/gitlab-runner/.cargo/registry/src/github.com-1ecc6299db9ec823/exonum-0.10.3/src/node/mod.rs:927
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: Any', src/libcore/result.rs:997:5
stack backtrace:
0: std::sys::unix::backtrace::tracing::imp::unwind_backtrace
at src/libstd/sys/unix/backtrace/tracing/gcc_s.rs:39
1: std::sys_common::backtrace::_print
at src/libstd/sys_common/backtrace.rs:70
2: std::panicking::default_hook::{{closure}}
at src/libstd/sys_common/backtrace.rs:58
at src/libstd/panicking.rs:200
3: std::panicking::default_hook
at src/libstd/panicking.rs:215
4: std::panicking::rust_panic_with_hook
at src/libstd/panicking.rs:478
5: std::panicking::continue_panic_fmt
at src/libstd/panicking.rs:385
6: rust_begin_unwind
at src/libstd/panicking.rs:312
7: core::panicking::panic_fmt
at src/libcore/panicking.rs:85
8: core::result::unwrap_failed
at /rustc/2aa4c46cfdd726e97360c2734835aa3515e8c858/src/libcore/macros.rs:16
9: <core::result::Result<T, E>>::unwrap
at /rustc/2aa4c46cfdd726e97360c2734835aa3515e8c858/src/libcore/result.rs:798
10: exonum::node::Node::run_handler
at /home/gitlab-runner/.cargo/registry/src/github.com-1ecc6299db9ec823/exonum-0.10.3/src/node/mod.rs:935
11: exonum::node::Node::run
at /home/gitlab-runner/.cargo/registry/src/github.com-1ecc6299db9ec823/exonum-0.10.3/src/node/mod.rs:1002
12: node::main
at src/main.rs:248
13: std::rt::lang_start::{{closure}}
at /rustc/2aa4c46cfdd726e97360c2734835aa3515e8c858/src/libstd/rt.rs:64
14: std::panicking::try::do_call
at src/libstd/rt.rs:49
at src/libstd/panicking.rs:297
15: __rust_maybe_catch_panic
at src/libpanic_unwind/lib.rs:92
16: std::rt::lang_start_internal
at src/libstd/panicking.rs:276
at src/libstd/panic.rs:388
at src/libstd/rt.rs:48
17: std::rt::lang_start
at /rustc/2aa4c46cfdd726e97360c2734835aa3515e8c858/src/libstd/rt.rs:64
18: main
19: __libc_start_main
20: _start
Config file
consensus_public_key = "cf25213e35897fd76ec736d8b0c771b3883642e6345d4606c91ccde1fe634fc9"
consensus_secret_key = "fd2d86a4e0ef91e9c69196ea80a593ae63de6ba5c75edb1cf3fab78868c8b350cf25213e35897fd76ec736d8b0c771b3883642e6345d4606c91ccde1fe634fc9"
listen_address = "0.0.0.0:2000"
external_address = "node1.test.mydomain.com:2000"
service_public_key = "73b3d35662379377ceada1bb5ae12cc17d27fd8d47286751d5d5496b7597baa0"
service_secret_key = "02df1fad9fe5b732553a6d919113e091f2b054b5d2a5546b14b3f64a30b860f373b3d35662379377ceada1bb5ae12cc17d27fd8d47286751d5d5496b7597baa0"
[api]
state_update_timeout = 10000
public_api_address = "0.0.0.0:8000"
private_api_address = "0.0.0.0:8010"
public_allow_origin = "*"
[[genesis.validator_keys]]
consensus_key = "cf25213e35897fd76ec736d8b0c771b3883642e6345d4606c91ccde1fe634fc9"
service_key = "73b3d35662379377ceada1bb5ae12cc17d27fd8d47286751d5d5496b7597baa0"
[[genesis.validator_keys]]
consensus_key = "677295c2154a73e8c73fa09000b8db4c26f88fa506acc0dc9f77c0c85a8d9e21"
service_key = "f7aa46ab611c1455862dba7665018e98d3f6e8521ceb84c99925e330fdec66fd"
[genesis.consensus]
peers_timeout = 10000
first_round_timeout = 3000
status_timeout = 5000
txs_block_limit = 1000
max_message_len = 1048576
min_propose_timeout = 10
max_propose_timeout = 200
propose_timeout_threshold = 500
[mempool]
[mempool.events_pool_capacity]
api_requests_capacity = 1024
network_events_capacity = 512
network_requests_capacity = 512
internal_events_capacity = 128
timeout_requests_capacity = 128
[network]
max_incoming_connections = 128
max_outgoing_connections = 128
tcp_nodelay = true
tcp_reconnect_timeout = 500
tcp_reconnect_timeout_max = 600000
tcp_connect_retry_timeout = 15000
tcp_connect_max_retries = 10
[services_configs]
[[connect_list.peers]]
public_key="cf25213e35897fd76ec736d8b0c771b3883642e6345d4606c91ccde1fe634fc9"
address="node1.test.mydomain.com:2000"
[[connect_list.peers]]
public_key="677295c2154a73e8c73fa09000b8db4c26f88fa506acc0dc9f77c0c85a8d9e21"
address="node2.test.mydomain.com:2000"
[database]
create_if_missing = true
@zakum1 Thanks for your report. We will try find out what's going on there.
I am closing this, because I haven't seen it re-occur after making some changes.
Actually I am still experiencing this, running my nodes on AWS (ubuntu v16.04). I am reopening and will look for more diagnostic information.
I am facing the same this, intermittently. Any clue?