etcd server connection issues in client
Dreamacro opened this issue · comments
Hi,
I'm having trouble with etcd server connections in client. Since the server is self-hosted and used for cross-region and cross-continent access, network connectivity is very unstable. Sometimes the server needs to be restarted and client cannot reconnect.
I tried using with_keep_alive
and with_connect_timeout
but it didn't help.
The problem mainly occurs in lease_keep_alive
. When the connection is interrupted or etcd is restarted, it cannot automatically reconnect. Sometimes it shows that the connection is refused and no longer retries.
What are some ways to handle client in this situation? How can I implement reasonable keep alive?
I used ConnectOptions::new
to create options, keep_alive_while_idle
should be true by default
Lines 816 to 828 in 3c46d0c
let opts = ConnectOptions::new()
.with_user(cfg.etcd.username, cfg.etcd.password)
.with_keep_alive(Duration::from_secs(20), Duration::from_secs(5));
BTW, client.watch
also has this problem
Currently, there are two types of parameters for etcd's auth_token, simple (the default) and jwt (recommended for production environments https://etcd.io/docs/v3.5/op-guide/configuration/#auth). These two modes require the functionality to refresh tokens in specific situations.
For simple, when the etcd server restarts, tonic returns an Unauthenticated status code, and the entire client cannot make any more requests after that.
For jwt, when the token's time limit exceeds the TTL (which is usually not very long), the entire client cannot make any more requests.
In the etcd-go client, there is some code for refreshing the token.
Same as etcd-rs
, etcd-client
does not refresh the token automatically.
Currently, there are two types of parameters for etcd's auth_token, simple (the default) and jwt (recommended for production environments https://etcd.io/docs/v3.5/op-guide/configuration/#auth). These two modes require the functionality to refresh tokens in specific situations.
For simple, when the etcd server restarts, tonic returns an Unauthenticated status code, and the entire client cannot make any more requests after that.
For jwt, when the token's time limit exceeds the TTL (which is usually not very long), the entire client cannot make any more requests.
In the etcd-go client, there is some code for refreshing the token.
Same as
etcd-rs
,etcd-client
does not refresh the token automatically.
see #45
@Dreamacro please see #65