it happens time out per 1minute when use etcd v3 watch method
bingchengxiaobing117 opened this issue · comments
problem
when I use etcd v3 watch method(default params),The following questions occurs:
- it happens time out per 1minute .
- There is a lot of connection building with etcd server in per worker progress
log info:
2020/04/15 15:05:34 [info] 9020#9020: *20628390 [lua] v3.lua:502: request_chunk(): http request method: POST path: /v3beta/watch body: {"create_request":{"range_end":"*********","start_revision":6647555,"key":"L2V0Y2QvY2ZnL2dsb2JhbA=="}} query: nil, context: ngx.timer
2020/04/15 15:06:34 [info] 9020#9020: *20628390 [lua] v3.lua:502: request_chunk(): http request method: POST path: /v3beta/watch body: {"create_request":{"range_end":"*********","start_revision":6647555,"key":"L2V0Y2QvY2ZnL2dsb2JhbA=="}} query: nil, context: ngx.timer
2020/04/15 15:07:34 [info] 9020#9020: *20628390 [lua] v3.lua:502: request_chunk(): http request method: POST path: /v3beta/watch body: {"create_request":{"range_end":"*********","start_revision":6647555,"key":"L2V0Y2QvY2ZnL2dsb2JhbA=="}} query: nil, context: ngx.timer
2020/04/15 15:08:34 [info] 9020#9020: *20628390 [lua] v3.lua:502: request_chunk(): http request method: POST path: /v3beta/watch body: {"create_request":{"range_end":"*********","start_revision":6647555,"key":"L2V0Y2QvY2ZnL2dsb2JhbA=="}} query: nil, context: ngx.timer
.......
other info:
etcd endpoints has three ip list such as (1.1.1.1:2379;1.1.1.2:2379;1.1.1.3:2379;)
connection info:
such as worker id: 9020 exits 58 establish connection
root@****:~# netstat -apn|grep 2379 |grep 'ESTABLISHED 9020' |wc -l
58
@YuanSheng Wang Have you met this problem? How to solve
@nic-chen do you have time to confirm this problem?
@YuanSheng Wang Have you met this problem? How to solve
I do not know how to fix it now. we mainly use etcd v2 now.
I has used etcd v2 don't has this problem, I think the reason is etcd v3 should be http response with no buffering( cause by the timing of buffer flush ) , you can see etcd office api guide such as watch
curl -N http://localhost:2379/v3/watch \
-X POST -d '{"create_request": {"key":"Zm9v"} }' &
so how to set the timing of buffer flush
in lua-rest-http module
Hi @bingchengxiaobing117 are you still facing the same problem. It seems I could not reproduce it.
Test code:
local etcd, err = require("resty.etcd").new({protocol = "v3"})
check_res(etcd, err)
local res, err = etcd:set("/test", "abc")
check_res(res, err)
ngx.timer.at(200, function ()
etcd:set("/test", "bcd3")
end)
local cur_time = ngx.now()
local body_chunk_fun, err = etcd:watch("/test", {timeout = 300})
if not body_chunk_fun then
ngx.say("failed to watch: ", err)
end
Log:
2020/09/06 14:47:21 [info] 28274#28274: *1 [lua] v3.lua:284: set(): v3 set body: {"header":{"raft_term":"2","cluster_id":"8925027824743593106","member_id":"13803658152347727308","revision":"7"}}, client: 127.0.0.1, server: localhost, request: "GET /t HTTP/1.1", host: "localhost"
2020/09/06 14:47:21 [info] 28274#28274: *1 [lua] v3.lua:501: request_chunk(): http request method: POST path: /v3/watch body: {"create_request":{"key":"L3Rlc3Q="}} query: nil, client: 127.0.0.1, server: localhost, request: "GET /t HTTP/1.1", host: "localhost"
2020/09/06 14:50:41 [info] 28274#28274: *3 [lua] v3.lua:284: set(): v3 set body: {"header":{"raft_term":"2","cluster_id":"8925027824743593106","member_id":"13803658152347727308","revision":"8"}}, context: ngx.timer, client: 127.0.0.1, server: 0.0.0.0:1984
2020/09/06 14:52:21 [notice] 28274#28274: signal 15 (SIGTERM) received from 28253, exiting
2020/09/06 14:52:21 [notice] 28274#28274: signal 3 (SIGQUIT) received from 28253, shutting down
2020/09/06 14:52:21 [info] 28274#28274: epoll_wait() failed (4: Interrupted system call)
2020/09/06 14:52:21 [notice] 28274#28274: exiting
2020/09/06 14:52:21 [notice] 28274#28274: exit