Dynomite Log File Grows Indefinitely
guruprasadb80 opened this issue · comments
I am seeing an issue in our production setup where Dynomite Log file grows indefinitely while printing the logs and system eventually reaches disk full condition and services are impacted.
[2019-06-19 09:51:01.905] dnode_accept:157 Accepting client connection from 10.237.85.33:40460 on sd 1023
[2019-06-19 09:51:01.905] dnode_accept:145 accept on <CONN_PEER_PROXY 0xd54260 9 listening on '10.237.92.98:5002'> failed: Too many open files
[2019-06-19 09:51:01.905] dnode_recv:214 <CONN_PEER_PROXY 0xd54260 9 listening on '10.237.92.98:5002'> Failed to accept a connection. Continuing
This happened once in the past on 19th June and same issue was seen again yesterday.
I have seen an earlier post 424. However, this issue seems to have been fixed and i am using a build v0.6.9.
Dynomite Config:
dyn_o_mite:
datacenter: dc-cak
rack: rack2
dyn_listen: 10.227.92.98:5002
dyn_seeds:
#Distribution to dynomite instances on CAK-Redis1 server
- 10.227.92.97:5001:rack1:dc-cak:0
- 10.227.92.97:5002:rack1:dc-cak:1073741824
- 10.227.92.97:5003:rack1:dc-cak:2147483648
- 10.227.92.97:5004:rack1:dc-cak:3221225472
#Distribution to dynomite instances on cak-Redis2 server (other than self) - 10.227.92.98:5001:rack2:dc-cak:1073741824
- 10.227.92.98:5003:rack2:dc-cak:2147483648
- 10.227.92.98:5004:rack2:dc-cak:3221225472
#Distribution to dynomite instances on rch-Redis1 server - 10.227.85.33:5001:rack1:dc-rch:0
- 10.227.85.33:5002:rack1:dc-rch:1073741824
- 10.227.85.33:5003:rack1:dc-rch:2147483648
- 10.227.85.33:5004:rack1:dc-rch:3221225472
#Distribution to dynomite instances on rch-Redis2 server - 10.227.85.34:5001:rack2:dc-rch:0
- 10.227.85.34:5002:rack2:dc-rch:1073741824
- 10.237.85.34:5003:rack2:dc-rch:2147483648
- 10.227.85.34:5004:rack2:dc-rch:3221225472
listen: 10.237.92.98:6312
servers: - 127.0.0.1:6302:1
tokens: '0'
data_store: 0
stats_listen: 127.0.0.1:6322
read_consistency : DC_ONE
write_consistency : DC_ONE
Any clues on what could have gone wrong?
Below logs are seen continuously.
[2019-08-05 16:48:53.637] client_unref_internal_try_put:94 <CONN_CLIENT 0x2299b20 -1 from '10.227.92.111:40284'> unref owner <POOL 0x222b0e0 'dyn_o_mite'>
[2019-08-05 16:48:53.637] client_unref_internal_try_put:94 <CONN_CLIENT 0x229a330 -1 from '10.227.92.111:40286'> unref owner <POOL 0x222b0e0 'dyn_o_mite'>
@guruprasadb80 We use internal tooling at Netflix to do log rotation, and that's why it's not implemented as part of Dynomite. It's fairly easy to set up yourself however.
Below logs are seen continuously.
[2019-08-05 16:48:53.637] client_unref_internal_try_put:94 <CONN_CLIENT 0x2299b20 -1 from '10.227.92.111:40284'> unref owner <POOL 0x222b0e0 'dyn_o_mite'>
[2019-08-05 16:48:53.637] client_unref_internal_try_put:94 <CONN_CLIENT 0x229a330 -1 from '10.227.92.111:40286'> unref owner <POOL 0x222b0e0 'dyn_o_mite'>
Does your problem solve? I've been in this situation, too. What does this log entry mean?
client_unref_internal_try_put:94 <CONN_CLIENT 0x2299b20 -1 from '10.227.92.111:40284'> unref owner <POOL 0x222b0e0 'dyn_o_mite'>