Giters

Netflix / dynomite

A generic dynamo implementation for different k-v storage engines

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Dynomite Log File Grows Indefinitely

guruprasadb80 opened this issue · comments

commented

I am seeing an issue in our production setup where Dynomite Log file grows indefinitely while printing the logs and system eventually reaches disk full condition and services are impacted.

[2019-06-19 09:51:01.905] dnode_accept:157 Accepting client connection from 10.237.85.33:40460 on sd 1023
[2019-06-19 09:51:01.905] dnode_accept:145 accept on <CONN_PEER_PROXY 0xd54260 9 listening on '10.237.92.98:5002'> failed: Too many open files
[2019-06-19 09:51:01.905] dnode_recv:214 <CONN_PEER_PROXY 0xd54260 9 listening on '10.237.92.98:5002'> Failed to accept a connection. Continuing

This happened once in the past on 19th June and same issue was seen again yesterday.

I have seen an earlier post 424. However, this issue seems to have been fixed and i am using a build v0.6.9.

Dynomite Config:

dyn_o_mite:
datacenter: dc-cak
rack: rack2
dyn_listen: 10.227.92.98:5002
dyn_seeds:
#Distribution to dynomite instances on CAK-Redis1 server

  • 10.227.92.97:5001:rack1:dc-cak:0
  • 10.227.92.97:5002:rack1:dc-cak:1073741824
  • 10.227.92.97:5003:rack1:dc-cak:2147483648
  • 10.227.92.97:5004:rack1:dc-cak:3221225472
    #Distribution to dynomite instances on cak-Redis2 server (other than self)
  • 10.227.92.98:5001:rack2:dc-cak:1073741824
  • 10.227.92.98:5003:rack2:dc-cak:2147483648
  • 10.227.92.98:5004:rack2:dc-cak:3221225472
    #Distribution to dynomite instances on rch-Redis1 server
  • 10.227.85.33:5001:rack1:dc-rch:0
  • 10.227.85.33:5002:rack1:dc-rch:1073741824
  • 10.227.85.33:5003:rack1:dc-rch:2147483648
  • 10.227.85.33:5004:rack1:dc-rch:3221225472
    #Distribution to dynomite instances on rch-Redis2 server
  • 10.227.85.34:5001:rack2:dc-rch:0
  • 10.227.85.34:5002:rack2:dc-rch:1073741824
  • 10.237.85.34:5003:rack2:dc-rch:2147483648
  • 10.227.85.34:5004:rack2:dc-rch:3221225472
    listen: 10.237.92.98:6312
    servers:
  • 127.0.0.1:6302:1
    tokens: '0'
    data_store: 0
    stats_listen: 127.0.0.1:6322
    read_consistency : DC_ONE
    write_consistency : DC_ONE

Any clues on what could have gone wrong?

commented

Below logs are seen continuously.

[2019-08-05 16:48:53.637] client_unref_internal_try_put:94 <CONN_CLIENT 0x2299b20 -1 from '10.227.92.111:40284'> unref owner <POOL 0x222b0e0 'dyn_o_mite'>
[2019-08-05 16:48:53.637] client_unref_internal_try_put:94 <CONN_CLIENT 0x229a330 -1 from '10.227.92.111:40286'> unref owner <POOL 0x222b0e0 'dyn_o_mite'>

commented

@guruprasadb80 We use internal tooling at Netflix to do log rotation, and that's why it's not implemented as part of Dynomite. It's fairly easy to set up yourself however.

Eg: https://linux.die.net/man/8/logrotate

commented

Below logs are seen continuously.

[2019-08-05 16:48:53.637] client_unref_internal_try_put:94 <CONN_CLIENT 0x2299b20 -1 from '10.227.92.111:40284'> unref owner <POOL 0x222b0e0 'dyn_o_mite'>
[2019-08-05 16:48:53.637] client_unref_internal_try_put:94 <CONN_CLIENT 0x229a330 -1 from '10.227.92.111:40286'> unref owner <POOL 0x222b0e0 'dyn_o_mite'>

Does your problem solve? I've been in this situation, too. What does this log entry mean?

client_unref_internal_try_put:94 <CONN_CLIENT 0x2299b20 -1 from '10.227.92.111:40284'> unref owner <POOL 0x222b0e0 'dyn_o_mite'>