Giters

pnxtech / hydra

A light-weight library for building distributed applications such as microservices

Home Page:https://www.hydramicroservice.com

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

if shutdown message is first message in health log then it never expires

rantecki opened this issue · comments

commented

Hi there,

I've been playing with a test setup of Hydra for a few days and doing a lot of starting/stopping of services in the process. I've noticed that Redis is slowly filling up with hydra:service:*:health:log keys from the beginning of time. These keys appear to be persistent.

E.g.

127.0.0.1:6379> keys *:health:log
1) "hydra:service:test-service:fbcef59e18874aa3993b6eff782285ff:health:log"

27.0.0.1:6379> llen hydra:service:test-service:fbcef59e18874aa3993b6eff782285ff:health:log
(integer) 1

27.0.0.1:6379> lindex hydra:service:test-service:fbcef59e18874aa3993b6eff782285ff:health:log 0
"{\"ts\":\"2017-10-24T08:29:27.609Z\",\"serviceName\":\"test-service\",\"type\":\"error\",\"processID\":6672,\"msg\":\"Service is shutting down.\"}"

127.0.0.1:6379> pttl hydra:service:test-service:fbcef59e18874aa3993b6eff782285ff:health:log
(integer) -1

Looking deeper into it, I notice that expire is called on the *:health:log keys as part of updatePresence(). This seems to be run on a timer. Nevertheless, my keys do not expire.

It's looking like I have a situation where the *:health:log key is created (i.e. the first log entry for the service is written) on service shutdown. In this case, the expire command never gets a chance to run on the key before the service shuts down, leaving the key in a persistent state.

I'm not sure if this situation is unique to my test setup, or if I've missed something somewhere. Does anybody else have this issue?

commented

@rantecki Thanks for submitting this! The health:log key is set to expire in one week.
https://github.com/flywheelsports/hydra/blob/master/index.js#L713
Please let us know if that's not working.

commented

Hi @cjus, as explained in the issue, this is not working for me.

commented

@rantecki will further look into this today.

commented

@rantecki I pushed hydra and hydra-express 1.4.26 which includes improved key expiration logic. However, that fix won't clean-up old keys. You may need to bring down services and issue a Redis flushdb command. Make sure you're comfortable doing that, and that you first issue a select 15 to switch to the DB that hydra uses.

commented

Thanks @cjus, that definitely seems to have fixed the problem.

I notice though that the health:log keys are now set to expire in KEY_EXPIRATION_TTL which is 3 seconds. That's a big jump from 1 week. Just wondering if that was intentional or perhaps a mistake?

commented

@rantecki You're correct. That's now fixed in hydra 1.4.27 and hydra-express 1.4.28