TTL being removed from Redis objects
tommymonk opened this issue · comments
We've been experiencing an issue where users are being blocked unexpectedly, it seems to us that the TTL of the redis objects is somehow being (re)set to -1 which means the counter becomes permanent.
We're using "Azure Cache For Redis" so I'm not sure the if this is a bug or incompatibility. When objects are created and we're watching they seem to have a valid TTL.
We haven't been able to reproduce the issue running locally, even pointing at the Azure cache.
I have also seen this periodically and have the exact same setup. I havn't been able to narrow down how this happens either.
I am also on Azure Redis. We had a client getting 429. The limit was applied over 4h. However, the client had been getting all requests denied for the last 2 days.
I have logs showing the count continuing to past the expected reset interval.
limit_logs.csv
@cristipufu
We observed the same issue, which resulted in some clients being blocked indefinitely, effectively causing a complete downtime for them even though they did not hit the specified API limits. We disabled rate limiting completely for now because of this - though I am aware we could use the provided custom strategy in the linked PR as a workaround.
Any chance the linked PR could be reviewed/merged in the near future? Let me know if I can help in any way.
Edit: my current best guess for why this is happening: Maybe the redis script execution gets aborted for whatever reason between calling INCRBYFLOAT
and calling TTL
. This would lead to the scenario where we never set the TTL at all for this key.