Blocking `ExtendLockLifetime` might cause `ThreadPool` starvation
foriequal0 opened this issue · comments
We hold per-user RedLock for every request. It works fine under a normal load.
But the problem starts when sometimes our server response time gets slower.
The request inbound rate remains the same for our scenario.
However the response time got longer, concurrent request count went higher.
Hence ExtendLockLifetime
is called more.
https://github.com/samcook/RedLock.net/blob/master/RedLockNet.SERedis/RedLock.cs#L314
ExtendLockLifetime
is called by Timer
, and Timer callbacks are dispatched to ThreadPool.
ThreadPool doesn't like blocking tasks, and it is crucial to have performant ThreadPool for performant async C#.
In our case, all threads in the ThreadPool are blocked by the sudden surge of ExtendLockLifetime
invocations, thus it causes ASP.NET server to almost stall.
.NET tries to spawn new threads to resolve starvation, but it cannot catch up with the rate of blocked threads with the default threading config. (and they do not recommend manually adjusting it)
https://learn.microsoft.com/en-us/dotnet/core/runtime-config/threading
Even if we change the config to spin up new threads quickly, it is not performant enough with that amount of threads due to the massive number of threads and context switches.