more info on race condition
jjb opened this issue · comments
Hi, thanks for an amazing project! Question about this:
Coverband on very high volume sites with many server processes reporting can have a race condition
- Does this mean that on a very high volume sites with many server processes reporting, there will be a somewhat frequent race condition, and on lower volume sites with fewer server processes, the same race condition can occur but much less often?
- How does this race condition present?
yeah, good call. I should try to improve this area of documentation... I will work on improving the docs and comments in the files.
For now:
Does this mean that on a very high volume sites with many server processes reporting, there will be a somewhat frequent race condition, and on lower volume sites with fewer server processes, the same race condition can occur but much less often?
Yes the race condition is when multiple servers are performing two operations against redis, the more servers you have talking to redis the higher the change that different servers will talk to redis between the two calls. So there is never an issue with a single server, but then the chance of a race increases as new servers are added. Honestly, I never noticed this error until running on systems with more than 250 reporting puma processes.
How does this race condition present?
When the race situation occurs one of the X servers will win the coverage update, resulting in some lost coverage data... This doesn't hurt the app or perf in anyway, but could result in hit lines not being reported as hit.
Solution:
We now have two Redis adapters the normal one which has this issue or a new one which doesn't have a race condition and moves some load from your application servers to the redis server. To use the new adapter you just set it in your config/coverband.rb like so.
config.store = Coverband::Adapters::HashRedisStore.new(Redis.new(url: redis_url), redis_namespace: 'coverband_data')
Note that this will increase memory and CPU needs on your Redis a little bit, if you use this depending on your utilization you might want a dedicated Redis... I run this on a dedicated redis for some apps with massive server clusters, which already put pretty high load on redis via background jobs and Rail.cache
OK, I think this is answered pretty well here... we could perhaps update the docs but maybe we just link to the issue and now it is always in google for search so calling this good.
okay, made a PR - feel free to reword or reject if you think google search is good enough :-D thanks
ok cool that is awesome I will take a look