Stats generation job fails entirely on deadlock detection
lladdy opened this issue · comments
Occasionally, a deadlock will occur when attempting to lock a competition participation.
Instead of the entire stats gen job falling over, it should at minimum skip the competition participation in question.
More ideally it should wait and then retry whatever caused the deadlock, with some max retry attempt.
This has been improved with an update but still results in bots not having their stats updated for up to ~a week.
Relevant stats job code is here, for anyone interested in trying to refine it:
https://github.com/aiarena/aiarena-web/blob/staging/aiarena/core/management/commands/generatestats.py