Add celery task distribution for user-provided mutation search
krassowski opened this issue · comments
If user provides a VCF file with thousands of mutations it locks one worker to await for search results. It may be better to use a task distribution system so we can prevent full website lock if too many users upload their VCF files at once.
We may use other task distribution system, but Celery seems to integrate well with Flask and was used for similar purpose by SPANR developers (http://tools.genes.toronto.edu/).
I implemented and deployed a basic celery task distribution. When searching for a bigger batch of mutations it shows a progress bar which is updated every 5 seconds. Everything seems to be working nicely and we don't need to be afraid that our server will hang due to too heavy load now (as additional search tasks will be queued and wait for a free worker to be executed).
I will make the progress page look nicer soon and close the issue if no problems emerge.