Docker Flask Celery Redis and Nginx
A basic Docker Compose template for orchestrating a Flask application & a Celery queue with Redis
Installation
git clone git@gitlab.com:sfcarroll/flask-celery-docker.git
cp .env.example .envBuild & Launch
docker-compose up -d --buildThis will expose the Flask application's endpoints on port 5000 as well as a Flower server for monitoring workers on port 5555
To add more workers:
docker-compose up -d --scale worker=5 --no-recreateTo shut down:
docker-compose downStart, stop, delete
stop all containers:
docker kill $(docker ps -q) && docker rm $(docker ps -a -q) && docker rmi $(docker images -q)
remove all containers
docker rm $(docker ps -a -q)
remove all docker images
docker rmi $(docker images -q)
To change the endpoints, update the code in api/app.py
Task changes should happen in queue/tasks.py
Initial build
git clone https://github.com/seanfcarroll/flask-celery-docker.git
docker-compose build
docker-compose up -d
Get status and logs
docker-compose ps
docker ps
docker logs 7b7ff37ed2d5
Rebuild a specific container
docker-compose ps
docker-compose stop web
docker-compose rm web
docker-compose up -d
Start, stop, delete
stop all containers:
docker kill $(docker ps -q) && docker rm $(docker ps -a -q) && docker rmi $(docker images -q)
remove all containers
docker rm $(docker ps -a -q)
remove all docker images
docker rmi $(docker images -q)
Shell into container
sudo docker exec -i -t 665b4a1e17b6 /bin/sh
Scaling
docker-compose scale worker=5
This will create 4 more containers each running a worker. http://your-dockermachine-ip:5555 should now show 5 workers waiting for some jobs!
Helpful
https://gist.github.com/amatellanes/a986f6babb9cf8556e36
https://stackoverflow.com/questions/50798281/register-multiple-tasks-with-celery
https://stackoverflow.com/questions/43920621/celery-tasks-on-multiple-machines
https://stackoverflow.com/questions/19853378/how-to-keep-multiple-independent-celery-queues