Why is my app stuck at booting new workers?
timpal0l opened this issue · comments
I have a rather simple python script that loads a large file in the beginning, before defining the fastapi endpoints.
app = FastAPI()
embeddings = np.load('embeddings.npy') # 15 sec to load
This takes about 15 seconds on a normal laptop. The app runs fine if i start it with vanilla uvicorn without a docker.
When using the fastapi-docker and the following settings are defined in thegunicorn_conf.py
{"loglevel": "info", "workers": 1, "bind": "0.0.0.0:80", "graceful_timeout": 300, "timeout": 300, "keepalive": 300, "errorlog": "-", "accesslog": "-", "workers_per_core": 1, "use_max_workers": null, "host": "0.0.0.0", "port": "80"}
[2020-05-08 08:41:27 +0000] [1] [INFO] Listening at: http://0.0.0.0:80 (1)
[2020-05-08 08:41:27 +0000] [1] [INFO] Using worker: uvicorn.workers.UvicornWorker
[2020-05-08 08:41:27 +0000] [8] [INFO] Booting worker with pid: 8
[2020-05-08 08:41:43 +0000] [19] [INFO] Booting worker with pid: 19
[2020-05-08 08:42:00 +0000] [30] [INFO] Booting worker with pid: 30
[2020-05-08 08:42:17 +0000] [41] [INFO] Booting worker with pid: 41
[2020-05-08 08:42:33 +0000] [52] [INFO] Booting worker with pid: 52
[2020-05-08 08:42:51 +0000] [63] [INFO] Booting worker with pid: 63
[2020-05-08 08:43:05 +0000] [74] [INFO] Booting worker with pid: 74
[2020-05-08 08:43:20 +0000] [85] [INFO] Booting worker with pid: 85
[2020-05-08 08:43:36 +0000] [96] [INFO] Booting worker with pid: 96
[2020-05-08 08:43:52 +0000] [107] [INFO] Booting worker with pid: 107
[2020-05-08 08:44:06 +0000] [118] [INFO] Booting worker with pid: 118
[2020-05-08 08:44:20 +0000] [129] [INFO] Booting worker with pid: 129
[2020-05-08 08:44:34 +0000] [140] [INFO] Booting worker with pid: 140
[2020-05-08 08:44:50 +0000] [151] [INFO] Booting worker with pid: 151
[2020-05-08 08:45:05 +0000] [162] [INFO] Booting worker with pid: 162
[2020-05-08 08:45:22 +0000] [173] [INFO] Booting worker with pid: 173
[2020-05-08 08:45:36 +0000] [184] [INFO] Booting worker with pid: 184
[2020-05-08 08:45:54 +0000] [195] [INFO] Booting worker with pid: 195
[2020-05-08 08:46:08 +0000] [206] [INFO] Booting worker with pid: 206
[2020-05-08 08:46:24 +0000] [217] [INFO] Booting worker with pid: 217
I set all kinds of timeout to 300 and workers to 1. The app is never reachable at http://0.0.0.0:80 since it's just spawning new workers and the startup never completes.
Started with docker run -p 80:80 -t my_app
Try increasing the GRACEFUL_TIMEOUT https://github.com/tiangolo/uvicorn-gunicorn-fastapi-docker#graceful_timeout
And decreasing the maximum number of workers (not workers per core, but absolute number of workers) https://github.com/tiangolo/uvicorn-gunicorn-fastapi-docker#web_concurrency
Also, monitor your machine's CPU and memory. If it takes 15 secs to load, I would imagine your embeddings are huge. You might be consuming all your RAM before it can finish loading. And with so many cores, and starting so many processes, each one of them will take the same amount of RAM each.
@tiangolo It turned out to be the Docker engines max ram was set to 2GB when the embeddings was 4GB!
@dwdewang it would probably be better to ask in a new GitHub issue providing a self-contained, minimal, reproducible, example that I can copy-paste to replicate it.
Sorry for the long delay! 🙈 I wanted to personally address each issue/PR and they piled up through time, but now I'm checking each one in order.