Giters

tiangolo / uvicorn-gunicorn-fastapi-docker

Docker image with Uvicorn managed by Gunicorn for high-performance FastAPI web applications in Python with performance auto-tuning.

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

[CRITICAL] WORKER TIMEOUT with basic fastapi and no load. (with example)

NielsDebrier opened this issue · comments

commented

First of all I want to say I love fastapi. It a blast to work with so big thanks to you and your team.

Now my problem.
We have a fastapi instance running on kubernetes. The weird part is that disk usage spikes until the point it is being evicted because lack of space. Upon further inspection we saw [CRITICAL] WORKER TIMEOUT. Now when we exec into the docker container we found a lot of files like these: core.gunicorn.3161.1590754193 which seem to be around 40M each. These are unreadable.

After long searching I didn't find anything related to the core.gunicorn files. But upon further inspection it became apparent that it had to do with the [CRITICAL] WORKER TIMEOUT. Because every file seems to correspond with one [CRITICAL] WORKER TIMEOUT. So I went after the problem to try and fix the Critical issue because it seems that will also fix my disk usage issue.

The weird thing is that we don't have crazy load or long processes running. Which often seems the problem with this [CRITICAL] WORKER TIMEOUT log.

So I made a simple basic fastapi app with about the same settings that we use on kubernetes. And local I get the same log [CRITICAL] WORKER TIMEOUT. Local I don't get the files but that is something I can look into when this critical issue is fixed.

The repo with the simple fastapi + docker can be found here: https://github.com/NielsDebrier/fastapiProblem
How to run the docker with the correct settings you can find in the readme.

If anyone could help me or have suggestions on what to do that would be amazing.
I have searched around a lot and was not able to find a solution for this. And because the example app is so simple I don't understand why there would be [CRITICAL] logs.

Thank you

Here are the debug logs I get local:

Checking for script in /app/prestart.sh
Running script /app/prestart.sh
Running inside /app/prestart.sh, you could add migrations to this file, e.g.:

#! /usr/bin/env bash

# Let the DB start
sleep 10;
# Run migrations
alembic upgrade head

{"loglevel": "debug", "workers": 6, "bind": "0.0.0.0:8080", "workers_per_core": 1.0, "host": "0.0.0.0", "port": "8080"}
[2020-05-29 11:51:35 +0000] [1] [DEBUG] Current configuration:
  config: /gunicorn_conf.py
  bind: ['0.0.0.0:8080']
  backlog: 2048
  workers: 6
  worker_class: uvicorn.workers.UvicornWorker
  threads: 1
  worker_connections: 1000
  max_requests: 0
  max_requests_jitter: 0
  timeout: 30
  graceful_timeout: 30
  keepalive: 120
  limit_request_line: 4094
  limit_request_fields: 100
  limit_request_field_size: 8190
  reload: False
  reload_engine: auto
  reload_extra_files: []
  spew: False
  check_config: False
  preload_app: False
  sendfile: None
  reuse_port: False
  chdir: /app
  daemon: False
  raw_env: []
  pidfile: None
  worker_tmp_dir: None
  user: 0
  group: 0
  umask: 0
  initgroups: False
  tmp_upload_dir: None
  secure_scheme_headers: {'X-FORWARDED-PROTOCOL': 'ssl', 'X-FORWARDED-PROTO': 'https', 'X-FORWARDED-SSL': 'on'}
  forwarded_allow_ips: ['127.0.0.1']
  accesslog: None
  disable_redirect_access_to_syslog: False
  access_log_format: %(h)s %(l)s %(u)s %(t)s "%(r)s" %(s)s %(b)s "%(f)s" "%(a)s"
  errorlog: -
  loglevel: debug
  capture_output: False
  logger_class: gunicorn.glogging.Logger
  logconfig: None
  logconfig_dict: {}
  syslog_addr: udp://localhost:514
  syslog: False
  syslog_prefix: None
  syslog_facility: user
  enable_stdio_inheritance: False
  statsd_host: None
  dogstatsd_tags:
  statsd_prefix:
  proc_name: None
  default_proc_name: fast.main:app
  pythonpath: None
  paste: None
  on_starting: <function OnStarting.on_starting at 0x7f5d40ba59e0>
  on_reload: <function OnReload.on_reload at 0x7f5d40ba5b00>
  when_ready: <function WhenReady.when_ready at 0x7f5d40ba5c20>
  pre_fork: <function Prefork.pre_fork at 0x7f5d40ba5d40>
  post_fork: <function Postfork.post_fork at 0x7f5d40ba5e60>
  post_worker_init: <function PostWorkerInit.post_worker_init at 0x7f5d40ba5f80>
  worker_int: <function WorkerInt.worker_int at 0x7f5d40b210e0>
  worker_abort: <function WorkerAbort.worker_abort at 0x7f5d40b21200>
  pre_exec: <function PreExec.pre_exec at 0x7f5d40b21320>
  pre_request: <function PreRequest.pre_request at 0x7f5d40b21440>
  post_request: <function PostRequest.post_request at 0x7f5d40b214d0>
  child_exit: <function ChildExit.child_exit at 0x7f5d40b215f0>
  worker_exit: <function WorkerExit.worker_exit at 0x7f5d40b21710>
  nworkers_changed: <function NumWorkersChanged.nworkers_changed at 0x7f5d40b21830>
  on_exit: <function OnExit.on_exit at 0x7f5d40b21950>
  proxy_protocol: False
  proxy_allow_ips: ['127.0.0.1']
  keyfile: None
  certfile: None
  ssl_version: 2
  cert_reqs: 0
  ca_certs: None
  suppress_ragged_eofs: True
  do_handshake_on_connect: False
  ciphers: None
  raw_paste_global_conf: []
  strip_header_spaces: False
[2020-05-29 11:51:35 +0000] [1] [INFO] Starting gunicorn 20.0.4
[2020-05-29 11:51:35 +0000] [1] [DEBUG] Arbiter booted
[2020-05-29 11:51:35 +0000] [1] [INFO] Listening at: http://0.0.0.0:8080 (1)
[2020-05-29 11:51:35 +0000] [1] [INFO] Using worker: uvicorn.workers.UvicornWorker
[2020-05-29 11:51:35 +0000] [8] [INFO] Booting worker with pid: 8
[2020-05-29 11:51:35 +0000] [9] [INFO] Booting worker with pid: 9
[2020-05-29 11:51:35 +0000] [8] [INFO] Started server process [8]
[2020-05-29 11:51:35 +0000] [8] [INFO] Waiting for application startup.
[2020-05-29 11:51:35 +0000] [8] [INFO] Application startup complete.
[2020-05-29 11:51:35 +0000] [10] [INFO] Booting worker with pid: 10
[2020-05-29 11:51:35 +0000] [11] [INFO] Booting worker with pid: 11
[2020-05-29 11:51:35 +0000] [9] [INFO] Started server process [9]
[2020-05-29 11:51:35 +0000] [9] [INFO] Waiting for application startup.
[2020-05-29 11:51:35 +0000] [9] [INFO] Application startup complete.
[2020-05-29 11:51:35 +0000] [12] [INFO] Booting worker with pid: 12
[2020-05-29 11:51:36 +0000] [10] [INFO] Started server process [10]
[2020-05-29 11:51:36 +0000] [10] [INFO] Waiting for application startup.
[2020-05-29 11:51:36 +0000] [10] [INFO] Application startup complete.
[2020-05-29 11:51:36 +0000] [13] [INFO] Booting worker with pid: 13
[2020-05-29 11:51:36 +0000] [11] [INFO] Started server process [11]
[2020-05-29 11:51:36 +0000] [11] [INFO] Waiting for application startup.
[2020-05-29 11:51:36 +0000] [11] [INFO] Application startup complete.
[2020-05-29 11:51:36 +0000] [1] [DEBUG] 6 workers
[2020-05-29 11:51:36 +0000] [12] [INFO] Started server process [12]
[2020-05-29 11:51:36 +0000] [12] [INFO] Waiting for application startup.
[2020-05-29 11:51:36 +0000] [12] [INFO] Application startup complete.
[2020-05-29 11:51:36 +0000] [13] [INFO] Started server process [13]
[2020-05-29 11:51:36 +0000] [13] [INFO] Waiting for application startup.
[2020-05-29 11:51:36 +0000] [13] [INFO] Application startup complete.
[2020-05-29 11:52:06 +0000] [1] [CRITICAL] WORKER TIMEOUT (pid:8)
[2020-05-29 11:52:06 +0000] [1] [CRITICAL] WORKER TIMEOUT (pid:9)
[2020-05-29 11:52:06 +0000] [1] [CRITICAL] WORKER TIMEOUT (pid:10)
[2020-05-29 11:52:06 +0000] [1] [CRITICAL] WORKER TIMEOUT (pid:11)
[2020-05-29 11:52:06 +0000] [1] [CRITICAL] WORKER TIMEOUT (pid:12)
[2020-05-29 11:52:06 +0000] [14] [INFO] Booting worker with pid: 14
[2020-05-29 11:52:06 +0000] [15] [INFO] Booting worker with pid: 15
[2020-05-29 11:52:06 +0000] [16] [INFO] Booting worker with pid: 16
[2020-05-29 11:52:06 +0000] [14] [INFO] Started server process [14]
[2020-05-29 11:52:06 +0000] [14] [INFO] Waiting for application startup.
[2020-05-29 11:52:06 +0000] [14] [INFO] Application startup complete.
[2020-05-29 11:52:06 +0000] [15] [INFO] Started server process [15]
[2020-05-29 11:52:06 +0000] [15] [INFO] Waiting for application startup.
[2020-05-29 11:52:06 +0000] [15] [INFO] Application startup complete.
[2020-05-29 11:52:06 +0000] [17] [INFO] Booting worker with pid: 17
[2020-05-29 11:52:06 +0000] [16] [INFO] Started server process [16]
[2020-05-29 11:52:06 +0000] [16] [INFO] Waiting for application startup.
[2020-05-29 11:52:06 +0000] [16] [INFO] Application startup complete.
[2020-05-29 11:52:06 +0000] [1] [DEBUG] 5 workers
[2020-05-29 11:52:06 +0000] [1] [CRITICAL] WORKER TIMEOUT (pid:13)
[2020-05-29 11:52:06 +0000] [18] [INFO] Booting worker with pid: 18
[2020-05-29 11:52:06 +0000] [17] [INFO] Started server process [17]
[2020-05-29 11:52:06 +0000] [17] [INFO] Waiting for application startup.
[2020-05-29 11:52:06 +0000] [17] [INFO] Application startup complete.
[2020-05-29 11:52:06 +0000] [19] [INFO] Booting worker with pid: 19
[2020-05-29 11:52:06 +0000] [18] [INFO] Started server process [18]
[2020-05-29 11:52:06 +0000] [18] [INFO] Waiting for application startup.
[2020-05-29 11:52:06 +0000] [18] [INFO] Application startup complete.
[2020-05-29 11:52:06 +0000] [1] [DEBUG] 6 workers
[2020-05-29 11:52:06 +0000] [19] [INFO] Started server process [19]
[2020-05-29 11:52:06 +0000] [19] [INFO] Waiting for application startup.
[2020-05-29 11:52:06 +0000] [19] [INFO] Application startup complete.
[2020-05-29 11:52:36 +0000] [1] [CRITICAL] WORKER TIMEOUT (pid:14)
[2020-05-29 11:52:36 +0000] [1] [CRITICAL] WORKER TIMEOUT (pid:15)
[2020-05-29 11:52:36 +0000] [1] [CRITICAL] WORKER TIMEOUT (pid:16)
[2020-05-29 11:52:36 +0000] [1] [CRITICAL] WORKER TIMEOUT (pid:17)
[2020-05-29 11:52:36 +0000] [1] [CRITICAL] WORKER TIMEOUT (pid:18)
[2020-05-29 11:52:36 +0000] [1] [CRITICAL] WORKER TIMEOUT (pid:19)
[2020-05-29 11:52:36 +0000] [38] [INFO] Booting worker with pid: 38
[2020-05-29 11:52:36 +0000] [39] [INFO] Booting worker with pid: 39
[2020-05-29 11:52:36 +0000] [38] [INFO] Started server process [38]
[2020-05-29 11:52:36 +0000] [38] [INFO] Waiting for application startup.
[2020-05-29 11:52:36 +0000] [38] [INFO] Application startup complete.
[2020-05-29 11:52:36 +0000] [40] [INFO] Booting worker with pid: 40
[2020-05-29 11:52:36 +0000] [39] [INFO] Started server process [39]
[2020-05-29 11:52:36 +0000] [39] [INFO] Waiting for application startup.
[2020-05-29 11:52:36 +0000] [39] [INFO] Application startup complete.
[2020-05-29 11:52:36 +0000] [41] [INFO] Booting worker with pid: 41
[2020-05-29 11:52:36 +0000] [1] [DEBUG] 4 workers
[2020-05-29 11:52:36 +0000] [42] [INFO] Booting worker with pid: 42
[2020-05-29 11:52:37 +0000] [43] [INFO] Booting worker with pid: 43
[2020-05-29 11:52:37 +0000] [40] [INFO] Started server process [40]
[2020-05-29 11:52:37 +0000] [40] [INFO] Waiting for application startup.
[2020-05-29 11:52:37 +0000] [40] [INFO] Application startup complete.
[2020-05-29 11:52:37 +0000] [1] [DEBUG] 6 workers
[2020-05-29 11:52:37 +0000] [41] [INFO] Started server process [41]
[2020-05-29 11:52:37 +0000] [41] [INFO] Waiting for application startup.
[2020-05-29 11:52:37 +0000] [41] [INFO] Application startup complete.
[2020-05-29 11:52:37 +0000] [42] [INFO] Started server process [42]
[2020-05-29 11:52:37 +0000] [42] [INFO] Waiting for application startup.
[2020-05-29 11:52:37 +0000] [42] [INFO] Application startup complete.
[2020-05-29 11:52:37 +0000] [43] [INFO] Started server process [43]
[2020-05-29 11:52:37 +0000] [43] [INFO] Waiting for application startup.
[2020-05-29 11:52:37 +0000] [43] [INFO] Application startup complete.
[2020-05-29 11:53:07 +0000] [1] [CRITICAL] WORKER TIMEOUT (pid:38)
[2020-05-29 11:53:07 +0000] [1] [CRITICAL] WORKER TIMEOUT (pid:39)
[2020-05-29 11:53:07 +0000] [1] [CRITICAL] WORKER TIMEOUT (pid:40)
[2020-05-29 11:53:07 +0000] [44] [INFO] Booting worker with pid: 44
[2020-05-29 11:53:07 +0000] [45] [INFO] Booting worker with pid: 45
[2020-05-29 11:53:07 +0000] [46] [INFO] Booting worker with pid: 46
[2020-05-29 11:53:07 +0000] [1] [CRITICAL] WORKER TIMEOUT (pid:41)
[2020-05-29 11:53:07 +0000] [1] [CRITICAL] WORKER TIMEOUT (pid:42)
[2020-05-29 11:53:07 +0000] [1] [CRITICAL] WORKER TIMEOUT (pid:43)
[2020-05-29 11:53:07 +0000] [47] [INFO] Booting worker with pid: 47
[2020-05-29 11:53:07 +0000] [44] [INFO] Started server process [44]
[2020-05-29 11:53:07 +0000] [44] [INFO] Waiting for application startup.
[2020-05-29 11:53:07 +0000] [44] [INFO] Application startup complete.
[2020-05-29 11:53:07 +0000] [45] [INFO] Started server process [45]
[2020-05-29 11:53:07 +0000] [45] [INFO] Waiting for application startup.
[2020-05-29 11:53:07 +0000] [45] [INFO] Application startup complete.
[2020-05-29 11:53:07 +0000] [46] [INFO] Started server process [46]
[2020-05-29 11:53:07 +0000] [46] [INFO] Waiting for application startup.
[2020-05-29 11:53:07 +0000] [46] [INFO] Application startup complete.
[2020-05-29 11:53:07 +0000] [48] [INFO] Booting worker with pid: 48
[2020-05-29 11:53:07 +0000] [47] [INFO] Started server process [47]
[2020-05-29 11:53:07 +0000] [47] [INFO] Waiting for application startup.
[2020-05-29 11:53:07 +0000] [47] [INFO] Application startup complete.
[2020-05-29 11:53:07 +0000] [49] [INFO] Booting worker with pid: 49
[2020-05-29 11:53:07 +0000] [48] [INFO] Started server process [48]
[2020-05-29 11:53:07 +0000] [48] [INFO] Waiting for application startup.
[2020-05-29 11:53:07 +0000] [48] [INFO] Application startup complete.
[2020-05-29 11:53:07 +0000] [49] [INFO] Started server process [49]
[2020-05-29 11:53:07 +0000] [49] [INFO] Waiting for application startup.
[2020-05-29 11:53:07 +0000] [49] [INFO] Application startup complete.
[2020-05-29 11:53:37 +0000] [1] [CRITICAL] WORKER TIMEOUT (pid:44)
[2020-05-29 11:53:37 +0000] [1] [CRITICAL] WORKER TIMEOUT (pid:45)
[2020-05-29 11:53:37 +0000] [1] [CRITICAL] WORKER TIMEOUT (pid:46)
[2020-05-29 11:53:37 +0000] [1] [CRITICAL] WORKER TIMEOUT (pid:47)
[2020-05-29 11:53:37 +0000] [1] [CRITICAL] WORKER TIMEOUT (pid:48)
[2020-05-29 11:53:37 +0000] [50] [INFO] Booting worker with pid: 50
...
commented

have you try setting the ENV TIMEOUT and GRACEFUL_TIMEOUT to 0?

commented

Have tried it, but didn't seem to fix it.
Bumping uvicorn fixed it for me locally. I will now test it online.
I will close it for now.

bumping to version 0.11.7

commented

Thanks for reporting back and closing the issue 👍