Gunicorn Async worker issue
vagarwal77 opened this issue · comments
I am using meinheld-gunicorn-flask base image and getting errors while using gunicorn asyn workers ("-k eventlet). If I remove “"-k eventlet”, (i.e. sync workers) every thing works fine.
Please suggest why I am having tough time with async workers? I have also attached requirement.txt for your referenmce.
Would really appreciate your help.
FROM tiangolo/meinheld-gunicorn-flask:python3.7
CMD ["--timeout=325", "--workers=5", "-k eventlet", "--bind", "0.0.0.0:5000", "wsgi:app"]
bash-4.2$ kubectl logs c3-dev1-semanticsearch-77fb799b6f-mmc5h -n pedatabase
[2019-12-11 00:27:31 +0000] [1] [INFO] Starting gunicorn 20.0.4
[2019-12-11 00:27:31 +0000] [1] [INFO] Listening at: http://0.0.0.0:5000 (1)
[2019-12-11 00:27:31 +0000] [1] [INFO] Using worker: eventlet
[2019-12-11 00:27:31 +0000] [10] [INFO] Booting worker with pid: 10
[2019-12-11 00:27:32 +0000] [11] [INFO] Booting worker with pid: 11
[2019-12-11 00:27:32 +0000] [12] [INFO] Booting worker with pid: 12
[2019-12-11 00:27:32 +0000] [13] [INFO] Booting worker with pid: 13
[2019-12-11 00:27:32 +0000] [14] [INFO] Booting worker with pid: 14
2019-12-11 00:27:33,464 - config_helper - MainThread - WARNING - [config_helper.py:38] redshift connection string not found in secret manager
[2019-12-11 00:27:33 +0000] [10] [ERROR] Exception in worker process
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/gunicorn/arbiter.py", line 583, in spawn_worker
worker.init_process()
File "/usr/local/lib/python3.7/site-packages/gunicorn/workers/geventlet.py", line 99, in init_process
super().init_process()
File "/usr/local/lib/python3.7/site-packages/gunicorn/workers/base.py", line 119, in init_process
self.load_wsgi()
File "/usr/local/lib/python3.7/site-packages/gunicorn/workers/base.py", line 144, in load_wsgi
self.wsgi = self.app.wsgi()
File "/usr/local/lib/python3.7/site-packages/gunicorn/app/base.py", line 67, in wsgi
self.callable = self.load()
File "/usr/local/lib/python3.7/site-packages/gunicorn/app/wsgiapp.py", line 49, in load
return self.load_wsgiapp()
File "/usr/local/lib/python3.7/site-packages/gunicorn/app/wsgiapp.py", line 39, in load_wsgiapp
return util.import_app(self.app_uri)
File "/usr/local/lib/python3.7/site-packages/gunicorn/util.py", line 358, in import_app
mod = importlib.import_module(module)
File "/usr/local/lib/python3.7/importlib/init.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "", line 1006, in _gcd_import
File "", line 983, in _find_and_load
File "", line 967, in _find_and_load_unlocked
File "", line 677, in _load_unlocked
File "", line 728, in exec_module
File "", line 219, in _call_with_frames_removed
File "/semanticsearch/app/wsgi.py", line 1, in
from runapp import app
File "/semanticsearch/app/runapp.py", line 29, in
from app import config
File "../app/config.py", line 25, in
SQL_ALCHEMY_ENGINE = create_engine(config_helper.getRedshiftConnectionString())
File "../util/config_helper.py", line 43, in getRedshiftConnectionString
'user': os.environ['REDSHIFT_USER'],
File "/usr/local/lib/python3.7/os.py", line 678, in getitem
raise KeyError(key) from None
KeyError: 'REDSHIFT_USER'
[2019-12-11 00:27:33 +0000] [10] [INFO] Worker exiting (pid: 10)
2019-12-11 00:27:33,541 - config_helper - MainThread - WARNING - [config_helper.py:38] redshift connection string not found in secret manager
[2019-12-11 00:27:33 +0000] [11] [ERROR] Exception in worker process
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/gunicorn/arbiter.py", line 583, in spawn_worker
worker.init_process()
File "/usr/local/lib/python3.7/site-packages/gunicorn/workers/geventlet.py", line 99, in init_process
super().init_process()
File "/usr/local/lib/python3.7/site-packages/gunicorn/workers/base.py", line 119, in init_process
self.load_wsgi()
File "/usr/local/lib/python3.7/site-packages/gunicorn/workers/base.py", line 144, in load_wsgi
self.wsgi = self.app.wsgi()
File "/usr/local/lib/python3.7/site-packages/gunicorn/app/base.py", line 67, in wsgi
self.callable = self.load()
File "/usr/local/lib/python3.7/site-packages/gunicorn/app/wsgiapp.py", line 49, in load
return self.load_wsgiapp()
File "/usr/local/lib/python3.7/site-packages/gunicorn/app/wsgiapp.py", line 39, in load_wsgiapp
return util.import_app(self.app_uri)
File "/usr/local/lib/python3.7/site-packages/gunicorn/util.py", line 358, in import_app
mod = importlib.import_module(module)
File "/usr/local/lib/python3.7/importlib/init.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "", line 1006, in _gcd_import
File "", line 983, in _find_and_load
File "", line 967, in _find_and_load_unlocked
return util.import_app(self.app_uri)
File "/usr/local/lib/python3.7/site-packages/gunicorn/util.py", line 358, in import_app
mod = importlib.import_module(module)
File "/usr/local/lib/python3.7/importlib/init.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "", line 1006, in _gcd_import
File "", line 983, in _find_and_load
File "", line 967, in _find_and_load_unlocked
File "", line 677, in _load_unlocked
File "", line 728, in exec_module
File "", line 219, in _call_with_frames_removed
File "/semanticsearch/app/wsgi.py", line 1, in
from runapp import app
File "/semanticsearch/app/runapp.py", line 29, in
from app import config
File "../app/config.py", line 25, in
SQL_ALCHEMY_ENGINE = create_engine(config_helper.getRedshiftConnectionString())
File "../util/config_helper.py", line 43, in getRedshiftConnectionString
'user': os.environ['REDSHIFT_USER'],
File "/usr/local/lib/python3.7/os.py", line 678, in getitem
raise KeyError(key) from None
KeyError: 'REDSHIFT_USER'
[2019-12-11 00:27:33 +0000] [10] [INFO] Worker exiting (pid: 10)
2019-12-11 00:27:33,541 - config_helper - MainThread - WARNING - [config_helper.py:38] redshift connection string not found in secret manager
[2019-12-11 00:27:33 +0000] [11] [ERROR] Exception in worker process
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/gunicorn/arbiter.py", line 583, in spawn_worker
worker.init_process()
File "/usr/local/lib/python3.7/site-packages/gunicorn/workers/geventlet.py", line 99, in init_process
super().init_process()
File "/usr/local/lib/python3.7/site-packages/gunicorn/workers/base.py", line 119, in init_process
self.load_wsgi()
File "/usr/local/lib/python3.7/site-packages/gunicorn/workers/base.py", line 144, in load_wsgi
self.wsgi = self.app.wsgi()
File "/usr/local/lib/python3.7/site-packages/gunicorn/app/base.py", line 67, in wsgi
self.callable = self.load()
File "/usr/local/lib/python3.7/site-packages/gunicorn/app/wsgiapp.py", line 49, in load
return self.load_wsgiapp()
File "/usr/local/lib/python3.7/site-packages/gunicorn/app/wsgiapp.py", line 39, in load_wsgiapp
return util.import_app(self.app_uri)
File "/usr/local/lib/python3.7/site-packages/gunicorn/util.py", line 358, in import_app
mod = importlib.import_module(module)
File "/usr/local/lib/python3.7/importlib/init.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "", line 1006, in _gcd_import
File "", line 983, in _find_and_load
File "", line 967, in _find_and_load_unlocked
File "", line 677, in _load_unlocked
File "", line 728, in exec_module
File "", line 219, in _call_with_frames_removed
File "/semanticsearch/app/wsgi.py", line 1, in
from runapp import app
File "/semanticsearch/app/runapp.py", line 29, in
from app import config
File "../app/config.py", line 25, in
SQL_ALCHEMY_ENGINE = create_engine(config_helper.getRedshiftConnectionString())
File "../util/config_helper.py", line 43, in getRedshiftConnectionString
'user': os.environ['REDSHIFT_USER'],
File "/usr/local/lib/python3.7/os.py", line 678, in getitem
raise KeyError(key) from None
KeyError: 'REDSHIFT_USER'
[2019-12-11 00:27:33 +0000] [11] [INFO] Worker exiting (pid: 11)
2019-12-11 00:27:33,615 - config_helper - MainThread - WARNING - [config_helper.py:38] redshift connection string not found in secret manager
[2019-12-11 00:27:33 +0000] [12] [ERROR] Exception in worker process
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/gunicorn/arbiter.py", line 583, in spawn_worker
worker.init_process()
File "/usr/local/lib/python3.7/site-packages/gunicorn/workers/geventlet.py", line 99, in init_process
super().init_process()
File "/usr/local/lib/python3.7/site-packages/gunicorn/workers/base.py", line 119, in init_process
self.load_wsgi()
File "/usr/local/lib/python3.7/site-packages/gunicorn/workers/base.py", line 144, in load_wsgi
self.wsgi = self.app.wsgi()
The whole idea of this image is not to use Gevent but Meinheld, as it shows the best performance in benchmarks.
If you need to use Gevent, then you probably should create a Dockerfile based on the official Python image and install Gunicorn and Gevent in it instead of using this image.
Assuming the original issue was solved, it will be automatically closed now. But feel free to add more comments or create new issues.