[BUG]wsgi多进程启动时同时创建日志文件夹出错

Question

[BUG]wsgi多进程启动时同时创建日志文件夹出错

devenami opened this issue 8 months ago · comments

提问题时，请尊重我！把必要的信息，什么环境，输入具体什么文本，运行什么函数讲清楚！
不要甩一句话，说的不清不楚，我无从定位，浪费时间。这样的提单我将直接close。

描述(Description)

描述你遇到了什么问题(Please describe your issue here)

在使用wsgi容器启动http服务时，如果wsgi容器启动多个worker进程会导致__init__.py文件中的logging调用多次，并且由于worker时同时启动的，set_logger中指定的文件夹路径可能会重复创建，在使用os.makedirs创建文件夹时，由于是多进程并行重复创建，函数内的判断文件夹是否存在的逻辑就可能出现无效判断的情况，从而导致makedirs函数报错。

版本(Version):

python 版本: (通过 python 可查) 3.10
jionlp 版本: (通过 jionlp.__version__ 可查) 1.3.47

jionlp的调用代码与输入文本(Code & Text):

假如我们使用gunicorn启动服务，并使用-w 9启动9个worker进程，在启动之前先删除用户目录下的~/.jionlp_logs文件夹，多尝试几次即可复现。

gunicorn -b 0.0.0.0:8080 -w 9 main:run_in_wsgi

调用报错日志如下(Log):

# jionlp - 微信公众号: JioNLP  Github: `https://github.com/dongrixinyu/JioNLP`.
# jionlp - 微信公众号: JioNLP  Github: `https://github.com/dongrixinyu/JioNLP`.
# jionlp - 微信公众号: JioNLP  Github: `https://github.com/dongrixinyu/JioNLP`.
# jionlp - 微信公众号: JioNLP  Github: `https://github.com/dongrixinyu/JioNLP`.
# jionlp - 微信公众号: JioNLP  Github: `https://github.com/dongrixinyu/JioNLP`.
# jionlp - 微信公众号: JioNLP  Github: `https://github.com/dongrixinyu/JioNLP`.
# jionlp - 微信公众号: JioNLP  Github: `https://github.com/dongrixinyu/JioNLP`.
# jionlp - 微信公众号: JioNLP  Github: `https://github.com/dongrixinyu/JioNLP`.
# jionlp - 微信公众号: JioNLP  Github: `https://github.com/dongrixinyu/JioNLP`.
# jiojio - `http://www.jionlp.com/jionlp_online/cws_pos` is available for online trial.
# jiojio - `http://www.jionlp.com/jionlp_online/cws_pos` is available for online trial.
# jiojio - `http://www.jionlp.com/jionlp_online/cws_pos` is available for online trial.
# jiojio - `http://www.jionlp.com/jionlp_online/cws_pos` is available for online trial.
# jiojio - `http://www.jionlp.com/jionlp_online/cws_pos` is available for online trial.
# jiojio - `http://www.jionlp.com/jionlp_online/cws_pos` is available for online trial.
# jiojio - `http://www.jionlp.com/jionlp_online/cws_pos` is available for online trial.
[2023-10-25 19:45:15 +0800] [73] [ERROR] Exception in worker process
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/gunicorn/arbiter.py", line 609, in spawn_worker
    worker.init_process()
  File "/usr/local/lib/python3.10/site-packages/gunicorn/workers/base.py", line 134, in init_process
    self.load_wsgi()
  File "/usr/local/lib/python3.10/site-packages/gunicorn/workers/base.py", line 146, in load_wsgi
    self.wsgi = self.app.wsgi()
  File "/usr/local/lib/python3.10/site-packages/gunicorn/app/base.py", line 67, in wsgi
    self.callable = self.load()
  File "/usr/local/lib/python3.10/site-packages/gunicorn/app/wsgiapp.py", line 58, in load
    return self.load_wsgiapp()
  File "/usr/local/lib/python3.10/site-packages/gunicorn/app/wsgiapp.py", line 48, in load_wsgiapp
    return util.import_app(self.app_uri)
  File "/usr/local/lib/python3.10/site-packages/gunicorn/util.py", line 371, in import_app
    mod = importlib.import_module(module)
  File "/usr/local/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "", line 1050, in _gcd_import
  File "", line 1027, in _find_and_load
  File "", line 1006, in _find_and_load_unlocked
  File "", line 688, in _load_unlocked
  File "", line 883, in exec_module
  File "", line 241, in _call_with_frames_removed
  File "/app/main.py", line 1, in 
    import apiserver
  File "/app/apiserver/__init__.py", line 13, in 
    from apiserver import sound_service, review_service, cdp_service
  File "/app/apiserver/review_service.py", line 12, in 
    from common.utils import redis_util, weixin_util
  File "/app/common/utils/weixin_util.py", line 17, in 
    from common.utils import es_util, str_util, qiyeweixin_util
  File "/app/common/utils/es_util.py", line 14, in 
    from common.utils.dete_util import get_current_hour, get_current_date, get_date_format, add_date
  File "/app/common/utils/dete_util.py", line 3, in 
    import jionlp as jio
  File "/usr/local/lib/python3.10/site-packages/jionlp/__init__.py", line 104, in 
    from jionlp.gadget import *
  File "/usr/local/lib/python3.10/site-packages/jionlp/gadget/__init__.py", line 16, in 
    from .location_recognizer import LocationRecognizer
  File "/usr/local/lib/python3.10/site-packages/jionlp/gadget/location_recognizer.py", line 39, in 
    import jiojio
  File "/usr/local/lib/python3.10/site-packages/jiojio/__init__.py", line 27, in 
    logging = set_logger(level='INFO', log_dir_name='.jiojio/jiojio_logs')
  File "/usr/local/lib/python3.10/site-packages/jiojio/util/logger.py", line 78, in set_logger
    os.makedirs(filename_directory)
  File "/usr/local/lib/python3.10/os.py", line 225, in makedirs
    mkdir(name, mode)
FileExistsError: [Errno 17] File exists: '/root/.jiojio/jiojio_logs'
[2023-10-25 19:45:15 +0800] [73] [INFO] Worker exiting (pid: 73)

期望行为(Expectation)

期望在多进程的环境下，正确创建日志文件