gunicorn duplicating scheduler jobs

[ethanopp]

Have a process in my flask app that refreshes data in a db every hour, works fine in Dev, but once I move to prod gunicorn, on startup the jobs get added multiple times... Did some research and came across this article, but not sure how I can pass the --preload flag to this through docker compose:
https://stackoverflow.com/questions/16053364/make-sure-only-one-worker-launches-the-apscheduler-event-in-a-pyramid-web-app-ru

Python code:

# Set cron job to pull data every hour
with app.app_context():
    try:
        scheduler = BackgroundScheduler()
        scheduler.add_job(func=refresh_database, trigger="cron", hour='*')
        dash_app.server.logger.info('Starting cron jobs')
        scheduler.start()
    except BaseException as e:
        dash_app.server.logger.error('Error starting cron jobs: {}'.format(e))

if __name__ == '__main__':
    dash_app.run_server(host='0.0.0.0', debug=False, port=80, ssl_context=('/keys/cert.crt', '/keys/certkey'))

Docker Compose:

  fitly:
    build:
      context: ./dockercontrol-master
      dockerfile: fitly-dockerfile
    container_name: fitly
    restart: always
    depends_on:
      - mariadb
      - letsencrypt
    ports:
      - "8050:80"
    environment:
      - MODULE_NAME=index
      - VARIABLE_NAME=app
      - TZ=America/New_York
      - PUID=1001
      - PGID=100
    volumes:
      - /share/CACHEDEV2_DATA/Container/Fitly/config.ini:/app/config.ini
      - /share/CACHEDEV2_DATA/Container/Fitly/log.log:/app/log.log
      - /share/CACHEDEV2_DATA/Container/LetsEncrypt/keys:/app/keys

Also...If there is a better approach then what I am trying to do above, please share!

[Tim McFadden]

Did you actually use preload=True in your config file? That would not be correct. The config file setting is named preload_app.

I'd give it another try with preload_app=True in the config file if so.

[ethanopp]

@mtik00 updated to the following, still duplicating the cron job 'start':

  fitly:
    build:
      context: ./dockercontrol-master
      dockerfile: fitly-dockerfile
    container_name: fitly
    restart: always
    depends_on:
      - mariadb
      - letsencrypt
    ports:
      - "8050:80"
    environment:
      - MODULE_NAME=index
      - VARIABLE_NAME=app
      - PRELOAD_APP=True
      - TZ=America/New_York
      - PUID=1001
      - PGID=100
    volumes:
      - /share/CACHEDEV2_DATA/Container/Fitly/config.ini:/app/config.ini
      - /share/CACHEDEV2_DATA/Container/Fitly/log.log:/app/log.log
      - /share/CACHEDEV2_DATA/Container/LetsEncrypt/keys:/app/keys

Is this where you were referring to update it?

Update I manually added preload_app=True to gunicorn_conf.py and it seems to be working... Is there a way to automatically do this though (env variable?) so every time the image is refreshed it gets set appropriately?

[ethanopp]

Alright I got this working with the following docker-compose. I went into the image and copy and pasted the gunicorn_conf.py code into a mounted file (named gunicorn_conf.py) and added the preload_app=True to that mounted file.

Then in the compose volumes I am overwriting the default images gunicorn_conf.py with the mounted version... The only issue I could foresee here is that if there is something in the images default gunicorn_conf.py that gets updated, it won't automatically get pulled through...

@mtik00 thanks for the tip with preload_app=True, you see any issues with this approach?

  fitly:
    build:
      context: ./dockercontrol-master
      dockerfile: fitly-dockerfile
    container_name: fitly
    restart: always
    depends_on:
      - mariadb
      - letsencrypt
    ports:
      - "8050:80"
    environment:
      - MODULE_NAME=index
      - VARIABLE_NAME=app
      - TZ=America/New_York
      - PUID=1001
      - PGID=100
    volumes:
      - /share/CACHEDEV2_DATA/Container/Fitly/gunicorn_conf.py:/gunicorn_conf.py
      - /share/CACHEDEV2_DATA/Container/Fitly/config.ini:/app/config.ini
      - /share/CACHEDEV2_DATA/Container/Fitly/log.log:/app/log.log
      - /share/CACHEDEV2_DATA/Container/LetsEncrypt/keys:/app/keys

[Tim McFadden]

Looks fine to me (although I'm still learning 😉).

Maybe you can define GUNICORN_CMD_ARGS='--preload=True' env var instead of using the config file? 🤷‍♂️

The documentation for gunicorn isn't super clear on what variables get evaluated when.
http://docs.gunicorn.org/en/stable/settings.html#settings

You don't have to overwrite the default gunicorn config, but the result will be the same either way.

https://github.com/tiangolo/meinheld-gunicorn-flask-docker/blob/master/README.md#gunicorn_conf

I'd probably just copy it to /app/gunicorn_conf.py, but again, it will be doing the same thing: fully overwriting the default one.

[ethanopp]

Thanks @mtik00! Got it working with the following (didn't need the =True portion):

  fitly:
    build:
      context: ./dockercontrol-master
      dockerfile: fitly-dockerfile
    container_name: fitly
    restart: always
    depends_on:
      - mariadb
      - letsencrypt
    ports:
      - "8050:80"
    environment:
      - MODULE_NAME=index
      - VARIABLE_NAME=app
      - TZ=America/New_York
      - GUNICORN_CMD_ARGS='--preload'
      - PUID=1001
      - PGID=100
    volumes:
      - /share/CACHEDEV2_DATA/Container/Fitly/config.ini:/app/config.ini
      - /share/CACHEDEV2_DATA/Container/Fitly/log.log:/app/log.log
      - /share/CACHEDEV2_DATA/Container/LetsEncrypt/keys:/app/keys

[Sebastián Ramírez]

Thanks for the help here @mtik00 ! 👏 🙇

Thanks for reporting back and closing the issue @ethanopp 👍