mumoshu / kube-airflow

A docker image and kubernetes config files to run Airflow on Kubernetes

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Python dependency management

gsemet opened this issue · comments

Let's imagine we have DAGs that does some math or connect to a database. This dags will probably use a python dependency, that should be fectched and installed on the python environment of the Dags.
If the scheduler + workers + web ui does not have these dependencies, they will crash (I think the scheduler can handle that).

My PR #16 uses git-sync to automatically synchronize the Dags with a git repository. But it will work until a new python dependency is needed

So there is two solutions:

  • force users to build there own docker image, and hope they do rolling update properly to deploy on their workers.
  • allow installation from the workers (from a requirements.txt defined in the Dag folders) in entrypoint.sh.