Python dependency management
gsemet opened this issue · comments
Let's imagine we have DAGs that does some math or connect to a database. This dags will probably use a python dependency, that should be fectched and installed on the python environment of the Dags.
If the scheduler + workers + web ui does not have these dependencies, they will crash (I think the scheduler can handle that).
My PR #16 uses git-sync to automatically synchronize the Dags with a git repository. But it will work until a new python dependency is needed
So there is two solutions:
- force users to build there own docker image, and hope they do rolling update properly to deploy on their workers.
- allow installation from the workers (from a requirements.txt defined in the Dag folders) in entrypoint.sh.