Airflow demo: Using the DockerOperator with Docker Compose

Most of tutorials just explains how to use the Airflow DockerOperator using the bare metal installation; and here we will use it with Airflow on top of Docker Compose.

Authors

First create a container with the webservice and create the airflow user, as described in the official docs:

$ docker-compose up airflow-init

or execute the following script below:

$ bash bin/initial_setup.sh

With this initial setup made, start the webservice and other components via docker-compose,

$ docker build -f dags/docker_job/Dockerfile -t docker_image_task . && \
docker-compose up -d

or execute the following script below that will do the same thing:

$ bash bin/start.sh

Finally when you're done with your experiment, stop all containers running the following command:

$  bash bin/stop.sh

Useful links that I used during this endeavor in importance order.