himewel / kind-airflow

Apache Airflow helm chart environment running on local kubernetes (kind) with docker abstraction

Kind Airflow

This repo presents a set of tools to start with Apache Airflow helm chart.

Repo structure

├── dags/               <- airflow python dag files
├── include/            <- python imports to dag files
├── plugins/            <- airflow plugins
├── tools/              <- bash scripts and Dockerfile to manipulate k8s
├── Dockerfile          <- docker container to release the dag files
├── packages.txt        <- apt packages to be installed in the docker build
├── requirements.txt    <- python packages to be installed in the docker build
└── settings.yaml       <- helm settings of airflow helm chart

How to start

• Check if your environment has docker installed and install it if not;
• To setup the k8s cluster and Apache Airflow helm chart run the following commands:

# build the docker image to manage the cluster
make build
# create a cluster and install the helm chart
make start \
    CLUSTER=<YOUR CLUSTER NAME> \
    RELEASE=<YOUR K8S NAMESPACE>

• Check the webserver status by forward the UI from the k8s pod:

make forward-webserver \
    PORT=<PORT TO FORWARD THE WEB UI> \
    RELEASE=<YOUR K8S NAMESPACE>

• The same can be done with flower:

make forward-flower \
    PORT=<PORT TO FORWARD THE WEB UI> \
    RELEASE=<YOUR K8S NAMESPACE>

• To delete the cluster and remove the k8s namespace:

make stop \
    RELEASE=<YOUR K8S NAMESPACE>

• To release a new version of the dag files, build the Dockerfile and load it to kind:

docker build . --tag <YOUR IMAGE NAME>:<YOUR IMAGE TAG>
make start \
    CLUSTER=<YOUR CLUSTER NAME> \
    DOCKERIMAGE=<YOUR IMAGE NAME>:<YOUR IMAGE TAG>
    RELEASE=<YOUR K8S NAMESPACE>