This repo shows what a minimal pipeline looks like in airflow. There are three versions,
- original.py shows the pipeline without any orchestration
- airflow_classic.py shows the pipeline as a classic airflow DAG
- airflow_taskflow.py shows the pipeline as a TaskFlow airflow DAG
- prefectified.py shows the pipeline as a Prefect 2 flow
Create a new virtual environment and install the dependencies,
make install
Now you can start a local instance of airflow,
make start
Open up http://0.0.0.0:8080 in your browser to access the UI.
The component services of airflow can be understood by looking at the official docker-compose.yaml. In a minimal setup you have,
- a database for metadata which is sqlite by default but typically postgres
- a webserver that serves the UI
- a scheduler that runs tasks using an executor which is typically
LocalExecutor
for local execution andCeleryExecutor
for distributed execution
In a more scalable setup using Celery you would add,
- one or more workers that executes tasks
- a redis instance to broker between scheduler and workers
- optionally a flower UI for monitoring Celery
Airflow also has extensive support for Kubernetes, including an official Helm chart.