This project demonstrates how to set up an ETL (Extract, Transform, Load) pipeline using Airflow and Docker Compose.
-
Clone this repository:
git clone https://github.com/cindyangelira/airflow-docker.git
-
Navigate to the project directory:
cd your-repo
-
Build and start the Docker containers:
docker-compose up -d
-
Access the Airflow web interface:
Open your web browser and go to
http://localhost:8080
.
-
Define your ETL tasks in the
dags
directory. -
Start the Airflow scheduler:
docker-compose exec webserver airflow scheduler
-
Trigger the ETL pipeline:
docker-compose exec webserver airflow trigger_dag your_dag_id
-
Monitor the progress and logs in the Airflow web interface.