Welcome to the Airflow Projects repository. This repository contains various projects and workflows created using Apache Airflow. It serves as a learning and experimentation platform for implementing and managing ETL processes with Airflow.
- Getting Started
- Prerequisites
- Installation
- Usage
- Using Docker
- Project Structure
- Contributing
- License
This repository provides several example projects to help you understand how to use Apache Airflow for different data processing tasks. Follow the instructions below to set up your environment and run the projects.
- Python 3.7+
- Apache Airflow 2.0+
- Docker
-
Clone the repository:
git clone https://github.com/python-dev-10/airflow-projects.git cd airflow-projects
-
Set up a virtual environment:
python -m venv venv source venv/bin/activate # On Windows use \`venv\\Scripts\\activate\`
-
Install the required dependencies:
pip install -r requirements.txt
Alternatively, you can use Docker to set up and run Airflow. Follow these steps:
-
Build and run the Docker containers:
docker-compose --up d
-
Access the Airflow web interface:
Open your web browser and navigate to
http://localhost:8080
.
-
Set up your Airflow configuration:
Edit the
airflow.cfg
file to configure your Airflow instance according to your needs. -
Add your DAGs:
Place your DAG files in the
airflow/dags
directory. Airflow will automatically detect and load them. -
Run your DAGs:
Trigger your DAGs from the Airflow web interface or using the Airflow CLI.
airflow-projects/
├── .venv/
├── airflow/
│ ├── config/
│ ├── dags/
│ │ ├── financial_data.py
│ │ └── forecast_etl.py
│ ├── data/
│ ├── logs/
│ └── plugins/
│ └── email_template.py
├── docker-compose.yaml
├── Dockerfile
├── requirements.txt
├── .gitignore
└── README.md
.venv/
: Virtual environment directory.airflow/
: Main directory for Airflow configurations and components.config/
: Configuration files for Airflow.dags/
: Contains all the DAG files.data/
: Directory for data files.logs/
: Directory for log files.plugins/
: Custom plugins for Airflow.
docker-compose.yaml
: Docker Compose configuration for running Airflow in containers.Dockerfile
: Dockerfile for building Airflow container.requirements.txt
: List of Python dependencies.
DAG: forecast_etl
- Create Table: Creates a table in PostgreSQL if it doesn't exist.
- Transform Data: Transforms the extracted data to a format suitable for insertion into the database.
- Generate SQL: Generates the SQL insert statement for the transformed data.
- Insert Data: Inserts the transformed data into the PostgreSQL table.
- open_weather_api: Your OpenWeatherMap API key stored as an Airflow Variable.
- city_name: The city for which to fetch the weather data, stored as an Airflow Variable.
- airflow_postgres_conn: Connection ID for connecting to the PostgreSQL database. This should be configured in Airflow with the appropriate connection details.
To successfully run this DAG, ensure the following prerequisites are met:
- Apache Airflow is installed and configured.
- PostgreSQL database is set up and accessible from Airflow.
- Airflow variables `open_weather_api` and `open_weather_city` are configured.