- Docker19.03 or greater
- Docker-compose 1.26 or greater
- dags - Path with DAG file
- plugins - Path with custom operators(MapOperator and ReduceOperator)
- input_data/tweets.csv - Tweets Dataset - Top 20 most followed users in Twitter social platform
- docker-compose.yaml - Docker-compose file
- requirements.txt - File with additional dependencies which will be installed in docker
docker-compose up --build -d airflow-init
docker-compose up
sudo chmod -R 777 minio
sudo chmod -R 777 postgres-data
- go to
localhost:8080
login with (user
: airflow,password
: airflow) - go to admin > connections
- add new connection with the following parameters:
Coonn Id = local_minio
Conn Type = S3
Extra:
{
"aws_access_key_id":"minio-access-key",
"aws_secret_access_key": "minio-secret-key",
"host": "http://minio:9000"
}
- go to
localhost:8080
login with (user
: airflow,password
: airflow) - go to admin > connections
- add new connection with the following parameters:
Coonn Id = postrgre_sql
Conn Type = Postgres
Host: db
Schema: tweets
Login: postgres
Password: postgres
docker-compose up