ranjanmanish / airflow-local

A Ubuntu (14.04) Vagrant Virtual Machine (VM) with Airflow, a data workflow management system from Airbnb

airflow-local

• Source code - Github
• Author - Gavin Noronha - [e-mail]

About

This project provides a Ubuntu (14.04) Vagrant Virtual Machine (VM) with Airflow, a data workflow management system from Airbnb.

There are Puppet scripts that automatically install the software when the VM is started.

Connect to the VM

1. To start the virtual machine(VM) type

   vagrant up
   

2. Connect to the VM

   vagrant ssh
   

Initialize Airflow

1. Setup the home directory

   export AIRFLOW_HOME=~/airflow
   

2. Initialize the sqlite database

   airflow initdb
   

3. Start the web server

   airflow webserver -p 8080
   

4. Open a web browser to the UI at http://192.168.33.10:8080

Run a task

1. List DAGS

   airflow list_dags
   

2. List tasks for example_bash_operator DAG

   airflow list_tasks example_bash_operator
   

3. List tasks for example_bash_operator in a tree view

   airflow list_tasks example_bash_operator -t
   

4. Run the runme_0 task on the example_bash_operator DAG today

   airflow run example_bash_operator runme_0 `date +%Y-%m-%d`
   

5. Backfill a DAG

   export START_DATE=$(date -d "-2 days" "+%Y-%m-%d")
   airflow backfill -s $START_DATE example_bash_operator
   

6. Clear the history of DAG runs

   airflow clear example_bash_operator
   

Add a new task

1. Go to the Airflow config directory

   cd ~/airflow
   

2. Set the airflow dags directory in airflow.cfg by change the line:

   dags_folder = /vagrant/airflow/dags
   

3. Restart the web server

   airflow webserver -p 8080
   

Documentation

1. Main documentation

   • https://pythonhosted.org/airflow/

2. Videos on Airflow

   • https://www.youtube.com/watch?v=dgaoqOZlvEA&feature=youtu.be
   • https://www.youtube.com/watch?v=dgaoqOZlvEA

3. Slides

   • http://www.slideshare.net/walterliu7/airflow-a-data-flow-engine
   • http://www.slideshare.net/Hadoop_Summit/airflow-an-open-source-platform-to-author-and-monitor-data-pipelines

4. Airflow reviews

   • http://bytepawn.com/airflow.html
   • https://www.pandastrike.com/posts/20150914-airflow

5. Airflow tips and tricks

   • https://medium.com/handy-tech/airflow-tips-tricks-and-pitfalls-9ba53fba14eb#.i2hu0syug
   • https://stlong0521.github.io/20161023%20-%20Airflow.html
   • https://databricks.com/blog/2016/12/08/integrating-apache-airflow-databricks-building-etl-pipelines-apache-spark.html
   • http://site.clairvoyantsoft.com/installing-and-configuring-apache-airflow/
   • https://gtoonstra.github.io/etl-with-airflow/principles.html
   • https://cwiki.apache.org/confluence/display/AIRFLOW/Common+Pitfalls

Disable logging

1. Change to the airflow directory

   cd /vagrant/airflow
   

2. Set airflow environment

   source set_airflow_env.sh
   

3. Run airflow without any logging messages

Setup airflow dags directory

1. Edit file ~/airflow/airflow.cfg

2. Set the following:

   dags_folder = /vagrant/airflow/dags
   load_examples = False
   

3. Start the scheduler by running the following

   airflow scheduler
   

Requirements

The following software is needed to get the software from github and run Vagrant to set up the Python development environment. The Git environment also provides an SSH client for Windows.

• Oracle VM VirtualBox
• Vagrant
• Git

How to copy the local files from laptop to Vagrant

• : By default, Vagrant will share your project directory (the directory with the Vagrantfile) to /vagrant Read more here: https://www.vagrantup.com/docs/synced-folders/