This is a small example of the workflow built with Apache Airflow.
You can find slides here and watch the talk here
The goal is to set up a data pipline to get a fresh portion of Stack Overflow questions with tag pandas to our mailbox daily.
A small python script could do the job, but for the learning purposes we choose to overengineer it.
By writing this workflow we will learn the main concepts of Apache Airflow, such as:
- Operators
- DAG
- Tasks
- Hooks
- Variables
- Connections
- XComs
Happy learning 🤓
📝 Apache Airflow Documentation
📝 Apache Airflow Tutorial for Data Pipelines
📝 Apache Airflow for the confused
📝 Airflow: Tutorial and Beginners Guide
📰 Managing Dependencies in Apache Airflow
📝 Getting Started with Airflow Using Docker
🎧 Putting Airflow Into Production
📝 How to configure SMTP server for apache airflow
If you have any questions or would like to get in touch with me, please drop me a message to hello@varya.io