This repo contains an Astronomer project with multiple example DAGs showing how to pass data between your Airflow tasks. A guide discussing the DAGs and concepts in depth can be found here.
This tutorial covers two methods for passing data between tasks:
- Using XCom
- Using Intermediate Data Storage
There are example DAGs for each. The XCom example also includes a DAG using the TaskFlow API, which is a new feature released with Airflow 2.0.
The easiest way to run these example DAGs is to use the Astronomer CLI to get an Airflow instance up and running locally:
- Install the Astronomer CLI
- Clone this repo somewhere locally and navigate to it in your terminal
- Initialize an Astronomer project by running
astro dev init
- Start Airflow locally by running
astro dev start
- Navigate to localhost:8080 in your browser and you should see the tutorial DAGs there