To get started with the code examples, start Airflow with Docker Compose with the following command:
docker-compose up -dThe webserver initializes a few things, so wait for a few seconds, and you should be able to access the Airflow webserver at http://localhost:8080.
To stop running the examples, run the following command:
docker-compose down -vLet's build a basic pipeline to fetch information about rocket launches, save them locally and then notify you when the task is complete.
- Use The Space Devs API, with the url, https://ll.thespacedevs.com/2.0.0/launch/upcoming/ will help you get all the data you need.
- Within the Docker container save the data to the postgres database or to a
tmpfile. You could save the images locally or to the database. - Send a basic message in the form of a bash command to notify you about the end of the pipeline.
If you have the time on your own try doing the following:
- Save the data to an S3 bucket
- Send notifications using email or slack