This data engineering project automates the extraction of data from two sources, a Postgres database, and a CSV file, and loads it into another Postgres database. The goal is to create a pipeline that can be run daily, while maintaining idempotency of tasks and clear isolation between steps.
To set up and run the data engineering pipeline, follow these steps:
-
Clone this repository to your local machine.
-
Make sure you have Docker Compose installed.
-
Open a terminal and navigate to the project directory.
-
Run the following command to setup and start everything:
make startAnd afterwards, you can open another terminal and stop everything with:
make stop
It can take 1-2 minutes for Airflow to fully start. You can monitor the progress in the same terminal you ran make start.
Once the containers are up and running, open a web browser and go to http://localhost:8080/ to access the Apache Airflow web interface. If the web interface doesn't load, it means Airflow is still starting, if it loads, it means Airflow has fully started and we're ready to authenticate.
To authenticate, use:
- User: airflow
- Password: airflow
In the Airflow web interface, you will find two DAGs representing the data pipeline:
- data_extraction_and_local_storage: This DAG handles Step 1 of the challenge, which involves extracting data from the Postgres database and CSV file.
- data_loading_to_final_database: This DAG manages Step 2, which loads the extracted data to Postgres. Click on the "play" button for the DAG you want to execute.
A calendar view will appear. Select a date as a parameter for the execution of the DAG and click "Trigger" to start the pipeline.
After you click you'll be redirected back to the home page, where you'll be able to see the status of the running DAG.
In case one of your DAG's tasks fail, you can check the logs to help you better understand what happened.
In the main.ipynb file, you can run the tasks locally, and check the query result.