The goal of this project is to perform Data Analytics ensuring Data Quality throughout the entire Extraction, Transformation, and Load (ETL) process using various tools and technologies. Additionally, the goal of this project is to assist data engineers in their daily tasks by monitoring data flow and quality. Help us with that questions:
- Is any data missing from the source?
- Are the source data within the previously reported standards?
- Was there any error during the reading, transformation, and final loading process into the report?
- Was there a change in the script or business rule during data transformation?
- Was there a change in the business rule?
- Programming Language - Python and SQL
- Database - PostgreSQL
- Ingestion - Airbyte
- Data Quality - Great Expectations
- Orchestration - Airflow
- Data Visualization - MetaBase
- Environment - Docker
- Transactional Database (Source)
- Data Warehouse ( Final Destination )
- Airbyte: To simplify data ingestion, ensuring smooth integration between the source and the staging area, monitoring schema/column changes, and ensuring the first full load and the subsequent incremental ones using Airbyte's own resources, it's fantastic.
- We set clear expectations for our data and ensure its quality every step of the way. Here, every time the data is moved or transformed, a check is carried out to see if it is as expected according to the business rules. Data Quality is directly linked to how the business works too.
- Checking null values
- Checking if the values in the Quantidade column follow the business rules and expected data
- Checking if the values in the Nome_tipo column follow the business rules and expected data
- Efficiently and automatically orchestration all steps of the data workflow. From ingestion to transformation and delivery, Airflow robustly provided end-to-end data process management and monitoring.
- Create visualizations and actionable insights from the processed data.
