This is a End-to-End Azure Data Engineering Project | Analysis on the entire ETL Pipeline - Azure Data Factory, Azure Lake Gen 2, Azure Databricks, Azure Synapse Analytics.
- Azure Data Factory-
![1 Data Factory Github Data Ingestion](https://private-user-images.githubusercontent.com/122162103/265088155-2b603073-752f-4b6d-921f-5c18e3566efe.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjQ3MzM1MjEsIm5iZiI6MTcyNDczMzIyMSwicGF0aCI6Ii8xMjIxNjIxMDMvMjY1MDg4MTU1LTJiNjAzMDczLTc1MmYtNGI2ZC05MjFmLTVjMThlMzU2NmVmZS5wbmc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjQwODI3JTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI0MDgyN1QwNDMzNDFaJlgtQW16LUV4cGlyZXM9MzAwJlgtQW16LVNpZ25hdHVyZT0zNTFhOTcxMjk4NWQ0NTVjYjk4ZDk3MmFlYTM2NjU2MjkwMGMwYTYzZTEwYmQ2ZmFmZWFlOTkxZjZmODA4ZjNjJlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCZhY3Rvcl9pZD0wJmtleV9pZD0wJnJlcG9faWQ9MCJ9.hCpGcFLFRwYOLDgOX-YLPmkjSNFPupGUj27SP9Sd6DY)
- Azure Data Lake Gen2 Raw Data Store-
![2 Data Lake Gen2- Raw Data Store](https://private-user-images.githubusercontent.com/122162103/265088333-5b8377fa-9271-4a7b-aa0b-b9bf9f051240.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjQ3MzM1MjEsIm5iZiI6MTcyNDczMzIyMSwicGF0aCI6Ii8xMjIxNjIxMDMvMjY1MDg4MzMzLTViODM3N2ZhLTkyNzEtNGE3Yi1hYTBiLWI5YmY5ZjA1MTI0MC5wbmc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjQwODI3JTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI0MDgyN1QwNDMzNDFaJlgtQW16LUV4cGlyZXM9MzAwJlgtQW16LVNpZ25hdHVyZT04N2NhMGM4ODBkOGYwYmVjZDdmYmIzMWYwYjFhNGRmYzdjNWM1MjdmZTI0OGY3MzE1MTBjMGQxNDEwYjM4YjFiJlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCZhY3Rvcl9pZD0wJmtleV9pZD0wJnJlcG9faWQ9MCJ9.XEg-maXDmROMqSqFEWEXHQW6qAzRIpZAxUim5HavOk8)
- Azure Data Bricks Transformation-
![3 Databricks Transformation](https://private-user-images.githubusercontent.com/122162103/265088429-d3505719-39b3-4e50-9bf6-2a43937427ca.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjQ3MzM1MjEsIm5iZiI6MTcyNDczMzIyMSwicGF0aCI6Ii8xMjIxNjIxMDMvMjY1MDg4NDI5LWQzNTA1NzE5LTM5YjMtNGU1MC05YmY2LTJhNDM5Mzc0MjdjYS5wbmc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjQwODI3JTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI0MDgyN1QwNDMzNDFaJlgtQW16LUV4cGlyZXM9MzAwJlgtQW16LVNpZ25hdHVyZT0yMGRkMDdjMjQ5MTBlYmYzNTk5NWNlYjM2NTI4YjU1ZTRmNzUyZTc1YjkwNDkwZDBiMDI0ZDQwNGZiNDkyY2VkJlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCZhY3Rvcl9pZD0wJmtleV9pZD0wJnJlcG9faWQ9MCJ9.8dzagpbKPqgkTC63YcBJe1WfmO_meddPWF9m6fGHbW8)
- Azure Data Lake Gen2 Transformed Data Store-
![4 Data Lake Gen2- Transformed Data Stored](https://private-user-images.githubusercontent.com/122162103/265088571-27e89c39-fe5b-441e-9a31-f578a994677b.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjQ3MzM1MjEsIm5iZiI6MTcyNDczMzIyMSwicGF0aCI6Ii8xMjIxNjIxMDMvMjY1MDg4NTcxLTI3ZTg5YzM5LWZlNWItNDQxZS05YTMxLWY1NzhhOTk0Njc3Yi5wbmc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjQwODI3JTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI0MDgyN1QwNDMzNDFaJlgtQW16LUV4cGlyZXM9MzAwJlgtQW16LVNpZ25hdHVyZT1kZDMwYjllOGVjNDliZDNmNjQ4YjIzNDgyNGQ0OTRlMmUxNWU3OGVkNTRkZGZlNTU5YjFkYzgwOWVlNTQ1NDI0JlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCZhY3Rvcl9pZD0wJmtleV9pZD0wJnJlcG9faWQ9MCJ9.uKY7jYf4jmSaLXIfQDHc4zH6jGNWeHHmjoxqMpwYwSw)
- Azure Synapse Analytics-
![5 Data Synapse Analytics using Queries](https://private-user-images.githubusercontent.com/122162103/265088672-7e37a003-2dce-48dd-a9ce-b2fbede1f31f.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjQ3MzM1MjEsIm5iZiI6MTcyNDczMzIyMSwicGF0aCI6Ii8xMjIxNjIxMDMvMjY1MDg4NjcyLTdlMzdhMDAzLTJkY2UtNDhkZC1hOWNlLWIyZmJlZGUxZjMxZi5wbmc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjQwODI3JTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI0MDgyN1QwNDMzNDFaJlgtQW16LUV4cGlyZXM9MzAwJlgtQW16LVNpZ25hdHVyZT1lZjlhNjhiYTk5ZTI1OTFhNzBjNDdlOGY4ZmQ1NTRlNGM5OGM0YTdjMzk4ODUwYTNiY2Q0Mjk5NDFhNzAyN2RhJlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCZhY3Rvcl9pZD0wJmtleV9pZD0wJnJlcG9faWQ9MCJ9.50AM9eIr95nnuDhcz1iJknTz1Hi1g9Nv8pToqsJngmA)