data-engineering-pipeline
There are 56 repositories under data-engineering-pipeline topic.
- Udacity-Data-Engineering-Projects
san089 / Udacity-Data-Engineering-Projects
Few projects related to Data Engineering including Data Modeling, Infrastructure setup on cloud, Data Warehousing and Data Lake development.
Language:Python 1402 - goodreads_etl_pipeline
An end-to-end GoodReads Data Pipeline for Building Data Lake, Data Warehouse and Analytics Platform.
Language:Python 1254 - versatile-data-kit
vmware / versatile-data-kit
One framework to develop, deploy and operate data workflows with Python and SQL.
Language:Python 417 alanchn31 / Movalytics-Data-Warehouse
Data pipeline performing ETL to AWS Redshift using Spark, orchestrated with Apache Airflow
Language:Python 123anna-geller / dataflow-ops
Project demonstrating how to automate Prefect 2.0 deployments to AWS ECS Fargate
Language:Python 109Code examples showing flow deployment to various types of infrastructure
Language:Python 97immu0001 / Udacity-Data-Engineer-nanodegree
Classwork projects and home works done through Udacity data engineering nano degree
Language:Jupyter Notebook 72- xontrib-pipeliner
anki-code / xontrib-pipeliner
Let your pipe lines flow thru the Python code in xonsh.
Language:Python 56 Deploy a Prefect flow to serverless AWS Lambda function
Language:Python 35mikeroyal / Apache-Spark-Guide
Apache Spark Guide
Language:Python 28InosRahul / f1-data-pipeline
F1 Data Pipeline
Language:Python 23VeraZab / nyc-stats
Analysis of 311 Service Requests for the City of NYC (from 2010 to 2023) Tech: Prefect cloud, dbt core, BigQuery, Compute Engine, CloudRun, Artifact Registry, Terraform, Docker
Language:Python 20- Language:Go 16
sanjeevai / disaster-response-pipeline
ETL pipeline combined with supervised learning and grid search to classify text messages sent during a disaster event
Language:Python 16kishlayjeet / Stock-Market-Real-Time-Data-Pipeline-with-Apache-Kafka-and-Cassandra
A end-to-end real-time stock market data pipeline with Python, AWS EC2, Apache Kafka, and Cassandra Data is processed on AWS EC2 with Apache Kafka and stored in a local Cassandra database.
Language:Python 15Learning from multiple companies in Silicon Valley. Netflix, Facebook, Google, Startups
brunocampos01 / predicting-retail-churn-with-azure-ml-studio
Challenge to job: Data Scientist
azureazure-pipelinesazure-machine-learning-studiomachine-learning-studiosoftplanchallengepython3pandasmachine-learningdeploy-machine-learningapi-restdata-engineeringdata-engineering-pipelinepowerbiazure-machine-learning-servicescheat-sheetscheat-sheet-machine-learningdata-scientistpythondata-scienceLanguage:Python 14dylanzenner / business_closures_de_pipeline
Data Engineering pipeline hosted entirely in the AWS ecosystem utilizing DocumentDB as the database
Language:Python 14Alero-Awani / Batch-data-engineering-project
A batch Data Pipeline that retrieves data from a user purchase table and a movie review table and is transformed to form a user behaviour metric table.
Language:HCL 13ketgo / marshmallow-pyspark
Marshmallow serializer integration with pyspark
Language:Python 12- Data-Engineering-Project-with-HDFS-and-Kafka
AhmetFurkanDEMIR / Data-Engineering-Project-with-HDFS-and-Kafka
Data Engineering Project with Hadoop HDFS and Kafka
Language:Python 10 antimoz-om / Antimoz
A data engineering pipeline for digital marketers.
Language:Shell 10An end-to-end Twitter Data Pipeline that extracts data from Twitter and loads it into AWS S3.
Language:Python 10koksang / social-media-analysis
Social Media Analysis, scalable solution, flexible deployment that analyses social media contents
Language:Jupyter Notebook 10datarootsio / notion-dbs-data-quality
Using Great Expectations and Notion's API, this repo aims to provide data quality for our databases in Notion.
Language:Python 9yashksaini-coder / Python-for-Data-Engineering
Data Engineering 🛠️ is like the backbone of data processing 📊, managing data pipelines 🚀, warehouses 🏢, and lakes 🌊. It's the bridge 🌉 between raw data and actionable insights, powering businesses 🚀 with efficient data management and analytics 📈.
Language:Jupyter Notebook 9AlphanAksoyoglu / tweeter-etl-pipeline
A streaming ETL pipeline for Realtime Tweet Collection, Analysis and Reporting
Language:Python 8Project demonstrating how to automate Prefect 2.0 deployments to AWS EKS
Language:Python 8benedekrozemberczki / AV_Ultimate_Student_Hunt
Solution for the Ultimate Student Hunt Challenge (1st place).
Language:R 8longNguyen010203 / Youtube-ETL-Pipeline
💜🌈📊 A Data Engineering Project that implements an ETL data pipeline using Dagster, Apache Spark, Streamlit, MinIO, Metabase, Dbt, Polars, Docker. Data from kaggle and youtube-api 🌺
Language:Jupyter Notebook 8siddharth271101 / Covid-19-and-Aviation-Industry
The goal of this project is to analyse the impact of Covid-19 on the Aviation industry through data engineering processes using technologies such as Apache Airflow, Apache Spark, Tableau and couple of AWS services
Language:Python 8Get started with Prefect by scheduling your Prefect flows with GitHub Actions
Language:Python 7DeleLinus / HFR-Data-Warehousing
End-to-end data engineering processes for the NIGERIA Health Facility Registry (HFR). The project leveraged Selenium, Pandas, PySpark, PostgreSQL and Airflow
Language:Python 7minhky2185 / healthcare_data_pipeline
An end-to-end data pipeline for building Data Lake and supporting report using Apache Spark.
Language:Python 7Yan-Luo-AU / Data_Engineer_Project_ETL_BI
This is an ETL project - extracting data from an ecommerce transactional database on RDS, transforming the data using AWS glue job, and loading it to a Redshift data warehouse, and connected it to Tableau for BI
Language:Python 7BayoAdejare / lightning-containers
Docker powered starter for geospatial analysis of lightning atmospheric data.
Language:Jupyter Notebook 6
