data-engineer
There are 42 repositories under data-engineer topic.
- data-engineering-wiki
data-engineering-community / data-engineering-wiki
The best place to learn data engineering. Built and maintained by the data engineering community.
Language:CSS 1042 wwwlike / vlife
企业级低代码快速开发平台，包含页面可视化配置、自定义表单、自定义报表、权限管理脚手架应用、前后端代码自动生成；主要特点是低代码开发，可实现复杂CRUD功能仅编写数据模型就能完成前后端开发
Language:TypeScript 646- versatile-data-kit
vmware / versatile-data-kit
One framework to develop, deploy and operate data workflows with Python and SQL.
Language:Python 411 unnati-xyz / scalable-data-science-platform
Content for architecting a data science platform for products using Luigi, Spark & Flask.
Language:Jupyter Notebook 162digitalghost-dev / premier-league
A Data Engineering project. Repository for backend infrastructure and Streamlit app files for a Premier League Dashboard.
Language:Python 158- Language:Jupyter Notebook 154
kelvins / awesome-dataops
:sunglasses: A curated list of awesome DataOps tools
Language:Python 121thedataengineeringbook / thedataengineeringbook
The Data Engineering Book - หนังสือวิศวกรรมข้อมูล ของคนไทย เพื่อคนไทย
Language:JavaScript 104kroudir / Data-Engineer-Nanodegree-Projects-Udacity
Projects done in the Data Engineer Nanodegree Program by Udacity.com
Language:Jupyter Notebook 77Jayvardhan-Reddy / Azure-Certification-DP-200
Road to Azure Data Engineer Part-I: DP-200 - Implementing an Azure Data Solution
Thomas-George-T / Thomas-George-T
Readme for my :octocat: Profile
deordie / deordie-meetups
DE or DIE meetup made by data engineers for data engineers. Currently in Russian only.
Language:Jupyter Notebook 53Keep-Current / web-miner
Crawls sites, to find new content and scrap it
Language:Python 33Data Engineering Digest
Language:SCSS 25nareshk1290 / dataquest-DE
Data Quest - Data Engineer Learning and Projects
Language:Jupyter Notebook 23sblack4 / google-data-engineering-coursera
For the Coursera specialization https://www.coursera.org/specializations/gcp-data-machine-learning
Road to Azure Data Engineer Part-II: DP-201 - Designing an Azure Data Solution
- Language:Python 17
nmolivo / dataquest_eng
Here's how to get DataQuest's Data Engineering Track missions' content to work on your localhost. Using data from my Valenbisi ARIMA modeling project, I document my steps using PostgreSQL, Postico, and the Command Line to get our DataQuest exercises running out of a Jupyter Notebook.
Language:Jupyter Notebook 16ortizfram / datacamp-Data-Engineer-with-Python-course
datacamp Data Engineer with Python course. 73 hours/ 19 Courses /2 Skill Assessments
Language:Python 16tspannhw / dws2017sydney
Code, Examples, Templates and Scripts for DataWorksSummit 2017 Sydney Talk
Language:Python 16camposvinicius / aws-etl
This is an ETL application on AWS with general open sales and customer data that you can find here: https://github.com/camposvinicius/data/blob/main/AdventureWorks.zip, it's a zipped file with some .csvs inside that we will apply transformations.
Language:Smarty 15san089 / data-engineer-roadmap
Learning from multiple companies in Silicon Valley. Netflix, Facebook, Google, Startups
tomcis / gcp-pro-de-certification
Google Cloud Platform Professional Data Engineer Certification resources ☁️☁️☁️
This is a pipeline of an ETL application in GCP with open airport code data, which you can find here: https://datahub.io/core/airport-codes/r/airport-codes_zip.zip, it's about a zipped .json, which let's apply transforms.
Language:Smarty 14HuemulSolutions / huemul-bigdatagovernance
Huemul BigDataGovernance, es una framework que trabaja sobre Spark, Hive y HDFS. Permite la implementación de una estrategia corporativa de dato único, basada en buenas prácticas de Gobierno de Datos. Permite implementar tablas con control de Primary Key y Foreing Key al insertar y actualizar datos utilizando la librería, Validación de nulos, largos de textos, máximos/mínimos de números y fechas, valores únicos y valores por default. También permite clasificar los campos en aplicabilidad de derechos ARCO para facilitar la implementación de leyes de protección de datos tipo GDPR, identificar los niveles de seguridad y si se está aplicando algún tipo de encriptación. Adicionalmente permite agregar reglas de validación más complejas sobre la misma tabla.
Language:Scala 11Wraps the DB by opening a REST API for storing and retrieving documents info & recommendations
Language:Python 11ZhiruiFeng / CarsMemory
A data engineering platform for maintaining a data ecosystem to support self-driving cars research.
Language:Python 10Jzbonner / dataengineering-db
Information relating to topics on Data Engineering, Data Infrastructure, Data Storing, Data Warehouses and Business Analysis. For those interested in both conceptual theory and use case examples for database design and development.
- Data-Engineering-Project-with-HDFS-and-Kafka
AhmetFurkanDEMIR / Data-Engineering-Project-with-HDFS-and-Kafka
Data Engineering Project with Hadoop HDFS and Kafka
Language:Python 8 DeleLinus / HFR-Data-Warehousing
End-to-end data engineering processes for the NIGERIA Health Facility Registry (HFR). The project leveraged Selenium, Pandas, PySpark, PostgreSQL and Airflow
Language:Python 7tuanx18 / data-engineer-portfolio
This is a repository to demonstrate my details, skills, projects and to keep track of my progression in Data Analytics and Data Science topics.
yashksaini-coder / Python-for-Data-Engineering
Data Engineering 🛠️ is like the backbone of data processing 📊, managing data pipelines 🚀, warehouses 🏢, and lakes 🌊. It's the bridge 🌉 between raw data and actionable insights, powering businesses 🚀 with efficient data management and analytics 📈.
Language:Jupyter Notebook 7BayoAdejare / lightning-containers
Docker powered starter for geospatial analysis of lightning atmospheric data.
Language:Jupyter Notebook 6
