dataengineering

There are 28 repositories under dataengineering topic.

• DataExpert-io / data-engineer-handbook

  This is a repo with links to everything you'd ever want to learn about data engineering

  apachesparkawesomebigdatadatadataengineeringsql
  9750
• OpenMetadata

  open-metadata / OpenMetadata

  OpenMetadata is a unified metadata platform for data discovery, data observability, and data governance powered by a central metadata repository, in-depth column level lineage, and seamless team collaboration.

  data-catalogdata-collaborationdata-contractsdata-discoverydata-governancedata-lineagedata-observabilitydata-profilingdata-qualitydata-quality-checksdata-sciencedata-validationdatacatalogdatadiscoverydataengineeringdataqualitydbtmetadatametadata-managementsnowflake
  Language:TypeScript 4751

• datafold / data-diff

  Compare tables within or across databases

  databasemysqlpostgresqlsnowflakerdbmstrinodata-engineeringdata-qualitydata-sciencedata-quality-monitoringdatabricks-sqldataengineeringdataqualityoracle-databasepostgressqldbtpythondatadata-diffing
  Language:Python 2910

• TobikoData / sqlmesh

  Efficient data transformation and modeling framework that is backwards compatible with dbt.

  dataopseltetlsqlpythondataengineeringtransformationdbt
  Language:Python 1473
• zingg

  zinggAI / zingg

  Scalable identity resolution, entity resolution, data mastering and deduplication using ML

  analyticsanalytics-engineeringdata-sciencedata-transformationdata-transformationsdataengineeringdatalakedataqualitydedupededuplicationentity-resolutionetlfuzzy-matchingfuzzymatchidentityidentity-resolutionmasterdatamlmodern-data-stackspark
  Language:Java 916
• automate-dv

  Datavault-UK / automate-dv

  A free to use dbt package for creating and loading Data Vault 2.0 compliant Data Warehouses (powered by dbt, an open source data engineering tool, registered trademark of dbt Labs)

  datavault20dbtdatawarehousingdatavaultetlsnowflakedataengineeringdatawarehousedatalakeeltdata-vaultsqlmetadata
  478

• grai-io / grai-core

  hacktoberfestdatadatalineagedbtopen-sourcepostgresqlpythonmssqlmysqlsnowflakedata-sciencedataengineeringfivetrandjangoparquetredshiftdata-lineage
  Language:Python 280
• snowpark-python-demos

  Snowflake-Labs / snowpark-python-demos

  This repository provides various demos/examples of using Snowpark for Python.

  pythonsnowparkdataengineeringdatasciencemachine-learning
  Language:Jupyter Notebook 250
• aws-ddk

  awslabs / aws-ddk

  An open source development framework to help you build data workflows and modern data architecture on AWS.

  awsdataengineeringdataopspython
  Language:TypeScript 248

• kevinheavey / modern-polars

  Code and data for the Modern Polars book

  data-analyticsdata-engineeringdata-sciencedataengineeringpandaspolarspython
  Language:Python 176
• Data-Engineering-Roadmap

  ErdemOzgen / Data-Engineering-Roadmap

  Roadmap for Data Engineering

  awesomeci-cdclouddata-sciencedataengineeringdevelopmentdevopsmachine-learningmlopsroadmapdatabasedatapipelinedatapreprocessingdatawarehousedeep-learninginterviewawesome-listawesome-resourcesguidelines
  Language:Java 167

• awslabs / aws-orbit-workbench

  A Data Platform built for AWS, powered by Kubernetes.

  kuberneteseksjupyterjupyterhubdata-analysisdatalakemachorbit-workbenchgpueks-clusterawsdataengineeringredshiftanalyticsworkbench
  Language:Python 128

• data-burst / data-engineering-roadmap

  data-engineeringdataengineeringroadmapdata-engineering-roadmap
  126

• ogbinar / DataEngineeringPilipinas

  Data Engineering Pilipinas is a community for data engineers, data analysts, data scientists, developers, AI / ML engineers, and users of closed and open source data tools and methods / techniques in the Philippines. Data Engineering Pilipinas is a PyData group.

  communitydatadatabasedataengineeringdatasciencedatasetsengineeringetlphilippinespythonsql
  Language:HTML 121

• kirilldikalin / myknowledge

  Все, о чем меня когда-либо спрашивали на собеседованиях, и другие полезные знания в кратком формате

  data-sciencepythondataengineeringsql
  96

• sparsh-ai / recohut

  Recohut - Learn data engineering, data science

  data-engineeringdataengineeringdatascience
  Language:MDX 93

• Eldar1205 / awesome-python-backend

  Index for online reading materials in order to learn Python and backend development/engineering concepts from scratch and develop a mastery sufficient for Senior/Principal Backend Engineers and Data Engineers

  backendbackendengineeringdataengineeringpython
  89

• josephmachado / beginner_de_project_stream

  Simple stream processing pipeline

  apache-flinkdataengineeringdatapipelinegraphanapostgresqlprometheus
  Language:Python 80

• mehd-io / pypi-duck-flow

  end-to-end data engineering project to get insights from PyPi using python, duckdb, MotherDuck & Evidence

  dataengineeringduckdbetlpython
  Language:Python 74

• TirendazAcademy / Awesome-Data-Science-Resources

  Resources about data science, machine learning, deep learning, data engineering, and SQL.

  aiartificial-intelligencedata-analysisdata-engineeringdata-sciencedataengineeringdatasciencedeep-learningdeeplearningmachine-learningmachinelearningmachinelearning-pythonsql
  70

• Finance-And-ML / US-Stock-Prediction-Using-ML-And-Spark

  Predict stock price based on financial news feeds

  stock-price-predictionsparkscalatableauwebscrapingdataengineering
  Language:Jupyter Notebook 59

• prodmodel / prodmodel

  Build, test, deploy, iterate - Dev and prod tool for data science pipelines

  build-toolbuild-automationbuild-systemdata-sciencedatadatascienceproductionproductivitydataengineeringdataeng
  Language:Python 59
• data-engineering-interviews

  kislerdm / data-engineering-interviews

  Data engineering interviews Q&A for data community by data community

  dataengineeringinterview-questionsopensourcepythonsqlkafkasparklinux
  Language:Python 58

• minhadona / data_engineer_interview_challenges

  Found a data engineering challenge or participated in a selection process ? Share with us!

  beginnerbeginner-friendlybeginner-projectbeginnerscode-challenge-practicecode-challengesdata-engineeringdata-pipelinedata-pipelinesdataengineeringfirst-timersgithubhacktoberfesthacktoberfest-2022hacktoberfest-acceptedhacktoberfest2022help-wantedinterview-practicemarkdown-onlygithub-events
  Language:Python 57

• abhishek-ch / data-machinelearning-the-boring-way

  Build & Learn Data Engineering,Machine Learning over Kubernetes. No Shortcut approach.

  data-infrastructuredataengineeringdatasciencekubernetesmachine-learningmlops
  Language:Python 53

• sbalnojan / run-a-data-team

  A guide for leading a data (engineering) team

  analyticsawesome-listdatadataengineeringdatascience
  53

• noahgift / data-engineering-and-dataops

  Duke MIDS: Data Engineering and DataOps Course

  dukeclouddataopsdataengineeringdatadata-sciencesoftware-engineeringmlopsbookcourse
  Language:HTML 52

• franloza / coches-net-dashboard

  Sample project that use Dagster, dbt, DuckDB and Dash to visualize car and motorcycle Spanish market

  dagsterdatadataengineeringduckdbeltpipelinescrapingvisualization
  Language:Python 50

• olist / work-at-olist-data

  Apply for a job at Olist's Data Team: https://olist.gupy.io/

  pythonpandassqldatasciencedataanalyticsdataengineeringmachinelearningrjuliadataset
  50
• apache-spark-docker

  Wittline / apache-spark-docker

  Dockerizing an Apache Spark Standalone Cluster

  pysparkapache-sparkdocker-composehive-metastorehivehdfshuedataengineeringdataengineerhadoop-clusterhadoop-dockerdocker
  Language:VBA 41

• CynthiaKoopman / Forecasting-Solar-Energy

  Forecasting Solar Power: Analysis of using a LSTM Neural Network

  deeplearningmachinelearningforecastinglstmneuralnetworksdata-sciencedataengineeringsolarpowersolar-energyrenewable-energyelectricity-gridforecasting-solar-power
  Language:Jupyter Notebook 40

• josephmachado / socialetl

  Project for "Data pipeline design patterns" blog.

  dataengineeringdesign-patternsetl-pipelinemakefilepythonsocial-media-datasqllite3reddit
  Language:Python 40

• sarahmk125 / airflow-docker-metrics

  dataengineeringairflow-metricsstatsd-formatgrafanaairflow-containersstatsd-metricsdataorchestrationworkflow-orchestration
  Language:Python 39

• danielsaban / data-scraping-sofascore

  Data Engineering/Scraping Project. Creating a detailed Sports Relational Database for the Top European Soccer Leagues.

  dataengineeringdataminingwebscrapingrelational-databasesparseretl-pipelinerest-api
  Language:Python 36

• aakashnand / trino-ranger-demo

  Tutorial on how to setup Trino and Apache Ranger using docker

  dockerrangertrinodataengineeringdatagovernancetutorial
  Language:Shell 35

• Spratiher9 / SparkDataset

  Instant search for and access to many datasets in Pyspark.

  standarddatasetspysparkpython3pythonsparkeasy-access-applicationinstantsearchrbenchmark-frameworkbenchmarkquickstartdatadataengineeringdatasetdata-miningdata-analysis
  Language:Jupyter Notebook 34