data-lake

There are 19 repositories under data-lake topic.

• lakeFS

  treeverse / lakeFS

  lakeFS - Data version control for your data lake | Git for data

  data-engineeringdata-versioninggoobject-storagedata-lakeaws-s3data-qualityazure-blob-storagegoogle-cloud-storagegolanggit-for-datalakefsapache-sparkapache-sparksqlhadoop-filesystemdatalakedatalakesdata-version-controlazure-storage
  Language:Go 4876

• dlt-hub / dlt

  data load tool (dlt) is an open source Python library that makes data loading easy 🛠️

  datapythondata-engineeringdata-lakedata-loadingdata-warehouseeltextractloadtransform
  Language:Python 4148

• apache / kyuubi

  Apache Kyuubi is a distributed and multi-tenant gateway to provide serverless SQL on data warehouses and lakehouses.

  sparkhivesqlthriftjdbcspark-sqldata-lakehadoopkuberneteshacktoberfest
  Language:Scala 2241
• Udacity-Data-Engineering-Projects

  san089 / Udacity-Data-Engineering-Projects

  Few projects related to Data Engineering including Data Modeling, Infrastructure setup on cloud, Data Warehousing and Data Lake development.

  datadata-engineeringdata-engineering-pipelineetl-pipelinecassandra-databasepostgresql-databasedata-modelingdata-warehousedata-lakeairflowairflow-operatorsclustercassandrainfrastructurepostgresawsaws-ec2aws-sdkaws-s3cloudformation
  Language:Python 1729

• bytedance / bitsail

  BitSail is a distributed high-performance data integration engine which supports batch, streaming and incremental scenarios. BitSail is widely used to synchronize hundreds of trillions of data every day.

  flinkbig-datadata-integrationdata-lakedata-pipelinedata-synchronizationhigh-performancereal-time
  Language:Java 1655
• goodreads_etl_pipeline

  san089 / goodreads_etl_pipeline

  An end-to-end GoodReads Data Pipeline for Building Data Lake, Data Warehouse and Analytics Platform.

  etl-pipelineetl-frameworksparkapache-sparkapache-airflowairflowredshiftemr-clusterlivys3warehousedata-lakeschedulerdata-migrationdata-engineeringdata-engineering-pipelinepythongoodreads-data-pipelineairflow-dagetl-job
  Language:Python 1416

• Teradata / kylo

  Kylo is a data lake management software platform and framework for enabling scalable enterprise-class data lakes on big data technologies such as Teradata, Apache Spark and/or Hadoop. Kylo is licensed under Apache 2.0. Contributed by Teradata Inc.

  sparknifikylodata-laketeradatahadoop
  Language:Java 1111

• alanchn31 / Data-Engineering-Projects

  Personal Data Engineering Projects

  airflowaws-redshiftcassandradata-engineeringdata-engineering-nanodegreedata-lakedata-modelingdata-warehouseingest-datamongodbpostgresscrapysparkstar-schema
  Language:Jupyter Notebook 946

• Canner / vulcan-sql

  Data API Framework for AI Agents and Data Apps

  api-builderdata-lakedata-warehousedatabasesqlanalyticsreportingspreadsheetbigqueryduckdbpostgresqlsnowflakerestful-apitypescriptclickhousevulcan-sqlvulcansqlksqldbaiai-agent
  Language:TypeScript 676

• pixelsdb / pixels

  An efficient storage and compute engine for both on-prem and cloud-native data analytics.

  cloud-databasedata-lakedata-warehousedatabaseolapcolumn-store
  Language:Java 610

• lakekeeper / lakekeeper

  Lakekeeper is an Apache-Licensed, secure, fast and easy to use Apache Iceberg REST Catalog written in Rust.

  catalogdata-lakeiceberglakehouselakehouse-governanceopen-lakehouserust
  Language:Rust 549

• uber / marmaray

  Generic Data Ingestion & Dispersal Library for Hadoop

  hadoopdata-lakeingest-dataavro-schemaschema-formatspark
  Language:Java 478
• data-lakes-on-aws

  aws-solutions-library-samples / data-lakes-on-aws

  Enterprise-grade, production-hardened, serverless data lake on AWS

  serverlessframeworkdata-lakeanalyticsawsetldata-engineeringlake-formationiacbest-practices
  Language:Python 471

• kaiwaehner / hivemq-mqtt-tensorflow-kafka-realtime-iot-machine-learning-training-inference

  Real Time Big Data / IoT Machine Learning (Model Training and Inference) with HiveMQ (MQTT), TensorFlow IO and Apache Kafka - no additional data store like S3, HDFS or Spark required

  kafkahivemqmqttkafka-streamskafka-connectksqltensorflowgrpcjavapythontiered-storagedata-lakeconfluentksqldbtensorflow-ioterraformgcpkubernetescloudmongodb
  Language:Jupyter Notebook 417

• cuebook / cuelake

  Use SQL to build ELT pipelines on a data lakehouse.

  apache-icebergdeltalakehousedatalakedata-lakeeltetldata-engineeringdata-integrationdata-ingestionapache-sparkspark-sqlupsertincremental-updatesdata-transferpipelinesdata-pipelinezeppelin-notebooksql
  Language:JavaScript 288
• wren-engine

  Canner / wren-engine

  🤖 The Semantic Engine for Model Context Protocol(MCP) Clients and AI Agents 🔥

  business-intelligencedatadata-analysisdata-analyticsdata-lakedata-warehousesqlsemanticsemantic-layerllmhacktoberfestagentagentic-aiaimcpmcp-server
  Language:Java 272
• amazon-s3-find-and-forget

  awslabs / amazon-s3-find-and-forget

  Amazon S3 Find and Forget is a solution to handle data erasure requests from data lakes stored on Amazon S3, for example, pursuant to the European General Data Protection Regulation (GDPR)

  data-lakeamazon-s3s3gdprawsparquetdata-erasureright-to-be-forgottenccpabig-dataprivacydata
  Language:Python 242

• maxi-k / btrblocks

  BtrBlocks: Efficient Columnar Compression for Data Lakes (SIGMOD 2023 Paper)

  compressiondata-lakedatabasesresearch
  Language:C++ 238

• Azure / usql

  U-SQL Examples and Issue Tracking

  big-datau-sqlazuredata-lake
  Language:C# 234

• garystafford / tickit-data-lake-demo

  Resources for video demonstrations and blog posts related to DataOps on AWS

  airflowawsdata-lakedataopsdevopsredshift
  Language:Python 174

• Azure / AzureDataLake

  Samples and Docs for Azure Data Lake Store and Analytics

  azurebig-datadata-lake
  139

• datopian / r2-bucket-uploader

  Cloudflare R2 bucket File Uploader with multipart upload enabled. Tested with files up to 10 GB size.

  blobblob-storagebucketcloudflaredata-lakeobject-storager2s3
  Language:TypeScript 122

• LearningJournal / Spark-Streaming-In-Python

  Apache Spark 3 - Structured Streaming Course Material

  apache-sparkbig-databigdatadata-lakepysparkpythonspark-sqlspark-streaming
  Language:Python 121

• smart-data-lake / smart-data-lake

  Smart Automation Tool for building modern Data Lakes and Data Pipelines

  data-lakescalasparksmart-data-lakehadoophivedeltalaketransform-datadata-pipelines
  Language:Scala 121

• GitDataAI / jzfs

  Git based Version Control File System for joint management of code, data, model and their relationship.

  data-collaborationdata-versioningdata-productdataopsmlopsdata-version-controlgit-for-datagit-filesystemversion-controlled-filesystemgitgit-interfacejzfsdatawareai-infrastructuredataset-managementmodel-management
  Language:TypeScript 109

• LearningJournal / SparkProgrammingInScala

  Apache Spark Course Material

  apache-sparkbig-databigdatadata-lakedatalakescalasparkspark-scalaspark-sql
  Language:Scala 88

• aws-samples / aws-dbs-refarch-datalake

  Reference Architectures for Datalakes on AWS

  data-lakedata-analyticsamazon-emringest-dataemr-clustergluehive-metastoredata-catalogdata-transformation
  Language:HTML 78

• Jayvardhan-Reddy / Azure-Certification-DP-200

  Road to Azure Data Engineer Part-I: DP-200 - Implementing an Azure Data Solution

  azurecertification-prepazure-certificationresourcesdp-200data-engineeringdata-engineerazure-storageazure-servicesdata-storagedata-lakeazure-portalsql-dwazure-databricksazure-data-factoryazure-cosmos-dbpolybaseexam-prepmicrosoft-azurebatch-processing
  68

• dominikhei / Local-Data-LakeHouse

  Sample Data Lakehouse deployed in Docker containers using Apache Iceberg, Minio, Trino and a Hive Metastore. Can be used for local testing.

  apache-icebergdata-lakedata-lakehousehive-metastorelakehouseminiotrino
  Language:Dockerfile 64

• camunda-community-hub / zeeqs

  GraphQL API for Zeebe data

  zeebegraphqldata-lakezeebe-tool
  Language:Kotlin 62

• datamindedbe / lighthouse

  Lighthouse is a library for data lakes built on top of Apache Spark. It provides high-level APIs in Scala to streamline data pipelines and apply best practices.

  data-lake
  Language:Scala 61

• MatsMoll / aligned

  The DBT of ML, as Aligned describes data dependencies in ML systems, and reduce technical data debt

  aifeature-engineeringfeature-storemlmlopsdbtdata-contractsdatacontractsml-opsdata-lake
  Language:Python 60
• querypal

  OElesin / querypal

  Web UI for Amazon Athena

  aws-athenadataanalyticssqlawsdata-lake
  Language:Vue 56

• KentHsu / Udacity-Data-Engineering-Nanodgree

  Udacity Data Engineering Nanodegree Program

  data-engineeringpostgresqlapache-cassandraaws-redshiftaws-s3apache-sparkapache-airflowdata-pipelinesdata-warehousesdata-lakedata-quality
  Language:Jupyter Notebook 52

• Wathon / data_engineering_with_python-track-datacamp

  Data Engineer with Python lecture notes from #datacamp.

  data-engineeringpythondatacamp-trackshellsqlscalaaws-botopysparkspark-sqlmongodbdata-warehousedata-lakeetl-pipelineoop-python
  Language:Jupyter Notebook 46

• realtimedatalake / rtdl

  rtdl makes it easy to build and maintain a real-time data lake

  apache-flinkapache-kafkabig-datacustomer-datadatadata-analysisdata-collectiondata-engineeringdata-lakedata-sciencedockerdocker-composedremiogolangmarketing-analyticsopen-sourceproduct-analyticsreal-time-data-lakertdlstateful-functions
  Language:Go 45