data-lake

There are 19 repositories under data-lake topic.

• lakeFS

  treeverse / lakeFS

  lakeFS - Data version control for your data lake | Git for data

  apache-sparkapache-sparksqlaws-s3azure-blob-storageazure-storagedata-engineeringdata-lakedata-qualitydata-version-controldata-versioningdatalakedatalakesgit-for-datagogolanggoogle-cloud-storagehadoop-filesystemlakefsobject-storage
  Language:Go 4354

• dlt-hub / dlt

  data load tool (dlt) is an open source Python library that makes data loading easy 🛠️

  datadata-engineeringdata-lakedata-loadingdata-warehouseeltextractloadpythontransform
  Language:Python 2319

• apache / kyuubi

  Apache Kyuubi is a distributed and multi-tenant gateway to provide serverless SQL on data warehouses and lakehouses.

  data-lakehacktoberfesthadoophivejdbckubernetessparkspark-sqlsqlthrift
  Language:Scala 2074

• bytedance / bitsail

  BitSail is a distributed high-performance data integration engine which supports batch, streaming and incremental scenarios. BitSail is widely used to synchronize hundreds of trillions of data every day.

  big-datadata-integrationdata-lakedata-pipelinedata-synchronizationflinkhigh-performancereal-time
  Language:Java 1614
• Udacity-Data-Engineering-Projects

  san089 / Udacity-Data-Engineering-Projects

  Few projects related to Data Engineering including Data Modeling, Infrastructure setup on cloud, Data Warehousing and Data Lake development.

  datadata-engineeringdata-engineering-pipelineetl-pipelinecassandra-databasepostgresql-databasedata-modelingdata-warehousedata-lakeairflowairflow-operatorsclustercassandrainfrastructurepostgresawsaws-ec2aws-sdkaws-s3cloudformation
  Language:Python 1475
• goodreads_etl_pipeline

  san089 / goodreads_etl_pipeline

  An end-to-end GoodReads Data Pipeline for Building Data Lake, Data Warehouse and Analytics Platform.

  airflowairflow-dagapache-airflowapache-sparkdata-engineeringdata-engineering-pipelinedata-lakedata-migrationemr-clusteretl-frameworketl-jobetl-pipelinegoodreads-data-pipelinelivypythonredshifts3schedulersparkwarehouse
  Language:Python 1273

• Teradata / kylo

  Kylo is a data lake management software platform and framework for enabling scalable enterprise-class data lakes on big data technologies such as Teradata, Apache Spark and/or Hadoop. Kylo is licensed under Apache 2.0. Contributed by Teradata Inc.

  data-lakehadoopkylonifisparkteradata
  Language:Java 1104

• alanchn31 / Data-Engineering-Projects

  Personal Data Engineering Projects

  data-lakeingest-datadata-engineeringdata-warehousecassandraaws-redshiftmongodbscrapysparkairflowpostgresdata-engineering-nanodegreestar-schemadata-modeling
  Language:Jupyter Notebook 820

• Canner / vulcan-sql

  Data API Framework for AI Agents and Data Apps

  api-builderdata-lakedata-warehousedatabasesqlanalyticsreportingspreadsheetbigqueryduckdbpostgresqlsnowflakerestful-apitypescriptclickhousevulcan-sqlvulcansqlksqldbaiai-agent
  Language:TypeScript 625

• uber / marmaray

  Generic Data Ingestion & Dispersal Library for Hadoop

  avro-schemadata-lakehadoopingest-dataschema-formatspark
  Language:Java 479
• aws-serverless-data-lake-framework

  awslabs / aws-serverless-data-lake-framework

  Enterprise-grade, production-hardened, serverless data lake on AWS

  analyticsawsbest-practicesdata-engineeringdata-lakeetlframeworkiaclake-formationserverless
  Language:Python 407

• kaiwaehner / hivemq-mqtt-tensorflow-kafka-realtime-iot-machine-learning-training-inference

  Real Time Big Data / IoT Machine Learning (Model Training and Inference) with HiveMQ (MQTT), TensorFlow IO and Apache Kafka - no additional data store like S3, HDFS or Spark required

  kafkahivemqmqttkafka-streamskafka-connectksqltensorflowgrpcjavapythontiered-storagedata-lakeconfluentksqldbtensorflow-ioterraformgcpkubernetescloudmongodb
  Language:Jupyter Notebook 402

• cuebook / cuelake

  Use SQL to build ELT pipelines on a data lakehouse.

  apache-icebergdeltalakehousedatalakedata-lakeeltetldata-engineeringdata-integrationdata-ingestionapache-sparkspark-sqlupsertincremental-updatesdata-transferpipelinesdata-pipelinezeppelin-notebooksql
  Language:JavaScript 285
• amazon-s3-find-and-forget

  awslabs / amazon-s3-find-and-forget

  Amazon S3 Find and Forget is a solution to handle data erasure requests from data lakes stored on Amazon S3, for example, pursuant to the European General Data Protection Regulation (GDPR)

  data-lakeamazon-s3s3gdprawsparquetdata-erasureright-to-be-forgottenccpabig-dataprivacydata
  Language:Python 238

• Azure / usql

  U-SQL Examples and Issue Tracking

  big-datau-sqlazuredata-lake
  Language:C# 234

• maxi-k / btrblocks

  BtrBlocks: Efficient Columnar Compression for Data Lakes (SIGMOD 2023 Paper)

  compressiondata-lakedatabasesresearch
  Language:C++ 213

• garystafford / tickit-data-lake-demo

  Resources for video demonstrations and blog posts related to DataOps on AWS

  dataopsawsdevopsairflowredshiftdata-lake
  Language:Python 166

• hansetag / iceberg-catalog

  A Rust implementation of the Iceberg REST Catalog specification.

  catalogdata-lakeiceberglakehouselakehouse-governanceopen-lakehouserust
  Language:Rust 147

• Azure / AzureDataLake

  Samples and Docs for Azure Data Lake Store and Analytics

  big-dataazuredata-lake
  140

• pixelsdb / pixels

  An efficient storage and compute engine for both on-prem and cloud-native data analytics.

  cloud-databasedata-lakedata-warehousedatabaseolapcolumn-store
  Language:Java 128

• LearningJournal / Spark-Streaming-In-Python

  Apache Spark 3 - Structured Streaming Course Material

  apache-sparkspark-streamingspark-sqlpysparkpythonbig-databigdatadata-lake
  Language:Python 120
• wren-engine

  Canner / wren-engine

  🤖 The semantic engine for LLMs, bringing semantic context to AI agents. 🔥

  business-intelligencedatadata-analysisdata-analyticsdata-lakedata-warehousesqlsemanticsemantic-layerllm
  Language:Java 115

• smart-data-lake / smart-data-lake

  Smart Automation Tool for building modern Data Lakes and Data Pipelines

  data-lakedata-pipelinesdeltalakehadoophivescalasmart-data-lakesparktransform-data
  Language:Scala 104

• datopian / r2-bucket-uploader

  Cloudflare R2 bucket File Uploader with multipart upload enabled. Tested with files up to 10 GB size.

  blobblob-storagebucketcloudflaredata-lakeobject-storager2s3
  Language:TypeScript 86

• LearningJournal / SparkProgrammingInScala

  Apache Spark Course Material

  apache-sparksparkspark-sqlspark-scalabig-databigdatadatalakedata-lakescala
  Language:Scala 84

• aws-samples / aws-dbs-refarch-datalake

  Reference Architectures for Datalakes on AWS

  data-lakedata-analyticsamazon-emringest-dataemr-clustergluehive-metastoredata-catalogdata-transformation
  Language:HTML 76

• GitDataAI / jiaozifs

  A Git-like version control file system for data lineage & data collaboration.

  data-collaborationdata-versioningaiopsdata-meshdata-productdataopsdigital-twinsenterprise-datahubmlopsfederated-learningdata-lake-managementdata-version-controldata-lakedatalakegit-for-datagit-filesystemjiaozifsversion-controlled-filesystemdata-lineage
  Language:Go 74

• Jayvardhan-Reddy / Azure-Certification-DP-200

  Road to Azure Data Engineer Part-I: DP-200 - Implementing an Azure Data Solution

  azurecertification-prepazure-certificationresourcesdp-200data-engineeringdata-engineerazure-storageazure-servicesdata-storagedata-lakeazure-portalsql-dwazure-databricksazure-data-factoryazure-cosmos-dbpolybaseexam-prepmicrosoft-azurebatch-processing
  65

• camunda-community-hub / zeeqs

  GraphQL API for Zeebe data

  zeebegraphqldata-lakezeebe-tool
  Language:Kotlin 62

• datamindedbe / lighthouse

  Lighthouse is a library for data lakes built on top of Apache Spark. It provides high-level APIs in Scala to streamline data pipelines and apply best practices.

  data-lake
  Language:Scala 60
• querypal

  OElesin / querypal

  Web UI for Amazon Athena

  aws-athenadataanalyticssqlawsdata-lake
  Language:Vue 54

• dominikhei / Local-Data-LakeHouse

  Sample Data Lakehouse deployed in Docker containers using Apache Iceberg, Minio, Trino and a Hive Metastore. Can be used for local testing.

  apache-icebergdata-lakedata-lakehousehive-metastorelakehouseminiotrino
  Language:Dockerfile 51

• KentHsu / Udacity-Data-Engineering-Nanodgree

  Udacity Data Engineering Nanodegree Program

  data-engineeringpostgresqlapache-cassandraaws-redshiftaws-s3apache-sparkapache-airflowdata-pipelinesdata-warehousesdata-lakedata-quality
  Language:Jupyter Notebook 51

• MatsMoll / aligned

  The DBT of ML, as Aligned describes data dependencies in ML systems, and reduce technical data debt

  aifeature-engineeringfeature-storemlmlopsdbtdata-contractsdatacontractsml-opsdata-lake
  Language:Python 49

• aws-samples / analyzing-reddit-sentiment-with-aws

  Learn how to use Kinesis Firehose, AWS Glue, S3, and Amazon Athena by streaming and analyzing reddit comments in realtime. 100-200 level tutorial.

  kinesis-firehosedelivery-streamaws-gluedata-streamamazon-athenadata-lakeredditsentiment-analysissentiment-classificationreal-timeself-learningtutorials
  Language:Python 44

• realtimedatalake / rtdl

  rtdl makes it easy to build and maintain a real-time data lake

  open-sourcedatadata-engineeringdata-analysisdata-sciencedata-collectionproduct-analyticsmarketing-analyticscustomer-databig-datadata-lakegolangapache-flinkstateful-functionsapache-kafkareal-time-data-lakertdldockerdocker-composedremio
  Language:Go 44