data-lake

There are 20 repositories under data-lake topic.

• lakeFS

  treeverse / lakeFS

  lakeFS - Data version control for your data lake | Git for data

  data-engineeringdata-versioninggoobject-storagedata-lakeaws-s3data-qualityazure-blob-storagegoogle-cloud-storagegolanggit-for-datalakefsapache-sparkapache-sparksqlhadoop-filesystemdatalakedatalakesdata-version-controlazure-storage
  Language:Go 4962

• dlt-hub / dlt

  data load tool (dlt) is an open source Python library that makes data loading easy 🛠️

  datapythondata-engineeringdata-lakedata-loadingdata-warehouseeltextractloadtransform
  Language:Python 4490

• apache / kyuubi

  Apache Kyuubi is a distributed and multi-tenant gateway to provide serverless SQL on data warehouses and lakehouses.

  sparkhivesqlthriftjdbcspark-sqldata-lakehadoopkuberneteshacktoberfest
  Language:Scala 2269
• Udacity-Data-Engineering-Projects

  san089 / Udacity-Data-Engineering-Projects

  Few projects related to Data Engineering including Data Modeling, Infrastructure setup on cloud, Data Warehousing and Data Lake development.

  datadata-engineeringdata-engineering-pipelineetl-pipelinecassandra-databasepostgresql-databasedata-modelingdata-warehousedata-lakeairflowairflow-operatorsclustercassandrainfrastructurepostgresawsaws-ec2aws-sdkaws-s3cloudformation
  Language:Python 1758

• bytedance / bitsail

  BitSail is a distributed high-performance data integration engine which supports batch, streaming and incremental scenarios. BitSail is widely used to synchronize hundreds of trillions of data every day.

  flinkbig-datadata-integrationdata-lakedata-pipelinedata-synchronizationhigh-performancereal-time
  Language:Java 1675
• goodreads_etl_pipeline

  san089 / goodreads_etl_pipeline

  An end-to-end GoodReads Data Pipeline for Building Data Lake, Data Warehouse and Analytics Platform.

  etl-pipelineetl-frameworksparkapache-sparkapache-airflowairflowredshiftemr-clusterlivys3warehousedata-lakeschedulerdata-migrationdata-engineeringdata-engineering-pipelinepythongoodreads-data-pipelineairflow-dagetl-job
  Language:Python 1430

• Teradata / kylo

  Kylo is a data lake management software platform and framework for enabling scalable enterprise-class data lakes on big data technologies such as Teradata, Apache Spark and/or Hadoop. Kylo is licensed under Apache 2.0. Contributed by Teradata Inc.

  data-lakehadoopkylonifisparkteradata
  Language:Java 1114

• apache / amoro

  Apache Amoro(incubating) is a Lakehouse management system built on open data lake formats.

  big-datadata-lakeflinkhudiicebergmanagement-systempaimonsparktrinolake-houseself-optimizing
  Language:Java 1065

• lakekeeper / lakekeeper

  Lakekeeper is an Apache-Licensed, secure, fast and easy to use Apache Iceberg REST Catalog written in Rust.

  catalogdata-lakeiceberglakehouserustlakehouse-governanceopen-lakehouse
  Language:Rust 1016

• alanchn31 / Data-Engineering-Projects

  Personal Data Engineering Projects

  data-lakeingest-datadata-engineeringdata-warehousecassandraaws-redshiftmongodbscrapysparkairflowpostgresdata-engineering-nanodegreestar-schemadata-modeling
  Language:Jupyter Notebook 961

• Canner / vulcan-sql

  Data API Framework for AI Agents and Data Apps

  api-builderdata-lakedata-warehousedatabasesqlanalyticsreportingspreadsheetbigqueryduckdbpostgresqlsnowflakerestful-apitypescriptclickhousevulcan-sqlvulcansqlksqldbaiai-agent
  Language:TypeScript 774

• pixelsdb / pixels

  An efficient storage and compute engine for both on-prem and cloud-native data analytics.

  cloud-databasedata-lakedata-warehousedatabaseolapcolumn-store
  Language:Java 710
• wren-engine

  Canner / wren-engine

  🤖 The Semantic Engine for Model Context Protocol(MCP) Clients and AI Agents 🔥

  business-intelligencedatadata-analysisdata-analyticsdata-lakedata-warehousesqlsemanticsemantic-layerllmhacktoberfestagentagentic-aiaimcpmcp-server
  Language:Java 493

• uber / marmaray

  Generic Data Ingestion & Dispersal Library for Hadoop

  avro-schemadata-lakehadoopingest-dataschema-formatspark
  Language:Java 478
• data-lakes-on-aws

  aws-solutions-library-samples / data-lakes-on-aws

  Enterprise-grade, production-hardened, serverless data lake on AWS

  serverlessframeworkdata-lakeanalyticsawsetldata-engineeringlake-formationiacbest-practices
  Language:Python 473

• kaiwaehner / hivemq-mqtt-tensorflow-kafka-realtime-iot-machine-learning-training-inference

  Real Time Big Data / IoT Machine Learning (Model Training and Inference) with HiveMQ (MQTT), TensorFlow IO and Apache Kafka - no additional data store like S3, HDFS or Spark required

  kafkahivemqmqttkafka-streamskafka-connectksqltensorflowgrpcjavapythontiered-storagedata-lakeconfluentksqldbtensorflow-ioterraformgcpkubernetescloudmongodb
  Language:Jupyter Notebook 417
• gigapi

  gigapi / gigapi

  GigAPI is a Timeseries lakehouse for real-time data and sub-second queries, powered by DuckDB OLAP + Parquet Query Engine, Compactor w/ Cloud-Native Storage. Drop-in FDAP alternative ⭐

  apiclickhouse-serverdata-lakedatabasedatalakeduckdbduckdb-apiduckdb-serverducklakefdapgigapipegolanglakehouseolapparquetqrynquery-enginerest-apis3sql
  Language:Go 358

• cuebook / cuelake

  Use SQL to build ELT pipelines on a data lakehouse.

  apache-icebergdeltalakehousedatalakedata-lakeeltetldata-engineeringdata-integrationdata-ingestionapache-sparkspark-sqlupsertincremental-updatesdata-transferpipelinesdata-pipelinezeppelin-notebooksql
  Language:JavaScript 288

• maxi-k / btrblocks

  BtrBlocks: Efficient Columnar Compression for Data Lakes (SIGMOD 2023 Paper)

  compressiondata-lakedatabasesresearch
  Language:C++ 265
• amazon-s3-find-and-forget

  awslabs / amazon-s3-find-and-forget

  Amazon S3 Find and Forget is a solution to handle data erasure requests from data lakes stored on Amazon S3, for example, pursuant to the European General Data Protection Regulation (GDPR)

  amazon-s3awsbig-dataccpadatadata-erasuredata-lakegdprparquetprivacyright-to-be-forgottens3
  Language:Python 242

• Azure / usql

  U-SQL Examples and Issue Tracking

  azurebig-datadata-lakeu-sql
  Language:C# 233

• garystafford / tickit-data-lake-demo

  Resources for video demonstrations and blog posts related to DataOps on AWS

  dataopsawsdevopsairflowredshiftdata-lake
  Language:Python 181

• fortiql / data-forge

  Data Forge — a modern data stack playground to practice flows and best practices, not just tools. Spark, Trino, Kafka, Iceberg, ClickHouse, Airflow, MinIO, Superset — all wired together locally with Docker Compose.

  bronze-silver-golddata-engineeringdata-lakedata-simulationlakehousemodern-data-stackopen-source
  Language:Jupyter Notebook 152

• Azure / AzureDataLake

  Samples and Docs for Azure Data Lake Store and Analytics

  azurebig-datadata-lake
  139

• datopian / r2-bucket-uploader

  Cloudflare R2 bucket File Uploader with multipart upload enabled. Tested with files up to 10 GB size.

  blobblob-storagebucketcloudflaredata-lakeobject-storager2s3
  Language:TypeScript 128

• smart-data-lake / smart-data-lake

  Smart Automation Tool for building modern Data Lakes and Data Pipelines

  data-lakescalasparksmart-data-lakehadoophivedeltalaketransform-datadata-pipelines
  Language:Scala 124

• LearningJournal / Spark-Streaming-In-Python

  Apache Spark 3 - Structured Streaming Course Material

  apache-sparkbig-databigdatadata-lakepysparkpythonspark-sqlspark-streaming
  Language:Python 123

• LearningJournal / SparkProgrammingInScala

  Apache Spark Course Material

  apache-sparksparkspark-sqlspark-scalabig-databigdatadatalakedata-lakescala
  Language:Scala 95

• aws-samples / aws-dbs-refarch-datalake

  Reference Architectures for Datalakes on AWS

  amazon-emrdata-analyticsdata-catalogdata-lakedata-transformationemr-clustergluehive-metastoreingest-data
  Language:HTML 78

• dominikhei / Local-Data-LakeHouse

  Sample Data Lakehouse deployed in Docker containers using Apache Iceberg, Minio, Trino and a Hive Metastore. Can be used for local testing.

  apache-icebergdata-lakedata-lakehousehive-metastorelakehouseminiotrino
  Language:Dockerfile 74

• Jayvardhan-Reddy / Azure-Certification-DP-200

  Road to Azure Data Engineer Part-I: DP-200 - Implementing an Azure Data Solution

  azurecertification-prepazure-certificationresourcesdp-200data-engineeringdata-engineerazure-storageazure-servicesdata-storagedata-lakeazure-portalsql-dwazure-databricksazure-data-factoryazure-cosmos-dbpolybaseexam-prepmicrosoft-azurebatch-processing
  67

• camunda-community-hub / zeeqs

  GraphQL API for Zeebe data

  data-lakegraphqlzeebezeebe-tool
  Language:Kotlin 62

• datamindedbe / lighthouse

  Lighthouse is a library for data lakes built on top of Apache Spark. It provides high-level APIs in Scala to streamline data pipelines and apply best practices.

  data-lake
  Language:Scala 62

• MatsMoll / aligned

  The DBT of ML, as Aligned describes data dependencies in ML systems, and reduce technical data debt

  aifeature-engineeringfeature-storemlmlopsdbtdata-contractsdatacontractsml-opsdata-lake
  Language:Python 60
• querypal

  OElesin / querypal

  Web UI for Amazon Athena

  analyticsawsaws-athenadatadata-lakesql
  Language:Vue 58

• KentHsu / Udacity-Data-Engineering-Nanodgree

  Udacity Data Engineering Nanodegree Program

  data-engineeringpostgresqlapache-cassandraaws-redshiftaws-s3apache-sparkapache-airflowdata-pipelinesdata-warehousesdata-lakedata-quality
  Language:Jupyter Notebook 52