data-lake

There are 18 repositories under data-lake topic.

• lakeFS

  treeverse / lakeFS

  lakeFS - Data version control for your data lake | Git for data

  data-engineeringdata-versioninggoobject-storagedata-lakeaws-s3data-qualityazure-blob-storagegoogle-cloud-storagegolanggit-for-datalakefsapache-sparkapache-sparksqlhadoop-filesystemdatalakedatalakesdata-version-controlazure-storage
  Language:Go 4087

• apache / kyuubi

  Apache Kyuubi is a distributed and multi-tenant gateway to provide serverless SQL on data warehouses and lakehouses.

  sparkhivesqlthriftjdbcspark-sqldata-lakehadoopkuberneteshacktoberfest
  Language:Scala 1946

• dlt-hub / dlt

  data load tool (dlt) is an open source Python library that makes data loading easy 🛠️

  datadata-engineeringdata-lakedata-loadingdata-warehouseeltextractloadpythontransform
  Language:Python 1765

• bytedance / bitsail

  BitSail is a distributed high-performance data integration engine which supports batch, streaming and incremental scenarios. BitSail is widely used to synchronize hundreds of trillions of data every day.

  big-datadata-integrationdata-lakedata-pipelinedata-synchronizationflinkhigh-performancereal-time
  Language:Java 1584
• Udacity-Data-Engineering-Projects

  san089 / Udacity-Data-Engineering-Projects

  Few projects related to Data Engineering including Data Modeling, Infrastructure setup on cloud, Data Warehousing and Data Lake development.

  datadata-engineeringdata-engineering-pipelineetl-pipelinecassandra-databasepostgresql-databasedata-modelingdata-warehousedata-lakeairflowairflow-operatorsclustercassandrainfrastructurepostgresawsaws-ec2aws-sdkaws-s3cloudformation
  Language:Python 1354
• goodreads_etl_pipeline

  san089 / goodreads_etl_pipeline

  An end-to-end GoodReads Data Pipeline for Building Data Lake, Data Warehouse and Analytics Platform.

  etl-pipelineetl-frameworksparkapache-sparkapache-airflowairflowredshiftemr-clusterlivys3warehousedata-lakeschedulerdata-migrationdata-engineeringdata-engineering-pipelinepythongoodreads-data-pipelineairflow-dagetl-job
  Language:Python 1229

• Teradata / kylo

  Kylo is a data lake management software platform and framework for enabling scalable enterprise-class data lakes on big data technologies such as Teradata, Apache Spark and/or Hadoop. Kylo is licensed under Apache 2.0. Contributed by Teradata Inc.

  data-lakehadoopkylonifisparkteradata
  Language:Java 1091

• alanchn31 / Data-Engineering-Projects

  Personal Data Engineering Projects

  data-lakeingest-datadata-engineeringdata-warehousecassandraaws-redshiftmongodbscrapysparkairflowpostgresdata-engineering-nanodegreestar-schemadata-modeling
  Language:Jupyter Notebook 733

• Canner / vulcan-sql

  Data API Framework for AI Agents and Data Apps

  api-builderdata-lakedata-warehousedatabasesqlanalyticsreportingspreadsheetbigqueryduckdbpostgresqlsnowflakerestful-apitypescriptclickhousevulcan-sqlvulcansqlksqldbaiai-agent
  Language:TypeScript 595

• uber / marmaray

  Generic Data Ingestion & Dispersal Library for Hadoop

  hadoopdata-lakeingest-dataavro-schemaschema-formatspark
  Language:Java 472
• aws-serverless-data-lake-framework

  awslabs / aws-serverless-data-lake-framework

  Enterprise-grade, production-hardened, serverless data lake on AWS

  serverlessframeworkdata-lakeanalyticsawsetldata-engineeringlake-formationiacbest-practices
  Language:Python 396

• kaiwaehner / hivemq-mqtt-tensorflow-kafka-realtime-iot-machine-learning-training-inference

  Real Time Big Data / IoT Machine Learning (Model Training and Inference) with HiveMQ (MQTT), TensorFlow IO and Apache Kafka - no additional data store like S3, HDFS or Spark required

  kafkahivemqmqttkafka-streamskafka-connectksqltensorflowgrpcjavapythontiered-storagedata-lakeconfluentksqldbtensorflow-ioterraformgcpkubernetescloudmongodb
  Language:Jupyter Notebook 390

• cuebook / cuelake

  Use SQL to build ELT pipelines on a data lakehouse.

  apache-icebergdeltalakehousedatalakedata-lakeeltetldata-engineeringdata-integrationdata-ingestionapache-sparkspark-sqlupsertincremental-updatesdata-transferpipelinesdata-pipelinezeppelin-notebooksql
  Language:JavaScript 283
• amazon-s3-find-and-forget

  awslabs / amazon-s3-find-and-forget

  Amazon S3 Find and Forget is a solution to handle data erasure requests from data lakes stored on Amazon S3, for example, pursuant to the European General Data Protection Regulation (GDPR)

  data-lakeamazon-s3s3gdprawsparquetdata-erasureright-to-be-forgottenccpabig-dataprivacydata
  Language:Python 232

• Azure / usql

  U-SQL Examples and Issue Tracking

  azurebig-datadata-lakeu-sql
  Language:C# 232

• maxi-k / btrblocks

  BtrBlocks: Efficient Columnar Compression for Data Lakes (SIGMOD 2023 Paper)

  compressiondata-lakedatabasesresearch
  Language:C++ 184

• garystafford / tickit-data-lake-demo

  Resources for video demonstrations and blog posts related to DataOps on AWS

  dataopsawsdevopsairflowredshiftdata-lake
  Language:Python 157

• Azure / AzureDataLake

  Samples and Docs for Azure Data Lake Store and Analytics

  azurebig-datadata-lake
  139

• LearningJournal / Spark-Streaming-In-Python

  Apache Spark 3 - Structured Streaming Course Material

  apache-sparkbig-databigdatadata-lakepysparkpythonspark-sqlspark-streaming
  Language:Python 113

• smart-data-lake / smart-data-lake

  Smart Automation Tool for building modern Data Lakes and Data Pipelines

  data-lakescalasparksmart-data-lakehadoophivedeltalaketransform-datadata-pipelines
  Language:Scala 96

• LearningJournal / SparkProgrammingInScala

  Apache Spark Course Material

  apache-sparkbig-databigdatadata-lakedatalakescalasparkspark-scalaspark-sql
  Language:Scala 82

• aws-samples / aws-dbs-refarch-datalake

  Reference Architectures for Datalakes on AWS

  data-lakedata-analyticsamazon-emringest-dataemr-clustergluehive-metastoredata-catalogdata-transformation
  Language:HTML 75

• Jayvardhan-Reddy / Azure-Certification-DP-200

  Road to Azure Data Engineer Part-I: DP-200 - Implementing an Azure Data Solution

  azurecertification-prepazure-certificationresourcesdp-200data-engineeringdata-engineerazure-storageazure-servicesdata-storagedata-lakeazure-portalsql-dwazure-databricksazure-data-factoryazure-cosmos-dbpolybaseexam-prepmicrosoft-azurebatch-processing
  62

• camunda-community-hub / zeeqs

  GraphQL API for Zeebe data

  data-lakegraphqlzeebezeebe-tool
  Language:Kotlin 60

• datamindedbe / lighthouse

  Lighthouse is a library for data lakes built on top of Apache Spark. It provides high-level APIs in Scala to streamline data pipelines and apply best practices.

  data-lake
  Language:Scala 60

• datopian / r2-bucket-uploader

  Cloudflare R2 bucket File Uploader

  blobblob-storagebucketcloudflaredata-lakeobject-storager2s3
  Language:TypeScript 54
• querypal

  OElesin / querypal

  Web UI for Amazon Athena

  aws-athenadataanalyticssqlawsdata-lake
  Language:Vue 52

• Canner / wren-engine

  Wren Engine is the backbone of the semantic layer - The semantic engine for LLMs; bringing business context to LLMs.

  business-intelligencedatadata-analysisdata-analyticsdata-lakedata-warehousellmsemanticsemantic-layersql
  Language:Java 49

• KentHsu / Udacity-Data-Engineering-Nanodgree

  Udacity Data Engineering Nanodegree Program

  apache-airflowapache-cassandraapache-sparkaws-redshiftaws-s3data-engineeringdata-lakedata-pipelinesdata-qualitydata-warehousespostgresql
  Language:Jupyter Notebook 48

• dominikhei / Local-Data-LakeHouse

  Sample Data Lakehouse deployed in Docker containers using Apache Iceberg, Minio, Trino and a Hive Metastore. Can be used for local testing.

  apache-icebergdata-lakedata-lakehousehive-metastorelakehouseminiotrino
  Language:Dockerfile 45

• aws-samples / analyzing-reddit-sentiment-with-aws

  Learn how to use Kinesis Firehose, AWS Glue, S3, and Amazon Athena by streaming and analyzing reddit comments in realtime. 100-200 level tutorial.

  kinesis-firehosedelivery-streamaws-gluedata-streamamazon-athenadata-lakeredditsentiment-analysissentiment-classificationreal-timeself-learningtutorials
  Language:Python 44

• GitDataAI / jiaozifs

  An Git-like version control file system for data lineage & data collaboration.

  data-collaborationdata-versioningaiopsdata-meshdata-productdataopsdigital-twinsenterprise-datahubmlopsfederated-learningdata-lake-managementdata-version-controldata-lakedatalakegit-for-datagit-filesystemjiaozifsversion-controlled-filesystemdata-lineage
  Language:Go 44

• realtimedatalake / rtdl

  rtdl makes it easy to build and maintain a real-time data lake

  open-sourcedatadata-engineeringdata-analysisdata-sciencedata-collectionproduct-analyticsmarketing-analyticscustomer-databig-datadata-lakegolangapache-flinkstateful-functionsapache-kafkareal-time-data-lakertdldockerdocker-composedremio
  Language:Go 43

• Wathon / data_engineering_with_python-track-datacamp

  Data Engineer with Python lecture notes from #datacamp.

  data-engineeringpythondatacamp-trackshellsqlscalaaws-botopysparkspark-sqlmongodbdata-warehousedata-lakeetl-pipelineoop-python
  Language:Jupyter Notebook 38
• cnfuzz

  suecodelabs / cnfuzz

  Breaking Cloud Native Web APIs in their natural habitat.

  cicdcloud-nativedata-lakefuzzinggolangkubernetesrestlersecurity-toolsmicrosoftopenapiopenapi-specrest-apirest-api-testservice-meshopensourceaws-s3aws
  Language:Go 36

• yahwang / Awesome-Data-Engineering

  📒(GitBook) A curated list of awesome Data Engineering resources

  data-engineeringdata-pipelinedata-lake
  32