delta-lake

There are 9 repositories under delta-lake topic.

• apache / doris

  Apache Doris is an easy-to-use, high performance and unified analytics database.

  agentaibigquerydatabasedbtdelta-lakeelthudiiceberglakehouseolappaimonquery-enginereal-timeredshiftsnowflakesparksql
  Language:Java 14269

• trinodb / trino

  Official repository of Trino, the distributed SQL query engine for big data, formerly known as PrestoSQL (https://trino.io)

  analyticsbig-datadata-sciencedatabasedatabasesdatalakedelta-lakedistributed-databasedistributed-systemshadoophiveicebergjavajdbcprestoprestodbquery-enginesqltrino
  Language:Java 11871

• StarRocks / starrocks

  The world's fastest open query engine for sub-second analytics both on and off the data lakehouse. With the flexibility to support nearly any scenario, StarRocks provides best-in-class performance for multi-dimensional analytics, real-time analytics, and ad-hoc queries. A Linux Foundation project.

  analyticsbig-datacloudnativedatabasedatalakedelta-lakedistributed-databasehudiicebergjoinlakehouselakehouse-platformmppolapreal-time-analyticsreal-time-updatesrealtime-databasesqlstar-schemavectorized
  Language:Java 10663

• delta-io / delta

  An open-source storage framework that enables building a Lakehouse architecture with compute engines including Spark, PrestoDB, Flink, Trino, and Hive and APIs

  acidanalyticsbig-datadelta-lakespark
  Language:Scala 8271

• roapi / roapi

  Create full-fledged APIs for slowly moving datasets without writing a single line of code.

  sqlgraphqlarrowrest-apianalyticsquerycolumnarrustin-memory-databasedatafusionblob-storagecloud-nativeparquetquery-frontendsdatasetsstatic-datasetss3delta-lake
  Language:Rust 3340

• delta-io / delta-rs

  A native Rust library for Delta Lake, with bindings into Python

  databricksdeltadelta-lakepandaspandas-dataframepythonrust
  Language:Rust 2946
• pg_mooncake

  Mooncake-Labs / pg_mooncake

  Real-time analytics on Postgres tables

  analyticscolumnstoredelta-lakeiceberglakehouseparquetpostgresql
  Language:Rust 1674

• databricks / LearningSparkV2

  This is the github repo for Learning Spark: Lightning-Fast Data Analytics [2nd Edition]

  apache-sparksparkstructured-streamingspark-sqlspark-mllibmllibmlflowdelta-lake
  Language:Scala 1339

• apache / incubator-xtable

  Apache XTable (incubating) is a cross-table converter for lakehouse table formats that facilitates interoperability across data processing systems and query engines.

  apache-hudiapache-icebergdelta-lake
  Language:Java 1101

• delta-io / delta-sharing

  An open protocol for secure data sharing

  big-datadata-sharingdelta-lakepandasspark
  Language:Scala 870
• koheesio

  Nike-Inc / koheesio

  Python framework for building efficient data pipelines. It promotes modularity and collaboration, enabling the creation of complex pipelines from simple, reusable components.

  data-engineeringdelta-lakepydanticpysparkpython
  Language:Python 630
• seafowl

  splitgraph / seafowl

  Analytical database for data-driven Web applications 🪶

  databasehttpsqlapiedgeserverlessvisualizationrustdatafusiondelta-lakedelta-rs
  Language:Rust 495

• aws-samples / amazon-sagemaker-local-mode

  Amazon SageMaker Local Mode Examples

  sagemakeramazon-sagemakerpytorchcatboostlightgbmpycharmtensorflow-trainingpytorch-trainingsagemaker-processingprophetscikit-learnprophet-modelhdbscan-clustering-algorithmhuggingfacehuggingface-transformersmachine-learningdelta-lakegensim-word2vecdasktensorflow
  Language:Python 259

• adidas / lakehouse-engine

  The Lakehouse Engine is a configuration driven Spark framework, written in Python, serving as a scalable and distributed engine for several lakehouse algorithms, data flows and utilities for Data Products.

  big-dataconfiguration-drivendata-engineeringdata-qualitydatabricksdelta-lakeframeworkgreat-expectationslakehousespark
  Language:Python 240

• josephmachado / data_engineering_best_practices

  Sample project to demonstrate data engineering best practices

  data-engineeringdelta-lakeetlgreat-expectationsminiopysparkspark
  Language:Python 196

• japila-books / delta-lake-internals

  The Internals of Delta Lake

  deltalakebookinternalsdelta-lakebooksdatalake
  183
• Real-time-Data-Warehouse

  izhangzhihao / Real-time-Data-Warehouse

  Real-time Data Warehouse with Apache Flink & Apache Kafka & Apache Hudi

  flinkdata-warehousereal-time-data-warehousedata-warehousingflink-sqldebeziumkafkaelasticsearchdelta-lakecdcchange-data-capturehudihoodieicebergsqldatalakedeltadeltalakesparkspark-sql
  Language:Dockerfile 117

• delta-incubator / delta-sharing-rs

  A Minimalistic Rust Implementation of Delta Sharing Server.

  axumdata-engineeringdelta-iodelta-lakerust
  Language:Rust 89

• anneglienke / 101_upsert-delta

  This repository exemplifies a simple ELT process using delta to perform upsert and remove data files that aren't in the latest state of the transaction log for the table.

  deltadelta-lakedeltalake
  Language:Python 86

• tikal-fuseday / delta-architecture

  Streaming data changes to a Data Lake with Debezium and Delta Lake pipeline

  data-pipelinedatabasesdebeziumdelta-lakekafkasparkstreams
  Language:HTML 75

• lhbench / lhbench

  Lakehouse storage system benchmark

  apache-hudiapache-iceberglakehousebenchmarkcidrdatabasedatabricksdelta-lake
  Language:Scala 72

• dask-contrib / dask-deltatable

  A Delta Lake reader for Dask

  daskdask-dataframesdelta-lakeparquetpython
  Language:Python 49

• neylsoncrepalde / edc-mod1-exercise-igti

  Exercícios do módulo 1 - Bootcamp EDC - IGTI 2021

  awsdelta-lakeemrsparkterraform
  Language:Python 49
• modern-data-lake-storage-layers

  dacort / modern-data-lake-storage-layers

  Jupyter notebooks and AWS CloudFormation template to show how Hudi, Iceberg, and Delta Lake work

  awsamazon-emrhudiicebergapache-hudiapache-icebergdelta-lake
  Language:Jupyter Notebook 47

• jeppe742 / DeltaLakeReader

  Read Delta tables without any Spark

  delta-tablesdelta-lake
  Language:Python 47

• ysfesr / Building-Data-LakeHouse

  Creation of a data lakehouse and an ELT pipeline to enable the efficient analysis and use of data

  delta-lakedockerhivelakehouseminioprestos3-storagespark
  Language:Python 45

• TatevKaren / free-resources-books-papers

  Books and Papers in Mathematics, Econometrics, Machine Learning, Finance etc for different levels that can be useful for Data Scientists, Developers and everyone whoo is interesting in STEM.

  free-resourcesfree-booksmachine-learningeconometricsmathematicsbooksdatabricksdelta-lakedevelopersdata-sciencestatistics
  41
• delta-go

  csimplestring / delta-go

  Native Delta Lake Implementation in Go

  bigdatadatabricksdataprocessingdelta-lakegolanginfrastructurespark
  Language:Go 40

• databrickslabs / delta-oms

  DeltaOMS is a solution that help build a centralized repository of Delta Transaction logs and associated operational metrics/statistics for your Delta Lakehouse. Unity Catalog supported in the v0.7.0-rc1 release.Documentation here - https://databrickslabs.github.io/delta-oms/v0.7.0-rc1/

  deltadelta-lakemetricscentralizedlakehousemonitoringdatabricks
  Language:Scala 39

• thanhENC / e2e-data-platform

  End-to-end data platform: A PoC Data Platform project utilizing modern data stack (Spark, Airflow, DBT, Trino, Lightdash, Hive metastore, Minio, Postgres)

  adventureworksairflowdata-pipelinedata-platformdbtdelta-lakedocker-composeend-to-endhive-metastorelightdashsparktrino
  Language:Python 34
• spark-movies-etl

  guidok91 / spark-movies-etl

  Spark data pipeline that processes movie ratings data.

  apache-airflowapache-icebergdata-engineeringdata-pipelineeltetlpysparksparkuv
  Language:Python 30

• jaehyeon-kim / dbt-on-aws

  dbt (data build tool) projects targeting AWS analytics services (redshift, glue, emr, athena) and open table formats

  athenadbtdelta-lakeemrgluehudiicebergredshift
  Language:HCL 29

• AndrewKuzmin / spark-structured-streaming-examples

  Spark structured streaming examples with using of version 3.5.1

  sparkspark-sqlspark-structured-streamingapache-sparkstructured-streamingdelta-lake
  Language:Scala 26

• harrydevforlife / building-lakehouse

  Building Data Lakehouse by open source technology. Support end to end data pipeline, from source data on AWS S3 to Lakehouse, visualize and recommend app.

  airflowdbtdelta-lakeflask-apihive-metastorelakehousemetabaseminiopythons3spark
  Language:Python 24

• apache / doris-streamloader

  Stream Loader for Apache Doris

  bigquerydatabasedbtdelta-lakeeltetlhadoophivehudiiceberglakehouseolapquery-enginereal-timeredshiftsnowflakesparksql
  Language:Go 23

• PFund-Software-Ltd / pfeed

  Data Engine for Manual/Algo Trading: Download/Stream -> Clean -> Store. Supports Data Lakehouse Architecture. Clean Once and Forget.

  algo-tradingbacktestingdata-lakehousedata-pipelinedata-storagedelta-lakehistorical-datapandaspolarsstreaming
  Language:Python 23