deltalake

There are 2 repositories under deltalake topic.

• paradedb / pg_analytics

  DuckDB-powered data lake analytics from Postgres

  analyticsarrowcolumnardatafusionlakehouseparadedbparquetpostgrespostgresqlduckdbolapbig-datadatabasedatalakedeltalakeicebergobject-storagerealtime-analyticssqllakehouse-platform
  Language:Rust 530

• databrickslabs / dbldatagen

  Generate relevant synthetic data quickly for your projects. The Databricks Labs synthetic data generator (aka `dbldatagen`) may be used to generate large simulated / synthetic data sets for test, POCs, and other uses in Databricks environments including in Delta Live Tables pipelines

  data-generationdatabricksdatagendatagenerationdatageneratordelta-live-tablesdeltalakefakerpysparkpythonsparkspark-streamingsynthetic-data
  Language:Python 432

• delta-io / kafka-delta-ingest

  A highly efficient daemon for streaming data from Kafka into Delta Lake

  deltalakedeltarustkafka
  Language:Rust 416

• MrPowers / mack

  Delta Lake helper methods in PySpark

  deltalakepyspark
  Language:Python 323

• japila-books / delta-lake-internals

  The Internals of Delta Lake

  bookbooksdatalakedelta-lakedeltalakeinternals
  186

• smart-data-lake / smart-data-lake

  Smart Automation Tool for building modern Data Lakes and Data Pipelines

  data-lakescalasparksmart-data-lakehadoophivedeltalaketransform-datadata-pipelines
  Language:Scala 124

• uname-n / deltabase

  a lightweight, comprehensive solution for managing delta tables built on polars and deltalake

  databasedelta-tablesdeltalakepolarssql
  Language:Python 122
• Real-time-Data-Warehouse

  izhangzhihao / Real-time-Data-Warehouse

  Real-time Data Warehouse with Apache Flink & Apache Kafka & Apache Hudi

  flinkdata-warehousereal-time-data-warehousedata-warehousingflink-sqldebeziumkafkaelasticsearchdelta-lakecdcchange-data-capturehudihoodieicebergsqldatalakedeltadeltalakesparkspark-sql
  Language:Dockerfile 118

• anneglienke / 101_upsert-delta

  This repository exemplifies a simple ELT process using delta to perform upsert and remove data files that aren't in the latest state of the transaction log for the table.

  deltadelta-lakedeltalake
  Language:Python 110

• WeBankFinTech / Streamis

  Streaming application development and management system, based on Linkis and DSS, planning to provide the workflow-like graphical drag-and-drop development capability.

  flinklinkisdataspherestudiowedataspherestreamisstreaminghudiicebergdatalakewarehousekafkadeltalake
  Language:Java 109
• ApacheSpark

  martandsingh / ApacheSpark

  This repository will help you to learn about databricks concept with the help of examples. It will include all the important topics which we need in our real life experience as a data engineer. We will be using pyspark & sparksql for the development. At the end of the course we also cover few case studies.

  apachesparkdata-analysisdata-engineeringdatabasedatabricksdatalakedeltalakeetl-pipelinehadoophivesparkspark-sqlspark-streamingtimetraveletlpysparksql
  Language:Python 103

• dacort / faker-cli

  Command-line interface to quickly generate fake CSV and JSON data

  awscsvfaker-providerjsondeltalakeparquetpyarrow
  Language:Python 83

• bhavink / databricks

  Databricks Platform - Architecture, Security, Automation and much more!!

  databricksazure-databricksdatabricks-rest-apidatabricks-dbconnectdatabricks-platformdatabricks-clideltalakesecurity
  Language:Jupyter Notebook 51

• buoyant-data / oxbow

  Collection of AWS Lambdas for creating and managing Delta tables

  deltalakeparquetdatalakelambdarust
  Language:Rust 46

• mrjsj / msfabricutils

  Spark-free Python utilities for Microsoft Fabric focused on Data Engineering using Polars and delta-rs

  dataframedeltalakeduckdbpythonfabricmicrosoft-fabricdelta-rspolarsdata-engineering
  Language:Python 36

• sankamuk / PysparkCheatsheet

  PySpark Cheatsheet

  apache-sparkdeltalakepythonstructured-streaming
  Language:Python 36

• DataTech-Solutions / Threat-Detection-and-Visualization

  Threat Detection and Visualization

  apidatafactorydatalakedefenderdeltalakekeyvaultparquet-filespostmanpowerbirolebasedpermissionssccmserverlesssqlpoolservicenowsiemsqltenablescdedicatedsqlpoolactive-directoryazuremlstudiomachine-learning
  Language:TSQL 32

• mrjsj / delta-lake-explorer

  azuredeltalakeduckdbsql-clientsteamlit
  Language:Python 29

• newfront / hitchhikers_guide_to_deltalake_streaming

  Don't Panic. This guide will help you when it feels like the end of the world.

  apacheapache-sparkdeltalakehitchhikers-guide
  Language:Jupyter Notebook 29

• leehuwuj / olh

  Open source stack lakehouse

  bigdatalakehousedataplatformkubernetessparkdeltalake
  Language:Python 25

• ognis1205 / delta-hub

  A platform and cloud-based service for data sharing based on the Delta Sharing protocol.

  chakra-uidelta-rsdeltalakenextjs13typescript
  Language:TypeScript 21
• awesome-databricks

  reisdebora / awesome-databricks

  A curated list of awesome Databricks resources, including Spark

  sparkdatabrickskoaladeltalakeawsazureawesome
  21

• aws-samples / amazon-emr-with-delta-lake

  Amazon EMR Notebook to show how to read from and write to Delta tables with Amazon EMR

  amazon-emrsparkdeltalake
  Language:Jupyter Notebook 17

• ismailhammounou / db2ixf

  db2ixf is a python package with a CLI that simplifies the parsing and processing of IBM Integration eXchange Format (IXF) files.

  conversionconvertercsvdb2db2-databaseibmibm-cloudixfjsonparquetparserparsingparsing-libraryprocessingpythonclideltalakepolarspyarrowjsonlines
  Language:Python 16

• taka-yayoi / public_repo

  Databricksのサンプルノートブックを格納しています。

  databrickspythonsparkdeltalakemlflow
  Language:Python 13

• aravinthsci / Spark_Delta_Lake

  Delta Lake Examples

  apachesparkdatalakedelta-lakedeltalakespark
  Language:Jupyter Notebook 12

• goodwillpunning / nodejs-sharing-client

  A Node.js connector for Delta Sharing.

  datasharingdeltalakedeltasharing
  Language:JavaScript 12

• bmsuisse / lakeapi

  API for distributing Data Lake Data

  apidata-lakeduckdbfastapipolarsrest-apisqldeltalakelakehousecolumnar
  Language:Python 11

• gerardwolf / blog

  Repository for all blog scripts and code

  deltalakedatabricksadlsgen2
  Language:TSQL 7

• roeap / flight-fusion

  arrowdata-sciencedatafusiondeltalakeflight
  Language:Rust 7
• spark_benchmark

  satyakommula96 / spark_benchmark

  Spark Performance Benchmark suite to evaluate all TPC-DS and TPC-H query times

  sparkbenchmarkingscalatpchtpcdsapache-sparkdeltalakebenchmark-suitecsvparquetorc
  Language:Scala 5

• yandex-cloud / yc-delta

  Delta Lake для Yandex Data Processing

  deltadelta-lakedeltalakesparkyandex-cloud
  Language:Java 5

• xbrianh / xdlake

  A loose implementation of the deltalake protocol, written in Python on top of pyarrow, focused on extensibility, customizability, and distributed data.

  databricksdelta-lakedeltalakedeltatableshiveparquetpyarrowpythonspark
  Language:Python 4

• alberttwong / opendatalakehouse

  The Data Lakehouse Readiness Score is a quantitative measure that assesses a database's vendor support of Apache Iceberg, Apache Hudi and Delta Lake.

  apachehudiapacheicebergdeltalakeopentableformat
  3

• cmackenzie1 / deltalake-examples-rs

  Examples of working with the DeltaLake in Rust!

  delta-lakedeltalakedatafusionrust
  Language:Rust 3

• LeoneGarage / StreamJoin

  A framework for incremental streaming joins and incremental streaming aggregations over change data feeds from Databricks Delta

  databricksdeltalakestructured-streaming
  Language:Python 3