apache-iceberg

There are 3 repositories under apache-iceberg topic.

• matano

  matanolabs / matano

  Open source security data lake for threat hunting, detection & response, and cybersecurity analytics at petabyte scale on AWS

  awscloudsecuritybig-dataserverlessapache-iceberglog-analyticslog-managementthreat-huntingrustalertingcloud-nativeaws-securitycloud-securitycybersecuritysecopssecurity-toolsdfirdetection-engineeringsiem
  Language:Rust 1447

• apache / incubator-xtable

  Apache XTable (incubating) is a cross-table converter for lakehouse table formats that facilitates interoperability across data processing systems and query engines.

  apache-hudiapache-icebergdelta-lake
  Language:Java 842

• cuebook / cuelake

  Use SQL to build ELT pipelines on a data lakehouse.

  apache-icebergdeltalakehousedatalakedata-lakeeltetldata-engineeringdata-integrationdata-ingestionapache-sparkspark-sqlupsertincremental-updatesdata-transferpipelinesdata-pipelinezeppelin-notebooksql
  Language:JavaScript 285

• lhbench / lhbench

  Lakehouse storage system benchmark

  apache-hudiapache-iceberglakehousebenchmarkcidrdatabasedatabricksdelta-lake
  Language:Scala 65

• dominikhei / Local-Data-LakeHouse

  Sample Data Lakehouse deployed in Docker containers using Apache Iceberg, Minio, Trino and a Hive Metastore. Can be used for local testing.

  apache-icebergdata-lakedata-lakehousehive-metastorelakehouseminiotrino
  Language:Dockerfile 51
• modern-data-lake-storage-layers

  dacort / modern-data-lake-storage-layers

  Jupyter notebooks and AWS CloudFormation template to show how Hudi, Iceberg, and Delta Lake work

  awsamazon-emrhudiicebergapache-hudiapache-icebergdelta-lake
  Language:Jupyter Notebook 47

• buster-so / buster-platform

  The open-source, AI-native data stack

  aianalyticsapache-icebergdbmslakehousestarrocksbusiness-intelligencedatadatabasewarehouse
  Language:HCL 32

• aws-samples / transactional-datalake-using-apache-iceberg-on-aws-glue

  Stream CDC into an Amazon S3 data lake in Apache Iceberg table format with AWS Glue Streaming and DMS

  apache-icebergaws-glueaws-dmsaws-athenaapache-spark
  Language:Python 24

• Bodo-inc / denali

  An open-source, community-driven REST catalog for Apache Iceberg!

  apache-icebergcataloggogolangiceberg
  Language:Go 24

• aws-samples / aws-glue-streaming-etl-with-apache-iceberg

  Streaming ETL job cases in AWS Glue to integrate Iceberg and creating an in-place updatable data lake on Amazon S3

  aws-glueapache-icebergaws-athenaapache-sparkaws-glue-streaming
  Language:Python 17

• tj--- / iceberg-demo

  A sample implementation of stream writes to an Iceberg table on GCS using Flink and reading it using Trino

  apache-flinkapache-icebergapache-kafkaflinkflink-stream-processingicebergkafkatrinogcsjava
  Language:Java 15

• tlepple / iceberg-intro-workshop

  Hands-on workshop with Apache Iceberg

  apache-icebergbig-datadelldell-object-storagelinuxminioobject-storagepysparksparkspark-sqlspark-sql-s3spark-streaming
  Language:Shell 13

• aws-samples / monitoring-apache-iceberg-table-metadata-layer

  Sample code to collect Apache Iceberg metrics for table monitoring

  apache-icebergawsaws-cloudwatchaws-glueaws-lambdadata-qualitymonitoringsam-cliapache-sparkpyiceberg
  Language:Python 12

• tlepple / data_origination_workshop

  Hands-on workshop with Iceberg, Redpanda, Debezium and Kafka-Connect

  apache-icebergdebeziumdebeziumkafkaconnectoricebergkafka-connectminiopostgresqlpysparkpythonredpandaredpanda-consolespark-streaming
  Language:Shell 12

• aws-samples / aws-glue-streaming-ingestion-from-kafka-to-apache-iceberg

  This is a collecton of Amazon CDK projects to show how to directly ingest streaming data from Amazon Mananged Service for Apache Kafka (MSK) and MSK Serverless into Apache Iceberg table in S3 with AWS Glue Streaming.

  aws-s3apache-icebergapache-kafkaaws-glue-streamingaws-mskaws-msk-serverlesspyspark
  Language:Python 10

• aws-samples / iceberg-streaming-examples

  This repo contains examples of high throughput ingestion using Apache Spark and Apache Iceberg. These examples cover IoT and CDC scenarios using best practices. The code can be deployed into any Spark compatible engine like Amazon EMR Serverless or AWS Glue. A fully local developer environment is also provided.

  apache-icebergapache-sparkstructured-streaming
  Language:Java 10

• YeonwooSung / MLOps

  Miscellaneous codes and writings for MLOps

  aiai-as-a-serviceawsllmllm-inferencellm-opsml-servingmlopsmultimodalbentomltriton-inference-serverapache-icebergdata-intensive-applicationsdockerkubernetessparkspark-nlpragvector-databasevectordb
  Language:Jupyter Notebook 10

• aws-samples / transactional-datalake-using-amazon-msk-serverless-and-apache-iceberg-on-aws-glue

  Stream CDC into an Amazon S3 data lake in Apache Iceberg format with AWS Glue Streaming using Amazon MSK Serverless and MSK Connect (Debezium)

  apache-icebergaws-glue-streamingaws-mskaws-msk-serverlessdebeziumkafkamsk-connect
  Language:Python 7

• gordonmurray / apache_flink_and_iceberg

  Using Apache Flink to write to s3 in Apache Iceberg format

  apache-flinkapache-icebergparquets3
  7
• lakehouse-poc

  fraibacas / lakehouse-poc

  Run an open-source data LakeHouse locally using Docker Compose

  apache-icebergapache-supersetdocker-composelakehouseprefect
  Language:Python 6

• davidvanegas2 / iceberg-s3-terraform-glue

  Automated setup of Apache Iceberg on Amazon S3 using Terraform and AWS Glue Data Catalog. Explore the power of a Lakehouse architecture for data management and analysis, featuring schema discovery, metadata management, and efficient querying with Amazon Athena.

  awsterraformapache-icebergdata-engineeringaws-lambdalakehouse
  Language:Python 5

• aws-samples / automation-of-building-a-transactional-data-lake

  apache-hudiapache-icebergdelta-laketransactional-data-lake
  Language:Python 4

• aws-samples / transactional-datalake-using-amazon-msk-and-apache-iceberg-on-aws-glue

  Stream CDC into an Amazon S3 data lake in Apache Iceberg format with AWS Glue Streaming using Amazon MSK and MSK Connect (Debezium)

  apache-icebergapache-sparkaws-athenaaws-glue-streamingdebeziumkafka-connectmysql
  Language:Python 3

• JesuFemi-O / iceberg-integration-framework

  A poc open framework to manage data ingestion into apache iceberg tables

  apache-iceberglakehouse-platformpyiceberg
  Language:Python 2

• joewood / react-iceberg

  React Components to visualize Apache Iceberg tables

  apache-icebergapache-sparkreactjsminios3avroapache-arrowdevcontainerdocker-compose
  Language:TypeScript 2

• MOBIN-F / iceberg-spark-tpcds-benchmark

  iceberg-spark-tpcds-benchmark

  apache-icebergapache-sparkicebergspark
  Language:Scala 2

• ev2900 / EMR_Studio_Iceberg

  Apache Icebery examples designed to be run on AWS Elastic Map Reduce (EMR) via. EMR Studio or EMR Notebooks

  apache-icebergawselastic-map-reduceemriceberg
  Language:Jupyter Notebook 1

• ev2900 / Iceberg_EMR_Athena

  Resources from an virtual tech talk / workshop - Set Up and Use Apache Iceberg Tables on Your Data Lake

  apache-icebergathenaawsemrspark
  Language:Jupyter Notebook 1

• j3-signalroom / apache_flink-kickstarter

  Examples of Apache Flink® applications showcasing the DataStream API and Table API in Java and Python, featuring AWS, GitHub, Terraform, and Apache Iceberg.

  apache-flinkflink-examplesflink-stream-processingflinkflink-kafkaapache-icebergicebergaws-parameter-storeaws-secrets-managergithub-actionssnowflaketerraform-cloudaws-s3
  Language:Java 1

• jordipuig37 / iceberg-schema-evolution

  A tool for learning Iceberg table format

  apache-icebergstreamlit
  Language:Python 1

• masterchief2007 / floeberg

  Experiments with Apache Iceberg

  apache-icebergjava
  Language:Java 1

• j3-signalroom / j3-techstack-lexicon

  J3's techStack Lexicon.

  apache-flinkflinkapache-icebergicebergterraformterraform-cloud
  0

• j3-signalroom / linux_flink_with_iceberg

  Apache Flink Docker image with Apache Iceberg support for Linux (i.e., non-Mac M1, M2, and M3 chips).

  apache-flinkflinkapache-icebergiceberg
  Language:Dockerfile

• j3-signalroom / mac_flink_with_iceberg

  Apache Flink Docker image with Apache Iceberg support for Mac M1, M2, or M3 chips.

  apache-flinkapache-icebergflinkiceberg
  Language:Dockerfile

• johnymontana / hands-on-havasu-geoparquet

  Notebook to accompany the "Hands-On With Havasu & GeoParquet" livestream

  apache-icebergapache-sedonageoparquetparquetsedonadb
  Language:Jupyter Notebook

• THeades / serverless-data-lakehouse

  This is an example project how to build a serverless data lakehouse on AWS using Terraform, Apache Iceberg and Spark.

  apache-icebergapache-sparkawsdata-engineeringdata-lakehouseterraform