titalk / LakeSoul

A Table Structure Storage on Data Lakes to Unify Batch and Streaming Data Processing

CN Doc

LakeSoul

LakeSoul is a unified streaming and batch table storage for fast data processing built on top of the Apache Spark engine by the DMetaSoul team, and supports scalable metadata management, ACID transactions, efficient and flexible upsert operation, schema evolution, and streaming & batch unification.

LakeSoul implements incremental upserts for both row and column and allows concurrent updates on the same partition. LakeSoul uses LSM-Tree like structure to support updates on hash partitioning table with primary key, and achieve very high write throughput (30MB/s/core) on cloud object store like S3 while providing optimized merge on read performance. LakeSoul scales meta data management by using PostgreSQL DB.

More detailed features please refer to our wiki page: Wiki Home

Usage Documentations

Please find usage documentations in project's wiki: Usage Doc

使用文档

Follow the Quick Start to quickly set up a test env.

Checkout the CDC Ingestion with Debezium and Kafka example on how to sync LakeSoul table with OLTP dbs like MySQL in a realtime manner.

Feature Roadmap

• Meta Management (#23)
  • Multiple Level Partitioning: Multiple range partition and at most one hash partition
  • Concurrent write with auto conflict resolution
  • MVCC with read isolation
  • Write transaction through Postgres Transaction
• Table operations
  • LSM-Tree style upsert for hash partitioned table
  • Merge on read for hash partition with upsert delta file
  • Copy on write update for non hash partitioned table
  • Compaction
• Spark Integration
  • Table/Dataframe API
  • SQL support with catalog except upsert
  • Query optimization
    • Shuffle/Join elimination for operations on primary key
  • Merge UDF (Merge operator)
  • Merge Into SQL support
    • Merge Into SQL with match on Primary Key (Merge on read)
    • Merge Into SQL with match on non-pk
    • Merge Into SQL with match condition and complex expression (Merge on read when match on PK) (depends on #66)
• Flink Integration (#57)
  • Table API
  • Flink CDC
    • Exactly Once Sink
    • Auto Schema Sync
    • Auto Table Creation (depends on #78)
• Hive Integration
  • Export to Hive partition after compaction
• Realtime Data Warehousing
  • CDC ingestion
  • Time Travel (Snapshot read)
  • Snapshot rollback
  • MPP Engine Integration (depends on #66)
    • Presto
    • Apache Doris
• Native IO (#66)
  • Object storage IO optimization
  • Native merge on read
• Cloud Native
  • Multi-layer storage classes support with data tiering

Community guidelines

Community guidelines

Feedback and Contribution

Please feel free to open an issue or dicussion if you have any questions.

Join our slack user group

Contact Us

Email us at [e-mail].

Opensource License

LakeSoul is opensourced under Apache License v2.0.