LakeSoul
LakeSoul is a cloud-native Lakehouse framework developed by DMetaSoul team, and supports scalable metadata management, ACID transactions, efficient and flexible upsert operation, schema evolution, and unified streaming & batch processing.
LakeSoul implements incremental upserts for both row and column and allows concurrent updates. LakeSoul uses LSM-Tree like structure to support updates on hash partitioning table with primary key, and achieve very high write throughput (30MB/s/core) on cloud object store like S3 while providing optimized merge on read performance. LakeSoul scales metadata management and achieves ACID control by using PostgreSQL. LakeSoul provides tools to ingest CDC and log streams automatically in a zero-ETL style.
More detailed features please refer to our doc page: Documentations
Quick Start
Follow the Quick Start to quickly set up a test env.
Tutorials
Please find tutorials in doc site: Tutorials
- Checkout LakeSoul Flink CDC Whole Database Synchronization Tutorial on how to sync an entire MySQL database into LakeSoul in realtime, with auto table creation, auto DDL sync and exactly once guarantee.
- Checkout Multi Stream Merge and Build Wide Table Tutorial on how to merge multiple stream with same primary key (and different other columns) concurrently without join.
- Checkout Upsert Data and Merge UDF Tutorial on how to upsert data and Merge UDF to customize merge logic.
- Checkout Snapshot API Usage on how to do snapshot read (time travel), snapshot rollback and cleanup.
- Checkout Incremental Query Tutorial on how to do incremental query in Spark in batch or stream mode.
Usage Documentations
Please find usage documentations in doc site: Usage Doc
Feature Roadmap
- Meta Management (#23)
- Multiple Level Partitioning: Multiple range partition and at most one hash partition
- Concurrent write with auto conflict resolution
- MVCC with read isolation
- Write transaction (two-stage commit) through Postgres Transaction
- Schema Evolution: Column add/delete supported
- Table operations
- LSM-Tree style upsert for hash partitioned table
- Merge on read for hash partition with upsert delta file
- Copy on write update for non hash partitioned table
- Compaction
- Data Warehousing
- Spark Integration
- Table/Dataframe API
- SQL support with catalog except upsert
- Query optimization
- Shuffle/Join elimination for operations on primary key
- Merge UDF (Merge operator)
- Merge Into SQL support
- Merge Into SQL with match on Primary Key (Merge on read)
- Merge Into SQL with match on non-pk
- Merge Into SQL with match condition and complex expression (Merge on read when match on PK) (depends on #66)
- Multiple Spark Versions Support
- Support Spark 3.3, 3.2 and 3.1
- Flink Integration and CDC Ingestion (#57)
- Table API
- Stream Sink
- Batch Sink
- Batch/Stream source
- Stream Source as ChangeLog Stream
- Flink CDC
- Exactly Once Sink
- Auto Schema Change (DDL) Sync
- Auto Table Creation (depends on #78)
- Support multiple source tables with different schemas (#84)
- Table API
- Hive Integration
- Export to Hive partition after compaction
- Apache Kyuubi (Hive JDBC) Integration
- Realtime Data Warehousing
- CDC ingestion
- Time Travel (Snapshot read)
- Snapshot rollback
- MPP Engine Integration (depends on #66)
- Presto
- Apache Doris
- Cloud and Native IO (#66)
- Object storage IO optimization
- Native merge on read
- Multi-layer storage classes support with data tiering
Community guidelines
Feedback and Contribution
Please feel free to open an issue or dicussion if you have any questions.
Join our slack user group
Contact Us
Email us at opensource@dmetasoul.com.
Opensource License
LakeSoul is opensourced under Apache License v2.0.