An implementation of Airbyte Protocol in Rust aiming to be simple, modular, extensible and resource efficient.
⭐⭐⭐Unstable and Work in progress⭐⭐⭐
Please try at your own risk. Feedback is much appreciated.
This is solely a personal project of Harsha Teja Kanna and not affiliated with Airbyte Inc. which owns Airbyte.
Note: This is NOT a Kubernetes Operator for Airbyte, only an
independent implementation of the Airbyte Protocol.
This is the partial mirror repository for an implementation of Airbyte Protocol data plane.
Why? Check the architecture reasoning here ADR
Why Rust? Check the decision record here ADR
Introduction blog: https://ekalavya.dev/why-and-how-i-implemented-airbyte-protocol-in-rust
Upcoming blog series here
Note: It will become the canonical repository once the implementation is ready(currently, in a rough state) for public
review and contributions.
- Standalone data
replication
Driverfrom AirbyteSourcetoDestination - Pluggable
Statemanagement layer - Pluggable progress
TrackingandMonitoring - Pluggable OCI Container/VM integration (containerd, Docker, Podman, Youki, Firecracker VM, ECS etc. )
- Multiple execution models (parallel/serial)
- Feature parity with official Airbyte Platform (
only data plane components )
- Mappers
- Validators
- Normalization/dbt? (Currently not in scope)
- Operators/Transformers? (Currently not in scope)
- Rust CDK (Connector Development Kit) 🚀
- Kubernetes Operator
- CLI
- Airbyte Desktop, Yes! 🚀
- Benchmarks
- Usage Examples
- KubeVela integration
- 100% Airbyte Protocol compatability and acceptance tests
- Automated tests
- CI
- Deployment and operations documentation
- Stable (GA) releases
- Setup SLSA3 Provenance for release artifacts.
- FAQs
- Strictly keep the driver binary to minimal size (ideally under 10MB). Very important 😃
- Start here for Quick Start on your machine.
- Start here for Replication Driver
- Start here for Kubernetes Operator
- Start here for CLI operations
- Documentation for existing and possible Integration ideas.
- Frequently asked questions (FAQs)
- https://h7kanna.medium.com/why-and-how-i-integrated-airbyte-and-apache-hudi-c18aff3af21a
- https://ekalavya.dev/why-and-how-i-implemented-airbyte-protocol-in-rust
- https://ekalavya.dev/let-us-copy-some-bytes-using-airbyte-replication-driver-in-rust
- https://ekalavya.dev/architecture-deep-dive-airbyte-replication-driver-in-rust
- https://ekalavya.dev/overview-of-airbyte-replication-operator-in-rust-use-cases
- https://ekalavya.dev/introduction-to-airbyte-replication-driver-in-rust-cli-operations
- https://ekalavya.dev/how-to-run-airbyte-replication-operator-kubevela-addon
- https://ekalavya.dev/overview-of-parallel-executions-in-airbyte-replication-driver
- https://ekalavya.dev/how-to-use-temporal-for-scheduling-airbyte-replication-driver
- https://ekalavya.dev/how-to-run-airbyte-replication-driver-using-podman
- https://ekalavya.dev/how-to-run-airbyte-replication-driver-using-youki
- https://ekalavya.dev/running-airbyte-replication-driver-on-amazon-ecs
- https://ekalavya.dev/running-airbyte-replication-driver-on-aws-fargate
- https://ekalavya.dev/firecracker-vm-for-airbyte-replications
- https://ekalavya.dev/how-i-integrated-airbyte-and-apache-hudi-again
