Barco is a lightweight, elastic, kubernetes-native event streaming system. It acts as a persistent buffer between services, supporting peaks by seamlessly scaling and allowing events to be consumed at a later time from the peak.
- Limited memory footprint: Memory usage can be capped at a few hundreds of Mb
- Production workloads with 0.5GiB of memory per pod
- 20K durable writes per second with 3
t3.nanobrokers - Small binary size, arm64 as first class citizen
- No additional dependencies, i.e., no Zookeeper
- New brokers can be started up in seconds, join the cluster and receive new data
- No need to operate it manually for scaling up/down cluster
- After a period of low usage, it scales down automatically
- Elastic in both computing and storage
- Basic setup using StatefulSets
- Volume setup, scaling, rolling upgrades is managed from Kubernetes
- Data distribution is K8s-aware: data placement based on the StatefulSet's pod ordinal
- Good K8s neighbor: Direct I/O, no OS page cache for log segments
⚠️ The project is still in early development ⚠️
Barco Streams is not production ready, expect bugs and things that don't work.
We honestly value your contribution to make this project ready for general availability. If you want to contribute, check out the Issues section on GitHub. The label "good first issue" marks tasks that are beginner-friendly.
Events are organized in topics. Topics in Barco are always multi-producer and multi-consumer. To achieve high availability and durability, topic events are persisted on disk on multiple Barco brokers.
Data is automatically distributed across brokers using consistent hashing in a similar way as Amazon DynamoDB and Apache Cassandra. Each broker is assigned a token based on the ordinal index within the cluster, which will be used to determine the data that naturally belongs to that broker.
Read the technical introduction in our documentation for more information.
You can install Barco on Kubernetes using kubectl by using our kustomize base.
Follow our guide to install on K8s.
You can use docker / docker compose to run Barco for application development and CI.
Read this guide to run it using docker compose.
Barco expects the cluster to be composed by multiple brokers. To simplify application development, we introduced "dev mode" setting to run a single-broker cluster for application development.
After building Barco, set BARCO_DEV_MODE environment variable before running:
BARCO_DEV_MODE=true BARCO_HOME=./barco-data go run .go build .
go test -v ./...The Contributing Guide has more information about environment setup and other topics that can help you to build Barco from source.
Persistent event streaming should act as a buffer between services, supporting peaks by seamlessly scaling, allowing events to be consumed at a later time from the peak. Once a certain amount of events have been produced and not consumed, the following events can be stored in object storage to be pulled once the consumers catch up, reducing costs for storage.
Message streaming should be highly available by default and only consume the minimum required resources to run effectively.
Regular maintenance operations like upgrading the cluster or decommissioning Kubernetes nodes are common and should be handled smoothly. For example, when a consumer is being restarted due to an upgrade, there's no need for re-balance the data assignment among consumers as it's very likely that the consumer will be ready in a few seconds. Querying the Kubernetes API can provide valuable insights to understand what is occurring and what to expect.
The system should elastically scale intelligently, to support the future web scale and without affecting running services.
We are always happy to have contributions to the project whether it is source code, documentation, bug reports, feature requests or feedback. To get started with contributing:
- Have a look through GitHub issues labeled "Good first issue".
- Read the contribution guide.
- See the build instructions, for details on building Barco.
- Create a fork of Barco and submit a pull request with your proposed changes.
Copyright (C) 2022 Jorge Bay
This program is free software: you can redistribute it and/or modify it under the terms of the GNU Affero General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.
This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Affero General Public License for more details.