MemQ: An efficient, scalable cloud native PubSub system
MemQ is a new PubSub system that augments Kafka at Pinterest. It uses a decoupled storage and serving architecture similar to Apache Pulsar and Facebook Logdevice; however, it relies on a pluggable replicated storage layer i.e. Object Store / DFS / NFS for storing data. The net result is a PubSub system that:
- Handles GB/s traffic
- Independently scales, writes, and reads
- Doesn’t require expensive rebalancing to handle traffic growth
- Is 90% more cost effective than our Kafka footprint
Getting Started
Checkout our Getting Started Guide
Usage
MemQ can be a valuable PubSub choice for the following use case families:
- Large scale data ingestion
- Bulk uploads
- Near Real Time Analytics
Design
Detailed architecture docs can be found here
Blog
Please checkout the original MemQ blog for details and background.
Authors
See contributors
License
MemQ distributed under Apache License, Version 2.0