Awesome Kafka
A curated list of awesome articles, tutorials, courses, and resources for the Apache Kafka ecosystem.
Contributing
Contributions are very welcome!
- Fork the repo
- Star the repo
- Raise Pull Request to branch 'main'
- Make sure to check for duplicacy before raising pull request.
Thank you!
Table of Contents
- Articles
- UI / Cluster management
- Configurations
- Clients
- CLI / Tools
- Schema
- Security
- Testing
- Talks / Presentations
- Books
Articles
- Introduction to Kafka
- All Kafka tutorials by Confluent
- Apache Kafka, Samza, and the Unix Philosophy of Distributed Data
- Event Sourcing: What it is and why it's awesome
- Introducing Kafka Streams: Stream Processing Made Simple
- SSL Authentication with Apache Kafka
- Kafka Cheet Sheet
- Apache Kafka for Beginners
- Introduction to Stream Processing using Kafka Streams
- Understanding Kafka with Factorio
- Using logs to build a solid data infrastructure (or: why dual writes are a bad idea)
- Bottled Water: Real-time integration of PostgreSQL and Kafka
- Deploying Apache Kafka on AWS Elastic Block Store (EBS)
- How We Monitor and Run Kafka At Scale
- Monitoring Kafka performance metrics
- Kafka for uSwitch's Event Pipeline
- Next Gen Real-time Streaming with Storm-Kafka Integration
- All netflix tech blog kafka tagged articles
- All linked in tech blog kafka tagged articles
- How to use Apache Kafka to transform a batch pipeline into a real-time one
- Maven quick start for Kafka Connect connectors
- Kafka Listeners - Explained
UI / Cluster management
- Kafdrop - Kafdrop is a web UI for viewing Kafka topics and browsing consumer groups. The tool displays information such as brokers, topics, partitions, consumers, and lets you view messages.
- Kafka Eagle - A easy and high-performance monitoring system, as well as offsets or metadata and other kafka information.
- Klustr - Monitoring tool and graphic visualizer for Apache Kafka that helps you track key metrics about your cluster in real-time.
- Remora - Kafka consumer lag-checking application for monitoring, written in Scala and Akka HTTP; a wrap around the Kafka consumer group command.Integrations with Cloudwatch and Datadog.Authentication recently added.
- Kowl - Kowl is a modern Kafka Web UI for exploring Kafka messages, configurations and more with a focus on a good UI/UX. Written in Go & React.
- AKHQ - Kafka GUI for Apache Kafka to manage topics, topics data, consumers group, schema registry, connect and more.
Configurations
- kafka-gitops - Manage Apache Kafka topics and generate ACLs through a desired state file.
- Kafka Specs - Tool to ease and automate Apache Kafka cluster configuration management.
- kattlo - Kattlo gives a way to manage topics, ACLs, users, schemas, ksqlDB, employing a evolutionary configuration. Easy to understand, easy to write and easy to apply.
- Julie - An operational manager for Apache Kafka (Automation, GitOps, SelfService)
Clients
- kafkajs - A modern Apache Kafka client for node.js
- Kafka Rust - Rust client for Apache Kafka.
- librdkafka - The Apache Kafka C/C++ library.
- Franz-go - franz-go is a feature complete client written in native Go.)
- KafkaFlow - A .NET framework to create Kafka based applications, simple to use and extend.
CLI / Tools
- Kafkacat - Generic command line non-JVM Apache Kafka producer and consumer.
- Kafkacli - CLI and Go Clients to manage Kafka components (Kafka Connect & SchemaRegistry).
- Kafka Shell - ⚡A supercharged, interactive Kafka shell built on top of the existing Kafka CLI tools.
Schema
- Javro is an interface allowing you to write Avro schemas with ease, to validate it, and to preview it into JSON format with samples.
- Confluent Schema Registry - Confluent Schema Registry for Kafka.
- Karapace - Karapace (Aiven) is the open source schema registry you can run on your own platforms. Alternative (compatible replacement) for Confluent Schema Registry, supports Avro and JSON Schema.
Security
- Kafka-Security-Manager - Manage your Kafka ACL at scale.
- Kafka-Encryption - About Kafka End to End Encryption.
Testing
- Kafka Junit - This library wraps Kafka's embedded test cluster, allowing you to more easily create and run integration tests using JUnit against a "real" kafka server running within the context of your tests.No need to stand up an external kafka cluster!
- Kafka-Streams-Test-Utils - Testing Kafka Streams.
- Mocked Streams - Scala DSL for Unit-Testing Processing Topologies in Kafka Streams.
- Test Container for Apache Kafka - Testcontainers can be used to automatically instantiate and manage Apache Kafka containers.More precisely Testcontainers uses the official Docker images for Confluent OSS Platform.
Talks / Presentations
- Introduction to Apache Kafka by Joe Stein
- Apache Kafka and the Next 700 Stream Processing Systems by Jay Kreps
- Using Logs To Build a Solid Data Infrastructure - Martin Kleppmann (LinkedIn)
- I ♥ Logs: Apache Kafka and Real-Time Data Integration
- Apache Kafka: Real-time Streaming and Data Pipelines with Apache Kafka by Joe Stein
- Chris Curtin - AJUG - Apache Kafka
- Getting started guide on Apache Kafka by Learning Journal
- Developing with the Go client for Apache Kafka
- Verisign - Apache Kafka 0.8 basic training
- Developing Real-Time Data Pipelines with Apache Kafka
- Infrastructure at Scale: Apache Kafka, Twitter Storm & Elastic Search (ARC303) | AWS re:Invent 2013
- Real-time streaming and data pipelines with Apache Kafka
- Building a Real-time Data Pipeline: Apache Kafka at LinkedIn
- Kafka replication apachecon_2013
- Introduction and Overview of Apache Kafka
Books
- Making sense of stream processing
- Designing Data-Intensive Applications
- Effective Kafka - a hands-on guide to building robust and scalable event-driven applications
- Kafka: The Definitive Guide
- A fast-paced introduction to every aspect of working with Kafka you need to really reap its benefits.
- By the end of the book, you'll be ready to use Kafka Streams in your projects to reap the benefits of the insight your data holds quickly and easily.
- Event Streaming with Kafka Streams and ksqlDB