A Clojure library for Kafka new client API.
Development is against the 0.9.0 or newer release of Kafka.
(use 'clj-kafka-client.producer)
(def producer (kafka-producer {"bootstrap.servers" "localhost:9092"}))
(let [r (record "My-Topic" "Hello world") ; Build a record to send
p (send-record producer r)] ; Send the record to Kafka server and get a promise for this record
@p) ; Deref the promise to get the metadata when the record was acknowledged by server
The returned value of send-record
implements the java.util.concurrent.Future
interface. So if you like you can use it as a Future
object instead of a promise in clojure.
You can use kafka consumer API directly with a shallow clojure wrapper like:
(use 'clj-kafka-client.consumer)
(def consumer (kafka-consumer {"bootstrap.servers" "localhost:9092"
"group.id" "My-Group"}))
(subscribe consumer)
(loop []
(let [records (poll consumer)]
(doseq [record records]
(println record))))
But we recommend you to use high level API to consume Kafka topic. It helps you to do three dirty works. First, create a kafka consumer subscribing to a topic; Second, create a thread pool to process consumed messages asynchronously with the handler provided by you; Third, pause the assigned partition to prevent polling more messages when consumer can not finish processing all the polled messages during a session timeout. And calling poll periodically to ping Kafka server to keep alive during pausing time. Then finally resume polling messages when all the previous polled messages is processed.
The third part is important and fallible due to the method poll
in KafkaConsumer
does two things: ping the Kafka server to keep alive and poll new messages from Kafka server. If having not called poll
over a session timeout the Kafka server will think the consumer is dead and start a unnecessary partition rebalance which may lead to consume some messages more than once. But sometimes your handler maybe slow and can not process all the polled messages within a session timeout. In this scenario you must call pause
in KafkaConsumer
to let you call poll
only to ping Kafka server without polling any new messages.
The high level API is as follows:
(use 'clj-kafka-client.consumer)
(defn msg-handler [msg]
(println msg))
(def consumer (create-high-level-consumer {"bootstrap.servers" "localhost:9092"
"group.id" "My-Group"}
"My-Topic"
msg-handler
:worker-pool-size 10))
;; Close consumer
(close-high-level-consumer consumer)
Copyright © 2017 ylgrgyq
Distributed under the Eclipse Public License either version 1.0 or (at your option) any later version.