tulios / kafkajs

A modern Apache Kafka client for node.js

Home Page:https://kafka.js.org

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

How to consume messages from multiple topics

Torchu opened this issue · comments

I'm consuming messages from different topics and for each subgroup of topics I want to process the messages in a different way.
Example:

const consumer = kafka.consumer({ groupId: 'my-group' });
await consumer.connect();
await consumer.subscribe({ topics: ['topic-A', 'topic-B', 'topic-C'] });
await consumer.run({
    eachMessage: async ({ topic, partition, message, heartbeat, pause }) => {
        if (topic === 'topic-A') {
            myFuncA(message);
        } else {
            myFuncB(message);
        }
    }
});

Is this the proper way to do it or should I create 2 different consuming groups: one for 'topic-A' and another one for 'topic-B' and 'topic-C' ?

yes, this is proper way how to do this. If you create multiple consumer groups - it will create 3x connections and 3x more requests to Kafka.