[Feature] Kafka client integration
vidursatija opened this issue · comments
Hello Mozilla-IoT team,
I was thinking that having Kafka client integration in the gateway might be awesome. A Kafka client will allow the gateway to publish IoT data that later can be used for processing and storing.
I would love to contribute and have studied the gateway backend codebase fairly. I have some experience in using Kafka with Node.js.
With little guidance, I'll be able to make this idea into a feature.
How soon can I start with this?
I know very little about Kafka. Would this be something running inside the gateway? What data would be published? Where would the data be published?
@vidursatija
Actually, I'm already working on it 😬
I didn't think there was much interest in it because a Kafka cluster tends to be a bit heavy for small IoT cases.
I can publish it later.
@mrstegeman
It's a highly scalable message broker who can handle vast amounts of data.
I'm currently building an addon which mirrors the property changes to a Kafka topic.
@mrstegeman I think this would be running inside the gateway kinda like how the addons plugins-manager is working. But the Kafka client won't be an addon(according to me). @tim-hellhake does mention that he's mirroring the property changes which is a smart way to go(I didn't know we could do that). But on the other hand, mirroring adds an overhead of mirroring(lol) thus I thought it might be better to integrate it straight into the gateway. We can publish event data such as rule engine events and other IoT data such as electric meter data, other sensor data. We can configure 2 kinds of topics. One for publishing IoT device data and the other for events
@tim-hellhake That's so awesome, man! I do agree that the Kafka cluster tends to be quite heavy for small IoT cases. I was thinking of just making a client and not the broker. The client will connect to a broker in the cloud. How are you going about the addon? Can I help in any way?
@vidursatija
Yeah, that's true, in the cloud it wouldn't be a big deal.
I need to clean up a few things.
It would be very if you could test it.
@tim-hellhake let me know when you want me to test it.
Thank you so much for this addon!
@vidursatija
The kafka bridge is available in the addon list now.