Giters

nodefluent / kafka-streams

equivalent to kafka-streams :octopus: for nodejs :sparkles::turtle::rocket::sparkles:

Home Page:https://nodefluent.github.io/kafka-streams/

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Contributing and Project Status

rob3000 opened this issue · comments

commented

Hi 👋

First of all thanks for this great package!

I was wondering what the development status of this project was and if there was plans to enhance some of the features such as the KStream <> KTable joins (inner, outer)? Would you be willing to allow PRs for this functionally and also if you had some ideas on how you'd like to see these features function?

Thanks again!

commented

Hi @rob3000 👋

I have not added any large updates to this package in the last months.
My focus has shifted from the Kafka eco-system to other stuff on AWS for a while now.
Nonetheless I would still love to see this package improve and I am sure glad to accept any PRs.

When looking at features especially the streaming DSL I always tried to implement them as lightweight as possible and built upon most.js operators https://github.com/nodefluent/kafka-streams/blob/master/docs/most-api.md

There have been a lot of requests in terms of KTable features, however getting these right under scale can be quite difficult, as for now these are not backed equally like Java kafka-streams handels them Message Key/Value based, backed by RocksDB and an off Kafka Topic. The kafka-streams js approach for KTables was built on key/value extraction from the message value stream, which is not 100% optimal - yet doable.

Also another thing that always quite bothered me was the quality of the Kafka Client's underneath, which is also why I spun up Sinek, which at least gave me a wrapper und different clients I had to replace on a regular basis.. for now this might have become easier with the new KafkaJS client - this also needs to be supported and possible streamlined in this package.

So mainly:

  • upgrading node-sinek once again
  • integrated latest version of node-sinke with KafkaJS as default client in kafka-streams
  • probably rewriting how kafka-streams handles key/value extraction for KTable
  • implement a proper backup-topic for every KTable operation, that is used to recreated KTables states when instances fail
  • add missing KTable features based on most.js operators as much as possible
  • enjoy a new released version of Kafka goodness 😄

BR,
Chris

commented

Hi @krystianity would you be willing do add extra maintainers to the project.
I have been using this library extensively and was looking if I can maybe do some simple PR's first, but if you are in different areas would you consider adding more maintainers?

I would love to fix documentation and build and then start with adding more features.

commented

Hi @wtrocki I am always open for PRs, makes sure to sync with @rob3000 though.
I am also fine with adding maintainers, if I see it going in the right direction - I have invited you two to the orga and main team.

commented

Thank you so much.
I will start with some simple issues over the coming days to get better understanding and help project.