Erlkaf as an optional kafka client

[Rafael Barbosa]

Hello there,

Would it be possible (and desireable) to offer an option to use erlkaf instead of brod as the underlying kafka client? We had to move to erlkaf from brod after some issues in our infrastructure and had to let go of broadway for that 😢

I can try to come up with a PR for that if there's no problem.

I'm sorry if this is the wrong place to ask such questions, you can close the issue if that's the case.

[José Valim]

Hi @escobera! It may make sense but at the moment I am not sure if it is better as a separate library or by encapsulating the different behaviours in a client module. If you give it a try, please let us know how it goes. :)

[Rafael Barbosa]

Oh, I see what you mean.

I noticed some of the main modules (i.e. BroadwayKafka.KafkaClient and BroadwayKafka.Producer) depend explicitly on :brod types and/or behaviours. And since brod's and erlkaf's api differs violently this would be much harder to achieve then just creating another client module. I don't think I'm able to code this in a timely fashion, so I'll give it a try on my free time but will close this as not to polute the issue list.

Thanks @josevalim =)