Enable the development of custom consumers

[Andrei Nesterov]

It is currently impossible to develop a custom consumer based on BaseConsumer because its queue property, which is necessary to receive notifications about new incoming messages, is private.

We can define set_nonempty_callback method on BaseConsumer similarly to how it has already been done for PartitionQueue. That will allow setting rdkafka_sys::rd_kafka_queue_cb_event_enable callback from within a custom consumer implementation.

Here is implementation for the proposal.

[Andrei Nesterov]

@benesch I'm sorry to bother you, but you seems to be the only active owner of the repo. There are two (1 & 2) issues that prevent us from using the library. Both have PRs, but they seem to be stuck on review process. Could you please take a look?

[Nikhil Benesch]

I’ve largely retired from maintaining this project due to time constraints—so sorry! @davidblewett and some other folks from Datadog have largely taken over maintenance. Hopefully one of them has time to jump in soon.

[Andrei Nesterov]

Noted, thank you for responding.

[Andrei Nesterov]

@davidblewett Can you please have a look at this?