3.0 release
bplommer opened this issue · comments
With Kakfa 3 having been released a while back and being a breaking change, I think we should plan to release fs2-kafka 3.0 in the next few months (say April or May) with whatever other improvements we get done by then. It might be good to start thinking now about what breaking changes we want to make before then.
I have some thoughts about 3.0.
This is the list of issues/tasks I want to include in 3.0:
- Consider moving commit method from CommitableOffset (and some others) to KafkaConsumer. Also in this issue I started a discussion about
CommittableOffsetBatchwith the support of multipleKafkaConsumers. It's a really important functionality. I faced an issue on production becauseCommittableOffsetBatchdoesn't support multiple consumers. Overall I want to say that commit-related stuff should be reworked in my opinion. - Rethink KafkaProducer/KafkaProducerConnection.
- Rework or remove CommitRecovery.
- Separate initialization effect in the constructors.
- New stream method with the full rebalance information. We could also backport this in 2.x branch.
There is also Refactor package structure. I have nothing against it, but it looks like a minor thing to me.
Besides the tasks I mentioned above I was thinking about KafkaConsumer's internals. At the current moment, these internals are pretty complex: we have aKafkaConsumer itself, a hand-maded KafkaConsumerActor, a queue for messages. Also, there is some complexity with different commit methods. And most of this complexity is to make all calls to a java consumer linearized. But actually, to achieve this we could use a simple Semaphore(1). Under the hood, this semaphore uses a queue (like an actor), but all low-level queue-related machinery is hidden. With this, we could dramatically reduce the complexity of library internals and make it more contributor-friendly. I think I have to create an issue about these thoughts.
So, that's how I see a perfect 3.0 release.
Unfortunately, due to my complex life and work situation (because of a political situation), I will not have time to work on these tasks. We could release 3.0 with some minimum improvements and all other stuff will be moved to 4.0.
I think the main thing that makes KafkaConsumerActor confusing is that it doesn't encapsulate its internals - the queues etc are instantiated in KafkaConsumer.scala and exposed there. If they were instantiated in KafkaConsumerActor.scala and enqueuing requests was done via a trait rather than by directly accessing the queue I think that would already simplify things quite a bit - and because this is all internal, we can do it on the 2.x branch.
and because this is all internal, we can do it on the 2.x branch.
Yes, that's true. We could do it in 2.0
About your suggestion — well, all you have said is true. But I think moving from actor to a semaphore would be better. This move will reduce more complexity. And also it could improve performance (less layers of abstractions = more performance).
I created a separate issue about simplification of consumer internals: #903. Let's continue to discuss this topic there.
Not from my side. Go on!
I might re-introduce support for Scala 2.12 in v3.x, to remove a reason for people to stay on v2.x - then we can drop it again for v4.x - what do you think?
If there is no obstacles on the way — let's do it.
Hey there, do we have any estimates for this release?
Thanks in advance!
Hey there, do we have any estimates for this release?
Thanks in advance!
Hi, sorry for the silence on this - it's now coming very soon!