jlandersen / vscode-kafka

When a consumer is started, a new Kafka JS client instance is created

Line 102 in b72c127

this.kafkaClient = await createKafka(this.options);

which means:

a new Kafka instance is created on each consumer start
this Kafka instance is never disposed.

To fix this problem, we should use clientAccessor (which is a pooling of Kafka client) but it requires some refactoring like adding consumer method for Client API like we have for producer

vscode-kafka/src/client/client.ts

Line 85 in b72c127

producer(): Promise<Producer>;

.

Adding this consumer method to client will be consistent with producer method.

But a matter that we must clarify is about Client API structure. @jlandersen if I understand the Client API goal is to have an API which doesn't depend on kafkajs. But if I take https://github.com/jlandersen/vscode-kafka/blob/master/src/client/client.ts#L85 the method return a Producer structure which is a kafkajs structure.

The question is how to fix that:

keep like this?
redefine Producer structure?

Any feedback are welcome!

Unfortunately the coupling between Client and the KafkaJS already exists (since the switch to KafkaJS). We'll probably need non-trivial refactoring to decouple them.
I think we choose the pragmatic solution and merge this PR anyways, and deal with a potential refactoring after the next release.

Unfortunately the coupling between Client and the KafkaJS already exists (since the switch to KafkaJS). We'll probably need non-trivial refactoring to decouple them.

An idea that I had is to do like vscode html language service in https://github.com/microsoft/vscode-html-languageservice/blob/106b074df01923e579d8ebc2429090abd339f219/src/htmlLanguageTypes.ts#L18

The main idea is that htmlLanguageTypes import LSP types and export them. After that other files imported types coming from htmlLanguageTypes and not from LSP.

If we follow this idea we could import all kafkajs from client.ts and export them. After that other files like producer.ts, etc import ProducerRecord from client.ts and not from kafkajs. @fbricon what do you think about that?

But as you said, it should be nice to do that in an another PR.

Unfortunately the coupling between Client and the KafkaJS already exists (since the switch to KafkaJS). We'll probably need non-trivial refactoring to decouple them.

I created the issue #166

Use client accessor inside consumer