.kfk.Metadata lists a topic even if it's deleted using .kfk.TopicDel
rpoply1 opened this issue · comments
Deleting a topic using .kfk.TopicDel doesn't remove that from the list of topics returned by .kfk.Metadata
client: .kfk.Consumer[`metadata.broker.list`group.id!`localhost:9092`0];
.kfk.Metadata[client]`topics;
topic: .kfk.Topic[client; `new_topic; ()!()];
.kfk.Metadata[client]`topics;
.kfk.TopicDel topic;
.kfk.Metadata[client]`topics;
```
Hi @rpoply1
As a general point, This appears to be somewhat of a limitation on the broker side as alluded to here. Although that's not at the heart of what I think the behavioural mismatch is here.
We're not really controlling the removal of the topic from the brokers perspective which is the information returned from .kfk.Metadata
rather the function .kfk.TopicDel
is removing our control over the global list of topics that your q session has knowledge of.
For context .kfk.Topic
creates a new topic associated with the client through invocation of the function rd_kafka_topic_new
on librdkafka side which we map to a list of topics that your session has knowledge of, the following code snippet alludes to this
rkt= rd_kafka_topic_new(rk, y->s, rd_topic_conf);
js(&topics, (S) rkt);
return ki(topics->n - 1);
Essentially this is the reason you're doing the assignment of topic
in the call to .kfk.Topic[client;`new_topic;()!()]
and using this down the line
In essence the deletion function is destroying the reference mapping the topic handle you created and added to the kdb topics
list in C. It does this by both destroying the client from the point of view of your client and removing any association with kafka to the kdb topic
list
rd_kafka_topic_destroy(rkt);
kS(topics)[x->i]= (S) 0;
We'll look into if there is something we can invoke to remove the topics from broker side but this would be an addition to the interface rather than an overhaul of the above functions from what I'm aware at this juncture.
This is done with a script:
./kafka-topics.sh --bootstrap-server localhost:9092 --topic test1 --delete
Note: that "if you have consumers up and running is that the topic will get auto-created if the cluster-wide property auto.create.topics.enable is true". Actually this property is necessary to create a topic with .kafka.newTopic