Giters

streamnative / kop

Kafka-on-Pulsar - A protocol handler that brings native Kafka protocol to Apache Pulsar

Home Page:https://streamnative.io/docs/kop

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

[FEATURE] Add a default protocol configuration for the unknown host

KamenRiderKuuga opened this issue · comments

commented

Is your feature request related to a problem? Please describe.
In our company's internal environment, we use LAN IP to connect to KoP, sometimes we need to proxy this address to the external network address, so that external developers can connect to our KoP.

Due to the external address must be set in kafkaAdvertisedListeners as first, so every time we adjust the address of the proxy server, we need to adjust the KoP configuration at the same time.

Describe the solution you'd like

After configuring the proxy, the external environment can actually access the KoP server, but the protocol corresponding to the address it uses cannot be found.

Because the configuration of Pulsar cannot be hot-updated, and in this case, there is actually no need to update the configuration frequently. It is hoped that there will be a default protocol configuration for unknown hosts to use.

kafkaDefaultProtocol=PLAINTEXT

In this way, no matter what address is used, as long as it can successfully connect to the server, the default protocol will be used

Describe alternatives you've considered
Maybe we can use 0.0.0.0 to represent all other IP addresses?

kafkaAdvertisedListeners=PLAINTEXT://0.0.0.0:9092
commented

In the sister project/fork https://github.com/datastax/starlight-for-kafka I am maintaining a Proxy component that allows you to let external clients to connect to KOP.
It is 100% compatible with KOP (so you can keep KOP on the brokers) is a proxy extension

commented

@eolivelli Recently there is an internal discussion in SN that whether should introduce your proxy solution again. As you know, the Pulsar proxy based solution was rejected long days ago. Would you mind us porting the proxy back to KoP?

Though at the moment we didn't reach the consensus internally, @ericsyh was investigating the proxy handler based proxy solution recently.

commented

@eolivelli Thanks, in the Starlight for Kafka proxy extension I find this:

kafkaAdvertisedListeners must contain the public address that clients will use to connect to the proxy. In the example above we are using pulsar-proxy:9092, but this address is available only inside the Kubernetes cluster. If you are exposing your service outside of the Kubernetes cluster, you must use the public name.

I'm not sure if this can achieve what I mentioned above, let me try.