ArroyoSystems / arroyo

Distributed stream processing engine in Rust

Home Page:https://arroyo.dev

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Add support for confluent schema basic auth

mwylde opened this issue · comments

Confluent cloud uses basic auth for authentication with the confluent schema registry. Currently, we do not allow setting this when fetching schemas from the schema registry.

Since this involves authentication details, it should probably be part of the connection profile configuration.

Hello @mwylde ,

I would like to give it a shot. Is this feature still in demand? If yes, could you please guide me on where to start?
For instance:

  1. What is the expected outcome?
  2. Which related components/files/directories should I explore first?
  3. I am new to computing engines, so any guidance you could provide would be immensely helpful.

My understanding is as follows:
A user (producer) registers a schema in the Confluent Schema Registry -> The producer begins to serialize data and send it to Kafka (or other types of storage/message queues) -> The Arroyo connector in arroyo_connectors retrieves the schema from the Schema Registry on Confluent Cloud -> The connector deserializes the byte-formatted data -> ...

My expected outcomes:

  1. Update create_connection_profile API to include basic information.
  2. In the kafka connector, retrieve the connection_profile, check if we need to fetch the schema from Confluent, and add basic authentication if necessary.

Any guidance would be greatly appreciated.

This has been completed with #416