Schema Registry and Bridge Deployment Scaling
djakupovic opened this issue · comments
Hi,
we need to make the http bridge compatible with the schema registry to ensure schema evolution and schema validity.
The config of the strimzi bridge is located at /tmp and the generating script at /opt/strimzi/bin/docker/kafka_bridge_config_generator.sh
BRIDGE_PROPERTIES=$(cat <<-EOF
#Bridge configuration
bridge.id=${KAFKA_BRIDGE_ID}
${BRIDGE_TRACING}
EOF
)
The id is fixed, so if we scale the spec replica the id stays the same for each replica. Does this need to be a unique id for each consumer?
We need more settings to set "schema.registry.url"
The schema.registry.ssl.* settings could be set with another loop
if [ "$KAFKA_SCHEMA_REGISTRY_TLS" = "true" ]; then
SCHEMA_REGISTRY_CONFIGURATION=$(cat <<EOF
#SCHEMA REGISTRY
schema.registry.ssl.key.password=
schema.registry.ssl.keystore.location=
schema.registry.ssl.keystore.password=
schema.registry.ssl.truststore.location=
schema.registry.ssl.truststore.password=
EOF
)
if http/amqp is disabled, do not print its configuration
PROPERTIES=$(cat <<EOF
$BRIDGE_PROPERTIES
$SCHEMA_REGISTRY_CONFIGURATION
...
EOF
)
Why do you think setting the schame regitry options is enough? I would expect it to be more complicated. I don't follow your point about the bridge.id it is as unique as you set it through the environment variable + it is anyway configured by the Strimzi Operator directly.
@scholzj
Isnt it the base image of confluent? If so, the setting is enough. You can play with the bridge and send a schema and the bridge will accept the POST body with the schema. Therefore setting the schema registry options should be enough.
Yes but the strimzi operator sets always the same id. Confluent documentation about the id:
id
Unique ID for the Confluent REST Proxy server instance. This is used in generating unique IDs for consumers that do not specify their ID. The ID is empty by default, which makes a single server setup easier to get up and running, but is not safe for multi-server deployments where automatic consumer IDs are used.
Type: string
Default: “”
Importance: high
No, Strimzi has nothing to do with Confluent. It is a CNCF open source project. This is the Strimzi Kafka Bridge which is our own bridge.
@scholzj Ok thank you for the reply, I assumed its a confluent rest proxy fork
I have then two wishes on the bridge :D
- Implementing the ApicurioRegistry for schema validation and evolution like the REST Proxy of confluent
- Being able to have several instances (consumers) to handle load on different consumers
e.g.
POST http://{{HOST}}/consumers/{{CONSUMER_GROUP}}
{
"name": "Instance1",
"format": "json",
"enable.auto.commit": false
...
}
POST http://{{HOST}}/consumers/{{CONSUMER_GROUP}}
{
"name": "Instance2",
"format": "json",
"enable.auto.commit": false
...
}
Schema registry is covered in #460. So it does not need another issue. As for the second point, I don't follow what you are asking for. Please explain the use-case and why does it not work today.
Use Case: Scaling the bridge
1 consumer group called group1
N consumers (different servers) with name consumer1 ... consumerN
each consumer creates an instance described like above
after each GET from a consumer a rebalance should occur to match the proper instance/consumer on the proxy to read from 1 or N topics.
Currently not possible with a heartbeat and rebalance failure
2022-08-30 08:36:00 INFO AbstractCoordinator:1123 - [Consumer clientId=instance1, groupId=group1] Attempt to heartbeat failed since group is rebalancing
I'm not sure how this helps with scalability. Also without the full logs it is not clear what the actual issue is. If you want two consumers use the same group, there might be rebalances which stop them, that is how the Kafka client works.
@djakupovic anything more to add to what Jakub was asking for? or we can close this one?
@djakupovic after a long time of inactivity I am going to close this one. Let us know if you want to reopen and still need help.