Expose Kafka Producer and Consumer Configuration
sglover opened this issue · comments
There are many configuration settings for Kafka Producers and Consumers (see https://kafka-python.readthedocs.io/en/master/apidoc/KafkaProducer.html). Currently, Ignition does not allow any of these to be configured on a per-deployment basis. Ignition should allow these to be set through its normal configuration mechanism, ideally as a set of properties in the configuration that can be used verbatim to configure the Producers and Consumers.
Kafka Producer, Consumer and AdminClient configuration can now be configured as follows in the configuration file:
messaging:
# Kafka connection url
connection_address: foundation-kafka:9092
config:
# timeout waiting for initial version check on Kafka producer/consumer initialisation
# 5000ms is usually sufficient, increase if problems with NoBrokersAvailable occur
api_version_auto_timeout_ms: 5000
Any properties under messaging.config
will be used to configure KafkaProducer, KafkaConsumer and KafkaAdminClient instances.