DataExpert-io / data-engineer-handbook

This is a repo with links to everything you'd ever want to learn about data engineering

Repository from Github https://github.comDataExpert-io/data-engineer-handbookRepository from Github https://github.comDataExpert-io/data-engineer-handbook

Issue : Inconsistent Environment Variable Usage for Kafka Credentials in Flink Jobs

PrinceSajjadHussain opened this issue · comments

The aggregation_job.py and start_job.py files use slightly different methods to pass kafka credentials to the flink connector. start_job.py creates the sasl_config variable, which is formatted in a manner that is different from the approach taken by aggregation_job.py. The formatting of these values could break if the env vars contain special characters, or if the underlying flink library has a quirk. This may lead to connection issues with Kafka. The configuration should be consistent across files to reduce the risk of errors and improve maintainability.