No platform events produced with standalone Kafka consumers
haeniya opened this issue · comments
Describe the bug
We are not seeing any EntityChangeEvent_v1 events being triggered in actions (e.g. the Hello World action).
It seems that the messages are not even produced for the platform event topic. Messages for other topics e.g. the metadata change log topic are produced and consumed successfully. I can see debug log messages in GMS about the KafkaEventProducer.java producing MetadataChangeLogs but there's nothing about platform events (e.g. method https://github.com/datahub-project/datahub/blob/e6d706687ab5af6b888176bb775c338ff9[…]java/com/linkedin/metadata/dao/producer/KafkaEventProducer.java).
After some investigation we think it's related to running standalone Kafka consumers. All the required ENV vars (PE_CONSUMER_ENABLED, MAE_CONSUMER_ENABLED) are set correctly and visible on the pods. Topics and consumer groups are set up correctly and the ENV vars referring to the topics also seem to be correct.
Switching back to integrated consumers shows EntityChangeEvent_v1 events being triggered again. Changing back to standalone consumers by setting datahub_standalone_consumers_enabled to true in the Helm values brings back the issue.
We're using the datahub Helm chart version 0.3.8 and Datahub 0.12.0
More information in this Slack thread: https://datahubspace.slack.com/archives/C029A3M079U/p1707290303707169?thread_ts=1653649465.225869&cid=C029A3M079U
@david-leifker already looked into this and if I understood correctly, he was able to reproduce the issue.
To Reproduce
Steps to reproduce the behavior:
- Switch to standalone consumers by setting
datahub_standalone_consumers_enabledtotruein the Helm values - Run the Hello World action
- Add an owner to a dataset an check in the action logs if the EntityChangeEvent_v1 was logged
- See that no platform event was triggered
Expected behavior
Action should log an EntityChangeEvent_v1 event.
