This Kafka Connect connector for Zeebe can do two things:
-
Send messages to a Kafka topic when a workflow instance reached a specific activity. Please note that a
messageis more precisely a kafkarecord, which is also often namedevent. This is a source in the Kafka Connect speak. -
Consume messages from a Kafka topic and correlate them to a workflow. This is a Kafka Connect sink.
It can work with Camunda Cloud or a standalone Zeebe broker.
See this blog post for some background on the implementation.
The following video walks you through an example connecting to Camunda Cloud:
You will find information on how to build the connector and how to run Kafka and Zeebe to get started quickly here:
The plugin comes with two connectors, a source and a sink connector.
The source connector activates Zeebe jobs, publishes them as Kafka records, and completes them once they have been committed to Kafka.
In a workflow model you can wait for certain events by name (extracted from the payload by messageNameJsonPath):
The sink connector consumes Kafka records and publishes messages constructed from those records to Zeebe. This uses the Zeebe Message Correlation features. So for example if no matching workflow instance is found, the message is buffered for its time-to-live (TTL) and then discarded. You could simply ingest all messages from a Kafka topic and check if they correlate to something in Zeebe.
In order to communicate with the Zeebe workflow engine, the connector has to create a Zeebe client.
If you want to connect to Camunda Cloud, you can use these properties:
zeebe.client.cloud.clusterId: Cluster ID you want to connect to. The Cluster must run on the public Camunda Cloudzeebe.client.cloud.clientId: Client ID for the connection. Ideally, create dedicated client credentials for this communication using the Camunda Cloud Console.zeebe.client.cloud.clientSecret: The Client Secret requiredzeebe.client.requestTimeout: timeout in milliseconds for requests to the Zeebe broker; defaults to10000(or 10 seconds)
If you want to connect to a Zeebe broker hosted yourself (e.g. running on localhost), use these properties:
zeebe.client.broker.contactPoint: the Zeebe broker address, specified ashost:port; defaults tolocalhost:26500zeebe.client.requestTimeout: timeout in milliseconds for requests to the Zeebe broker; defaults to10000(or 10 seconds)zeebe.client.security.plaintext: disable secure connections to the gateway for local development setups
For client and job worker configuration, we reuse the system properties as used by Zeebe, so if you already have a properties file for those they should simply work.
The connector does not yet support schemas, and currently expect all records to be JSON. Therefore, in order to properly construct a message, we use JSON path to extract properties from the Kafka record. These paths are all configurable per connector.
message.path.correlationKey: JSONPath query to use to extract the correlation key from the record; defaults to$.correlationKeymessage.path.messageName: JSONPath query to use to extract the message name from the record; defaults to$.messageNamemessage.path.timeToLive: JSONPath query to use to extract the time to live from the record; defaults to$.timeToLivemessage.path.variables: JSONPath query to use to extract the variables from the record; defaults to$.variables
You can find sample properties for the sink connector here.
Similar to receiving a message, in a workflow model you can also throw messages (i.e. the message throw event). Zeebe does not yet support this BPMN feature; what we can do to allow communication with external systems through Kafka, however, is use service tasks.
In a workflow you can then add a ServiceTask with a configurable task type which will create a record on the configured Kafka topic:
Under the hood, the connector will create one job worker per configured task type, consume their jobs, and publish records to Kafka based on those. As we do not yet support schemas, the record values are a JSON representation of the job itself, and the record keys are the job key.
In order to communicate with Zeebe, the connector has to create a Zeebe client, which must be configured with the following properties:
zeebe.client.broker.contactPoint: the Zeebe broker address, specified ashost:port; defaults tolocalhost:26500zeebe.client.requestTimeout: timeout in milliseconds for requests to the Zeebe broker; defaults to10000(or 10 seconds)
For client and job worker configuration, we reuse the system properties as used by Zeebe, so if you already have a properties file for those they should simply work.
zeebe.client.worker.maxJobsActive: the maximum number of jobs that the worker can activate in a single request; defaults to100zeebe.client.job.worker: the worker name; defaults tokafka-connectorzeebe.client.job.timeout: how long before a job activated by the worker is made activatable again to others, in milliseconds; defaults to5000(or 5 seconds)job.types: a comma-separated list of job types that should be consumed by the connector; defaults tokafkajob.header.topics: the custom service task header which specifies to which topics the message should be published to; defaults tokafka-topic
You can find sample properties for the source connector here.
You can filter the variables being sent to Kafka by adding a custom header to the "sendMessage" task with the configuration option "job.variables".
Set the value of this key to a comma-separated list of variables to pass to Kafka.
If this custom header is not present, then all variables in the scope will be sent to Kafka by default.
Kafka Connect allows you to configure what happens if a message cannot be processed. A great explanation can be found in Kafka Connect Deep Dive – Error Handling and Dead Letter Queues. This of course also applies to this connector.
This project is set up to be released on Confluent Hub.
When
- Building this project via
mvn package - You will find the plugin package as ZIP file under
target/components/packages, e.g.target/components/packages/zeebe-io-kafka-connect-zeebe-1.0.0.zip - Which can be installed onto the Confluent Hub