Webfleet Kafka Connector
Introduction
This connector consumes messages from webfleet API (popQueueMessagesExtern) and publish to a Kafka topic.
Messages are produced using Schema of webfleet (Schema Registry is required).
You can check com.flixtech.kafka.WebfleetSchema
for more details on the Webfleet message Schema.
Parameters to be passed while starting the Webfleet Kafka connector:
Basic required parameters:
- topic: Output topic of Kafka (Sink)
- webfleet.api.endpoint_url: URL to connect to webfleet api
- webfleet.api.account: Account name to connect to webfleet api
- webfleet.api.user: User name to connect to webfleet api
Password related parameters:
We provide implementation to use credstash
for retrieving api password or
adding password directly as a parameter. Only one of the below options should
be used:
Credstash parameters:
- webfleet.api.password.credstash_table: Credstash table where the key is stored
- webfleet.api.password.credstash_key: Credstash secret key
Password as a parameter:
- webfleet.api.password: Password to connecto to webfleet api.
Example:
curl -i -X POST \
-H "Accept:application/json" \
-H "Content-Type:application/json" \
localhost:8083/connectors/ \
-d '{
"name": "<NAME OF CONNECTOR>",
"config": {
"connector.class": "com.flixtech.kafka.WebfleetSourceConnector",
"tasks.max": "1",
"topic": "<Output Kafka Topic>",
"webfleet.api.endpoint_url": "<ENDPOINT URL>",
"webfleet.api.account": "<ACCOUNT NAME>",
"webfleet.api.user": "<USER NAME>",
"webfleet.api.password": "<PASSWORD>"
}
}'
Note:
-
Webfleet API after each pull requires to be acknowledged per user. Thus having multiple tasks does not inherently make sense. Thus
task.max
is always defaulted to1
. -
Current implementation only support one user. If you want to use multiple user on multiple queues you can start multiple connectors.
-
Webfleet limits the number of pull and acknowledgment you can send per minute. Generally this limit is 10 per minute.