Example source connector which sends Wikipedia edits into Kafka. There are two implementations:
WikiSourceConnector
/WikiSourceTask
which is a all-in-one solution: all the logic is in the taskWikiSourceTextConnector
/WikiSourceTextTask
which only copies records from SSE; the filtering on data records and Wiki language is delegated to Single Message Transforms.
This second example will store the edits in AVRO format; the AVRO schema is autogenerated and stored in the schema registry which is provided in the Docker compose.
The JSON payloads to install the plugins are in src/test/resources
and show how the connectors and transforms are
configured.
- Java 11 or up
- Docker desktop
jq
(optional, but convenient for formatting json output)kafkacat
(orkcat
if you are on a Mac; can be installed by Homebrew)
Instructions in telegram style:
- Build the project
Check that
mvn clean verify
target/connect-plugin
contains a-jar-with-dependencies.jar
which will contain the plugins. - Start the docker compose
docker-compose up # or 'docker compose up', depending on your docker desktop version
- Verify that the connect plugins are found, in another terminal
You should see
curl localhost:9001/connector-plugins | jq
org.tonvanbart.wikipedia.connect.WikiSourceConnector
andorg.tonvanbart.wikipedia.connect.WikiSourceTextConnector
among the listed classes. - Try to create the connector
curl -X POST -H "Content-Type: application/json" -d @src/test/resources/add-source-connector.json localhost:9001/connectors | jq
- Check if messages are produced; updates to be published will be logged at DEBUG level in the
bitconnect
container log:You can usedocker logs bitconnect -f
kafkacat
(kcat
) to check the topic:kafkacat -b localhost:9092 -C -t wikipedia_edits
- To create the "text only" connector and its transform chain, post the second payload:
In this case, open the included Kowl in a browser on http://localhost:8080 - the "schema registry" tab should show a schema
curl -X POST -H "Content-Type: application/json" -d @src/test/resources/add-source-text-connector.json localhost:9001/connectors | jq
wikievents-value
being registered, on the "topics" tab you should see a topicwikievents
with some update events.
Note: schema registration won't happen until an event is received; if you picked a quiet Wiki this might take a few soconds!
Messages produced to Kafka by WikiSourceConnector
are JSON formatted and contain the old and new page size (note: can be 0 in some cases),
the username of the editing user, Wikipedia page title and edit comment and an indication if the user is a bot.
See example payload below:
{
"bot": false,
"sizeOld": 5139,
"sizeNew": 5163,
"timestamp": "2023-09-19T21:43:55Z",
"user": "Guss",
"title": "Gebruiker:Guss/Kladblok",
"comment": "/* Ontspoorde vriendschap / Beschuldiging van misbruik (titel??) */Stijl"
}
The wiki language to follow, and topic to produce to can be configured.
property | meaning | default |
---|---|---|
wiki.language |
Two letter language code of the wiki to follow | nl |
target.topic |
Topic where wiki edits are produced to | none |
For convenience, there is a Makefile which can perform some of the curl
commands for you:
Command | result |
---|---|
make add-connector |
installs the compiled connector |
make log-debug |
set log level to DEBUG for the connector classes |
make log-info |
set log level to INFO for the connector classes |
make update-config |
Update connector config using values in update-config.json |
make list-status |
list connector status (needs jq ) |