Up and running with Kafka and Go
This is a quick and dirty cradle to grave walk-through to get up and running with Go and Kafka producer/consumers. I assume that you have at least a basic understanding of Kafka as I will not be going over Kafka in large outside of what you see in the repo. We will utilize segment's kafka-go client to create our producers and consumers. We will be using avro and confluents schema registry to encode/decode our data. To work with avro we will be utilizing linkedin's goavro library and for our schema registry client will be the landoop schema registry client.
Setup
You'll need the following:
- Go 1.7 or newer required.
- go dep dependency manager
- docker version 17.05 or newer
- edward tool for managing microservice instances
- landoop's fast-data-dev we will use the docker image to create an environment that is both portable and reliable. As it is docker, the setup is painless.
Note: I will be speaking in the rest of the write up as an OSX user
Setting up Landoop
If you want to bend fast-data-dev to your will, go straight to their github repo.
Note: when accessing the UI, it is using the OSX hostname created by docker-machine, if you are using linux just replace it with localhost
LINUX
As with most things software, linux is the easiest to setup. Just run:
docker run --rm --name ldp --net=host landoop/fast-data-dev
Pay attention to the --name ldp
, as this is how you will enter the container to create topics or whatever else you want to run inside the container.
OSX and Windows
To run this properly in Mac you'll have a few hoops to jump through. First you'll need to run the following to create an environment for your docker containers to run in:
docker-machine create --driver virtualbox --virtualbox-memory 6000 landoop
You can verify that your new environment was successful by running:
docker-machine ls
You should see the name landoop. Now you will need to get into that environment with the following command:
eval $(docker-machine env landoop)
Now you can run the following to start up your fast-data-dev container:
docker run --rm --name ldp -p 2181:2181 -p 3030:3030 -p 8081-8083:8081-8083 \
-p 9581-9585:9581-9585 -p 9092:9092 -e ADV_HOST=192.168.99.100 \
-e SAMPLEDATA=0 -e RUNTESTS=0 \
landoop/fast-data-dev:latest
Pay attention to the --name ldp
, as this is how you will enter the container to create topics or whatever else you want to run inside the container.
Verify setup
open a browser and go to http://192.168.99.100:3030
. You should see the landoop UI.
Create the zip Topics
Make sure the fast-data-dev container is up and running. Eval into the correct docker-machine environment with the following:
eval $(docker-machine env landoop)
Now run the following to create the zip topic first:
docker exec -it ldp kafka-topics --create --zookeeper 192.168.99.100:2181 --topic zip --replication-factor 1 --partitions 3
the zip topic will have 3 partitions and a replication factor of 1
Now create the address enriched zipcode topic:
docker exec -it ldp kafka-topics --create --zookeeper 192.168.99.100:2181 --topic zip.address --replication-factor 1 --partitions 1
Verify you have created the two new topics with the following:
docker exec -it ldp kafka-topics --list --zookeeper 192.168.99.100:2181
NOTE: You can also do all this with the bash script included, if on OSX, run the eval command above before the script
bash topic_setup.sh
Add the necessary schemas to confluent's schema registry
This part is fairly straight forward. Jump back to your browser and go to http://192.168.99.100:3030/schema-registry-ui/. Click the NEW button in the upper right.
One quick word before continuing, make sure to name your subjects with this format topicname-value
or topicname-key
. Now you can't say I didn't warn you...
Next create the schema for the zip topic messages as you see below
Subject Name = zipcode-value
Schema:
{
"type": "record",
"name": "zipcode",
"namespace": "com.landoop",
"doc": "simple zipcode producer schema",
"fields": [
{
"name": "zipcode",
"type": "string"
},
{
"name": "timestamp",
"type": "long",
"doc": "Time the message was encoded to avro (nanoseconds since epoch). May be used for ordering."
}
]
}
click validate and viola, schema created
now create the schema for the enrichedZip topic messages
Subject Name = zip_address-value
Schema:
{
"type": "record",
"name": "addressEnrichedZip",
"namespace": "com.landoop",
"doc": "Schema to define enriched zip data, enriched with addresses",
"fields": [
{
"name": "zipcode",
"type": "string"
},
{
"name": "timestamp",
"type": "long",
"doc": "Time the message was encoded to avro (nanoseconds since epoch). May be used for ordering."
},
{
"name": "addresses",
"type": {
"type": "array",
"items": {
"type": "record",
"name": "address",
"fields": [
{
"name": "zipcode",
"type": "string"
},
{
"name": "street",
"type": "string"
},
{
"name": "city",
"type": "string"
},
{
"name": "state",
"type": "string"
},
{
"name": "country",
"type": "string"
}
]
}
}
}
]
}
again click validate and schema should have been created.
We are not going to create any keys for this demo, but you would follow the same syntax as you see above.
Verify everything is working
Just run the following command from the root level of the repo:
edward start -t kafka-pipeline
you should see the producer logs and the consumer will start logging in like after 5-10 seconds or so. If that does not work, you should see some helpful logs to help you debug the issue.
SETUP COMPLETE