WebEngDUS UG Talk - Nov. 16th, 2017
Hey folks, this is the demo project from the talk at the WebEngDUS on Nov. 16th, 2017.
How to start everything up
We use a lot of different stuff here, so getting it started can be overwhelming at first. Just follow the instructions and "trust me" for a moment.
All services assume, that you execute them in a separate terminal session, either with tabs, tmux or screen, or you use something like iTerm2's split feature.
Important
- The base directory for all commands is
kafka-dist. - The
kafka-dist/bindirectory contains scripts prefixed withdemo-. These scripts are custom additions from us and intend to ease the demo process. - All services we start here are
CTRL-C-able without doing harm.
MySQL
We assume, that a MySQL server is running on localhost, port 3306. We further assume, that there is a root user with no password.
Please execute this in the MySQL CLI:
CREATE DATABASE IF NOT EXISTS `shop`;
Elasticsearch
Please start an Elasticsearch instance somewhere. The current configuration
assumes it on localhost, but you can change that in kafka-dist/connectors/elasticsearch.properties
Redis
Please start a Redis server somewhere. The current configuration expects it
on localhost, but you can change that in kafka-dist/connectors/redis_cache.properties and
then you need to tell the RedisTemplate in Spring Boot to use your custom connection (that
can be done by providing a RedisConnection bean with the appropriate connection string. See
webengdus.shopservice.RedisConfiguration for the right place to start). The last place
to change the Redis address is in the Stream Processor.
Zookeeper
We run ZK in foreground. Alternatively you can add -daemon and move it
into the background. There is a corresponding stop command to remove the
process later on.
bin/zookeeper-server-start.sh config/zookeeper.properties
Kafka
We run Kafka in foreground. Alternatively you can add -daemon and move it
into the background. There is a corresponding stop command to remove the
process later on.
bin/kafka-server-start.sh config/server.properties
Kafka Topics
Before we start more of the components, we need to add a few topics to the broker.
bin/demo-create_topics.sh
Kafka Connect
Kafka Connect is, without using systemd or daemon tools, a foreground-only
tool. We've tried to commit working version of each plugin into the
kafka-dist/plugins directory, so you can start Kafka Connect right away.
We've assembled already the correct command in the demo scripts for you.
bin/demo-start_connect.sh
Landoop Kafka Connect UI (Optional)
This one needs Docker and gives you a nice little UI for the Kafka Connect installation.
The start script is "made for macOS", because it assumes that your primary
network interface is named en0. Depending on your OS, this might need some adjustment.
Change the following line in bin/demo-docker_connect_ui.sh according to your needs:
MY_IP=$(ifconfig en0 | grep inet | grep -v inet6 | awk '{print $2}')
Then run:
bin/demo-docker_connect_ui.sh
Shop Service
We don't provide the shop service as jar, because it would make the repository even bigger.
Instead, we give you the command to build and run it from your shell. Gradle will download the dependencies on demand and then start the Spring Boot application.
Important: Don't run this for the first time on a mobile connection! Gradle will leave your with the impression of downloading half of the internet.
Change the directory to the root of the repository!
cd ..
./gradlew shop-service:bootRun
The stream processor
The SP tracks the sold items per time and reports it directly to the Redis instance. You can start it by using gradle again:
Important: Don't run this for the first time on a mobile connection! Gradle will leave your with the impression of downloading half of the internet.
Change the directory to the root of the repository!
cd ..
./gradlew streamproc:run
Additional scripts
Delete topics
This script removes the topics from the broker. The broker is already configured to enable topic deletion.
bin/demo-delete_topics.sh
Reset sink offsets
This script resets the offset of the redis sink to force it to read the topic from the earliest offset.
bin/demo-reset_sink_offsets.sh