az3 / kinesis-three

Java Kinesis Consumer Sample with KCL

kinesis-three

This is a sample Kinesis Consumer Application, with basic implementation of amazon-kinesis-client library.

• It demonstrates the ingestion part of incoming stream.
• The output should be properly formatted for your own use case.

Run

• To run, first you need a valid AWS auth; with that you should be able to read/write DynamoDB and read from Kinesis stream.
• Sample AWS policy is shown here. You can test it on AWS Policy Simulator.
• Then, you need to prepare a "properties" file to connect to AWS. Sample file is here.

$ cat <<EOF >/tmp/app.prop
appName = kinesis-three-app1
kinesisInputStream = mystream
regionName = us-east-1
kinesisEndpoint = https://kinesis.us-east-1.amazonaws.com
workerId = myworker1a
idleTimeBetweenReadsInMillis = 1000
prometheusPort = 9751

EOF

• Finally, you can run application with the following command.

$ git clone --depth 1 https://github.com/az3/kinesis-three.git
$ cd kinesis-three
$ mvn clean install
$ java -DCONFIG_FILE=/tmp/app.prop -jar target/kinesis-three.jar

• It will parse incoming stream data to console (stdout) and persist position in DynamoDB.