hotsen / kafka-twitter-relatedwords

Part of a data processing pipeline example using Apache Kafka on Heroku

kafka-twitter-relatedwords

This app is one component of a data processing pipeline example using Apache Kafka on Heroku. See the Getting Started Guide for more details.

Running Locally

Make sure you have Node.js > 5.x, Heroku CLI installed, and an Apache Kafka on Heroku cluster running.

You'll also need to copy .env.sample to .env and provide values for all of the environment variables listed.

• KAFKA_URL: Comma-separated list of Kafka broker URLs
• KAFKA_CLIENT_CERT: Contents of the Kafka client certificate. This is set on a Heroku app when the Apache Kafka on Heroku add-on is attached.
• KAFKA_CLIENT_CERT_KEY: Contents of the Kafka client certificate key. This is set on a Heroku app when the Apache Kafka on Heroku add-on is attached.
• KAFKA_CONSUMER_TOPIC: Kafka topic name from which to consume messages.

And you'll need to update Procfile to include one process per keyword being tracked.

git clone [e-mail]:heroku/kafka-twitter-relatedwords.git # or clone your own fork
cd kafka-twitter-relatedwords
npm install
heroku local

Your app should now be running on localhost:5000.

Deploying to Heroku

heroku create
heroku addons:attach my-kafka-app::KAFKA
# where my-kafka-app is the original app the kafka cluster was attached to

heorku config:set KAFKA_CONSUMER_TOPIC= # Kafka topic from which to consume messages

git push heroku master

After the deploy is successful, scale up each process to 1. Scale the web process to zero. You can do this from either the CLI (e.g. heroku ps:scale dog=1) or the Heroku dashboard.

Documentation

For more information about using Apache Kafka on Heroku, see these resources:

• Full demo Getting Started Guide
• Apache Kafka on Heroku on Dev Center