Custom Kafka Producer and Consumer apps.
- Article: https://www.javaworld.com/article/3060078/big-data/big-data-messaging-with-kafka-part-1.html
Goal:
- Create your own CustomConsumer.
- Create your own CustomProducer to auto-generate and send at least 10 messages to a topic using the Twitter API.
- Use a custom namespace to avoid naming conflicts with other "CustomProducer" classes.
Alternate:
- Use Facebook API, or create a random word generator, choose a random noun-verb-noun, or cycle through a list sending one message every second.
- Experiment - the goal is to be comfortable adding external messaging to any Java app.
- Twitter Developer Account
- OpenJDK or JDK (8 or up)
- Apache Maven
- Apache Zookeeper
- Apache Kafka
Also:
- Configure system environment variables including Path.
- Configure Zookeeper
- Configure Kafka
- Determine your namespace.
- Follow the Maven standard project directory structure.
- Use src\main\java for your code (with namespace folders)
- Use src\main\resources for your proporties files.
- Write your custom Kafka Producer in your namespace.
- Write your custome Kafka Consumer in your namespace.
- Add pom.xml
- Add README.md
- Add .gitignore
Open PowerShell as Administrator in the root project folder, compile the code using Maven and create an executable jar file. Generated artificacts can be found in the new 'target' folder.
Recommendation: Run this command each time you make code changes.
mvn clean compile assembly:single
Start and keep running the Zookeeper service.
Start and keep running the Kafka service.
Open PowerShell as Administrator in the root project folder, start the original consumer app:
java -cp target/kafka-case-1.0-SNAPSHOT-jar-with-dependencies.jar edu.nwmissouri.isl.professorcase.kafka.CustomConsumer
Open a new PowerShell as Administrator in the root project folder, start the Producer app using topic test:
java -cp target/kafka-case-1.0-SNAPSHOT-jar-with-dependencies.jar edu.nwmissouri.isl.professorcase.kafka.CustomProducer
Verify messages are output by the Consumer.
See http://www.baeldung.com/twitter4j
For help registering your Twitter API see this Tweepy article:
Register app:
Developer Agreement:
Twitter App Attributes Example:
- Name: KafkaCaseCustomProducer
- Description: Sample app to illustrate working with the Twitter API and using Kafka from a Java application
- Website: https://bitbucket.org/professorcase/h08
- Callback URL: (blank)