Set up Kafka with Intellij idea
I. Download latest version of Kafka from the official docs:
II. Extract the file and run Zookeper and kafka-server
![Screenshot from 2024-04-29 10-25-40](https://private-user-images.githubusercontent.com/96888594/326399607-3b81fe56-6737-447a-9cc9-a969c1283723.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjI2NjMzOTYsIm5iZiI6MTcyMjY2MzA5NiwicGF0aCI6Ii85Njg4ODU5NC8zMjYzOTk2MDctM2I4MWZlNTYtNjczNy00NDdhLTljYzktYTk2OWMxMjgzNzIzLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA4MDMlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwODAzVDA1MzEzNlomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTZiNzYxODM5ZDc2MTA4OWU1ZDdlZjQ4NDI3YzUwZGY3OTFlMmFhOWRlM2M4MTg1ZWY2ODI1NWJlMDg4NjA3YWEmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.NSTHf-u_FNZZ-x2kSyU6Ddw1lHBQzbPsee-jAlZoApk)
open the extracted folder:
![Ubuntu](https://camo.githubusercontent.com/b51b672b44d8445dbcc388e0beb6122800b2620264b029c1d0d985f94f6e9732/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f5562756e74752d4539353432303f7374796c653d666f722d7468652d6261646765266c6f676f3d7562756e7475266c6f676f436f6c6f723d7768697465)
bin/zookeeper-server-start.sh config/zookeeper.properties
bin/kafka-server-start.sh config/server.properties
![Windows 11](https://camo.githubusercontent.com/fa44730048dfbce3cb3dbeec73419442294bef870b85cc4e9dfbd50eed714b6a/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f57696e646f777325323031312d2532333030373964352e7376673f7374796c653d666f722d7468652d6261646765266c6f676f3d57696e646f77732532303131266c6f676f436f6c6f723d7768697465)
start bin\windows\zookeeper-server-start config\zookeeper.properties
start bin\windows\kafka-server-start config\server.properties
bin\windows\kafka-topics --create --topic quickstart-events --bootstrap-server localhost:9092
bin\windows\kafka-topics --create --topic quickstart-events2 --bootstrap-server localhost:9092 --replication-factor 1 --partitions 1
Start bin\windows\kafka-console-producer –topic quickstart-events --bootstrap-server localhost:9092
Start bin\windows\kafka-console-consumer –topic quickstart-events --bootstrap-server localhost:9092
III. Create a maven project:
![Screenshot from 2024-04-29 10-27-58](https://private-user-images.githubusercontent.com/96888594/326399670-c8084f3b-6c45-43b7-94cc-c9ffd38474d2.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjI2NjMzOTYsIm5iZiI6MTcyMjY2MzA5NiwicGF0aCI6Ii85Njg4ODU5NC8zMjYzOTk2NzAtYzgwODRmM2ItNmM0NS00M2I3LTk0Y2MtYzlmZmQzODQ3NGQyLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA4MDMlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwODAzVDA1MzEzNlomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTc3OTdiNWMyYzM2MmJhNmQzYWY4ZGE5MTllNmNmNjliMWE3NjY5ZDZkYTk1NjU0YTIyOTQ5N2QyNGQyMzFmNTEmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.DSR4r6nZhr3ys4bygpPfIRUTnSbwunnPl5McjJZl0kc)
Use Kafka Plugin in the Marketplace to create Topics directly from the IDE:
![Screenshot from 2024-04-29 11-59-51](https://private-user-images.githubusercontent.com/96888594/326413995-90307447-c7e6-4a0c-b3ba-1735f4620ed8.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjI2NjMzOTYsIm5iZiI6MTcyMjY2MzA5NiwicGF0aCI6Ii85Njg4ODU5NC8zMjY0MTM5OTUtOTAzMDc0NDctYzdlNi00YTBjLWIzYmEtMTczNWY0NjIwZWQ4LnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA4MDMlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwODAzVDA1MzEzNlomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTdlMDE1ZmUyOTU4MjQ4ZmJjNGVjYzFkZjNkMTEzMWZlMmIyN2U4NGZmODE0MTA1NGZhNTY1MzI3M2NhNmNkOWEmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.Kb_XMKui0qg5BpUNEhSorQm3AKcBX1ppFVfnCgRoUpQ)
![Screenshot from 2024-04-29 12-03-43](https://private-user-images.githubusercontent.com/96888594/326414482-ec3ff313-b29f-48ea-a646-053b20522f71.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjI2NjMzOTYsIm5iZiI6MTcyMjY2MzA5NiwicGF0aCI6Ii85Njg4ODU5NC8zMjY0MTQ0ODItZWMzZmYzMTMtYjI5Zi00OGVhLWE2NDYtMDUzYjIwNTIyZjcxLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA4MDMlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwODAzVDA1MzEzNlomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTU4MDljM2VkNWI0MzdmNmNhZmJjNDcxMTBlNmZmYjZkZmNmNDExYzA5MzQxZjJmMjZiNDhjNDYzYWUxYTA5NmUmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.x9GzxyafK6vQ1zbbxoNgld0VWrF9Uj9qA7M7nYCCmFU)
1. Add the necessary Dependencies from Apache Kafka docs (Client & Streams)
<dependency>
<groupId>org.apache.kafka</groupId>
<artifactId>kafka-streams</artifactId>
<version>3.7.0</version>
</dependency>
<dependency>
<groupId>org.apache.kafka</groupId>
<artifactId>kafka_2.13</artifactId>
<version>3.7.0</version>
</dependency>
<dependency>
<groupId>org.apache.kafka</groupId>
<artifactId>kafka-clients</artifactId>
<version>3.7.0</version>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-log4j12</artifactId>
<version>1.7.25</version>
</dependency>
<dependency>
<groupId>log4j</groupId>
<artifactId>log4j</artifactId>
<version>1.2.17</version>
</dependency>
2. Create Producer class:
public class ProducerApp {
public static void main(String[] args) {
ProducerApp producerApp = new ProducerApp();
producerApp.sendMessage();
}
public void sendMessage() {
Properties props = new Properties();
props.put("bootstrap.servers", "localhost:9092");
props.put("acks", "all");
props.put("retries", 0);
props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");
props.put("GROUP_ID_CONFIG", "test-group-1");
KafkaProducer<String, String> producer = new KafkaProducer<>(props);
int i = 0;
while(true){
ProducerRecord<String, String> message = new ProducerRecord<>("test-topic", Integer.toString(i));
producer.send(message);
try {
i++;
System.out.println("message" + i);
Thread.sleep(1000);
}catch (Exception e) {
System.out.println(e.getMessage());
}
}
}
}
3. Create Consumer class:
public class ConsumerApp {
public static void main(String[] args) {
ConsumerApp newConsumer = new ConsumerApp();
newConsumer.consume();
}
public static void consume() {
Properties props = new Properties();
props.put("bootstrap.servers", "localhost:9092");
props.put("group.id", "test");
props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName());
props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName());
props.put(ConsumerConfig.GROUP_ID_CONFIG, "test");
props.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");
props.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, "false");
props.put(ConsumerConfig.AUTO_COMMIT_INTERVAL_MS_CONFIG, "1000");
props.put(ConsumerConfig.SESSION_TIMEOUT_MS_CONFIG, "30000");
props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");
props.put("GROUP_ID_CONFIG", "test-group-1");
KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props);
consumer.subscribe(Collections.singletonList("test-topic"));
Executors.newScheduledThreadPool(1).scheduleAtFixedRate(()-> {
ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(1000));
records.forEach(record -> {
System.out.println("Key "+record.key()+" value "+record.value()+" topic:"+record.topic());
});
},1000, 1000, TimeUnit.MILLISECONDS);
}
}
![Screenshot from 2024-04-29 11-28-05](https://private-user-images.githubusercontent.com/96888594/326405974-24fb916f-42d0-4028-86b4-bc6daa0afce8.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjI2NjMzOTYsIm5iZiI6MTcyMjY2MzA5NiwicGF0aCI6Ii85Njg4ODU5NC8zMjY0MDU5NzQtMjRmYjkxNmYtNDJkMC00MDI4LTg2YjQtYmM2ZGFhMGFmY2U4LnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA4MDMlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwODAzVDA1MzEzNlomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTc0YjQwNDUxNzU4MGQ4NDQ5OWYzYjQwODgxOGJlZjY4NWM5OWMzZTdlMmRlZDM3NDNmZWE5M2ZlYWU4NDdmZDYmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.1pgRJziJhg0Oc5L9l1-_aL2rgGMIQ9t-d-xvqY_JNnM)