This sample project demonstrates how to use Spring Cloud Stream connected with Apache Kafka managed by Strimzi in a Kubernetes and OpenShift.
By end of this tutorial you'll have a simple Spring Boot based Greetings microservice running that
- Deploy Kafka Cluster on Kubernetes or OpenShift Cluster
- Takes a message from a REST API
- Writes it to a Kafka topic
- Reads it from the topic
- Outputs it to the console
Spring Cloud Stream is a framework built upon Spring Boot for building message-driven microservices.
Apache Kafka is a popular high performance and horizontally scalable messaging platform originally developed by LinkedIn.
Strimzi provides a way to run an Apache Kafka cluster on OpenShift and Kubernetes in various deployment configurations.
Kubernetes is an open-source system for automating deployment, scaling, and management of containerized applications.
OpenShift is combines application lifecycle management - including image builds, continuous integration, deployments, and updates - with Kubernetes.
This tutorial requires a Kubernetes or OpenShift platform available. If you do not have one, you could use one of the following resources to deploy locally a Kubernetes or OpenShift Cluster:
- minishift - run OpenShift locally
- Red Hat Container Development Kit
- minikube - Running Kubernetes Locally
Strimzi includes a set of Kubernetes Operators to deploy a full Kafka Cluster on a Kubernetes or OpenShift platform.
You can follow the instructions from Community Documentation or you could use my Ansible Playbook to do it. In both cases it is very easy to do it.
src/main/strimzi
folder includes a set of custom resource definitions to deploy a Kafka Cluster
and a Kafka Topic using the Strimzi Operators.
To deploy the Kafka Cluster:
- Kubernetes:
$ kubectl apply -f src/main/strimz/kafka.yml -n namespace
kafka.kafka.strimzi.io/my-kafka created
- OpenShift:
$ oc apply -f src/main/strimz/kafka.yml -n namespace
kafka.kafka.strimzi.io/my-kafka created
To deploy the Kafka Topic:
- Kubernetes:
$ kubectl apply -f src/main/strimz/kafkatopic.yml -n namespace
kafkatopic.kafka.strimzi.io/greetings-sample created
- OpenShift:
$ oc apply -f src/main/strimz/kafkatopic.yml -n namespace
kafkatopic.kafka.strimzi.io/greetings-sample created
After some minutes Kafka Cluster will be deployed:
$ kubectl get pod
NAME READY STATUS RESTARTS AGE
my-kafka-entity-operator-8474bb6769-xqzt9 3/3 Running 0 1m
my-kafka-kafka-0 2/2 Running 0 2m
my-kafka-kafka-1 2/2 Running 0 2m
my-kafka-kafka-2 2/2 Running 0 2m
my-kafka-kafka-exporter-5b4dff4858-8z9gw 1/1 Running 0 30s
my-kafka-zookeeper-0 2/2 Running 0 3m
my-kafka-zookeeper-1 2/2 Running 0 3m
my-kafka-zookeeper-2 2/2 Running 0 3m
strimzi-cluster-operator-c8d786dcb-8rt9v 1/1 Running 2 5d
Go to Spring Initializer to create a Maven project:
- Add necessary dependencies:
Web
,Spring Cloud Stream
,Kafka
,Devtools
,Actuator
,Lombok
- Click the Generate Project button to download the project as a zip file
- Extract zip file and import the maven project to your favourite IDE
Notice the maven dependencies in the pom.xml
file:
- Spring Boot and Spring Cloud Dependencies
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-web</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-actuator</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-stream</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-starter-stream-kafka</artifactId>
</dependency>
<!-- hot reload - press Ctrl+F9 in IntelliJ after a code change while application is running -->
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-devtools</artifactId>
<optional>true</optional>
</dependency>
- Other Dependencies
<!-- Also install the Lombok plugin in your IDE -->
<dependency>
<groupId>org.projectlombok</groupId>
<artifactId>lombok</artifactId>
<optional>true</optional>
</dependency>
These dependencies are managed by <dependencyManagement>
section:
<dependencyManagement>
<dependencies>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-dependencies</artifactId>
<version>${spring-cloud.version}</version>
<type>pom</type>
<scope>import</scope>
</dependency>
</dependencies>
</dependencyManagement>
In order for our application to be able to communicate with Kafka, we'll need to define an outbound stream to write messages to a Kafka topic, and an inbound stream to read messages from a Kafka topic.
Spring Cloud provides a convenient way to do this by simply creating an interface that defines a separate method for each stream.
package com.jromanmartin.kafka.streams.stream;
import org.springframework.cloud.stream.annotation.Input;
import org.springframework.cloud.stream.annotation.Output;
import org.springframework.messaging.MessageChannel;
import org.springframework.messaging.SubscribableChannel;
public interface GreetingsStreams {
String INPUT = "greetings-in";
String OUTPUT = "greetings-out";
@Input(INPUT)
SubscribableChannel inboundGreetings();
@Output(OUTPUT)
MessageChannel outboundGreetings();
}
The inboundGreetings()
method defines the inbound stream to read from Kafka and outboundGreetings()
method defines
the outbound stream to write to Kafka.
During runtime Spring will create a java proxy based implementation of the GreetingsStreams
interface that
can be injected as a Spring Bean anywhere in the code to access our two streams.
Our next step is to configure Spring Cloud Stream to bind to our streams in the GreetingsStreams
interface.
This can be done by creating a @Configuration
class com.jromanmartin.kafka.streams.config.StreamsConfig
with
below code:
package com.jromanmartin.kafka.streams.config;
import com.jromanmartin.kafka.streams.stream.GreetingsStreams;
import org.springframework.cloud.stream.annotation.EnableBinding;
@EnableBinding(GreetingsStreams.class)
public class StreamsConfig {
}
Binding the streams is done using the @EnableBinding
annotation where the GreatingsService
interface is passed to.
By default, the configuration properties are stored in the src/main/resources/application.yaml
file.
spring:
cloud:
stream:
kafka:
binder:
brokers: my-kafka-kafka-bootstrap:9092
auto-create-topics: false
configuration:
auto.offset.reset: latest
bindings:
greetings-in:
destination: greetings
group: greetings-in-group
contentType: application/json
greetings-out:
destination: greetings
contentType: application/json
The above configuration properties configure the address of the Kafka server to connect to, and the Kafka topic we use for both the inbound and outbound streams in our code. They both must use the same Kafka topic!
Kafka brokers are defined by a Kubernetes or OpenShift service created by Strimzi when the Kafka cluster is deployed. This service, called cluster-name-kafka-bootstrap exposes 9092 port for plain traffic and 9093 for encrypted traffic.
The contentType
properties tell Spring Cloud Stream to send/receive our message objects as String
s in the streams.
Create a simple com.jromanmartin.kafka.streams.model.Greetings
class with below code that will represent
the message object we read from and write to the greetings
Kafka topic:
package com.jromanmartin.kafka.streams.model;
// lombok autogenerates getters, setters, toString() and a builder (see https://projectlombok.org/):
import lombok.Builder;
import lombok.Getter;
import lombok.Setter;
import lombok.ToString;
@Getter @Setter @ToString @Builder
public class Greetings {
private long timestamp;
private String message;
}
Notice how the class doesn't have any getters and setters thanks to the Lombok annotations. The @ToString
will
generate a toString()
method using the class' fields and the @Builder
annotation will allow us
creating Greetings
objects using fluent builder (see below).
Let's create the com.jromanmartin.kafka.streams.service.GreetingsService
class with below code that will
write a Greetings
object to the greetings
Kafka topic:
package com.jromanmartin.kafka.streams.service;
import com.jromanmartin.kafka.streams.model.Greetings;
import com.jromanmartin.kafka.streams.stream.GreetingsStreams;
import lombok.extern.slf4j.Slf4j;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.messaging.MessageChannel;
import org.springframework.messaging.MessageHeaders;
import org.springframework.messaging.support.MessageBuilder;
import org.springframework.stereotype.Service;
import org.springframework.util.MimeTypeUtils;
@Service
@Slf4j
public class GreetingsService {
@Autowired
private GreetingsStreams greetingsStreams;
public void sendGreeting(final Greetings greetings) {
log.info("Sending greetings {}", greetings);
MessageChannel messageChannel = greetingsStreams.outboundGreetings();
boolean sent = messageChannel.send(MessageBuilder
.withPayload(greetings)
.setHeader(MessageHeaders.CONTENT_TYPE, MimeTypeUtils.APPLICATION_JSON)
.build());
log.info("Sent {} greetings {}", sent, greetings);
}
}
The @Service
annotation will configure this class as a Spring Bean and inject the GreetingsService
dependency via
@Autowired
annotation.
The @Slf4j
annotation will generate an SLF4J logger field that we can use for logging.
In the sendGreeting()
method we use the injected GreetingsStream
object to send a message represented by the Greetings
object.
Now we'll be creating a REST API endpoint that will trigger sending a message to Kafka using the GreetingsService
Spring Bean:
package com.jromanmartin.kafka.streams.web;
import com.jromanmartin.kafka.streams.model.Greetings;
import com.jromanmartin.kafka.streams.service.GreetingsService;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.http.HttpStatus;
import org.springframework.http.ResponseEntity;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RequestParam;
import org.springframework.web.bind.annotation.ResponseStatus;
import org.springframework.web.bind.annotation.RestController;
@RestController
public class GreetingsController {
@Autowired
private GreetingsService greetingsService;
@GetMapping("/greetings")
@ResponseStatus(HttpStatus.ACCEPTED)
public ResponseEntity<Greetings> greetings(@RequestParam("message") String message) {
Greetings greetings = Greetings.builder()
.message(message)
.timestamp(System.currentTimeMillis())
.build();
greetingsService.sendGreeting(greetings);
return ResponseEntity.ok(greetings);
}
}
The @RestController
annotation tells Spring that this is a Controller bean. The greetings()
method defines
an HTTP GET /greetings
endpoint that takes a message
request param and passes it to
the sendGreeting()
method in GreetingsService
.
Let's create a com.jromanmartin.kafka.streams.service.GreetingsListener
class that will listen to messages
on the greetings
Kafka topic and log them on the console:
package com.jromanmartin.kafka.streams.service;
import com.jromanmartin.kafka.streams.model.Greetings;
import com.jromanmartin.kafka.streams.stream.GreetingsStreams;
import lombok.extern.slf4j.Slf4j;
import org.springframework.cloud.stream.annotation.StreamListener;
import org.springframework.kafka.support.KafkaHeaders;
import org.springframework.messaging.handler.annotation.Headers;
import org.springframework.messaging.handler.annotation.Payload;
import org.springframework.stereotype.Component;
import java.util.Map;
@Component
@Slf4j
public class GreetingsListener {
@StreamListener(GreetingsStreams.INPUT)
public void handleGreetings(@Payload Greetings greetings, @Headers Map<String, Object> headers) {
log.info("Received greetings: {}. Partition: {}. Offset: {}", greetings,
headers.get(KafkaHeaders.RECEIVED_PARTITION_ID), headers.get(KafkaHeaders.OFFSET));
}
}
The @Component
annotation similarly to @Service
and @RestController
defines a Spring Bean.
GreetingsListener
has a single method, handleGreetings()
that will be invoked by Spring Cloud Stream with
every new Greetings
message object on the greetings
Kafka topic. This is thanks to the @StreamListener
annotation
configured for the handleGreetings()
method.
@Headers
annotation inject Kafka record headers from the Kafka Topic. This map includes additional information from
the record as: partition id, offset, ...
The last piece of the puzzle is the com.jromanmartin.kafka.streams.StreamKafkaApplication
class that was
auto-generated by the Spring Initializer:
package com.jromanmartin.kafka.streams;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
@SpringBootApplication
public class StreamKafkaApplication {
public static void main(String[] args) {
SpringApplication.run(StreamKafkaApplication.class, args);
}
}
No need to make any changes here. You can either run this class as a Java application from your IDE, or run the application from the command line using the Spring Boot maven plugin:
$ mvn spring-boot:run
Or you can deploy into Kubernetes or OpenShift platform using Eclipse JKube Maven Plug-ins:
For Kubernetes:
$ mvn k8s:build k8s:resource k8s:apply -Pkubernetes
For OpenShift:
$ mvn oc:build oc:resource oc:apply -Popenshift
Once the application is running, go to http://<KUBERNETES_OPENSHIFT_HOST>/greetings?message=hello
in the browser
and check your console.
To get the route the following command in Kubernetes give you the host:
$ kubectl get route spring-cloud-stream-kafka-sample -o jsonpath='{.spec.host}'
In OpenShift:
$ oc get route spring-cloud-stream-kafka-sample -o jsonpath='{.spec.host}'
This command will send a message to the Kafka Topic:
curl http://$(oc get route spring-cloud-stream-kafka-sample -o jsonpath='{.spec.host}')//greetings?message=hello; echo
{"timestamp":1581086660762,"message":"hello"}
2020-02-07 14:44:20.762 INFO 1 --- [nio-8080-exec-2] c.j.k.streams.service.GreetingsService : Sending greetings Greetings(timestamp=1581086660762, message=hello)
2020-02-07 14:44:20.763 INFO 1 --- [nio-8080-exec-2] c.j.k.streams.service.GreetingsService : Sent true greetings Greetings(timestamp=1581086660762, message=hello)
2020-02-07 14:44:20.770 INFO 1 --- [container-0-C-1] c.j.k.streams.service.GreetingsListener : Received greetings: Greetings(timestamp=1581086660762, message=hello). Partition: 1. Offset: 1
I hope you enjoyed this tutorial. Feel free to ask any questions and leave your feedback.