Playing with Kafka on Windows

This repository gathers Kafka code examples coming from various websites or written by myself.
In particular it includes several build scripts (bash scripts, batch files) for experimenting with the Kafka system on a Windows machine.

☛ Read the document "Getting Started" from the Kafka documentation to know more about the Kafka ecosystem.

Ada, Akka, C++, Dart, Deno, Docker, Flix, Golang, GraalVM, Haskell, Kotlin, LLVM, Modula-2, Node.js, Rust, Scala 3, Spark, Spring, TruffleSqueak and WiX Toolset are other trending topics we are continuously monitoring.

Project dependencies ▴

This project depends on the following external software for the Microsoft Windows platform:

Git 2.44 (release notes)
Kafka 3.7 ¹ (release notes)
Temurin OpenJDK 17 LTS ² (release notes, bug fixes)

Optionally one may also install the following software:

Apache Maven 3.9 (requires Java 8+) (release notes)
Gradle 8.6 (requires Java 8+) (release notes)
Offset Explorer 2.3 ³ (change history)
Scala 2.13 (requires Java 8+) (release notes, Scala API)
Temurin OpenJDK 11 LTS (release notes, bug fixes)
Temurin OpenJDK 21 LTS (release notes, Java 21 API)
Visual Studio Code 1.86 (release notes)

☛ Installation policy
When possible we install software from a Zip archive rather than via a Windows installer. In our case we defined C:\opt\ as the installation directory for optional software tools (in reference to the /opt/ directory on Unix).

For instance our development environment looks as follows (March 2024) ⁴:

C:\opt\apache-maven\               ( 10 MB)
C:\opt\Git\                        (367 MB)
C:\opt\gradle\                     (135 MB)
C:\opt\jdk-temurin-11.0.22_7\      (302 MB)
C:\opt\jdk-temurin-17.0.10_7\      (301 MB)
C:\opt\jdk-temurin-21.0.2_13\      (325 MB)
C:\opt\kafka_2.13-3.7.0\           (118 MB)
C:\opt\scala-2.13.13\              ( 24 MB)
C:\opt\VSCode\                     (341 MB)
C:\Program Files\OffsetExplorer2\  (112 MB)

🔎 Git for Windows provides a BASH emulation used to run git from the command line (as well as over 250 Unix commands like awk, diff, file, grep, more, mv, rmdir, sed and wc).

Directory structure ▴

This project is organized as follows:

docs\
examples\{README.md, quickstart, ..}
kafka\ (git module)
README.md
RESOURCES.md
setenv.bat

where

directory docs\ contains Kafka related papers/articles.
directory examples\ contains Kafka examples grabbed from various websites (see file examples\README.md).
file README.md is the Markdown document for this page.
file RESOURCES.md gathers Kafka related informations.
file setenv.bat is the batch command for setting up our environment.

We also define a virtual drive – e.g. drive K: – in our working environment in order to reduce/hide the real path of our project directory (see article "Windows command prompt limitation" from Microsoft Support).

🔎 We use the Windows external command subst to create virtual drives; for instance:
> subst K: %USERPROFILE%\workspace\kafka-examples

In the next section we give a brief description of the batch files present in this project.

Batch commands ▴

`setenv.bat`

We execute command setenv.bat once to setup our development environment; it makes external tools such as git.exe, gradle.bat and sh.exe directly available from the Windows command prompt (see section Project dependencies).

   > setenv -verbose
    Tool versions:
      java 17.0.10, javac 17.0.10, scalac 2.13.13,
      gradle 8.6, kafka-configs 3.7.0, mvn 3.9.6,
      git 2.44.0.windows.1, diff 3.10, bash 5.2.26(1)-release
    Tool paths:
      C:\opt\jdk-temurin-17.0.10_7\bin\java.exe
      C:\opt\jdk-temurin-17.0.10_7\bin\javac.exe
      C:\opt\scala-2.13.13\bin\scalac.bat
      C:\opt\gradle\bin\gradle.bat
      C:\opt\kafka_2.13-3.7.0\bin\windows\kafka-configs.bat
      C:\opt\apache-maven\bin\mvn.cmd
      C:\opt\Git\bin\git.exe
      C:\opt\Git\usr\bin\diff.exe
      C:\opt\Git\bin\bash.exe
    Environment variables:
      "GIT_HOME=C:\opt\Git"
      "GRADLE_HOME=C:\opt\gradle"
      "JAVA_HOME=C:\opt\jdk-temurin-17.0.10_7"
      "KAFKA_HOME=C:\opt\kafka_2.13-3.7.0"
      "MAVEN_HOME=C:\opt\apache-maven"
      "SCALA_HOME=C:\opt\scala-2.13.13"
    
   > where git gradle
    C:\opt\Git\bin\git.exe
    C:\opt\Git\mingw64\bin\git.exe
    C:\opt\gradle\bin\gradle
    C:\opt\gradle\bin\gradle.bat
    C:\opt\Git\bin\sh.exe
    C:\opt\Git\usr\bin\sh.exe

Footnotes ▴

[1] Kafka components ↩

Kakfa	ZooKeeper	Jetty	Java
3.7.0		9.4.54 ^b)	8+
3.6.1	3.8.3 ^a)	9.4.53 ^b)	8+
3.6.0	3.8.2	9.4.52	8+
3.5.1	3.6.4	9.4.51	8+
3.5.0	3.6.4	9.4.51	8+
3.4.1	3.6.4	9.4.51	8+
3.4.0	3.6.3	9.4.48	8+
3.3.x	3.6.3	?	?
3.2.x	3.6.3	?	?

^a) ZooKeeper current stable version is 3.9.1 as of October 3, 2023 (MVN Repository).
^b) Jetty current version is 12.0.6 as of January 31, 2024 (MVN Repository).
^c) Jetty 9.4.x is at End of Community Support as of June 1, 2023 (#7958).

[2] Java compatibility ↩

KIP-750: Java 8 support is deprecating in Apache Kafka 3.0 and will be dropped in Apache Kafka 4.0.

[3] Offset Explorer ↩

Offset Explorer (formerly named Kalfa Tool) provides the following features :

The browser tree allows us to easily view and navigate the objects (brokers, topices, particitions, consumers) in our Apache Kafka cluster.
The explorer allows us to view messages (and their keys) in the partitions of the topics we are interested in.
If the string-based data is either in JSON or XML format, we can view it in a pretty-printed manner.
The browser tree also allows us to quickly view all offsets of our Kafka consumers.

[4] Downloads ↩

In our case we downloaded the following installation files (see section 1):

apache-maven-3.9.6-bin.zip                         ( 10 MB)
gradle-8.6-bin.zip                                 (118 MB)
kafka_2.13-3.7.0.tgz                               ( 82 MB)
offsetexplorer_64bit.exe                           ( 37 MB)
OpenJDK11U-jdk_x64_windows_hotspot_11.0.22_7.zip   ( 99 MB)
OpenJDK17U-jdk_x64_windows_hotspot_17.0.10_7.zip   ( 99 MB)
OpenJDK21U-jdk_x64_windows_hotspot_21.0.2_13.zip   (191 MB)
PortableGit-2.44.0-64-bit.7z.exe                   ( 41 MB)

mics/March 2024 ▲