Fork notes
Compiling in Xcode
To compile librdkafka with Xcode, do these two things:
brew install librdkafka
- add this to your
dependencies
section ofPackage.swift
dependencies: [ // ... .package(url: "https://github.com/TelemetryDeck/SwiftKafka.git", branch: "main"), // ... ],
- add this to the
target
section of yourPackage.swift
:
linkerSettings: [
.unsafeFlags(["-Xlinker", "-L/usr/local/lib"], .when(platforms: [.macOS]))
]
source: Kitura#11 (comment)
Compression / Buffer things
SwiftKafka
A swift implementation of Kafka for producing and consuming from event streams.
This works by wrapping the librdkafka C library.
Swift version
The latest version of SwiftKafka requires Swift 5.0 or later. You can download this version of the Swift binaries by following this link.
Usage
Swift Package Manager
Add dependencies
Add the SwiftKafka
package to the dependencies within your application’s Package.swift
file. Substitute "x.x.x"
with the latest SwiftKafka
release.
.package(url: "https://github.com/IBM-Swift/SwiftKafka.git", from: "x.x.x")
Add SwiftKafka
to your target's dependencies:
.target(name: "example", dependencies: ["SwiftKafka"]),
Import package
import SwiftKafka
Getting Started
To use SwiftKafka you will need to install the librdkafka
package:
macOS
brew install librdkafka
Linux
Install librdkafka from the Confluent APT repositories - see instructions here (following steps 1 and 2 to add the Confluent package signing key and apt repository), and then install librdkafka:
sudo apt install librdkafka
Running a Kafka instance locally
To experiment locally, you can set up your own Kafka server to produce/consume from.
On macOS you can follow this guide on Kafka Installation using Homebrew to run a local server.
On Linux, you can follow this guide for a manual install on Ubuntu.
KafkaConfig
The KafkaConfig
class contains your configuration settings for a KafkaConsumer
/KafkaProducer
.
The class is initialized with default values which can then be changed using the helper functions. For example, to enable all logging you would set the debug variable:
let config = KafkaConfig()
config.debug = [.all]
Alternatively, you can access the configuration dictionary directly on the KafkaConfig
object:
let config = KafkaConfig()
config["debug"] = "all"
The list of configuration keys and descriptions can be found in the librdkafka CONFIGURATION.md.
When you pass this class to a producer/consumer, a copy is made so further changes to the instance will not affect existing configurations.
KafkaProducer:
The KafkaProducer
class produces messages to a Kafka server.
You can initialize a KafkaProducer
using a KafkaConfig
instance or with the default configuration.
The producer sends a KafkaProducerRecord
with the following fields:
- topic: The topic where the record will be sent. If this topic doesn't exist the producer will try to create it.
- value: The message body that will be sent with the record.
- partition: The topic partition the record will be sent to. If this is not set the partition will be automatically assigned.
- key: If the partition is not set, records with the same key will be sent to the same partition. Since order is guaranteed within a partition, these records will be read in order they were produced.
The send()
function is asynchronous. The result is returned in a callback which contains a KafkaConsumerRecord
on success or a KafkaError
on failure.
The following example produces a message with the value "Hello World" to a "test" topic of a Kafka server running on localhost.
do {
let producer = try KafkaProducer()
guard producer.connect(brokers: "localhost:9092") == 1 else {
throw KafkaError(rawValue: 8)
}
producer.send(producerRecord: KafkaProducerRecord(topic: "test", value: "Hello world", key: "Key")) { result in
switch result {
case .success(let message):
print("Message at offset \(message.offset) successfully sent")
case .failure(let error):
print("Error producing: \(error)")
}
}
} catch {
print("Error creating producer: \(error)")
}
KafkaConsumer:
The KafkaConsumer
class consumes messages from a Kafka server.
You can initialize a KafkaConsumer
using a KafkaConfig
instance or with the default configuration.
You can then subscribe to topics using subscribe()
.
This will distribute the topic partitions evenly between consumers with the same group id.
If you do not set a group id, a random UUID will be used.
Alternatively to can use assign()
to manually set the partition and offset for the consumer.
Both subscribe()
and assign()
are asynchronous and will return immediately, however they may take up to sessionTimeoutMs (Default 10 seconds) * 2 before the consumer completely connects.
To consume messages from Kafka you call poll(timeout:)
. This will poll Kafka, blocking for timeout
seconds. When it completes, it returns an array of KafkaConsumerRecord
with the following fields:
- value: The message value if it can be UTF8 decoded to a String.
- valueData: The message value as raw data.
- key: The message key if it can be utf8 decoded to a String.
- keyData: The message key as raw data.
- offset: The message offset.
- topic: The topic that the message was consumed from.
- partition: The partition that the message was consumed from.
When you have finished consuming, you can call close()
to close the connection and unassigns the consumer.
The unassigned partitions will then be rebalanced between other consumers in the group.
If close()
is not called, the consumer will be closed when the class is deallocated.
The following example consumes and print all unread messages from the "test" topic of the Kafka server.
do {
let config = KafkaConfig()
config.groupId = "Kitura"
config.autoOffsetReset = .beginning
let consumer = try KafkaConsumer(config: config)
guard consumer.connect(brokers: "localhost:9092") == 1 else {
throw KafkaError(rawValue: 8)
}
try consumer.subscribe(topics: ["test"])
while(true) {
let records = try consumer.poll()
print(records)
}
} catch {
print("Error creating consumer: \(error)")
}
API Documentation
For more information visit our API reference.
Community
We love to talk server-side Swift, and Kitura. Join our Slack to meet the team!
License
This library is licensed under Apache 2.0. Full license text is available in LICENSE.