aghyad / flink-connector-dynamodb

Apache Flink connector sink for AWS DynamoDB

Flink Connector DynamoDB

Java library provides Apache Flink connector sink for AWS DynamoDB database that can be used with Flink 1.11.1 runtime version.

At Klarna we use streaming applications extensively. Amazon Kinesis Data Analytics with Flink 1.11.3 is starting to be one of the choices for the development of new streaming analytics applications at Klarna. Unfortunately, Apache Flink does not provide a connector sink for AWS DynamoDB database out of the box at the moment. This project is to solve this gap.

Usage example

final FlinkDynamoDBClientBuilder dynamoDBBuilder = new DynamoDBBuilder() {
    @Override
    public DynamoDBClient build() {
        return DynamoDBClient.builder().withRegion(Region.EU_WEST_1).build();
    }
};

final DynamoDBSinkWriteRequestMapper<String> mapper = new DynamoDBSinkWriteRequestMapper<>() {
    @Override
    public WriteRequest map(String in) {
        return WriteRequest.builder().putRequest(PutRequest.builder().build());
    }
};

final DynamoDBSinkConfig dynamoDBSinkConfig = DynamoDBSinkBaseConfig.builder()
    .batchSize(25)
    .queueLimit(10)
    .build();

final FlinkDynamoDBSink<String> dynamoDbSink = new FlinkDynamoDBSink<>(
        dynamoDBBuilder,
        "table_name"
        dynamoDBSinkConfig,
        mapper
);

env.addSource(createKafkaConsumer())
        .addSink(dynamoDbSink)
        .execute();

Development setup

This project is available from Maven Central Repository.

Maven

<dependency>
    <groupId>com.klarna</groupId>
    <artifactId>flink-connector-dynamodb</artifactId>
    <version>1.0.0</version>
</dependency>

Gradle

implementation com.klarna:flink-connector-dynamodb:1.0.0

How to contribute

See our guide on contributing.

Release History

See our changelog.

License

Copyright © 2020 Klarna Bank AB

For license details, see the LICENSE file in the root of this project.