Kafka Connect SAP is a generic set of connectors, using the Apache Kafka Connect framework for reliably connecting Kafka with SAP systems
To install the connector from source,
Clone this repository to your local desktop, and then bring up a command prompt in the directory.
and use the following command.
mvn clean install -DskipTests
Include the Jdbc Jar
- Follow the steps in http://help.sap.com/saphelp_hanaplatform/helpdata/en/ff/15928cf5594d78b841fbbe649f04b4/frameset.htm guide to access the SAP HANA Jdbc jar.
- Place it in the same directory as the
Kafka Connector
jar or under theCLASSPATH
directory.
For getting started with this connector, the following steps need to be completed.
- Create the config file for sink named
kafka-connect-sink.properties
.
name=test-sink
connector.class=com.sap.kafka.connect.sink.hana.HANASinkConnector
tasks.max=1
topics=test_topic
connection.url=jdbc:sap://<url>/
connection.user=<username>
connection.password=<password>
auto.create=true
schema.registry.url=<schema registry url>
test_topic.table.name="SYSTEM"."DUMMY_TABLE"
- Start the kafka-connect sink connector using the following command.
./bin/connect-standalone ./etc/schema-registry/connect-avro-standalone.properties ./etc/kafka/kafka-connect-sink.properties
- Create the config file for source named
kafka-connect-source.properties
.
name=kafka-connect-source
connector.class=com.sap.kafka.connect.source.hana.HANASourceConnector
tasks.max=1
topics=kafka_source_1,kafka_source_2
connection.url=jdbc:sap://<url>/
connection.user=<username>
connection.password=<password>
kafka_source_1.table.name="SYSTEM"."com.sap.test::hello"
-
- Start the kafka-connect source connector using the following command.
./bin/connect-standalone ./etc/schema-registry/connect-avro-standalone.properties ./etc/kafka/kafka-connect-source.properties
In a production environment, it is suggested to run the Kafka Connector on distributed mode
The kafka connector for SAP Systems
provides a wide set of configuration options both for source & sink.
The full list of configuration options for kafka connector for SAP Systems
is as follows:
-
Sink
-
topics
- This setting can be used to specifya comma-separated list of topics
. Must not have spaces. -
auto.create
- This setting allows creation of a new table in SAP DBs if the table specified in{topic}.table.name
does not exist. Should be aBoolean
. Default isfalse
. -
batch.size
- This setting can be used to specify the number of records that can be pushed into SAP DB table in a single flush. Should be anInteger
. Default is3000
. -
max.retries
- This setting can be used to specify the maximum no. of retries that can be made to re-establish the connection to SAP DB in case the connection is lost. Should be anInteger
. Default is10
. -
{topic}.table.name
- This setting allows specifying the SAP DBs table name where the data needs to be written to. Should be aString
. Must be compatible to SAP DB Table name like"SCHEMA"."TABLE"
. -
{topic}.table.type
- This is a DB specific configuration setting which allows creation of Row & Column tables ifauto.create
is set to true. Default value iscolumn
. And supported values arecolumn, row
. -
{topic}.pk.mode
- This setting can be used to specify the primary key mode required whenauto.create
is set totrue
& the table name specified in{topic}.table.name
does not exist in SAP DB. Default isnone
. And supported values arerecord_key, record_value
. -
{topic}.pk.fields
- This setting can be used to specifya comma-separated list of primary key fields
when{topic}.pk.mode
is set torecord_key
orrecord_value
. Must not have spaces. -
{topic}.table.partition.mode
- This is a SapDB Sink specific configuration setting which determines the table partitioning in SAP DB. Default value isnone
. And supported values arenone, hash, round_robin
. -
{topic}.table.partition.count
- This is a SapDB Sink specific configuration setting which determines the number of partitions the table should have. Required whenauto.create
is set totrue
and table specified in{topic}.table.name
does not exist in SAP DBs. Should be anInteger
. Default value is0
.
-
-
Source
-
topics
- This setting can be used to specifya comma-separated list of topics
. Must not have spaces. -
mode
- This setting can be used to specify the mode in which data should be fetched from SAP DB table. Default isbulk
. And supported values arebulk, incrementing
. -
queryMode
- This setting can be used to specify the query mode in which data should be fetched from SAP DB table. Default istable
. And supported values aretable, query ( to support sql queries )
. When usingqueryMode: query
it is also required to havequery
parameter defined. This query parameter needs to be prepended by TopicName. If theincrementing.column.name
property is used together, then it can be omitted from where clause in query sql. -
{topic}.table.name
- This setting allows specifying the SAP DB table name where the data needs to be written to. Should be aString
. Must be compatible to SAP DB Table name like"SCHEMA"."TABLE"
. -
{topic}.poll.interval.ms
- This setting allows specifying the poll interval at which the data should be fetched from SAP DB table. Should be anInteger
. Default value is60000
. -
{topic}.incrementing.column.name
- In order to fetch data from a SAP DB table whenmode
is set toincrementing
, an incremental ( or auto-incremental ) column needs to be provided. The type of the column can beInt, Float, Decimal, Timestamp
. This considers SAP DB Timeseries tables also. Should be a valid clumn name ( respresented as aString
) present in the table. -
{topic}.partition.count
- This setting can be used to specify the no. of topic partitions that the Source connector can use to publish the data. Should be anInteger
. Default value is1
.
-
The unit tests
provide examples on every possible mode in which the connector can be configured.
Raise Github issues to obtain support from project committers: Santosh Addanki (santosh.kumar.addanki@sap.com) Subhobrata Dey (subhobrata.dey@sap.com)
Contributions are accepted by sending Pull Requests to this repo.
Currently only SAP Hana is supported.
Copyright (c) 2017 SAP SE or an SAP affiliate company. All rights reserved. This file is licensed under the Apache Software License, v. 2 except as noted otherwise in the LICENSE file.