rogeriob2br / camel-kafka-connector

Camel Kafka Connector allows you to use all Camel components as Kafka Connect connectors

Camel Kafka Connector

Introduction

Note	The project is WIP!

This is a "Camel Kafka connector adapter" that aims to provide a user friendly way to use all Apache Camel components in Kafka Connect. For more information about Kafka Connect take a look here.

Build the project

mvn clean package

Build the project and run integration tests

To build the project it is sufficient to:

mvn clean install

To run the integration tests it is required to:

• have Docker version 17.05 or higher running

• run:

  mvn -DskipIntegrationTests=false clean verify package

It is also possible to point the tests to use an external services. To do so, you must set properties for the services that you want to run. This causes the tests to not launch the local container and use existing remote instances. At the moment, the following properties can be set for remote testing:

• kafka.instance.type

  • kafka.bootstrap.servers

• aws-service.instance.type

  • access.key: AWS access key (mandatory for remote testing)

  • secret.key: AWS secret key (mandatory for remote testing)

  • aws.region: AWS region (optional)

  • aws.host: AWS host (optional)

• aws-service.kinesis.instance.type

  • access.key: AWS access key (mandatory for remote testing)

  • secret.key: AWS secret key (mandatory for remote testing)

  • aws.region: AWS region (optional)

  • aws.host: AWS host (optional)

• elasticsearch.instance.type

  • elasticsearch.host

  • elasticsearch.port

• cassandra.instance.type

  • cassandra.host

  • cassandra.cql3.port

• jdbc.instance.type

  • jdbc.connection.url

• jms-service.instance.type

  • jms.broker.address

• hdfs.instance.type

  • hdfs.host

  • hdfs.port

• mongodb.instance.type

  • mongodb.url

Additionally, a few manual tests can be enabled and executed with adequate configuration on the accounts and environments used by those services. This is very specific to the nature of each of those services, therefore please consult the comments on each of those test cases for the details related to their setup.

The following manual or semi-automated tests are available under these circumstances. They can be enabled with these flags along with others that are specific to that service:

• it.test.salesforce.enable

• it.test.slack.enable

mvn -Dkafka.bootstrap.servers=host1:port -Dkafka.instance.type=remote -DskipIntegrationTests=false clean verify package

It’s possible to use a properties file to set these properties. To do so use -Dtest.properties=/path/to/file.properties.

Try it out locally

You can use Camel Kafka connectors with local Apache Kafka installation. For more details see the local guide.

Try it out on OpenShift with Strimzi

You can also use Camel Kafka connectors on OpenShift. For more details see the OpenShift guide.

Known limitations

In order to use other Camel components you need to add them as dependencies in the project pom.xml.