rsilvestre / kafka-connect

equivalent to kafka-connect :wrench: for nodejs :sparkles::turtle::rocket::sparkles:

node-kafka-connect

What can I do with this?

The framework can be used to build connectors, that transfer data to and from Apache Kafka and Databases, very easily. If you are looking for already implemented connectors for you favorite datastore, take a look at the Available Connector Implementations below.

Info

• node-kafka-connect is a framework to implement large kafka -> datastore & datastore -> kafka data movements.
• it can be used to easily built connectors from/to kafka to any kind of datastore/database.
• a connector might consist of a SourceConnector + SourceTask to poll data from a datastore into a kafka topic.
• a connector might consist of a SinkConnector + SinkTask to put data from a kafka topic into a datastore.
• Converters might be used to apply alteration to any data-stream.
• any operation in node-kafka-connect is asynchronous
• ships with auto http server (health-checks, kafka-stats)
• ships with auto metrics (prometheus)

Available Connector Implementations

• Sequelize (MySQl, Postgres, SQLite, MSSQL)
• Google BigQuery
• Salesforce
• Google PubSub

Creating custom Connectors

npm install --save kafka-connect

const source = new TestSourceConfig(config, 
    TestSourceConnector, 
    TestSourceTask, 
    [TestConverter]);
    
source.run().then();

const sink = new TestSinkConfig(config,
    TestSinkConnector, 
    TestSinkTask, 
    [TestConverter]);
 
sink.run().then();

Docs

• Implementation-Helper Overview
• Framework Events

FAQ

• Q: it is running slow / only synchronous / 1 by 1 messages ?
• A: just set the config.batch object as it is described here