Salesforce Multi Org Syn Using Kafka & Postgres

IN PROGRESS

needs: https://github.com/amiel/heroku-buildpack-apt#feature/support-adding-keys and heroku/jvm

This proof of concept is intended to demonstrate the use of Kafka Connect to sync the data from Heroku Postgres to Heroku Kafka and f

Usage

Create Kafka Connect Connectors

Source Connector

curl -X POST https://<HEROKU_APP_URL>/connectors -H "Content-Type: application/json" --data @- << EOF
{
    "name": "postgres-source",
    "config": {
      "connector.class": "io.confluent.connect.jdbc.JdbcSourceConnector",
      "tasks.max": "1",      
      "connection.url": "jdbc:postgresql://<Postgres-Host>:5432/<Database-Name>?user=<Database-UserName>&password=<DatabasePassword>",
      "mode": "incrementing",
      "incrementing.column.name": "id",
      "topic.prefix": "connect_"      
    }
}
EOF

Sink Connector

curl -X POST https://<HEROKU_APP_URL>/connectors -H "Content-Type: application/json" --data @- << EOF
{
    "name": "redshift-sink",
    "config": {
      "connector.class": "io.confluent.connect.jdbc.JdbcSinkConnector",
      "tasks.max": "1",
      "topics": "connect_demo",
      "connection.url": "jdbc:redshift://<Redshift-Host>:5439/<Redshift-Database>?user=<Redshift-UserName>&password=<Redshift-Password>",
      "auto.create": "false"
    }
}
EOF

Delete Kafka Connect Connectors

curl -X DELETE https://<HEROKU_APP_URL>/connectors/<CONNECTOR_NAME>

Insert some data into Heroku Postgres

JDBC Connector Info

About

Languages

Language:Shell 100.0%