ots-m2m / pg-kinesis-bridge

Listen for events from PostgreSQL and send them to AWS Kinesis

pg-kinesis-bridge

A daemon that will listen to a PostgreSQL notification channel and send the data to an AWS Kinesis Stream.

Installation

Install with npm or yarn:

npm install -g pg-kinesis-bridge

Usage

usage: pg-kinesis-bridge [-h] -c CHANNEL -s STREAMNAME

Listen for events from PostgreSQL and send them to AWS Kinesis

Optional arguments:
  -h, --help            Show this help message and exit.
  -c CHANNEL, --channel CHANNEL
                        PostgreSQL channel name
  -s STREAMNAME, --streamName STREAMNAME
                        Kinesis stream name

The libraries used by pg-kinesis-bridge respect the normal environment variables used by postgres clients and aws clients. This can be used as a simple way to pass configuration. e.g.

PGHOST=localhost PGPORT=1234 PGDATABASE=postgres AWS_PROFILE=work AWS_SDK_LOAD_CONFIG=y pg-kinesis-bridge -c CHANNEL -s STREAMNAME

Also added the environment variable LOCAL_KINESIS which can be set to true in which case, it will use the following kinesis config

kinesis_config = { endpoint: 'http://KINESIS_HOST:4567' }

where KINESIS_HOST is the environment variable passed to the docker container

Similar Projects

• pg-amqp-bridge to send postgres notifications to RabbitMQ or services such as Azure Event Hub
• postgresql-to-amqp an alternative for AMQP
• pgsql-listen-exchange an alternative for AMQP
• skeeter to send postgres notifications to 0MQ
• pg-bridge for AWS SNS or webhooks
• postgrest-ws for websockets
• postgresql2websocket for websockets

Implementation Notes

The PostgreSQL notification channel name is used as the Kinesis PartitionKey.