AWS SQS River Plugin for ElasticSearch

The AWS SQS plugin uses Amazon's SQS as a river by pulling messages from a given queue. Right after a message is indexed it gets deleted from the queue.

Messages are in the following JSON format:

{
  "_id": "123",
  "_index": "es_index_name",
  "_type": "es_data_type",
  "_data": { "key1": "value1" ...}
}

If _data is missing the data with this id will be deleted from the index.

If _index is missing it will fallback to the index that was initially configured, otherwise the _index property overrides the default configuration and allows you to dynamically switch between indexes.

The fields _id and _type are required.

To configure put this in your elasticsearch.yml:

cloud.aws.region: AWS REGION
cloud.aws.access_key: AWS ACCESS KEY
cloud.aws.secret_key: AWS SECRET KEY
cloud.aws.sqs.queue_url: AWS QUEUE URL

Or use river configuration:

curl -XPUT 'localhost:9200/_river/my_sqs_river/_meta' -d '{
  "type": "amazonsqs",
  "amazonsqs": {
      "region": "AWS REGION",
      "access_key": "AWS ACCESS KEY",
      "secret_key": "AWS SECRET KEY",
      "queue_url": "AWS QUEUE URL"
    },
    "index": {
      "max_messages": 10,
      "timeout_seconds": 10,
      "index": "es_index_name"
    }

}'

In order to install the plugin, simply run: bin/plugin -install aleski/elasticsearch-river-amazonsqs/1.2.

mrmx / elasticsearch-river-amazonsqs

AWS SQS River Plugin for ElasticSearch

About