psilospore / kafka2http

Subscribe to a topic and make http calls. Supports retry, dead letter queues, and templatized request bodies.

kafka2http

Simple Example

You just have some topic and you want to hit some API endpoint when you get a message.

1. Get a message from the users topic
2. POST to /users/create

kafka:
    topic: ping
    group: a
http:
    method: POST
    url: http://myapi.com/users/pong

Templated requests based on body

You can template a http request based on the contents of the body.

http:
    method: POST
    url: http://myapi.com/users/create
    body: |
        {
            userId: ${msgBody.user.id},
            lovesCats: True
        }

More sophisticated example

The above examples can be useful but often there are other concerns.

1. What if you want to keep retrying?
2. What if we want to only retry in speific cases?
3. What about authentication?

The following example demonstrates:

1. Get a message from the users topic
2. POST to /users/create
3. Templatize body of request based on message contents
4. Retry but only on 500 errors
5. If either there is a failure or the retry limit is reached put it in another topic for retrivial later (dead letter queue)
6. Throttle service so that it doesn't overwhelm the receiving service

kafka:
    topic: users
    group: a
http:
    method: POST
    url: http://myapi.com/users/create
    body: |
        {
            userId: ${msgBody.user.id},
            lovesCats: True
        }
    headers:
        Content-Type: application/json
    authorization: Bearer
retry:
    onlyRetryOn:
        - statusCode: 5xx
onlyRetryOn:
    5**
deadLetterQueue: users-dlq

Limitations

• Only supports json encoded messages for now.
• This is meant to be a simple service and you might have more sophisticated needs.
  • If we are not able to accommodate you may want to build your own service or make a fork.