rx-aws

This project contains reactive stream bindings for AWS services including Kinesis and SQS.

It consumes SQS Queues and Kinesis Streams and publishes them onto the Reactor Event Bus.

From there the events can be consumed and processed by Reactive Streams compliant operators, including RxJava.

Project Reactor

Reactor is a second-generation Reactive library for building non-blocking applications on the JVM based on the Reactive Streams Specification.

Reactor's EventBus is a very elegant pub-sub implementation.

EventBus

Using the EventBus is very easy:

EventBus bus = EventBus.create(Environment.initializeIfEmpty(), Environment.THREAD_POOL);

To subscribe to all events publsihed on the bus:

bus.on(Selectors.matchAll(),e-> {
  System.out.println("Hello, "+e.getData());
});

To publish an event:

bus.notify("greeting",Event.wrap("World!"));

Will yield the following output:

Hello, World!

Nice!

SQS

The following code will consume the given queue and publish events onto the given EventBus:

new SQSReactorBridge.Builder()
  .withUrl("https://sqs.us-east-1.amazonaws.com/111122223333/myqueue")
  .withEventBus(bus)
  .build()
  .start();

Consuming messages from the queue is then as easy as subscribing to the EventBus:

bus.on(SQSMessageSelectors.anySQSMessage(),event -> {
  System.out.println(event);
});

SNS via SQS

SNS messages are commonly delivered over an SQS queue. The SQSReactorBridge has special processing for this. When building the bridge, just call withSNSSupport(true):

new SQSReactorBridge.Builder()
  .withUrl("https://sqs.us-east-1.amazonaws.com/111122223333/myqueue")
  .withEventBus(bus)
  .withSNSSupport(true)   // << enable SNS support
  .build()
  .start();

This will cause SNS messages to be parsed and re-emitted as Event<SNSMessage>.

Kinesis

The following code will start a Kinesis Consumer Libarary (KCL) worker instance that will read from the stream named mystream located in the us-west-1 region. It will publish events onto the specified EventBus.

new KinesisReactorBridge.Builder()
	.withRegion("us-east-1")
	.withStreamName("mystream")
	.withAppName("myapp")
	.withEventBus(bus)
	.build()
  .start();

This can subscribed to similarly:

bus.on(Selectors.type(KinesisRecord.cass), it -> {
    System.out.println(it);
});

Selectors and Predicates

When Message objects are publshed onto the event bus, they are wrapped in an SQSMessage object. This SQSMessage object is used as the key for the publish operation.

This makes it possible to filter messages for subscription. This is an example of a Lambda Predicate that matches any SQSMessage that comes from the test stream:

Predicate<SQSMessage> p = msg -> {
  return msg.getUrl().endsWith("/test");
}

This could then be applied like so:

bus.on(Selectors.predicate(p), it -> {
    System.out.println(it);
});

There is a convenience method in MoreSelectors that allows this to be condensed into one line with no intermediate variables:

bus.on(
  MoreSelectors.typedPredicate( (SQSMessage msg) -> msg.getUrl().endsWith("/test")
), 
  it -> System.out.println(it)
);

The following table describes some generic Selectors that are independent of AWS:

Selector	Description
MoreSelectors.typedPredicate(Predicate predicate)	Similar to Selectors.predicate(), but it uses generics properly to that the return value is properly typed.
MoreSelectors.jsonPredicate(Predicate predicate)	MoreSelectors.typedPredicate() convenience method for JsonNode payloads
MoreSelectors.all(Selector ...selectors)	Composes selectors together. All must evaluate to true.

And there are several Selectors that apply only to SQS:

Selector	Description
SQSReactorBridge.eventsFromBridgeSelector()	Selects all events that were generated by the given bridge.
SQSMessageSelectors.anySQSMessage()	Selects any SQSMessage event
SQSMessageSelectors.queueName(String name)	Matches the queue name via URL

And a number that are speicific to Kinesis:

Selector	Description
KinesisRecordSelectors.anyKinesisRecord()	Convenience for Selectors.type(KinesisRecord.class)
KinesisRecordSelectors.streamName(String name)	Matches all KinesisRecord events from the given stream
KinesisRecordSelectors.streamArn(String arn)	Matches all KinesisRecord events for the given ARN. You must prrovide the ARN via withStreamArn() when constructing the bridge for this to work.
KinesisReactorBridge.eventsFromBridgeSelector()	Matches all KinesisRecord events originating from the given bridge

RxJava

It is straightforward to bridge the reactive streams API into an RxJava Observable using the RxReactiveStreams Adapter:

Observable<Event<SQSMessage>> observable = 
    (Observable<Event<SQSMessage>>) 
        RxReactiveStreams.toObservable( 
            bus.on(SQSMessageSelectors.anySQSMessage())
        );

You can then apply operators and subscribe to the observable.

Since the types can get quite complicated, we have provided a few wrapper methods that simplify things.

For instance, if you want to get straight to the String payload:

Observable<String> observable = 

  SQSReactiveStreamAdapters.toObservableString(
    bus.on(SQSMessageSelectors.anySQSMessage())
  );

bjoernw / rx-aws