Rhapsody is an asynchronous message processing library that builds on the Reactive Streams Specification to deliver reusable functionalities under the following categories:
- At-Least-Once Processing Guarantee
- Quality of Service
- Observability
While delivering features that fall in to the above categories, Rhapsody aims to maintain the inherent developmental attributes afforded by the Reactive Streams Specification:
- Non-blocking backpressure
- Arbitrary parallelism
- Infrastructural Interoperability (Kafka, RabbitMQ, etc.)
- Implementation Interoperability (ReactiveX/RxJava, Project Reactor, etc.)
Where appropriate, Rhapsody is backed by Project Reactor to build Transformers and inject side-effect behaviors (like Logging and Metric collection).
Rhapsody started off as an Inner Source library at Vrbo, an Expedia Group brand. At time-of-inception, Vrbo was heavily expanding its usage of Kafka as the de facto asynchronous messaging system of choice.
The need for an alternative asynchronous streaming framework arose out of Vrbo's growing diversity of streaming use cases. Traditionally available streaming libraries, like Kafka Streams, were not quite flexible enough to support the superset of functionalites that developers were trying to build on top of Kafka. In particular, traditional streaming libraries make assumptions about what infrastructures are in play, what processing topologies should be available, and how easy it should be to design for failure. Vrbo's needs for asynchronous streaming use cases include(d) heterogeneous infrastructure endpoints (like RabbitMQ-to-Kafka), I/O-bound processing (message processing that requires API or Database interaction), Extended Architecture (XA) / Two Phase Commit Processing, Batch Processing, Message Deduplication, and others.
And so, Rhapsody was born to attempt providing a streaming framework that provides At-Least-Once Processing Guarantees (such as what we get from other traditional streaming libraries), arbitrary parallelism (to address I/O-bound processing scalability), and arbitrary interoperability with today's and the future's streaming infrastructures.
Before getting started, it's important to note that Rhapsody does not aim to be a replacement for any fluent Reactive Streams libraries. The goal of Rhapsody is to build on existing Reactive Streams implementations and provide functionality that primarily addresses usage with infinite asynchronous flows of messages while adhering to the Reactive Streams API. In particular, Rhapsody heavily integrates with Project Reactor and related projects, like Reactor Kafka and Reactor RabbitMQ, to avoid re-implementation of existing Publisher and Subscriber implementations.
We highly recommend getting familiar with Project Reactor via its Learning Resources if you are not already familiar with Reactive Streams or any of its implementations.
The key abstraction around which Rhapsody builds "At Least Once Processing Guarantees" is Acknowledgeable. The goal behind "acknowledgeability" is to restore a lightweight form of the bi-directional communication in control flow that we lose when moving from synchronous to asynchronous code.
In synchronous control flows, we have the (dubious) benefit of being able to tightly couple the processing of any given input/message to its successful completion or abnormal termination (Errors/Exceptions). The same goes for asynchronous control flows where there is neither the presence of backpressure or thread boundaries. In either case, when the controlling thread completes the processing/emission of a unit of data without erroneous termination, there is a reasonable implication that the corresponding data has been successfully processed, or, at worst, any errors resulting from the processing of that data were gracefully handled. An opposite, mutually-exclusive implication is made when that processing results in an Error/Exception being raised/thrown.
In contrast, asynchronous control flows that may incorporate backpressure and/or arbitrary numbers of asynchronous boundaries do not typically have a semantic for communicating "successful completion" or "abnormal termination" to the emission sources of processed data. Acknowledgeability aims to address this by providing "channels" for "acknowledgement" and "nacknowledgement" (negative acknowledgement) that are logically coupled to the originating data emitted by a Publisher and propagated with that data's downstream transformations. For example, negatively acknowledging the processing/transformation of a Kafka Record allows us to emit the corresponding error (and hence not committing past its offset) and subsequently resubscribe such that the error-inducing Record is eventually reprocessed.
Rhapsody has incrementally evolved to include commonly desired Quality of Service functionalities, like Rate Limiting, Deduplication, and Maximum In-Flight Acknowledgeability. Like most of the features provided by Rhapsody, these are implemented as Publisher Transformers.
Rhapsody aims to provide observability in to Reactive Streams by leveraging existing Project Reactor integrations and integrating with standard observability APIs, like SLF4J for logging, Micrometer for metrics, and OpenTracing
Rhapsody is a Java library that requires at least JDK 1.8+ for building and integration.
Rhapsody is built using maven
mvn clean verify
Check out the Samples to see Rhapsody in action
Please refer to CONTRIBUTING for information on how to contribute to Rhapsody