This repository provides a collection of components that help with various data integration use cases. It provides standalone Java functions that can be reused in end-user applications.
In addition to providing these functions, this repository is also used to generate out of the box Spring Cloud Stream applications those using the functions as a baseline. These applications can be run standalone as Spring Cloud Stream applications or as part of a data flow such as the one orchestrated using Spring Cloud Data Flow.
The repository is divided into two sections - Functions and Applications. The former is used for hosting the various functions and the latter is for generating the apps.
The following are the four major components of this repository.
-
Standalone Java Functions as java.util.function.Function/Consumer/Supplier
-
Core components for the out of the box applications
-
Out of the box Spring Cloud Stream applications as Source/Sink/Processor
-
Aggregator for the out of the box applications.
| Supplier | Consumer | Function |
|---|---|---|
| Source | Sink | Processor |
|---|---|---|
By default, the source applications are auto-configured with functions which may optionally be included in a composite function definition.
This feature enables the sources to do things like
- execute SpEL transformations
- enrich message headers
- filter events
- produce task launch requests
or any combination of the above, without requiring a separate processor.
For example, the time source executed, as shown below, will perform a series of transformations to publish a task launch request every second to the rabbit exchange time-test.
java -jar target/time-source-rabbit-3.0.0-SNAPSHOT.jar --spring.cloud.stream.bindings.output.destination=time-test --spring.cloud.stream.function.definition="timeSupplier|spelFunction|headerEnricherFunction|taskLaunchRequestFunction" --spel.function.expression="payload.length()" --header.enricher.headers=task-id=payload*2 --task.launch.request.task-name-expression="'task-'+headers['task-id']"The transformed message looks like this:
headers:
task-id: 34
content_type: application/json
Payload
49 bytes
Encoding: string
{"args":[],"deploymentProps":{},"name":"task-34"}Let’s upack this function definition:
timeSupplier|spelFunction|headerEnricherFunction|taskLaunchRequestFunction
This creates a composed Supplier beginning with the default timeSupplier which is the foundation for time-source.
This produces a String like 06/18/20 16:01:38. We transform this using the SpEL expression payload.length().
The spelFunction applies to a Message from which we can extract and transform the payload or headers, in accordance with common Spring Integration conventions.
The output of spelFunction is the length of the date-time String, 17.
From here we apply the header enricher to add a Message header, task-id with the value of payload*2 or ,34.
We use the task-id header to generate the task name for the task launch request using the SpEL expression "'task-'+headers['task-id']", or task-34.
This somewhat contrived example is intended to show the power of function composition.
Even so, if you have task-34 as a task definition in Data Flow, you could build a simple pipeline time | tasklauncher to launch the task every second.
Prior to 3.0 release of Stream Applications, this composition required extensive customization.
Please see the individual function documentation for an explanation of its configuration properties.
|
Note
|
Support for composite functions includes auto-configuration for conventional binding name mappings (input and output) derived from the function definition and the presence of spring.cloud.stream.bindings.output….
In this example, --spring.cloud.stream.bindings.output.destination=time-test is enabled behind the scenes by the auto-configured property
--spring.cloud.stream.function.bindings.timeSupplierspelFunctionheaderEnricherFunctiontaskLaunchRequestFunction-out-0=output.
|
You can build everything from the root of the repository.
./mvnw clean install
But, this may not be what you are interested in doing since you are probably interested in a single application or a few of them. In order to build the functions and applications that you are interested in, you need to build them selectively as shown below.
./mvnw clean install -f functions
You can also build a single function or group of functions. For e.g if you are only interested in jdbc-supplier and log-consumer, do the following.
./mvnw clean install -pl :jdbc-suppler,:log-consumer
Let’s assume that you want to build JDBC Source application based on Kafka Binder in Spring Cloud Stream and Log Sink application based on Rabbit binder. Here is what you need to do. Assuming that you built both functions and stream-applications-core as above.
./mvnw clean package -pl :jdbc-source
cd applications/source/jdbc-source/apps/jdbc-source-kafka
./mvnw clean packageThis will generate the Kafka binder based uber jar in the target folder.
Similarly for the log sink, do the following.
./mvnw clean package -pl :log-sink
cd applications/sink/log-sink/apps/log-sink-rabbit
./mvnw clean packagePlease see our Code of Conduct