This is an example of a custom source module using the Spring Integration S3 inbound channel adapter to download files from a S3 bucket. This is built and packaged for installation in a Spring XD runtime environment using maven. The project includes sample unit and integration tests, including the ability to test the module in an embedded single node container. It also illustrates how to define module options using simple property descriptors.
In order to install the module and run it in your Spring XD installation, you will need to have installed:
- Spring XD version 1.1.x (Instructions)
Ensure you have the following AWS credentials on hand:
- accessKey the accessKey for AWS
- bucket the bucket for S3
- remoteDirectory the remoteDirectory for S3 - secretKey the secretKey for AWS
- localDirectory the localDirectory where files will be downloaded to
- fixedRate the fixed rate polling interval specified in milliseconds
Update the ConfigurationTest-context.xml
file with credentials above to run the unit tests.
Create the localDirectory on your machine
This implements a source module which downloads files from a s3 bucket from given AWS API credentials using an existing Spring Integration S3 inbound channel adapter. The example demonstrates the use of the spring-xd-module-parent
pom and an integration test to test the module registered and deployed in an embedded Spring XD single node container.
$ mvn package
The project's [pom][] declares spring-xd-module-parent
as its parent. This adds the dependencies needed to test the module and also configures the [Spring Boot Maven Plugin][] to package the module as an uber-jar, packaging any dependencies that are not already provided by the Spring XD container. See the [Modules][] section in the Spring XD Reference for a more detailed explanation of module class loading.
$./gradlew clean test bootRepackage
The project's [build.gradle][] applies the spring-xd-module
plugin, providing analagous build and packaging support for gradle. This plugin also applies the [Spring Boot Gradle Plugin][] as well as the [propdeps plugin][].
The uber-jar will be in [project-build-dir]/s3-source-1.0.0.BUILD-SNAPSHOT.jar
. To install and register the module to your Spring XD distribution, use the module upload
Spring XD shell command. Start Spring XD and the shell:
_____ __ _______
/ ___| (-) \ \ / / _ \
\ `--. _ __ _ __ _ _ __ __ _ \ V /| | | |
`--. \ '_ \| '__| | '_ \ / _` | / ^ \| | | |
/\__/ / |_) | | | | | | | (_| | / / \ \ |/ /
\____/| .__/|_| |_|_| |_|\__, | \/ \/___/
| | __/ |
|_| |___/
eXtreme Data
1.1.0.BUILD-SNAPSHOT | Admin Server Target: http://localhost:9393
Welcome to the Spring XD shell. For assistance hit TAB or type "help".
xd:>module upload --file [path-to]/s3-source-1.0.0.BUILD-SNAPSHOT.jar --name s3Source --type source
Successfully uploaded module 'source:s3Source'
xd:>
You can also get information about the available module options:
xd:>module info source:s3Source
Information about source module 's3Source':
Option Name Description Default Type
------------------ --------------------------------------------------------- ------- ---------
fixedRate the fixed rate polling interval specified in milliseconds 5000 int
accessKey the accessKey for AWS RANDOMSTRING java.lang.String
bucket the bucket for S3 testboom java.lang.String
remoteDirectory the remoteDirectory for S3 / java.lang.String
maxMessagesPerPoll the maximum number of messages per poll 100 int
secretKey the secretKey for AWS RANDOMString java.lang.String
localDirectory the localDirectory /tmp/nec java.lang.String
outputType how this module should emit messages it produces <none> org.springframework.util.MimeType
Now create and deploy a stream:
xd:>stream create S3Test --definition "s3Source --accessKey=[accessKey] --secretKey=[secretKey] --bucket=[bucket] --remoteDirectory=[remoteDirectory] --localDirectory=[localDirectory] | log" --deploy
You should see the stream output in the Spring XD log