tags | projects | ||||
---|---|---|---|---|---|
|
|
This guide walks you through testing a simple Spring YARN project.
What you’ll build
You’ll use an existing sample application and create a simple JUnit test to verify application is executed properly.
What you’ll need
https://raw.githubusercontent.com/spring-guides/getting-started-macros/master/prereq_editor_jdk_buildtools.adoc - Local single-node instance based on Hadoop 2.2.0 or later
Note
|
Testing this sample application don’t need existing or running Hadoop instance. |
If you’re not familiar with gradle, refer to Building Java Projects with Gradle.
Set up the project
Focus on this guide is to use JUnit
to test your sample application. We have already prepared a project for gradle and application files based on sample YARN Basic Sample. What is left for you is to create a JUnit
test class.
Create a JUnit Test Class
gs-yarn-testing-dist/src/test/java/hello/AppIT.java
link:complete/gs-yarn-testing-dist/src/test/java/hello/AppIT.java[]
Let’s go through step by step what’s happening in this JUnit
class. As already mentioned earlier we don’t need any existing or running Hadoop instances, instead testing framework from Spring YARN provides an easy way to fire up a mini cluster where your tests can be run in an isolated environment.
We named a class as AppIT
which plays nice with both gradle and
maven when tests are executed during the build. Tests will need a real
artifacts to be build which means test can only run after jar files
has been build and processed by a Spring Boot’s
repackage plugin.
In gradle build file we make sure artifacts are build before tests are
run. In maven build file we use failsafe
plugin which is meant for
integration testing and tests handled by it are executed after all
project artifacts has been build.
-
We used
spring.yarn.client.files
property to override settings defined inapplication.yml
file because test is run fromgs-yarn-testing-dist
project. -
@MiniYarnClusterTest
is a composed annotation telling Spring to start a Hadoop’s mini cluster having components forHDFS
andYARN
. Hadoop’s configuration from this minicluster is automatically injected into your testing context. -
AbstractBootYarnClusterTests
is a class containing a lot of base functionality what you need in your tests.
Then it’s time to deploy the application into a running minicluster
-
submitApplicationAndWait()
method simply runs yourClientApplication
and expects it to do an application deployment. On default it will wait 60 seconds an application to finish and returns a current state. -
We make sure that we have a correct application state
We use ContainerLogUtils
to find our container logs files from a minicluster.
-
We assert count of a log files
-
We expect some specified content from log file
-
We expect
stderr
files to be empty
Run Test
If using gradle:
./gradlew clean build
If using maven:
mvn clean package
If test is executed succesfully you’d be able to see YARN container logs within the minicluster logging directory:
$ find gs-yarn-testing-dist/target/|grep std
target/yarn--1502101888/yarn--1502101888-logDir-nm-0_0/application_1392395851515_0001/container_1392395851515_0001_01_000002/Container.stdout
target/yarn--1502101888/yarn--1502101888-logDir-nm-0_0/application_1392395851515_0001/container_1392395851515_0001_01_000002/Container.stderr
target/yarn--1502101888/yarn--1502101888-logDir-nm-0_0/application_1392395851515_0001/container_1392395851515_0001_01_000001/Appmaster.stdout
target/yarn--1502101888/yarn--1502101888-logDir-nm-0_0/application_1392395851515_0001/container_1392395851515_0001_01_000001/Appmaster.stderr
$ grep Hello gs-yarn-testing-dist/target/yarn--1502101888/yarn--1502101888-logDir-nm-0_0/application_1392395851515_0001/container_1392395851515_0001_01_000002/Container.stdout
[2014-02-14 16:37:54.278] boot - 18453 INFO [main] --- HelloPojo: Hello from HelloPojo
[2014-02-14 16:37:54.278] boot - 18453 INFO [main] --- HelloPojo: About to list from hdfs root content
[2014-02-14 16:37:55.157] boot - 18453 INFO [main] --- HelloPojo: FileStatus{path=hdfs://localhost:33626/; isDirectory=true; modification_time=1392395854968; access_time=0; owner=jvalkealahti; group=supergroup; permission=rwxr-xr-x; isSymlink=false}
[2014-02-14 16:37:55.157] boot - 18453 INFO [main] --- HelloPojo: FileStatus{path=hdfs://localhost:33626/app; isDirectory=true; modification_time=1392395854968; access_time=0; owner=jvalkealahti; group=supergroup; permission=rwxr-xr-x; isSymlink=false}
Summary
Congratulations! You have now created a simple yet effective JUnit
test for building Spring YARN projects.
See Also
The following guides may also be helpful: