mkuthan / example-spark

Spark, Spark Streaming and Spark SQL unit testing strategies

Spark, Spark Streaming and Spark SQL unit testing strategies

Features

• ClockWrapper for efficient clock management in Spark Streaming jobs.
• Base traits for testing Spark, Spark Streaming and Spark SQL to eliminate boilerplate code.
• Sample applications to show how to make your code testable.
• All tests can be run or debugged directly from IDE, or using SBT.
• All test fixtures are prepared as in-memory data structures.
• SBT is configured to avoid problems with multiple Spark contexts in the same JVM SPARK-2243.
• SBT is configured to prepare project assembly for deployment on the cluster.

References

• http://mkuthan.github.io/blog/2015/03/01/spark-unit-testing/
• https://github.com/holdenk/spark-testing-base