Vader Sentiment in Scala with Spark 
This is a Scala conversion of https://github.com/cjhutto/vaderSentiment.
It contains UDF wrappers for the functions, and outputs a map of the sentiments corresponding to a text.
Build with Java 1.8, Scala 2.11.8, SBT 0.13.15 and Spark 2.1
Map => { positivity polarity, negative polarity, neutral polarity, compound polarity }
Compound polarity:
- positive sentiment: compound score >= 0.5
- neutral sentiment: (compound score > -0.5) and (compound score < 0.5)
- negative sentiment: compound score <= -0.5
Direct calling in Scala:
SentimentIntensityAnalyser.polarityScores("sentimental text here") or as UDF:
import io.witlox.sentiment.Vader._
...
// using a UDF you would have to unwrap the map to it's respective columns, here the functions
// are split by element of the map, so simply add them column by column
val dfWithPositiveSentiment = df.withColumn("positive", positive($"text"))
val dfWithCompoundSentiment = dfWithPositiveSentiment.withColumn("sentiment", compound($"text"))
Also included is a bucketing function for dates. In order to group sentiments (for example in Tweets) we need to group them.
import io.witlox.utils.TimePartition
// create a TimePartition on year (smallest available is milliseconds)
val tp = TimePartition("year")
val bucket = tp.bucketize("2012-01-01T01:01:01.000Z")
or as UDF:
val tp = TimePartition("year")
val dfWithBucket = df.withColumn("bucket", tp.bucket($"ISODateTimeFormatString"))
