curtishoward / sparkudfexamples

Spark SQL UDF examples

Spark UDF Examples

Simple examples of Spark SQL user-defined functions. Tested with CDH 5.13.1, Spark 2.1.0 (see the spark1.6 branch for Spark 1.x examples)

Load the sample data

hdfs dfs -put data/temperatures.json temperatures.json
hdfs dfs -put data/inventory.json    inventory.json

Build the Java and Scala examples

Under each example root (java-udf/, scala-udf/, ...):

mvn package

Run them

Python UDF:

spark2-submit --master local python-udf-example.py

Scala UDF:

spark2-submit --class com.cloudera.fce.curtis.sparkudfexamples.scalaudf.ScalaUDFExample --master local target/scalaudf-0.0.1-jar-with-dependencies.jar

Java UDF:

spark2-submit --class com.cloudera.fce.curtis.sparkudfexamples.javaudf.JavaUDFExample  --master local target/javaudf-0.0.1-jar-with-dependencies.jar

Scala UDAF:

spark2-submit --class com.cloudera.fce.curtis.sparkudfexamples.scalaudaf.ScalaUDAFExample --master local target/scalaudaf-0.0.1-jar-with-dependencies.jar

Hive UDF:

spark2-submit --jars target/hiveudf-0.0.1-jar-with-dependencies.jar  hive-udf-example.py

Scala UDAF From PySpark:

spark2-submit --jars target/scalaudaffrompython-0.0.1.jar --driver-class-path target/scalaudaffrompython-0.0.1.jar scala-udaf-from-python.py