An example of docker-compose to set up a single Apache Spark node connecting to MongoDB via MongoDB Spark Connector
** For demo purposes only **
You can start by running command :
docker-compose run spark bash
Which would run the spark node and the mongodb node, and provides you with bash shell for the spark.
From the spark instance, you could reach the MongoDB instance using mongodb hostname.
You can find a small dataset example in /home/ubuntu/times.json which you can load using initDocuments.scala :
spark-shell --conf "spark.mongodb.input.uri=mongodb://mongodb:27017/spark.times" --conf "spark.mongodb.output.uri=mongodb://mongodb/spark.output" --packages org.mongodb.spark:mongo-spark-connector_${SCALA_VERSION}:${MONGO_SPARK_VERSION} -i ./initDocuments.scala
For example, please see examples.scala to query from mongodb, run a simple aggregation, dataframe SQL and finally write output back to mongodb. This file will also be available inside of the spark container in /home/ubuntu/examples.scala
Run the spark shell by executing:
spark-shell --conf "spark.mongodb.input.uri=mongodb://mongodb:27017/spark.times" --conf "spark.mongodb.output.uri=mongodb://mongodb/spark.output" --packages org.mongodb.spark:mongo-spark-connector_${SCALA_VERSION}:${MONGO_SPARK_VERSION}You can also append -i <file.scala> to execute a scala file via the spark shell. For example:
spark-shell --conf "spark.mongodb.input.uri=mongodb://mongodb:27017/spark.times" --conf "spark.mongodb.output.uri=mongodb://mongodb/spark.output" --packages org.mongodb.spark:mongo-spark-connector_${SCALA_VERSION}:${MONGO_SPARK_VERSION} -i ./examples.scala See related article: