Uber Remote Shuffle Service (RSS)
Uber Remote Shuffle Service provides the capability for Apache Spark applications to store shuffle data on remote servers. See more details on Spark community document: [SPARK-25299][DISCUSSION] Improving Spark Shuffle Reliability.
Please contact us (remoteshuffleservice@googlegroups.com) for any question or feedback.
Supported Spark Version
- The master branch supports Spark 2.4.x. The spark30 branch supports Spark 3.0.x.
How to Build
Make sure JDK 8+ and maven is installed on your machine.
Build RSS Server
- Run:
mvn clean package -Pserver -DskipTests
This command creates remote-shuffle-service-xxx-server.jar file for RSS server, e.g. target/remote-shuffle-service-0.0.9-server.jar.
Build RSS Client
- Run:
mvn clean package -Pclient -DskipTests
This command creates remote-shuffle-service-xxx-client.jar file for RSS client, e.g. target/remote-shuffle-service-0.0.9-client.jar.
How to Run
Step 1: Run RSS Server
- Pick up a server in your environment, e.g.
server1
. Run RSS server jar file (remote-shuffle-service-xxx-server.jar) as a Java application, for example,
java -Dlog4j.configuration=log4j-rss-prod.properties -cp target/remote-shuffle-service-0.0.9-server.jar com.uber.rss.StreamServer -port 12222 -serviceRegistry standalone -dataCenter dc1
Step 2: Run Spark application with RSS Client
-
Upload client jar file (remote-shuffle-service-xxx-client.jar) to your HDFS, e.g.
hdfs:///file/path/remote-shuffle-service-0.0.9-client.jar
-
Add configure to your Spark application like following (you need to adjust the values based on your environment):
spark.jars=hdfs:///file/path/remote-shuffle-service-0.0.9-client.jar
spark.executor.extraClassPath=remote-shuffle-service-0.0.9-client.jar
spark.shuffle.manager=org.apache.spark.shuffle.RssShuffleManager
spark.shuffle.rss.serviceRegistry.type=standalone
spark.shuffle.rss.serviceRegistry.server=server1:12222
spark.shuffle.rss.dataCenter=dc1
- Run your Spark application
Run with High Availability
Remote Shuffle Service could use a Apache ZooKeeper cluster and register live service instances in ZooKeeper. Spark applications will look up ZooKeeper to find and use active Remote Shuffle Service instances.
In this configuration, ZooKeeper serves as a Service Registry for Remote Shuffle Service, and we need to add those parameters when starting RSS server and Spark application.
Step 1: Run RSS Server with ZooKeeper as service registry
- Assume there is a ZooKeeper server
zkServer1
. Pick up a server in your environment, e.g.server1
. Run RSS server jar file (remote-shuffle-service-xxx-server.jar) as a Java application onserver1
, for example,
java -Dlog4j.configuration=log4j-rss-prod.properties -cp target/remote-shuffle-service-0.0.9-server.jar com.uber.rss.StreamServer -port 12222 -serviceRegistry zookeeper -zooKeeperServers zkServer1:2181 -dataCenter dc1
Step 2: Run Spark application with RSS Client and ZooKeeper service registry
-
Upload client jar file (remote-shuffle-service-xxx-client.jar) to your HDFS, e.g.
hdfs:///file/path/remote-shuffle-service-0.0.9-client.jar
-
Add configure to your Spark application like following (you need to adjust the values based on your environment):
spark.jars=hdfs:///file/path/remote-shuffle-service-0.0.9-client.jar
spark.executor.extraClassPath=remote-shuffle-service-0.0.9-client.jar
spark.shuffle.manager=org.apache.spark.shuffle.RssShuffleManager
spark.shuffle.rss.serviceRegistry.type=zookeeper
spark.shuffle.rss.serviceRegistry.zookeeper.servers=zkServer1:2181
spark.shuffle.rss.dataCenter=dc1
- Run your Spark application