`randomSplit()` produces incorrect results for the Git Dataset on Spark 1.4
agude opened this issue · comments
The user vectors for the Git dataset are being "mangled" by this line:
hermes/src/hermes_run_script.py
Line 68 in 0fd69ed
If you recombine the test and train vectors, they new combined RDD has the right number of rows, but the number of distinct users and items changes. It looks like some duplicate rows are introduced and some rows are dropped.
This problem does not seem to effect any other datasets (we tested them all), and is not present when run locally on my laptop using Spark 1.6. It only shows up when running on the cluster with Spark 1.4.
We have temporarily worked around this problem by saving the test and training vectors on my laptop and moving them to the cluster.
The bug also persists on Spark 1.5.2.