Error: Java heap space - pylrmr

[Abdul H Khan]

Hi all,

I hoping to get some help..I have exhausted all resources with "Java heap space" when running bind.cols input with plyrmr.

My code

Sys.setenv(HADOOP_HOME='/opt/cloudera/parcels/CDH/lib/hadoop')
Sys.setenv(HADOOP_CMD='/usr/bin/hadoop')
Sys.setenv(HADOOP_STREAMING='/opt/cloudera/parcels/CDH/lib/hadoop-mapreduce/hadoop-streaming.jar')

indir <- "/home/akhan/data_R/"

library(rhdfs)
library(plyrmr)
options (java.parameters="-Xmx4096m")
hdfs.init()

bind.cols(mtcars, carb.per.cyl = carb/cyl)
options (java.parameters="-Xmx4096m")
bind.cols(input("/tmp/mtcars"), carb.per.cyl = carb/cyl)

I have looked all the post and googled around to figure out what maybe causing this issue and no luck. PLYRMR is installed on all of the hadoop cluster nodes.

16/06/22 18:27:55 INFO mapreduce.Job: Running job: job_1466530531134_0121
16/06/22 18:28:00 INFO mapreduce.Job: Job job_1466530531134_0121 running in uber mode : false
16/06/22 18:28:00 INFO mapreduce.Job: map 0% reduce 0%
16/06/22 18:28:03 INFO mapreduce.Job: Task Id : attempt_1466530531134_0121_m_000001_0, Status : FAILED
Error: Java heap space
16/06/22 18:28:03 INFO mapreduce.Job: Task Id : attempt_1466530531134_0121_m_000000_0, Status : FAILED
Error: Java heap space
16/06/22 18:28:07 INFO mapreduce.Job: Task Id : attempt_1466530531134_0121_m_000000_1, Status : FAILED
Error: Java heap space
16/06/22 18:28:07 INFO mapreduce.Job: Task Id : attempt_1466530531134_0121_m_000001_1, Status : FAILED
Error: Java heap space
16/06/22 18:28:10 INFO mapreduce.Job: Task Id : attempt_1466530531134_0121_m_000000_2, Status : FAILED
Error: Java heap space
16/06/22 18:28:10 INFO mapreduce.Job: Task Id : attempt_1466530531134_0121_m_000001_2, Status : FAILED
Error: Java heap space
16/06/22 18:28:15 INFO mapreduce.Job: map 100% reduce 100%
16/06/22 18:28:16 INFO mapreduce.Job: Job job_1466530531134_0121 failed with state FAILED due to: Task failed task_1466530531134_0121_m_000000
Job failed as tasks failed. failedMaps:1 failedReduces:0

16/06/22 18:28:16 INFO mapreduce.Job: Counters: 13
Job Counters
Failed map tasks=8
Launched map tasks=8
Other local map tasks=6
Data-local map tasks=1
Rack-local map tasks=1
Total time spent by all maps in occupied slots (ms)=33624
Total time spent by all reduces in occupied slots (ms)=0
Total time spent by all map tasks (ms)=16812
Total vcore-seconds taken by all map tasks=16812
Total megabyte-seconds taken by all map tasks=34430976
Map-Reduce Framework
CPU time spent (ms)=0
Physical memory (bytes) snapshot=0
Virtual memory (bytes) snapshot=0
16/06/22 18:28:16 ERROR streaming.StreamJob: Job not successful!
Streaming Command Failed!
Error in mr(map = map, reduce = reduce, combine = combine, vectorized.reduce, :
hadoop streaming failed with error code 1
Calls: ... mrexec -> as.big.data -> do.call -> -> mr
Execution halted