error reading collection

[Stefano Spada]

Hi,
I'm trying to read in R, in chunks of 50k rows at a time, a collection of about 300k rows.

  dt <- dbGetQuery(mongo, arg, querystr,(i*chunk_size),chunk_size)

This is the error I get back
Error in .jcall(rmongo.object@javaMongo, "S", "dbGetQuery", collection, :
java.lang.OutOfMemoryError: Java heap space

The system has still plenty of memory available and the user I'm running the has the ulimit set to unlimited memory allocation.

[Tommy Chheng]

Which version of Java do you have on the system?

You can increase the java heap space by setting a environment variable

see:
http://stackoverflow.com/questions/417152/how-do-i-set-javas-min-and-max-heap-size-through-environment-variables
http://stackoverflow.com/questions/12577437/how-to-set-a-java-system-property-so-that-it-is-effective-whenever-i-start-jvm-w

[Stefano Spada]

java version "1.7.0_55"

OpenJDK Runtime Environment (rhel-2.4.7.1.el6_5-x86_64 u55-b13)

OpenJDK 64-Bit Server VM (build 24.51-b03, mixed mode)

On Tue, Jun 3, 2014 at 9:34 AM, Tommy Chheng [e-mail]
wrote:

Which version of Java do you have on the system?

You can increase the java heap space by setting a environment variable

see:

http://stackoverflow.com/questions/417152/how-do-i-set-javas-min-and-max-heap-size-through-environment-variables

http://stackoverflow.com/questions/12577437/how-to-set-a-java-system-property-so-that-it-is-effective-whenever-i-start-jvm-w

—
Reply to this email directly or view it on GitHub
#30 (comment).

Stefano Spada
CEO & Founder
GRANDSLA, Inc.
http://www.grandsla.com
skype: stefano.spada
g+:[e-mail]
[phone]

[Tommy Chheng]

Did increasing the heap space by setting the ENV variable resolve the issue?

On Tue, Jun 3, 2014 at 10:54 AM, StefanoSpada [e-mail]
wrote:

java version "1.7.0_55"

OpenJDK Runtime Environment (rhel-2.4.7.1.el6_5-x86_64 u55-b13)

OpenJDK 64-Bit Server VM (build 24.51-b03, mixed mode)

On Tue, Jun 3, 2014 at 9:34 AM, Tommy Chheng [e-mail]
wrote:

Which version of Java do you have on the system?

You can increase the java heap space by setting a environment variable

see:

http://stackoverflow.com/questions/417152/how-do-i-set-javas-min-and-max-heap-size-through-environment-variables

http://stackoverflow.com/questions/12577437/how-to-set-a-java-system-property-so-that-it-is-effective-whenever-i-start-jvm-w

—
Reply to this email directly or view it on GitHub
#30 (comment).

Stefano Spada
CEO & Founder
GRANDSLA, Inc.
http://www.grandsla.com
skype: stefano.spada
g+:[e-mail]
[phone]

—
Reply to this email directly or view it on GitHub
#30 (comment).

Tommy Chheng

[Stefano Spada]

I increased the java heap as suggested by setting
JAVA_OPTS="-Xms512m -Xmx1g"

still I get the error ...

Error in .jcall(rmongo.object@javaMongo, "S", "dbGetQuery", collection, :
java.lang.OutOfMemoryError: Java heap space
16
stop(structure(list(message = "java.lang.OutOfMemoryError: Java heap
space", call = .jcall(rmongo.object@javaMongo, "S", "dbGetQuery",
collection, query, keys, skip, limit), jobj = <S4 object of class
structure("jobjRef", package = "rJava")>), .Names = c("message", "call",
"jobj"), class = c("OutOfMemoryError", "VirtualMachineError", "Error",
"Throwable", "Object", "Exception", "error", "condition" )))
15
.jcheck(silent = FALSE)
14
.jcall(rmongo.object@javaMongo, "S", "dbGetQuery", collection, query, keys,
skip, limit)
13
dbGetQueryForKeys(rmongo.object, collection, query, "{}", skip, limit)
12
dbGetQueryForKeys(rmongo.object, collection, query, "{}", skip, limit)
11
dbGetQuery(mongo, arg, querystr, (i * chunk_size), chunk_size)
10
dbGetQuery(mongo, arg, querystr, (i * chunk_size), chunk_size) at
rserveSource.R#55
9
connect_rmongo(collection) at
rserveSource.R#97
8
eval(expr, envir, enclos)
7
eval(ei, envir)
6
withVisible(eval(ei, envir))
5
source("/home/stefano/git/server/server-mgmt/rserveSource.R") at
2mongo.R#3
4
eval(expr, envir, enclos)
3
eval(ei, envir)
2
withVisible(eval(ei, envir))
1
source("2mongo.R")

On Tue, Jun 3, 2014 at 3:25 PM, Tommy Chheng [e-mail]
wrote:

Did increasing the heap space by setting the ENV variable resolve the
issue?

On Tue, Jun 3, 2014 at 10:54 AM, StefanoSpada [e-mail]
wrote:

java version "1.7.0_55"

OpenJDK Runtime Environment (rhel-2.4.7.1.el6_5-x86_64 u55-b13)

OpenJDK 64-Bit Server VM (build 24.51-b03, mixed mode)

On Tue, Jun 3, 2014 at 9:34 AM, Tommy Chheng [e-mail]
wrote:

Which version of Java do you have on the system?

You can increase the java heap space by setting a environment variable

see:

http://stackoverflow.com/questions/417152/how-do-i-set-javas-min-and-max-heap-size-through-environment-variables

http://stackoverflow.com/questions/12577437/how-to-set-a-java-system-property-so-that-it-is-effective-whenever-i-start-jvm-w

—
Reply to this email directly or view it on GitHub
#30 (comment).

Stefano Spada
CEO & Founder
GRANDSLA, Inc.
http://www.grandsla.com
skype: stefano.spada
g+:[e-mail]
[phone]

—
Reply to this email directly or view it on GitHub
#30 (comment).

Tommy Chheng

—
Reply to this email directly or view it on GitHub
#30 (comment).

Stefano Spada
CEO & Founder
GRANDSLA, Inc.
http://www.grandsla.com
skype: stefano.spada
g+:[e-mail]
[phone]

[Tommy Chheng]

According to http://stackoverflow.com/questions/12577437/how-to-set-a-java-system-property-so-that-it-is-effective-whenever-i-start-jvm-w

You need to set JAVA_TOOL_OPTIONS or _JAVA_TOOL_OPTIONS, not JAVA_OPTS