High memory consumption

Question

High memory consumption

honza opened this issue 10 years ago · comments

I created the default project with lein new chestnut <xyz>, started the repl with lein repl followed by (run) and (browser-repl). Before doing this, I removed all of my plugins from ~/.lein/profiles.clj.

I am now left with 3 java processes. Together, these processes use almost 2GB of memory.

This is on OSX 10.10 and here is the repl session if it helps.

~/Code/front on master  $ lein repl
nREPL server started on port 53644 on host 127.0.0.1 - nrepl://127.0.0.1:53644
REPL-y 0.3.5, nREPL 0.2.6
Clojure 1.6.0
Java HotSpot(TM) 64-Bit Server VM 1.7.0_60-b19
    Docs: (doc function-name-here)
          (find-doc "part-of-name-here")
  Source: (source function-name-here)
 Javadoc: (javadoc java-object-or-class-here)
    Exit: Control+D or (exit) or (quit)
 Results: Stored in vars *1, *2, *3, an exception in *e

front.server=> (run)
2014-11-23 10:25:39.806:INFO:oejs.Server:jetty-7.6.13.v20130916
2014-11-23 10:25:39.856:INFO:oejs.AbstractConnector:Started SelectChannelConnector@0.0.0.0:10555
Starting web server on portStarting figwheel.
 10555 .
#<Server org.eclipse.jetty.server.Server@6f7714c8>
front.server=> Compiling ClojureScript.
Figwheel: Starting server at http://localhost:3449
Figwheel: Serving files from '(dev-resources|resources)/public'
front.server=> (browser-repl)
<< started Weasel server on ws://0.0.0.0:9001 >>
Type `:cljs/quit` to stop the ClojureScript REPL
nil

Is this normal? Can something be done about this?

Arne Brasseur · Answer 1 · Sun Nov 23 2014 18:05:37 GMT+0800 (China Standard Time)

Hi, thanks for reporting!

The three processes are leiningen, the repl, and figwheel. You can bring it down to two by using lein trampoline repl. This will replace the leiningen process with the repl once it starts.

Unfortunately Figwheel can't be used as a library, which means we have to invoke it through leiningen, so that creates an extra JVM. See also #47.

Apart from that this behavior is indeed "normal", although 2GB seems like a lot. For me it's more around 500MB.

There are various flags to control or limit the amount of memory the JVM uses, you could experiment with those.

Honza Pokorný · Answer 2 · Mon Nov 24 2014 03:01:54 GMT+0800 (China Standard Time)

Would you be willing to share some details about your setup? I'm having a hard time understanding why someone people have such low memory numbers...

Arne Brasseur · Answer 3 · Mon Nov 24 2014 03:09:08 GMT+0800 (China Standard Time)

This is what I use

$  java -version
java version "1.8.0_25"
Java(TM) SE Runtime Environment (build 1.8.0_25-b17)
Java HotSpot(TM) 64-Bit Server VM (build 25.25-b02, mixed mode)

$  lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 14.04.1 LTS
Release:        14.04
Codename:       trusty

$  lein --version
Leiningen 2.5.0 on Java 1.8.0_25 Java HotSpot(TM) 64-Bit Server VM

Not much special for the rest. ~/lein/profiles.clj only contains the cider-nrepl plugin. I'm not sure what else could be relevant here.

Is it only with chestnut projects that you have this? Maybe compare with some other configs to see what part is so memory hungry. So do a lein trampoline repl in an empty directory, or in a new project, compare memory before/after (run), do lein figwheel separately and see how that behaves, etc.

Honza Pokorný · Answer 4 · Mon Nov 24 2014 05:10:05 GMT+0800 (China Standard Time)

Thank you for your help!

I have upgraded to JDK 1.8 and now my setup matches what you described (apart from the OS, of course).

A couple of notes:

When using lein trampoline repl, I can't successfully execute the (browser-repl) call.
When using the cider-nrepl middleware in my profile, I only get 2 java processes instead of 3 and save about 500mb of memory.

Yes, I have noticed this with other projects as well but this case was unfortunately especially bad.

Thanks again.

Arne Brasseur · Answer 5 · Mon Nov 24 2014 05:14:40 GMT+0800 (China Standard Time)

Glad it's better now. It's good to know this is a pain point. I hope we can make it all work seemlessly in a single process eventually.

Bruce Hauman · Answer 6 · Sun Nov 30 2014 03:05:35 GMT+0800 (China Standard Time)

I've been watching this and been thinking a lot about making a library thats executable from the repl.

Martin Klepsch · Answer 7 · Sun Nov 30 2014 03:42:11 GMT+0800 (China Standard Time)

I think lively could be embedded in the process easily. You'd need to handle Clojurescript compilation on your own then though.

Arne Brasseur · Answer 8 · Sun Nov 30 2014 03:46:27 GMT+0800 (China Standard Time)

@bhauman that would be absolutely fabulous!

@martinklepsch how does lively relate to figwheel? Are they completely separate implementations?

Martin Klepsch · Answer 9 · Sun Nov 30 2014 04:48:07 GMT+0800 (China Standard Time)

@plexus They're separate. Lively only implements the pushing of new code to the the browser. And does not care about compiling Clojurescript.

Bruce Hauman · Answer 10 · Sat Dec 20 2014 05:53:41 GMT+0800 (China Standard Time)

Just want to let you know that I just released lein-figwheel 0.2.0-SNAPSHOT and you can now launch the autobuilder/change server from the new figwheel-sidecar library.

https://github.com/bhauman/lein-figwheel/blob/master/sidecar/src/figwheel_sidecar/auto_builder.clj#L38

Unfortunately config options are not yet validated at this level. The config validation is all still up in the leiningen plugin.

Arne Brasseur · Answer 11 · Fri Dec 26 2014 20:18:06 GMT+0800 (China Standard Time)

Thank you @bhauman , that's great news!