Building madoko from source requires too much memory with GHC 8

[Maël Valais]

Hi,

While trying to hack the excellent madoko processor (see #6), I noticed that the memory needed for running koka-0.6.0-dev is huge (using ghc 8.0.2). I only have 4GB of RAM here, and koka wants over 3.8GB during the build:

If I let the process running, the whole system freezes. My workaround is simply to launch koka, kill it as soon as it reaches ~2GB and the relaunch it and so on.

Do you have any idea what could be causing that?

Thanks!

[Maël Valais]

I just tested on a linux box with 64GB of RAM and ghc 8.0.2: at the end of the build, koka is at 20GB of resident memory (whatever this means). Also, 1TB of virtual memory is asked:

---

Nevertheless, the 20GB memory footprint is propably due to a memory leak somewhere (ghc? koka?)

Update: the 1TB of virtual memory seems normal on 64-bits arch according to the official 8.0.1 notes.
Update2: this problem also seems to exist on other projects like ghc-mod (see issue here) and seems to be related to ghc >= 8. This bug might not exist on ghc 7.

---

Update3: I just built koka-0.6.0-dev using ghc 7.10.3. But same problem: when compiling madoko, koka-0.6.0-dev takes a huge amount of memory. @daanx, what version of ghc are you building koka-0.6 on? Maybe I am doing it all wrong?

[Maël Valais]

Hi,
I did some digging using GHC 7.10.2. It seems that some substitute function has an issue and never de-allocates for some reason. I think it relates to the substitute in koka-0.6/src/Type/TypeVar.hs.

Here are my steps to produce this graph (I followed a guide):

1. First, compile koka using the profile mode:

   cd koka-0.6
   git checkout v0.6.0-dev
   jake variant=profile
   

2. Then, compile madoko using the koka compiler we just compiled:

   cd madoko
   ../koka-0.6/out/profile/koka-0.6.0-dev -isrc -i../koka-0.6/lib  -c -olib --outname=madoko  -v  main +RTS -hc
   

   You can ctrl+c when you feel it's enough; then

    hp2ps -c koka-0.6.0-dev.hp
   

which will create a postscript fill koka-0.6.0-dev.ps. To convert it to png:

convert -density 300 -rotate -90 koka-0.6.0-dev.ps koka-0.6.0-dev.png

Here is koka-0.6.0-dev.hp.zip

[Mingye Wang]

Reproduced with ghc 7.10.3 too. My version sees the substitute/|->/subCom... hill divided into many colors. The -hr graph gives a bunch of substitute. The -hm graph unsurprisingly shows all TypeVars. The -hd graph blames a certain closure Data.Map.Base.sat_sjGv, but that is meaningful only for my current distribution of ghc; there's some bytes in the trace info (DWARF?) that says ->>Map^@^@<map> after it. -hy shows a huge * in the postscript graph.

Which "subCom..." is it though? subCommon? subCompose?

[Daan]

Hi all, apologies for the late reaction :-)
As a quick fix, you can restart "jake" a few times and each time it should make more progress.

One problem that causes this is that Madoko currently relies on a rather old version of the Koka compiler (0.6) that was before Koka started to have algebraic effects. I hope to upgrade Madoko to a newer version soon as it has been lagging for too long.

Secondly, there is indeed some kind of space leak inside the compiler but I have as yet not been able to fix it yet -- but the space profile up there gives a big hint! I guess the problem is in Type/TypeVar/subCompose -- perhaps a few seq calls will alleviate some of these problems?

[Sean Leather]

For what's it worth, I saw this on macOS, too. Memory consumption went over 18.3 GB.