Memory leak
smacker opened this issue · comments
Code to reproduce the issue: https://gist.github.com/smacker/1f01afa3002cdd706e0eb08e38352546
Start bblfsh & run the script:
$ docker run -it --rm --privileged -p 9432:9432 -d bblfsh/bblfshd:v2.11.8-drivers
bc538df7847372580ad4c1e0c144d381c64970b0a21f183a824a6da628f4d4ac
$ go run leak.go
$ docker stats --no-stream
CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS
bc538df78473 unruffled_goldberg 0.05% 169MiB / 3.855GiB 4.28% 366kB / 1.88MB 0B / 0B 55
$ docker stats --no-stream
CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS
bc538df78473 unruffled_goldberg 191.95% 803.5MiB / 3.855GiB 20.35% 9.18MB / 15.8MB 0B / 0B 83
$ docker exec bc538df78473 bblfshctl status
+------------+----------------+---------------+---------+--------+
| LANGUAGE | SUCCESS/FAILED | STATE/DESIRED | WAITING | EXITED |
+------------+----------------+---------------+---------+--------+
| ruby | 2/0 | 1/1 | 0 | 0 |
| python | 61/0 | 1/1 | 0 | 0 |
| go | 433/0 | 2/2 | 8 | 0 |
| javascript | 72/0 | 1/1 | 0 | 0 |
| java | 13/0 | 1/1 | 0 | 0 |
+------------+----------------+---------------+---------+--------+
Response time 24.6µs
Drivers started. All good.
Let's wait a little.
$ docker stats --no-stream
CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS
bc538df78473 unruffled_goldberg 191.68% 3.604GiB / 3.855GiB 93.49% 12.2MB / 40.4MB 6.23MB / 0B 83
$ docker exec bc538df78473 bblfshctl status
+------------+----------------+---------------+---------+--------+
| LANGUAGE | SUCCESS/FAILED | STATE/DESIRED | WAITING | EXITED |
+------------+----------------+---------------+---------+--------+
| python | 61/0 | 1/1 | 0 | 0 |
| go | 906/0 | 2/2 | 8 | 0 |
| javascript | 95/0 | 1/1 | 0 | 0 |
| java | 13/0 | 1/1 | 0 | 0 |
| ruby | 2/0 | 1/1 | 0 | 0 |
+------------+----------------+---------------+---------+--------+
Response time 3.0495ms
$ time docker stats --no-stream
CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS
bc538df78473 unruffled_goldberg 58.36% 3.666GiB / 3.855GiB 95.08% 35.6MB / 145MB 3.05GB / 2.6MB 83
real 0m23.468s
user 0m0.048s
sys 0m0.028s
Docker almost got frozen at this point. 23 seconds to get stats.
And it's getting worse after a while and only killing docker itself helps because docker stop doesn't respond.
update: I run it once again and got similar behavior but at one point client received: rpc error: code = Unavailable desc = transport is closing and memory was freed.
I see 1 in "EXITED" column for go driver in bblfshctl status.
In server logs I see:
[2019-03-16T19:24:46Z] ERROR request processed content 7132538 bytes error: rpc error: code = Unavailable desc = transport is closing elapsed=9m57.187631s filename=frequencies.go language=go
most probably related: #226
Just to clarify - most probably all above is using latest v2.11.8 and with high probability is expected to be fixed in master, so should be tested on v2.12.0 as soon as it's out (today?).
It's using v2.11.8-drivers as you can see in the issue description.
I'm not sure it's going to be fixed in 2.12. It looks like the leak is in go driver.
(I see memory grows slowly when I just parse the same small file in the loop & as described above after bblfsh restarted go-driver memory was freed)
Not sure if it's fixed.
root@d79d70c61f08:/# bblfshctl status
+------------+----------------+---------------+---------+--------+
| LANGUAGE | SUCCESS/FAILED | STATE/DESIRED | WAITING | EXITED |
+------------+----------------+---------------+---------+--------+
| javascript | 128/5 | 1/1 | 0 | 4 |
| java | 14/0 | 1/1 | 0 | 3 |
| ruby | 3/0 | 1/1 | 0 | 1 |
| python | 77/0 | 1/1 | 0 | 4 |
| go | 1288/4 | 2/2 | 8 | 2 |
+------------+----------------+---------------+---------+--------+
$ time docker stats --no-stream
CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS
d79d70c61f08 zen_tu 27.82% 3.689GiB / 3.855GiB 95.69% 214MB / 639MB 216MB / 81.9kB 102
real 0m30.798s
user 0m0.045s
sys 0m0.029s
After few more minutes:
$ time docker stats --no-stream
CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS
d79d70c61f08 zen_tu 40.29% 3.676GiB / 3.855GiB 95.37% 214MB / 639MB 3.25GB / 4.72MB 107
real 1m29.567s
user 0m0.045s
sys 0m0.036s
After some more time, similar to the previous report at one point it produced:
rpc error: code = Unavailable desc = transport is closing
and the memory went down:
$ time docker stats --no-stream
CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS
d79d70c61f08 zen_tu 190.65% 673.8MiB / 3.855GiB 17.07% 218MB / 649MB 5.08GB / 16MB 114
real 0m2.290s
user 0m0.055s
sys 0m0.036s
and started climbing slowly again.
bblfsh/bblfshd:v2.13.0-drivers
After some more investigation, it looks like actually fixed.
First I tried to parse the same small file in loop and the memory is stable.
Then I was using this script: https://gist.github.com/smacker/ac5a1457bfd088b21c5b1c3181bcf670
which parses the same big file in the loop and print stat after each successful parsing.
$ go run main.go
CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS
779ec6d8716f gracious_burnell 0.08% 2.25GiB / 3.855GiB 58.37% 3.93MB / 23.3MB 45.1kB / 0B 17
CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS
779ec6d8716f gracious_burnell 0.11% 2.413GiB / 3.855GiB 62.60% 7.86MB / 46.7MB 45.1kB / 0B 18
CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS
779ec6d8716f gracious_burnell 0.10% 2.555GiB / 3.855GiB 66.28% 11.8MB / 70MB 45.1kB / 0B 18
CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS
779ec6d8716f gracious_burnell 0.12% 2.576GiB / 3.855GiB 66.81% 15.7MB / 93.4MB 45.1kB / 0B 18
CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS
779ec6d8716f gracious_burnell 0.10% 2.59GiB / 3.855GiB 67.19% 19.6MB / 117MB 45.1kB / 0B 18
CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS
779ec6d8716f gracious_burnell 0.12% 2.59GiB / 3.855GiB 67.20% 23.6MB / 140MB 45.1kB / 0B 18
CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS
779ec6d8716f gracious_burnell 0.11% 2.656GiB / 3.855GiB 68.89% 27.5MB / 163MB 45.1kB / 0B 18
CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS
779ec6d8716f gracious_burnell 0.11% 2.659GiB / 3.855GiB 68.98% 31.4MB / 187MB 45.1kB / 0B 18
CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS
779ec6d8716f gracious_burnell 0.13% 2.66GiB / 3.855GiB 68.99% 35.4MB / 210MB 45.1kB / 0B 18
CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS
779ec6d8716f gracious_burnell 0.10% 2.66GiB / 3.855GiB 68.99% 39.3MB / 233MB 45.1kB / 0B 18
CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS
779ec6d8716f gracious_burnell 0.10% 2.664GiB / 3.855GiB 69.09% 43.2MB / 257MB 45.1kB / 0B 18
CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS
779ec6d8716f gracious_burnell 0.08% 2.664GiB / 3.855GiB 69.10% 47.1MB / 280MB 45.1kB / 0B 18
memory doesn't grow infinitely anymore.
The behavior from above is caused by the file https://github.com/src-d/enry/blob/master/data/frequencies.go, parsing of which consumes more memory than is avalible on my system.
But it's kind of expected.