bblfshd crashed and does not start after parsing millions of files

[Vadim Markovtsev]

I ran massive file parsing on 4 machines:

• typos-1.infra.mining.prod.srcd.host
• typos-2.infra.mining.prod.srcd.host
• typos-3.infra.mining.prod.srcd.host
• typos-4.infra.mining.prod.srcd.host

The pipeline was the same as in #270 (comment) How I started bblfshd:

docker run -d --name bblfshd --privileged -p 9432:9432 bblfsh/bblfshd
docker exec -it bblfshd bblfshctl driver install --all

All the versions correspond to #281

Those instances ran for several days and then crashed. docker start bblfshd starts the server and it immediately stops. Each of the 4 machines has the same symptoms. The free space is fine: 75% of a 200GB volume is free.

Here are the logs:

• typos-1
• typos-2
• typos-3
• typos-4

Each file is ~4 gigs uncompressed.

I noticed that #270 persists. The access to my machines can be granted by @rporres upon request.

[Denys Smirnov]

error creating control listener: listen unix /var/run/bblfshctl.sock: bind: address already in use

Try removing the /var/run/bblfshctl.sock file on the server.

[Denys Smirnov]

It also shows a giant stack trace at the end. From a quick look, it may have been affected by #264.

[Denys Smirnov]

bblfshd v2.13.0 is out and should fix the crash, memory consumption and hopefully the CPU consumption as well.

Leaving this issue open, since we still need to solve an issue with the socket left after bblfshd crash.