bblfshd crashed and does not start after parsing millions of files
vmarkovtsev opened this issue · comments
I ran massive file parsing on 4 machines:
typos-1.infra.mining.prod.srcd.host
typos-2.infra.mining.prod.srcd.host
typos-3.infra.mining.prod.srcd.host
typos-4.infra.mining.prod.srcd.host
The pipeline was the same as in #270 (comment) How I started bblfshd:
docker run -d --name bblfshd --privileged -p 9432:9432 bblfsh/bblfshd
docker exec -it bblfshd bblfshctl driver install --all
All the versions correspond to #281
Those instances ran for several days and then crashed. docker start bblfshd
starts the server and it immediately stops. Each of the 4 machines has the same symptoms. The free space is fine: 75% of a 200GB volume is free.
Here are the logs:
Each file is ~4 gigs uncompressed.
I noticed that #270 persists. The access to my machines can be granted by @rporres upon request.
error creating control listener: listen unix /var/run/bblfshctl.sock: bind: address already in use
Try removing the /var/run/bblfshctl.sock
file on the server.
It also shows a giant stack trace at the end. From a quick look, it may have been affected by #264.
bblfshd v2.13.0 is out and should fix the crash, memory consumption and hopefully the CPU consumption as well.
Leaving this issue open, since we still need to solve an issue with the socket left after bblfshd crash.