accept: Too many open files (Errno)
blueway opened this issue · comments
Description
accept: Too many open files (Errno)
| 20181106 10:44 +08:00: from /crystal/src/socket.cr:246:11 in 'accept_impl'
| 20181106 10:44 +08:00: from /crystal/share/crystal/src/socket/tcp_server.cr:102:20 in 'accept?'
| 20181106 10:44 +08:00: from /crystal/share/crystal/src/http/server.cr:353:13 in '->'
| 20181106 10:44 +08:00: from /crystal/share/crystal/src/fiber.cr:255:3 in 'run'
| 20181106 10:44 +08:00: from /crystal/share/crystal/src/fiber.cr:75:34 in '->'
| 20181106 10:44 +08:00: from ???
| 20181106 10:44 +08:00: accept: Too many open files (Errno)
Steps to Reproduce
- [First Step]
- [Second Step]
- [and so on...]
Expected behavior: [What you expect to happen]
Actual behavior: [What actually happens]
Reproduces how often: [What percentage of the time does it reproduce?]
Versions
crystal 0.27.0
kemal 0.25.0
open files (-n) 65536
Os error, your client open too many connections (ulimit -n) to kemal, and not close it.
how to close?
This could very well be a Kemal error. In order to investigate, you need to provide reproducible steps.
no special steps,
centos7 with nginx
Kemal.config do |cfg|
cfg.serve_static = true
cfg.logging = true
cfg.env= "production"
end
Kemal.run { |cfg| cfg.server.not_nil!.bind_tcp("0.0.0.0",6000, true) }
Okay, then it's unlikely that kemal is causing such many open files. They're probably caused by some other software on the system. You'll need to investigate using lsof or similar tool, which program uses thousands of handles.
Ok, it may be working when i update nginx.conf configs, thanks
No, it has happened again, i have update nginx.conf file.
It seems to be irrelevant to nginx, the nginx have no error logs. I will watch it more times
Did you find out which program opens so many files? And which ones?
Now find /proc -print | grep -P '/proc/\d+/fd/'| awk -F '/' '{print $3}' | uniq -c | sort -rn | head show:
the crystal proc open files are the most
and the num is increasing until 8192
How to fix it ? My program crashs twice every day
what is your clients? browser requests? maybe you have very slow response, so many clients reach limit fast?
What kind of file descriptor does the crystal process keep open? lsof +p [PID] shows this (it's obviously gonna be a long list).
no, the response is fast, this is android request, lsof +p [PID] show most:
test 0000 root xxx IPv4 xxx 0t0 TCP xx:irdmi->x.x.x.x:xxxx (ESTABLISHED)
This has nothing to do with sys environment, my python3 program is running normally . The two programs have the same logic
Maybe because I didnt use nginx to proxy the program request, the kemal program cant support all requests
probably your clients not close connection after requests, you can exactly try nginx and set inactivity timeout to close it force.
The clients have closed the connection
We cant find the problem , so I decide to close the issue. My program is running normally when i use nginx to proxy the kemal program.
Thanks