health-checker dies after fails to open any additional TCP connections after a period of time.
josh-padnick opened this issue · comments
I experienced some puzzling behavior where health-checker would run fine for a while, dutifully making its TCP checks. But then all TCP checks just started dieing and resulted in the following log output:
time="2018-02-05T00:15:50Z" level=info msg="Received inbound request. Beginning health checks..."
time="2018-02-05T00:15:50Z" level=info msg="Attempting to connect to port 8083 via TCP..."
time="2018-02-05T00:15:50Z" level=warning msg="TCP connection to port 8083 FAILED: dial tcp 0.0.0.0:8083: socket: too many open files"
time="2018-02-05T00:15:50Z" level=info msg="Attempting to connect to port 8081 via TCP..."
time="2018-02-05T00:15:50Z" level=warning msg="TCP connection to port 8081 FAILED: dial tcp 0.0.0.0:8081: socket: too many open files"
time="2018-02-05T00:15:50Z" level=info msg="Attempting to connect to port 8082 via TCP..."
time="2018-02-05T00:15:50Z" level=warning msg="TCP connection to port 8082 FAILED: dial tcp 0.0.0.0:8082: socket: too many open files"
time="2018-02-05T00:15:50Z" level=info msg="At least one health check failed. Returning HTTP 504 response.\n"
The issue is that net.DialTimeout() should close its connection, but I accidentally omitted this .
Resolved by #6.