Add timeout
langalex opened this issue · comments
Our site is having problems at the moment resolving logs.papertrailapp.com. This results in the following message being printed when trying to write to the logger:
RemoteSyslogLogger::UdpSender error: SocketError: getaddrinfo: Temporary failure in name resolution
The problem is that this only happens after 20s (system/Ruby socket timeout?) essentially bringing our entire app down.
Before I start implementing anything: what do you think about adding a 100ms (or less) timeout around sending logs?
I'm not sure off the top of my head the best way to deal with timeouts there.
Unfortunately the easy approach of using Timeout is really bad to use in any situation that requires good performance: Each invocation of Timeout spawns a new Thread.
@langalex: https://github.com/papertrail/remote_syslog might also be worth investigating. It's completely standalone and decoupled from your app.
thanks @troy. one other idea i had - since the problem was dns, not sending the actual logs - was to resolve the configured host name to an ip address once (and with a timeout) and use the ip address for sending the logs. do you see any problems with that approach?
That would work too, though at the expense of any redundancy presented by dynamic DNS or otherwise tied to the receiving IP. However, it sure beats the problem you encountered, and just in general seems like a reasonable approach.
You could also define it in /etc/hosts or equivalent if that was preferable.
going with remote_syslog