Too many logs on network error
babolivier opened this issue · comments
One of our hosts experienced a failure during a reboot recently, causing it to lose its public IP, resulting in an inability to communicate over the Internet. This caused beamium to constantly check for network availability, logging at each try. Here's an excerpt:
Aug 16 18:50:52.073 ERRO HTTP error: Network is unreachable (os error 101), sink: ovh-metrics-plan-m
Aug 16 18:50:52.075 ERRO HTTP error: Network is unreachable (os error 101), sink: ovh-metrics-plan-m
Aug 16 18:50:52.077 ERRO HTTP error: Network is unreachable (os error 101), sink: ovh-metrics-plan-m
Aug 16 18:50:52.079 ERRO HTTP error: Network is unreachable (os error 101), sink: ovh-metrics-plan-m
As shown above, beamium tries and logs every 2ms, which caused it to fill up the entirety of the host's available disk space. There should be a delay between two tries, ideally (imho) matching the parameters:scan-period config parameter (since that's the one determining the minimum delay between two pushes to Warp10). On that host, beamium has been installed using the Debian/Ubuntu package, and is at version 1.8.0-stretch.