sflow-to-rrd-handler: spikes when API key disappears
nickhilliard opened this issue · comments
If the API key disappears, then sflow-to-rrd-handler will not zero out the matrix on each write, causing the graph output to increase monotonically.
Probably the process needs to trap authorization problems in reload_mactable() and abort if authorization is unsuccessful, as that is a terminal error that cannot be resolved without operator intervention.
We are experiencing a similar, but maybe more generic issue here. [Or do you consider this a different issue?]
Whenever the API is unavailable (database, webserver, ...), sflow-to-rrd-handler will block in reload_mactable() and eventually produce spikes.
It'd be cool to decouple reload_mactable() from the main loop, e.g. short timeout and improved error handling.
PRs welcome @agbcix if you have perl skillz. See this page on contributing.
I looked into this more deeply now. It seems like reload_mactable() is dealing well with the timeout etc.
But: Upon any error (timeout, authentication issue, ...) the $newmactable will be undefined. Only if $newmactable is defined, we are running matrix_init() which is actually flushing the in-memory data.
Proposed fix (no PR yet):
run $matrix = matrix_init($mactable, $infraid); unconditionally.