panic in prometheus input due to nil
skorpy2009 opened this issue · comments
prometheus.go:154: interface conversion: interface {} is nil, not string
cpthook assumes that there is always an "instance" label, but this is not always the case.
Oct 22 16:57:21 prometheus1.aixit.off.de.ffffm.net CptHook[318]: 2018/10/22 16:57:21 Got http event for /prometheus
Oct 22 16:57:21 prometheus1.aixit.off.de.ffffm.net CptHook[318]: 2018/10/22 16:57:21 http: panic serving 127.0.0.1:49184: interface conversion: interface {} is nil, not string
Oct 22 16:57:21 prometheus1.aixit.off.de.ffffm.net CptHook[318]: goroutine 349 [running]:
Oct 22 16:57:21 prometheus1.aixit.off.de.ffffm.net CptHook[318]: net/http.(*conn).serve.func1(0xc4201b6d20)
Oct 22 16:57:21 prometheus1.aixit.off.de.ffffm.net CptHook[318]: /usr/lib/go-1.8/src/net/http/server.go:1721 +0xd0
Oct 22 16:57:21 prometheus1.aixit.off.de.ffffm.net CptHook[318]: panic(0x878540, 0xc42025dc80)
Oct 22 16:57:21 prometheus1.aixit.off.de.ffffm.net CptHook[318]: /usr/lib/go-1.8/src/runtime/panic.go:489 +0x2cf
Oct 22 16:57:21 prometheus1.aixit.off.de.ffffm.net CptHook[318]: main.PrometheusModule.getHandler.func1(0xaebaa0, 0xc420278540, 0xc42046a900)
Oct 22 16:57:21 prometheus1.aixit.off.de.ffffm.net CptHook[318]: /usr/local/go/src/github.com/fleaz/CptHook/prometheus.go:154 +0x879
Oct 22 16:57:21 prometheus1.aixit.off.de.ffffm.net CptHook[318]: net/http.HandlerFunc.ServeHTTP(0xc420056d80, 0xaebaa0, 0xc420278540, 0xc42046a900)
Oct 22 16:57:21 prometheus1.aixit.off.de.ffffm.net CptHook[318]: /usr/lib/go-1.8/src/net/http/server.go:1942 +0x44
Oct 22 16:57:21 prometheus1.aixit.off.de.ffffm.net CptHook[318]: net/http.(*ServeMux).ServeHTTP(0xb1f540, 0xaebaa0, 0xc420278540, 0xc42046a900)
Oct 22 16:57:21 prometheus1.aixit.off.de.ffffm.net CptHook[318]: /usr/lib/go-1.8/src/net/http/server.go:2238 +0x130
Oct 22 16:57:21 prometheus1.aixit.off.de.ffffm.net CptHook[318]: net/http.serverHandler.ServeHTTP(0xc42009eb00, 0xaebaa0, 0xc420278540, 0xc42046a900)
Oct 22 16:57:21 prometheus1.aixit.off.de.ffffm.net CptHook[318]: /usr/lib/go-1.8/src/net/http/server.go:2568 +0x92
Oct 22 16:57:21 prometheus1.aixit.off.de.ffffm.net CptHook[318]: net/http.(*conn).serve(0xc4201b6d20, 0xaec1e0, 0xc42025da00)
Oct 22 16:57:21 prometheus1.aixit.off.de.ffffm.net CptHook[318]: /usr/lib/go-1.8/src/net/http/server.go:1825 +0x612
Oct 22 16:57:21 prometheus1.aixit.off.de.ffffm.net CptHook[318]: created by net/http.(*Server).Serve
Oct 22 16:57:21 prometheus1.aixit.off.de.ffffm.net CptHook[318]: /usr/lib/go-1.8/src/net/http/server.go:2668 +0x2ce
Oct 22 16:57:21 prometheus1.aixit.off.de.ffffm.net CptHook[318]: 2018/10/22 16:57:21 http: panic serving 127.0.0.1:49182: interface conversion: interface {} is nil, not string
This probably comes from https://github.com/fleaz/CptHook/blob/master/input/prometheus.go#L154
The alert is:
POST /prometheus HTTP/1.1
Host: 127.0.0.1:8086
User-Agent: Alertmanager/0.15.2
Content-Length: 1012
Content-Type: application/json
{"receiver":"irc","status":"firing","alerts":[{"status":"firing","labels":{"alertname":"BGP6TransitSessionDown","severity":"page"},"annotations":{"description":"\u003e=2 BGP IPv6 Transit session are down for more than 1 minute","summary":"Instance : BGP6 down","value":""},"startsAt":"2018-10-21T12:50:32.490632004+02:00","endsAt":"2018-10-22T17:13:32.490632004+02:00","generatorURL":"https://prometheus.ffm.freifunk.net/graph?g0.expr=count%28%28avg_over_time%28bird_bgp6_session_up%7Bname%3D~%22er.%2Aof_aixit_net%7CHurricane_Electric_KLEYREX%22%7D%5B1m%5D%29+%3D%3D+0%29%29+%3E+1\u0026g0.tab=1"}],"groupLabels":{"alertname":"BGP6TransitSessionDown"},"commonLabels":{"alertname":"BGP6TransitSessionDown","severity":"page"},"commonAnnotations":{"description":"\u003e=2 BGP IPv6 Transit session are down for more than 1 minute","summary":"Instance : BGP6 down","value":""},"externalURL":"https://prometheus.ffm.freifunk.net/alertmanager","version":"4","groupKey":"{}/{}:{alertname=\"BGP6TransitSessionDown\"}"}