OTEL Dependency Panic

Question

OTEL Dependency Panic

karmops opened this issue 3 months ago · comments

karmops commented 3 months ago

Migrating from 2.5.0 to 2.6.1

the issue also happens on: 2.6.0

targeting ARM machines

just ran the same deployment I had, updating the OTEL config to:

    "telemetry/opentelemetry": {
      "metric_reporting_period": 0,
      "exporters": {
        "prometheus": [
          {
            "port": 9091,
            "name": "krakend"
          }
        ]
      }
    }

Stack Trace

Parsing configuration file: /etc/krakend/krakend.json
2024/04/10 22:09:17 KRAKEND INFO: Starting KrakenD v2.6.1
2024/04/10 22:09:17 KRAKEND INFO: Working directory is /etc/krakend
2024/04/10 22:09:17 KRAKEND DEBUG: [SERVICE: Gin] Debug enabled
2024/04/10 22:09:17 KRAKEND DEBUG: [SERVICE: Gin][HTTPsecure] Successfuly loaded module
2024/04/10 22:09:17 KRAKEND DEBUG: [SERVICE: Gin][Botdetector] The bot detector has been registered successfully
2024/04/10 22:09:17 KRAKEND INFO: Starting the KrakenD instance

... for brevity, it was removed

2024/04/10 22:02:08 http2: panic serving 10.1.21.208:36858: interface conversion: *http.http2responseWriter is not http.Hijacker: missing method Hijack
goroutine 131196 [running]:
net/http.(*http2serverConn).runHandler.func1()
/usr/local/go/src/net/http/h2_bundle.go:6104 +0x138
panic({0x2d73560, 0x4027f6a510})
/usr/local/go/src/runtime/panic.go:884 +0x1f4
github.com/krakend/krakend-otel/http/server.newTrackingResponseWriter(...)
/go/pkg/mod/github.com/krakend/krakend-otel@v0.2.0/http/server/response_writer.go:90
github.com/krakend/krakend-otel/http/server.(*trackingHandler).ServeHTTP(0x4000ee03c0, {0x3989540, 0x400012a980}, 0x4008a63500)
/go/pkg/mod/github.com/krakend/krakend-otel@v0.2.0/http/server/server.go:37 +0x418
net/http.serverHandler.ServeHTTP({0x0?}, {0x3989540, 0x400012a980}, 0x4008a63500)
/usr/local/go/src/net/http/server.go:2936 +0x2c0
net/http.initALPNRequest.ServeHTTP({{0x398b520?, 0x4027f6a270?}, 0x402ab7e000?, {0x4000768c30?}}, {0x3989540, 0x400012a980}, 0x4008a63500)
/usr/local/go/src/net/http/server.go:3545 +0x1d0
net/http.(*http2serverConn).runHandler(0x4b4?, 0x31590f5?, 0x16?, 0x4027f65eb0?)
/usr/local/go/src/net/http/h2_bundle.go:6111 +0xc0
created by net/http.(*http2serverConn).scheduleHandler
/usr/local/go/src/net/http/h2_bundle.go:6046 +0x21c
2024/04/10 22:02:14 http2: panic serving 10.1.21.208:46688: interface conversion: *http.http2responseWriter is not http.Hijacker: missing method Hijack
goroutine 139410 [running]:
net/http.(*http2serverConn).runHandler.func1()
/usr/local/go/src/net/http/h2_bundle.go:6104 +0x138
panic({0x2d73560, 0x4027f6ad20})
/usr/local/go/src/runtime/panic.go:884 +0x1f4
github.com/krakend/krakend-otel/http/server.newTrackingResponseWriter(...)
/go/pkg/mod/github.com/krakend/krakend-otel@v0.2.0/http/server/response_writer.go:90
github.com/krakend/krakend-otel/http/server.(*trackingHandler).ServeHTTP(0x4000ee03c0, {0x3989540, 0x400012a9a0}, 0x4008a63e00)
/go/pkg/mod/github.com/krakend/krakend-otel@v0.2.0/http/server/server.go:37 +0x418
net/http.serverHandler.ServeHTTP({0x402a6c4d38?}, {0x3989540, 0x400012a9a0}, 0x4008a63e00)
/usr/local/go/src/net/http/server.go:2936 +0x2c0
net/http.initALPNRequest.ServeHTTP({{0x398b520?, 0x402adda420?}, 0x402266ce00?, {0x4000768c30?}}, {0x3989540, 0x400012a9a0}, 0x4008a63e00)
/usr/local/go/src/net/http/server.go:3545 +0x1d0
net/http.(*http2serverConn).runHandler(0x0?, 0x11dbdf4?, 0x4000ef0380?, 0x0?)
/usr/local/go/src/net/http/h2_bundle.go:6111 +0xc0
created by net/http.(*http2serverConn).scheduleHandler
/usr/local/go/src/net/http/h2_bundle.go:6046 +0x21c
2024/04/10 22:02:14 http2: panic serving 10.1.21.208:46690: interface conversion: *http.http2responseWriter is not http.Hijacker: missing method Hijack
goroutine 139398 [running]:
net/http.(*http2serverConn).runHandler.func1()
/usr/local/go/src/net/http/h2_bundle.go:6104 +0x138
panic({0x2d73560, 0x40288f9fb0})
/usr/local/go/src/runtime/panic.go:884 +0x1f4
github.com/krakend/krakend-otel/http/server.newTrackingResponseWriter(...)
/go/pkg/mod/github.com/krakend/krakend-otel@v0.2.0/http/server/response_writer.go:90
github.com/krakend/krakend-otel/http/server.(*trackingHandler).ServeHTTP(0x4000ee03c0, {0x3989540, 0x4000010910}, 0x400095fd00)
/go/pkg/mod/github.com/krakend/krakend-otel@v0.2.0/http/server/server.go:37 +0x418
net/http.serverHandler.ServeHTTP({0x402a6c1d38?}, {0x3989540, 0x4000010910}, 0x400095fd00)
/usr/local/go/src/net/http/server.go:2936 +0x2c0
net/http.initALPNRequest.ServeHTTP({{0x398b520?, 0x4027f6a600?}, 0x402266d180?, {0x4000768c30?}}, {0x3989540, 0x4000010910}, 0x400095fd00)
/usr/local/go/src/net/http/server.go:3545 +0x1d0
net/http.(*http2serverConn).runHandler(0x402ab75560?, 0x402a6c1f88?, 0x4000648360?, 0x402a6c1f98?)
/usr/local/go/src/net/http/h2_bundle.go:6111 +0xc0
created by net/http.(*http2serverConn).scheduleHandler
/usr/local/go/src/net/http/h2_bundle.go:6046 +0x21c
2024/04/10 22:02:24 http2: panic serving 10.1.21.208:48160: interface conversion: *http.http2responseWriter is not http.Hijacker: missing method Hijack
goroutine 129153 [running]:
net/http.(*http2serverConn).runHandler.func1()
/usr/local/go/src/net/http/h2_bundle.go:6104 +0x138
panic({0x2d73560, 0x402addb2c0})
/usr/local/go/src/runtime/panic.go:884 +0x1f4
github.com/krakend/krakend-otel/http/server.newTrackingResponseWriter(...)
/go/pkg/mod/github.com/krakend/krakend-otel@v0.2.0/http/server/response_writer.go:90
github.com/krakend/krakend-otel/http/server.(*trackingHandler).ServeHTTP(0x4000ee03c0, {0x3989540, 0x400a1a8430}, 0x402addc800)
/go/pkg/mod/github.com/krakend/krakend-otel@v0.2.0/http/server/server.go:37 +0x418
net/http.serverHandler.ServeHTTP({0x40288fd1e0?}, {0x3989540, 0x400a1a8430}, 0x402addc800)
/usr/local/go/src/net/http/server.go:2936 +0x2c0
net/http.initALPNRequest.ServeHTTP({{0x398b520?, 0x402addab70?}, 0x4021a61500?, {0x4000768c30?}}, {0x3989540, 0x400a1a8430}, 0x402addc800)
/usr/local/go/src/net/http/server.go:3545 +0x1d0
net/http.(*http2serverConn).runHandler(0x0?, 0x11dbdf4?, 0x40288f9d10?, 0x0?)
/usr/local/go/src/net/http/h2_bundle.go:6111 +0xc0
created by net/http.(*http2serverConn).scheduleHandler
/usr/local/go/src/net/http/h2_bundle.go:6046 +0x21c
2024/04/10 22:02:24 http2: panic serving 10.1.21.208:48186: interface conversion: *http.http2responseWriter is not http.Hijacker: missing method Hijack
goroutine 139402 [running]:
net/http.(*http2serverConn).runHandler.func1()
/usr/local/go/src/net/http/h2_bundle.go:6104 +0x138
panic({0x2d73560, 0x402addb350})
/usr/local/go/src/runtime/panic.go:884 +0x1f4
github.com/krakend/krakend-otel/http/server.newTrackingResponseWriter(...)
/go/pkg/mod/github.com/krakend/krakend-otel@v0.2.0/http/server/response_writer.go:90
github.com/krakend/krakend-otel/http/server.(*trackingHandler).ServeHTTP(0x4000ee03c0, {0x3989540, 0x4000010920}, 0x402ae30200)
/go/pkg/mod/github.com/krakend/krakend-otel@v0.2.0/http/server/server.go:37 +0x418
net/http.serverHandler.ServeHTTP({0x402a6c1d38?}, {0x3989540, 0x4000010920}, 0x402ae30200)
/usr/local/go/src/net/http/server.go:2936 +0x2c0
net/http.initALPNRequest.ServeHTTP({{0x398b520?, 0x4027f6adb0?}, 0x402266d500?, {0x4000768c30?}}, {0x3989540, 0x4000010920}, 0x402ae30200)
/usr/local/go/src/net/http/server.go:3545 +0x1d0
net/http.(*http2serverConn).runHandler(0x0?, 0x11dbdf4?, 0x402ae32000?, 0x0?)
/usr/local/go/src/net/http/h2_bundle.go:6111 +0xc0
created by net/http.(*http2serverConn).scheduleHandler
/usr/local/go/src/net/http/h2_bundle.go:6046 +0x21c

karmops · Answer 1 · Thu Apr 11 2024 06:20:28 GMT+0800 (China Standard Time)

now I am blind =D I can't see any metrics because I had to disable it

karmops · Answer 2 · Thu Apr 11 2024 06:52:49 GMT+0800 (China Standard Time)

sounds like some package is using some different package version which it doesn't implement Hijacker interface

Daniel Ortiz · Answer 3 · Thu Apr 11 2024 20:17:49 GMT+0800 (China Standard Time)

Hi @karmops
Opentelemetry didn't exist in v2.5, only 2.6.
That configuration works fine for me, can you share instructions to generate that panic? I've tried on amd64 and arm64 hosts.

karmops · Answer 4 · Fri Apr 12 2024 01:32:50 GMT+0800 (China Standard Time)

I didn't do anything too special. I just increased the version of my Krakend from 2.5.0 to 2.6.1 and replaced the settings as suggested in the documentation... nothing more.

my krakend-settings

{
  "$schema": "https://www.krakend.io/schema/v2.6/krakend.json",
  "version": 3,
  "name": "KrakenD Open Source API Gateway",
  "port": "{{ env "KRAKEND_PORT" }}",
  "host": ["{{ env "HOST" }}"],
  "debug_endpoint": true,
  "timeout": "6000ms",
  "tls": {
    "public_key": "/etc/krakend/ssl/certs/tls.crt",
    "private_key": "/etc/krakend/ssl/certs/tls.key",
    "min_version": "TLS12"
  },
  "endpoints": [
    {{ template "some-template.tmpl" }}
  ],
  "extra_config": {
    "router": {
      "return_error_msg": true
    },
    "security/cors": {
      "allow_origins": ["*"],
      "allow_methods": ["GET", "POST", "PATCH", "DELETE"],
      "allow_headers": [
        "Origin",
        "Authorization",
        "Content-Type",
        "Content-Length",
        "User-Agent",
        "X-Forwarded-For",
        "Accept-Language",
        "trackingId"
      ],
      "expose_headers": ["Content-Length", "Content-Type"],
      "allow_credentials": true,
      "debug": false,
      "max_age": "12h"
    },
    "security/bot-detector": {
      "empty_user_agent_is_bot": true,
      "cache_size": 1000
    },
    "security/http": {
      "frame_deny": true,
      "custom_frame_options_value": "DENY",
      "referrer_policy": "same-origin",
      "content_type_nosniff": true,
      "browser_xss_filter": true,
      "content_security_policy": "default-src 'self';",
      "is_development": false
    },
    "telemetry/opentelemetry": {
      "metric_reporting_period": 0,
      "exporters": {
        "prometheus": [
          {
            "port": 9091,
            "name": "krakend"
          }
        ]
      }
    }
  }
}

karmops · Answer 5 · Fri Apr 12 2024 01:35:53 GMT+0800 (China Standard Time)

2024/04/10 22:02:08 http2: panic serving 10.1.21.208:36858: interface conversion: *http.http2responseWriter is not http.Hijacker: missing method Hijack
goroutine 131196 [running]:
net/http.(*http2serverConn).runHandler.func1()
/usr/local/go/src/net/http/h2_bundle.go:6104 +0x138
panic({0x2d73560, 0x4027f6a510})
/usr/local/go/src/runtime/panic.go:884 +0x1f4
github.com/krakend/krakend-otel/http/server.newTrackingResponseWriter(...)
/go/pkg/mod/github.com/krakend/krakend-otel@v0.2.0/http/server/response_writer.go:90
github.com/krakend/krakend-otel/http/server.(*trackingHandler).ServeHTTP(0x4000ee03c0, {0x3989540, 0x400012a980}, 0x4008a63500)
/go/pkg/mod/github.com/krakend/krakend-otel@v0.2.0/http/server/server.go:37 +0x418
net/http.serverHandler.ServeHTTP({0x0?}, {0x3989540, 0x400012a980}, 0x4008a63500)
/usr/local/go/src/net/http/server.go:2936 +0x2c0
net/http.initALPNRequest.ServeHTTP({{0x398b520?, 0x4027f6a270?}, 0x402ab7e000?, {0x4000768c30?}}, {0x3989540, 0x400012a980}, 0x4008a63500)
/usr/local/go/src/net/http/server.go:3545 +0x1d0
net/http.(*http2serverConn).runHandler(0x4b4?, 0x31590f5?, 0x16?, 0x4027f65eb0?)
/usr/local/go/src/net/http/h2_bundle.go:6111 +0xc0
created by net/http.(*http2serverConn).scheduleHandler
/usr/local/go/src/net/http/h2_bundle.go:6046 +0x21c

but this sounds like some library version in some component should upgrade to a more recent version of golang.

A. Lombarte · Answer 6 · Tue Apr 16 2024 21:23:49 GMT+0800 (China Standard Time)

Fix: krakend/krakend-otel#14

A. Lombarte · Answer 7 · Wed Apr 17 2024 20:03:43 GMT+0800 (China Standard Time)

Hi @karmops,
This should be fixed now, please use v2.6.2

Thank you for reporting it

karmops · Answer 8 · Fri Apr 19 2024 05:01:26 GMT+0800 (China Standard Time)

gonna try and let you know

karmops · Answer 9 · Sat Apr 20 2024 04:06:25 GMT+0800 (China Standard Time)

@alombarte Thank you very much, it is fixed. no panic anymore and krakend is generating the open telemetry metrics.