Backoff Uptrace pods restart with error `panic: reflect.Set: value of type uint16 is not assignable to type string`
Dima-Diachenko opened this issue · comments
Setup: we're using Uptrace Helm deployment 1.7.4.
Today we've faced with a weird issue, that I don't know how to debug or analyze. Uptrace works as usual, but at some random time Uptrace pods goes to backoff restart with the same error in logs:
panic: reflect.Set: value of type uint16 is not assignable to type string
goroutine 203255 [running]:
github.com/uptrace/uptrace/pkg/httputil.PanicHandler.ServeHTTP.func1()
github.com/uptrace/uptrace/pkg/httputil/middleware.go:25 +0x5b
panic({0x171cc00?, 0xc02a4fddb0?})
runtime/panic.go:770 +0x132
go.opentelemetry.io/otel/sdk/trace.(*recordingSpan).End.deferwrap1()
go.opentelemetry.io/otel/sdk@v1.26.0/trace/span.go:381 +0x25
go.opentelemetry.io/otel/sdk/trace.(*recordingSpan).End(0xc019517500, {0x0, 0x0, 0xc02790c780?})
go.opentelemetry.io/otel/sdk@v1.26.0/trace/span.go:419 +0xa82
panic({0x171cc00?, 0xc02a4fddb0?})
runtime/panic.go:770 +0x132
reflect.Value.assignTo({0x171cd00?, 0x428d540?, 0x1a088ba?}, {0x1a088ba, 0xb}, 0x171cc00, 0x0)
reflect/value.go:3356 +0x299
reflect.Value.Set({0x171cc00?, 0xc012f45b00?, 0x4dabbb?}, {0x171cd00?, 0x428d540?, 0x0?})
reflect/value.go:2325 +0xe6
github.com/uptrace/go-clickhouse/ch/chschema.NumericColumnOf[...].ConvertAssign(0x35ff620?, 0x8d6d7a?, {0x171cc00?, 0xc012f45b00, 0xc01659a270?})
github.com/uptrace/go-clickhouse@v0.3.1/ch/chschema/column.go:138 +0x153
github.com/uptrace/go-clickhouse/ch.scanRow(0xc0000b48c0, 0xc017ded340, {0x187ea20?, 0xc012f45b00?, 0xc012d6be60?}, 0xc01308f700?, 0x0)
github.com/uptrace/go-clickhouse@v0.3.1/ch/model_table_struct.go:88 +0x130
github.com/uptrace/go-clickhouse/ch.(*sliceTableModel).ScanBlock(0xc012d6be00, 0xc01308f700)
github.com/uptrace/go-clickhouse@v0.3.1/ch/model_table_slice.go:42 +0x7b
github.com/uptrace/go-clickhouse/ch.(*baseQuery).query(0x0?, {0x3605370, 0xc012d6be30}, {0x7f9d0f8e9aa0, 0xc012d6be00}, {0xc0082da000?, 0x3605ed0?})
github.com/uptrace/go-clickhouse@v0.3.1/ch/query_base.go:122 +0x1d1
github.com/uptrace/go-clickhouse/ch.(*SelectQuery).scan(0xc01d7c1180, {0x3605370, 0xc012d6bbc0}, 0x0, {0xc0248b2760?, 0x50?, 0xc0248b25f8?})
github.com/uptrace/go-clickhouse@v0.3.1/ch/query_select.go:589 +0x1ad
github.com/uptrace/go-clickhouse/ch.(*SelectQuery).Scan(...)
github.com/uptrace/go-clickhouse@v0.3.1/ch/query_select.go:561
github.com/uptrace/uptrace/pkg/tracing.SelectAttrValues({0x3605370, 0xc012d6bbc0}, 0xc0001f8000, 0xc00eef4160, {0xc0262362e6, 0x19})
github.com/uptrace/uptrace/pkg/tracing/attr_handler.go:165 +0x8e5
github.com/uptrace/uptrace/pkg/tracing.(*AttrHandler).AttrValues(0xc0008940d0, {0x3601ce0, 0xc00920c500}, {0xc0192cb8c0, {{0xc0262362c4, 0x3b}, 0xc0005c8b80, 0xc0006a6dc0, 0x0}})
github.com/uptrace/uptrace/pkg/tracing/attr_handler.go:108 +0x1be
github.com/uptrace/uptrace/pkg/org.(*Middleware).UserAndProject-fm.(*Middleware).UserAndProject.func1({0x3601ce0, 0xc00920c500}, {0xc0192cb320, {{0xc0262362c4, 0x3b}, 0xc0005c8b80, 0xc0006a6dc0, 0x0}})
github.com/uptrace/uptrace/pkg/org/auth.go:105 +0x24f
github.com/uptrace/bunrouter/extra/bunrouterotel.(*config).Middleware-fm.(*config).Middleware.func1({0x3601ce0, 0xc00920c500}, {0xc0192cb320, {{0xc0262362c4, 0x3b}, 0xc0005c8b80, 0xc0006a6dc0, 0x0}})
github.com/uptrace/bunrouter/extra/bunrouterotel@v1.0.21/bunrouterotel.go:70 +0xa99
github.com/uptrace/uptrace/pkg/bunapp.(*App).httpErrorHandler-fm.(*App).httpErrorHandler.func1({0x3601ce0, 0xc00920c500}, {0xc0192cb320, {{0xc0262362c4, 0x3b}, 0xc0005c8b80, 0xc0006a6dc0, 0x0}})
github.com/uptrace/uptrace/pkg/bunapp/app.go:231 +0x90
github.com/uptrace/bunrouter/extra/reqlog.(*middleware).Middleware-fm.(*middleware).Middleware.func1({0x35fe920?, 0xc02790c780?}, {0xc0192cb320, {{0xc0262362c4, 0x3b}, 0xc0005c8b80, 0xc0006a6dc0, 0x0}})
github.com/uptrace/bunrouter/extra/reqlog@v1.0.21/middleware.go:77 +0x124
github.com/uptrace/bunrouter.(*Router).ServeHTTPError(0x79355c?, {0x35fe920, 0xc02790c780}, 0xc0192cb320)
github.com/uptrace/bunrouter@v1.0.21/router.go:52 +0x108
github.com/uptrace/bunrouter.(*Router).ServeHTTP(0x42e0960?, {0x35fe920?, 0xc02790c780?}, 0x4?)
github.com/uptrace/bunrouter@v1.0.21/router.go:45 +0x1d
github.com/klauspost/compress/gzhttp.NewWrapper.func1.1({0x3601d40, 0xc006ac4c00}, 0xc0192cb320)
github.com/klauspost/compress@v1.17.6/gzhttp/compress.go:495 +0x54f
net/http.HandlerFunc.ServeHTTP(0xc0192cb320?, {0x3601d40?, 0xc006ac4c00?}, 0x3605370?)
net/http/server.go:2166 +0x29
github.com/uptrace/uptrace/pkg/httputil.DecompressHandler.ServeHTTP({{0x35f1b20?, 0xc000ce1ba0?}}, {0x3601d40, 0xc006ac4c00}, 0xc0192cb320)
github.com/uptrace/uptrace/pkg/httputil/middleware.go:53 +0x1a3
github.com/uptrace/uptrace/pkg/httputil.(*TraceparentHandler).ServeHTTP(0xc000ce1bc0, {0x3601d40, 0xc006ac4c00}, 0xc0192cb320)
github.com/uptrace/uptrace/pkg/httputil/middleware.go:90 +0xb4
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.(*middleware).serveHTTP(0xc000cbb040, {0x35fe830, 0xc01d8d4700}, 0xc0192cb200, {0x35f1b60, 0xc000ce1bc0})
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp@v0.48.0/handler.go:225 +0x1243
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.NewMiddleware.func1.1({0x35fe830?, 0xc01d8d4700?}, 0x52a04f?)
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp@v0.48.0/handler.go:83 +0x35
net/http.HandlerFunc.ServeHTTP(0xc0061a81c0?, {0x35fe830?, 0xc01d8d4700?}, 0xc0192cb200?)
net/http/server.go:2166 +0x29
main.init.func2.(*Cors).Handler.11({0x35fe830, 0xc01d8d4700}, 0xc0192cb200)
github.com/rs/cors@v1.10.1/cors.go:281 +0x184
net/http.HandlerFunc.ServeHTTP(0xc028a9ce60?, {0x35fe830?, 0xc01d8d4700?}, 0x44a58b?)
net/http/server.go:2166 +0x29
github.com/uptrace/uptrace/pkg/httputil.PanicHandler.ServeHTTP({{0x35f1b20?, 0xc000ce1d00?}}, {0x35fe830?, 0xc01d8d4700?}, 0x4a630f?)
github.com/uptrace/uptrace/pkg/httputil/middleware.go:31 +0x51
net/http.serverHandler.ServeHTTP({0xc0008419b0?}, {0x35fe830?, 0xc01d8d4700?}, 0x6?)
net/http/server.go:3137 +0x8e
net/http.(*conn).serve(0xc01b3d3050, {0x3605370, 0xc006413350})
net/http/server.go:2039 +0x5e8
created by net/http.(*Server).Serve in goroutine 185
net/http/server.go:3285 +0x4b4
It happened again and again:
I'm ready to provide any additional information if it's needed.
Hope for help since I don't know what to do. All obvious debug steps I've passed and didn't detect the root cause.
Should be fixed by e61d284