Problems running gRPC server, misteriously led to failure attaching `event_exit_acct_process` on latest version

Question

Problems running gRPC server, misteriously led to failure attaching `event_exit_acct_process` on latest version

inliquid opened this issue 5 months ago · comments

What happened?

This issue stated as a discussion with several participants who had mostly the same problem with Tetragon v1.0.0-v1.0.2.

Symptoms are:

Tetragon can't run due to liveness probe failure, which is based on tetra being connected to Tetragon over gRPC
When liveness probe is disabled Tetragon starts, but gRPC server not run despite logs say that it should:

time="2024-03-23T15:19:59Z" level=info msg="Starting gRPC server" address="localhost:54321" protocol=tcp

When this line is present there should be goroutine started with gRPC server instance.
However, what we observed is that for some mysterious reason it's not run, and stack trace shows no such goroutine - try search for "grpc" in the stack trace output:

# gops stack 1
goroutine 14 [running]:
runtime/pprof.writeGoroutineStacks({0x2e63600, 0xc00086a310})
        /usr/local/go/src/runtime/pprof/pprof.go:703 +0x6a
runtime/pprof.writeGoroutine({0x2e63600?, 0xc00086a310?}, 0xc0000a9400?)
        /usr/local/go/src/runtime/pprof/pprof.go:692 +0x25
runtime/pprof.(*Profile).WriteTo(0x2a6f74e?, {0x2e63600?, 0xc00086a310?}, 0x231f?)
        /usr/local/go/src/runtime/pprof/pprof.go:329 +0x146
github.com/google/gops/agent.handle({0x7c842f4de5c8?, 0xc00086a310}, {0xc0007a6000?, 0x1?, 0x1?})
        /go/src/github.com/cilium/tetragon/vendor/github.com/google/gops/agent/agent.go:200 +0xd2
github.com/google/gops/agent.listen({0x2e83de8, 0xc0002b9520})
        /go/src/github.com/cilium/tetragon/vendor/github.com/google/gops/agent/agent.go:144 +0x1a5
created by github.com/google/gops/agent.Listen in goroutine 1
        /go/src/github.com/cilium/tetragon/vendor/github.com/google/gops/agent/agent.go:122 +0x379

goroutine 1 [chan send]:
github.com/cilium/tetragon/pkg/sensors.(*Manager).RemoveAllSensors(...)
        /go/src/github.com/cilium/tetragon/pkg/sensors/manager.go:278
github.com/cilium/tetragon/pkg/observer.RemoveSensors({0x2e98248?, 0xc002d7e910})
        /go/src/github.com/cilium/tetragon/pkg/observer/observer.go:420 +0xc7
main.tetragonExecute.func5()
        /go/src/github.com/cilium/tetragon/cmd/tetragon/main.go:303 +0x1e
main.tetragonExecute()
        /go/src/github.com/cilium/tetragon/cmd/tetragon/main.go:440 +0x1c65
main.execute.func1(0xc000670700?, {0x2a63272?, 0x4?, 0x2a63276?})
        /go/src/github.com/cilium/tetragon/cmd/tetragon/main.go:733 +0xcb
github.com/spf13/cobra.(*Command).execute(0xc00056b200, {0xc000072050, 0x1, 0x1})
        /go/src/github.com/cilium/tetragon/vendor/github.com/spf13/cobra/command.go:944 +0x863
github.com/spf13/cobra.(*Command).ExecuteC(0xc00056b200)
        /go/src/github.com/cilium/tetragon/vendor/github.com/spf13/cobra/command.go:1068 +0x3a5
github.com/spf13/cobra.(*Command).Execute(...)
        /go/src/github.com/cilium/tetragon/vendor/github.com/spf13/cobra/command.go:992
main.execute()
        /go/src/github.com/cilium/tetragon/cmd/tetragon/main.go:746 +0x16f
main.main()
        /go/src/github.com/cilium/tetragon/cmd/tetragon/tetragon.go:11 +0x13

goroutine 138 [chan receive, 1 minutes]:
k8s.io/client-go/tools/cache.(*controller).Run.func1()
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/controller.go:132 +0x25
created by k8s.io/client-go/tools/cache.(*controller).Run in goroutine 134
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/controller.go:131 +0xab

goroutine 16 [syscall, 1 minutes]:
os/signal.signal_recv()
        /usr/local/go/src/runtime/sigqueue.go:152 +0x29
os/signal.loop()
        /usr/local/go/src/os/signal/signal_unix.go:23 +0x13
created by os/signal.Notify.func1.1 in goroutine 1
        /usr/local/go/src/os/signal/signal.go:151 +0x1f

goroutine 49 [chan receive, 1 minutes]:
main.tetragonExecute.func3(0x0?)
        /go/src/github.com/cilium/tetragon/cmd/tetragon/main.go:255 +0x50
created by main.tetragonExecute in goroutine 1
        /go/src/github.com/cilium/tetragon/cmd/tetragon/main.go:253 +0xed7

goroutine 50 [chan receive, 1 minutes]:
github.com/cilium/tetragon/pkg/sensors.startSensorManager.func1()
        /go/src/github.com/cilium/tetragon/pkg/sensors/manager.go:65 +0x65
created by github.com/cilium/tetragon/pkg/sensors.startSensorManager in goroutine 1
        /go/src/github.com/cilium/tetragon/pkg/sensors/manager.go:60 +0xc5

goroutine 51 [IO wait, 1 minutes]:
internal/poll.runtime_pollWait(0x7c842fb4cea8, 0x72)
        /usr/local/go/src/runtime/netpoll.go:343 +0x85
internal/poll.(*pollDesc).wait(0xc002b18000?, 0xc002b13c50?, 0x0)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x27
internal/poll.(*pollDesc).waitRead(...)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Accept(0xc002b18000)
        /usr/local/go/src/internal/poll/fd_unix.go:611 +0x2ac
net.(*netFD).accept(0xc002b18000)
        /usr/local/go/src/net/fd_unix.go:172 +0x29
net.(*TCPListener).accept(0xc002b16080)
        /usr/local/go/src/net/tcpsock_posix.go:152 +0x1e
net.(*TCPListener).Accept(0xc002b16080)
        /usr/local/go/src/net/tcpsock.go:315 +0x30
net/http.(*Server).Serve(0xc002d9c000, {0x2e83de8, 0xc002b16080})
        /usr/local/go/src/net/http/server.go:3056 +0x364
net/http.(*Server).ListenAndServe(0xc002d9c000)
        /usr/local/go/src/net/http/server.go:2985 +0x71
net/http.ListenAndServe(...)
        /usr/local/go/src/net/http/server.go:3239
github.com/cilium/tetragon/pkg/metrics.EnableMetrics({0xc000589144, 0x5})
        /go/src/github.com/cilium/tetragon/pkg/metrics/metrics.go:25 +0x1cf
created by main.tetragonExecute in goroutine 1
        /go/src/github.com/cilium/tetragon/cmd/tetragon/main.go:323 +0x109e

goroutine 81 [select]:
k8s.io/client-go/util/workqueue.(*delayingType).waitingLoop(0xc002b98060)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/util/workqueue/delaying_queue.go:276 +0x305
created by k8s.io/client-go/util/workqueue.newDelayingQueue in goroutine 31
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/util/workqueue/delaying_queue.go:113 +0x21f

goroutine 31 [sync.Cond.Wait, 1 minutes]:
sync.runtime_notifyListWait(0xc0038ecb50, 0x0)
        /usr/local/go/src/runtime/sema.go:527 +0x159
sync.(*Cond).Wait(0xc000b808a0?)
        /usr/local/go/src/sync/cond.go:70 +0x85
k8s.io/client-go/util/workqueue.(*Type).Get(0xc000b808a0)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/util/workqueue/queue.go:200 +0x99
github.com/cilium/tetragon/pkg/metrics.StartPodDeleteHandler()
        /go/src/github.com/cilium/tetragon/pkg/metrics/metricwithpod.go:126 +0x44
created by main.tetragonExecute in goroutine 1
        /go/src/github.com/cilium/tetragon/cmd/tetragon/main.go:325 +0x10b6

goroutine 143 [select]:
github.com/cilium/tetragon/pkg/process.(*Cache).cacheGarbageCollector.func1()
        /go/src/github.com/cilium/tetragon/pkg/process/cache.go:47 +0xf1
created by github.com/cilium/tetragon/pkg/process.(*Cache).cacheGarbageCollector in goroutine 1
        /go/src/github.com/cilium/tetragon/pkg/process/cache.go:43 +0xe7

goroutine 137 [chan receive, 1 minutes]:
k8s.io/client-go/tools/cache.(*sharedProcessor).run(0xc002d7e190, 0xc001215dd0?)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/shared_informer.go:803 +0x47
k8s.io/client-go/tools/cache.(*sharedIndexInformer).Run.(*Group).StartWithChannel.func4()
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:55 +0x1e
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1()
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:72 +0x4f
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start in goroutine 134
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:70 +0x73

goroutine 96 [chan receive, 1 minutes]:
k8s.io/client-go/tools/cache.(*processorListener).run.func1()
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/shared_informer.go:969 +0x4b
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x30?)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:226 +0x33
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc002b90f38?, {0x2e66c80, 0xc002bf9170}, 0x1, 0xc002ab0e40)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:227 +0xaf
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0x0?, 0x3b9aca00, 0x0, 0xc0?, 0xc002b90f88?)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:204 +0x7f
k8s.io/apimachinery/pkg/util/wait.Until(...)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:161
k8s.io/client-go/tools/cache.(*processorListener).run(0xc002586ab0)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/shared_informer.go:968 +0x69
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1()
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:72 +0x4f
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start in goroutine 111
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:70 +0x73

goroutine 86 [sync.Cond.Wait, 1 minutes]:
sync.runtime_notifyListWait(0xc002c2c028, 0x0)
        /usr/local/go/src/runtime/sema.go:527 +0x159
sync.(*Cond).Wait(0xc000943be0?)
        /usr/local/go/src/sync/cond.go:70 +0x85
k8s.io/client-go/tools/cache.(*DeltaFIFO).Pop(0xc002c2c000, 0xc002bcf9f0)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/delta_fifo.go:575 +0x236
k8s.io/client-go/tools/cache.(*controller).processLoop(0xc002c2a140)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/controller.go:188 +0x30
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x30?)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:226 +0x33
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0x4466c0?, {0x2e66c80, 0xc002bf9080}, 0x1, 0xc0000d6420)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:227 +0xaf
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc002bfb6f0?, 0x3b9aca00, 0x0, 0x0?, 0xdf8475800?)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:204 +0x7f
k8s.io/apimachinery/pkg/util/wait.Until(...)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:161
k8s.io/client-go/tools/cache.(*controller).Run(0xc002c2a140, 0xc0000d6420)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/controller.go:159 +0x393
k8s.io/client-go/tools/cache.(*sharedIndexInformer).Run(0xc00370e210, 0x0?)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/shared_informer.go:504 +0x2c8
github.com/cilium/tetragon/pkg/k8s/client/informers/externalversions.(*sharedInformerFactory).Start.func1()
        /go/src/github.com/cilium/tetragon/vendor/github.com/cilium/tetragon/pkg/k8s/client/informers/externalversions/factory.go:119 +0x5c
created by github.com/cilium/tetragon/pkg/k8s/client/informers/externalversions.(*sharedInformerFactory).Start in goroutine 1
        /go/src/github.com/cilium/tetragon/vendor/github.com/cilium/tetragon/pkg/k8s/client/informers/externalversions/factory.go:117 +0x205

goroutine 139 [select, 1 minutes]:
k8s.io/client-go/tools/cache.watchHandler({0x0?, 0x0?, 0x4704d60?}, {0x2e7ae40, 0xc00250c500}, {0x7c842f8c6a20, 0xc00370e4d0}, {0x2ec3af0?, 0x2a1c760}, 0x0, ...)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/reflector.go:714 +0x17b
k8s.io/client-go/tools/cache.(*Reflector).watch(0xc002ba4540, {0x0?, 0x0?}, 0xc0000d6420, 0x61bf?)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/reflector.go:433 +0x545
k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch(0xc002ba4540, 0xc0000d6420)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/reflector.go:358 +0x36c
k8s.io/client-go/tools/cache.(*Reflector).Run.func1()
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/reflector.go:291 +0x25
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x10?)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:226 +0x33
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc002514420?, {0x2e66ca0, 0xc002d7e640}, 0x1, 0xc0000d6420)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:227 +0xaf
k8s.io/client-go/tools/cache.(*Reflector).Run(0xc002ba4540, 0xc0000d6420)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/reflector.go:290 +0x1cd
k8s.io/client-go/tools/cache.(*controller).Run.(*Group).StartWithChannel.func2()
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:55 +0x1e
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1()
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:72 +0x4f
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start in goroutine 134
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:70 +0x73

goroutine 104 [chan receive]:
k8s.io/client-go/tools/cache.(*processorListener).run.func1()
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/shared_informer.go:969 +0x4b
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x30?)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:226 +0x33
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc000948f38?, {0x2e66c80, 0xc003056090}, 0x1, 0xc00305a000)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:227 +0xaf
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0x0?, 0x3b9aca00, 0x0, 0x30?, 0xc0023da8a0?)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:204 +0x7f
k8s.io/apimachinery/pkg/util/wait.Until(...)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:161
k8s.io/client-go/tools/cache.(*processorListener).run(0xc002507950)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/shared_informer.go:968 +0x69
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1()
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:72 +0x4f
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start in goroutine 47
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:70 +0x73

goroutine 105 [select]:
k8s.io/client-go/tools/cache.(*processorListener).pop(0xc002507950)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/shared_informer.go:939 +0x10d
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1()
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:72 +0x4f
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start in goroutine 47
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:70 +0x73

goroutine 120 [sync.Cond.Wait, 1 minutes]:
sync.runtime_notifyListWait(0xc0031902e8, 0x0)
        /usr/local/go/src/runtime/sema.go:527 +0x159
sync.(*Cond).Wait(0xc002528be0?)
        /usr/local/go/src/sync/cond.go:70 +0x85
k8s.io/client-go/tools/cache.(*DeltaFIFO).Pop(0xc0031902c0, 0xc002585f30)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/delta_fifo.go:575 +0x236
k8s.io/client-go/tools/cache.(*controller).processLoop(0xc001762e60)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/controller.go:188 +0x30
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x30?)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:226 +0x33
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0x4466c0?, {0x2e66c80, 0xc00297b8f0}, 0x1, 0xc0000d6420)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:227 +0xaf
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc002994f90?, 0x3b9aca00, 0x0, 0x0?, 0x0?)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:204 +0x7f
k8s.io/apimachinery/pkg/util/wait.Until(...)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:161
k8s.io/client-go/tools/cache.(*controller).Run(0xc001762e60, 0xc0000d6420)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/controller.go:159 +0x393
k8s.io/client-go/tools/cache.(*sharedIndexInformer).Run(0xc003190210, 0x0?)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/shared_informer.go:504 +0x2c8
github.com/cilium/tetragon/pkg/k8s/client/informers/externalversions.(*sharedInformerFactory).Start.func1()
        /go/src/github.com/cilium/tetragon/vendor/github.com/cilium/tetragon/pkg/k8s/client/informers/externalversions/factory.go:119 +0x5c
created by github.com/cilium/tetragon/pkg/k8s/client/informers/externalversions.(*sharedInformerFactory).Start in goroutine 118
        /go/src/github.com/cilium/tetragon/vendor/github.com/cilium/tetragon/pkg/k8s/client/informers/externalversions/factory.go:117 +0x205

goroutine 115 [IO wait]:
internal/poll.runtime_pollWait(0x7c842fb4cdb0, 0x72)
        /usr/local/go/src/runtime/netpoll.go:343 +0x85
internal/poll.(*pollDesc).wait(0xc002cddd00?, 0xc002562000?, 0x0)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x27
internal/poll.(*pollDesc).waitRead(...)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc002cddd00, {0xc002562000, 0xa000, 0xa000})
        /usr/local/go/src/internal/poll/fd_unix.go:164 +0x27a
net.(*netFD).Read(0xc002cddd00, {0xc002562000?, 0xc00256603a?, 0x1a5?})
        /usr/local/go/src/net/fd_posix.go:55 +0x25
net.(*conn).Read(0xc0004d6008, {0xc002562000?, 0x6f33b7?, 0xc0031421b8?})
        /usr/local/go/src/net/net.go:179 +0x45
crypto/tls.(*atLeastReader).Read(0xc0005e9698, {0xc002562000?, 0xc0005e9698?, 0x0?})
        /usr/local/go/src/crypto/tls/conn.go:805 +0x3b
bytes.(*Buffer).ReadFrom(0xc0031422a8, {0x2e65a40, 0xc0005e9698})
        /usr/local/go/src/bytes/buffer.go:211 +0x98
crypto/tls.(*Conn).readFromUntil(0xc003142000, {0x2e63520?, 0xc0004d6008}, 0x5fcb?)
        /usr/local/go/src/crypto/tls/conn.go:827 +0xde
crypto/tls.(*Conn).readRecordOrCCS(0xc003142000, 0x0)
        /usr/local/go/src/crypto/tls/conn.go:625 +0x250
crypto/tls.(*Conn).readRecord(...)
        /usr/local/go/src/crypto/tls/conn.go:587
crypto/tls.(*Conn).Read(0xc003142000, {0xc003195000, 0x1000, 0x98cbc9?})
        /usr/local/go/src/crypto/tls/conn.go:1369 +0x158
bufio.(*Reader).Read(0xc003196000, {0xc003180200, 0x9, 0x25aecc0?})
        /usr/local/go/src/bufio/bufio.go:244 +0x197
io.ReadAtLeast({0x2e63320, 0xc003196000}, {0xc003180200, 0x9, 0x9}, 0x9)
        /usr/local/go/src/io/io.go:335 +0x90
io.ReadFull(...)
        /usr/local/go/src/io/io.go:354
golang.org/x/net/http2.readFrameHeader({0xc003180200, 0x9, 0xa8400?}, {0x2e63320?, 0xc003196000?})
        /go/src/github.com/cilium/tetragon/vendor/golang.org/x/net/http2/frame.go:237 +0x65
golang.org/x/net/http2.(*Framer).ReadFrame(0xc0031801c0)
        /go/src/github.com/cilium/tetragon/vendor/golang.org/x/net/http2/frame.go:498 +0x85
golang.org/x/net/http2.(*clientConnReadLoop).run(0xc00315ff98)
        /go/src/github.com/cilium/tetragon/vendor/golang.org/x/net/http2/transport.go:2275 +0x11f
golang.org/x/net/http2.(*ClientConn).readLoop(0xc003192000)
        /go/src/github.com/cilium/tetragon/vendor/golang.org/x/net/http2/transport.go:2170 +0x65
created by golang.org/x/net/http2.(*Transport).newClientConn in goroutine 114
        /go/src/github.com/cilium/tetragon/vendor/golang.org/x/net/http2/transport.go:821 +0xcbe

goroutine 84 [select]:
k8s.io/client-go/tools/cache.(*Reflector).startResync(0xc002ba4540, 0xc0000d6420, 0xc002ab0840, 0x209?)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/reflector.go:370 +0x10f
created by k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch in goroutine 139
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/reflector.go:357 +0x348

goroutine 44 [sync.Cond.Wait]:
sync.runtime_notifyListWait(0xc00370e448, 0x19)
        /usr/local/go/src/runtime/sema.go:527 +0x159
sync.(*Cond).Wait(0xc0036f92c0?)
        /usr/local/go/src/sync/cond.go:70 +0x85
k8s.io/client-go/tools/cache.(*DeltaFIFO).Pop(0xc00370e420, 0xc0025047b0)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/delta_fifo.go:575 +0x236
k8s.io/client-go/tools/cache.(*controller).processLoop(0xc0031741e0)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/controller.go:188 +0x30
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x30?)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:226 +0x33
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0x4466c0?, {0x2e66c80, 0xc002508ae0}, 0x1, 0xc0000d6420)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:227 +0xaf
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc0025021d0?, 0x3b9aca00, 0x0, 0x0?, 0xdf8475800?)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:204 +0x7f
k8s.io/apimachinery/pkg/util/wait.Until(...)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:161
k8s.io/client-go/tools/cache.(*controller).Run(0xc0031741e0, 0xc0000d6420)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/controller.go:159 +0x393
k8s.io/client-go/tools/cache.(*sharedIndexInformer).Run(0xc00370e0b0, 0x0?)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/shared_informer.go:504 +0x2c8
k8s.io/client-go/informers.(*sharedInformerFactory).Start.func1()
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/informers/factory.go:150 +0x5c
created by k8s.io/client-go/informers.(*sharedInformerFactory).Start in goroutine 1
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/informers/factory.go:148 +0x205

goroutine 85 [select, 1 minutes]:
golang.org/x/net/http2.(*clientStream).writeRequest(0xc002c01680, 0xc002aa7f00)
        /go/src/github.com/cilium/tetragon/vendor/golang.org/x/net/http2/transport.go:1464 +0xac7
golang.org/x/net/http2.(*clientStream).doRequest(0xc0025024e0?, 0xc002522fb8?)
        /go/src/github.com/cilium/tetragon/vendor/golang.org/x/net/http2/transport.go:1326 +0x18
created by golang.org/x/net/http2.(*ClientConn).RoundTrip in goroutine 139
        /go/src/github.com/cilium/tetragon/vendor/golang.org/x/net/http2/transport.go:1232 +0x308

goroutine 109 [select, 1 minutes]:
golang.org/x/net/http2.(*clientStream).writeRequest(0xc0028ac780, 0xc0027f4700)
        /go/src/github.com/cilium/tetragon/vendor/golang.org/x/net/http2/transport.go:1464 +0xac7
golang.org/x/net/http2.(*clientStream).doRequest(0xc0025021d0?, 0xc002529fb8?)
        /go/src/github.com/cilium/tetragon/vendor/golang.org/x/net/http2/transport.go:1326 +0x18
created by golang.org/x/net/http2.(*ClientConn).RoundTrip in goroutine 129
        /go/src/github.com/cilium/tetragon/vendor/golang.org/x/net/http2/transport.go:1232 +0x308

goroutine 47 [chan receive, 1 minutes]:
k8s.io/client-go/tools/cache.(*sharedProcessor).run(0xc002d7e140, 0x0?)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/shared_informer.go:803 +0x47
k8s.io/client-go/tools/cache.(*sharedIndexInformer).Run.(*Group).StartWithChannel.func4()
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:55 +0x1e
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1()
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:72 +0x4f
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start in goroutine 44
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:70 +0x73

goroutine 48 [chan receive, 1 minutes]:
k8s.io/client-go/tools/cache.(*controller).Run.func1()
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/controller.go:132 +0x25
created by k8s.io/client-go/tools/cache.(*controller).Run in goroutine 44
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/controller.go:131 +0xab

goroutine 129 [select]:
k8s.io/client-go/tools/cache.watchHandler({0x0?, 0x0?, 0x4704d60?}, {0x2e7ae40, 0xc00250c300}, {0x7c842f8c6a20, 0xc00370e420}, {0x2ec3af0?, 0x2a1b8e0}, 0x0, ...)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/reflector.go:714 +0x17b
k8s.io/client-go/tools/cache.(*Reflector).watch(0xc002ba40e0, {0x0?, 0x0?}, 0xc0000d6420, 0xc003c4fd80?)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/reflector.go:433 +0x545
k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch(0xc002ba40e0, 0xc0000d6420)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/reflector.go:358 +0x36c
k8s.io/client-go/tools/cache.(*Reflector).Run.func1()
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/reflector.go:291 +0x25
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x10?)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:226 +0x33
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc002514090?, {0x2e66ca0, 0xc002d7e2d0}, 0x1, 0xc0000d6420)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:227 +0xaf
k8s.io/client-go/tools/cache.(*Reflector).Run(0xc002ba40e0, 0xc0000d6420)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/reflector.go:290 +0x1cd
k8s.io/client-go/tools/cache.(*controller).Run.(*Group).StartWithChannel.func2()
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:55 +0x1e
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1()
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:72 +0x4f
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start in goroutine 44
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:70 +0x73

goroutine 108 [select]:
k8s.io/client-go/tools/cache.(*Reflector).startResync(0xc002ba40e0, 0xc0000d6420, 0xc00305a8a0, 0xf?)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/reflector.go:370 +0x10f
created by k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch in goroutine 129
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/reflector.go:357 +0x348

goroutine 106 [chan receive]:
k8s.io/client-go/tools/cache.(*processorListener).run.func1()
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/shared_informer.go:969 +0x4b
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x30?)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:226 +0x33
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc003c53738?, {0x2e66c80, 0xc0030560c0}, 0x1, 0xc00305a060)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:227 +0xaf
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0x0?, 0x3b9aca00, 0x0, 0x0?, 0xc003c53788?)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:204 +0x7f
k8s.io/apimachinery/pkg/util/wait.Until(...)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:161
k8s.io/client-go/tools/cache.(*processorListener).run(0xc0025079e0)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/shared_informer.go:968 +0x69
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1()
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:72 +0x4f
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start in goroutine 47
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:70 +0x73

goroutine 107 [select]:
k8s.io/client-go/tools/cache.(*processorListener).pop(0xc0025079e0)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/shared_informer.go:939 +0x10d
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1()
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:72 +0x4f
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start in goroutine 47
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:70 +0x73

goroutine 133 [sync.Cond.Wait]:
sync.runtime_notifyListWait(0xc0028ac7c8, 0x18)
        /usr/local/go/src/runtime/sema.go:527 +0x159
sync.(*Cond).Wait(0x2e79fe0?)
        /usr/local/go/src/sync/cond.go:70 +0x85
golang.org/x/net/http2.(*pipe).Read(0xc0028ac7b0, {0xc00216a001, 0x7dff, 0x7dff})
        /go/src/github.com/cilium/tetragon/vendor/golang.org/x/net/http2/pipe.go:76 +0xdf
golang.org/x/net/http2.transportResponseBody.Read({0x2a62177?}, {0xc00216a001?, 0x2a73ede?, 0xa?})
        /go/src/github.com/cilium/tetragon/vendor/golang.org/x/net/http2/transport.go:2558 +0x65
encoding/json.(*Decoder).refill(0xc001717e00)
        /usr/local/go/src/encoding/json/stream.go:165 +0x188
encoding/json.(*Decoder).readValue(0xc001717e00)
        /usr/local/go/src/encoding/json/stream.go:140 +0x9d
encoding/json.(*Decoder).Decode(0xc001717e00, {0x25ff1c0, 0xc0005e9dd0})
        /usr/local/go/src/encoding/json/stream.go:63 +0x75
k8s.io/apimachinery/pkg/util/framer.(*jsonFrameReader).Read(0xc002509560, {0xc002192000, 0x8000, 0xa000})
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/framer/framer.go:152 +0x19c
k8s.io/apimachinery/pkg/runtime/serializer/streaming.(*decoder).Decode(0xc002d7e550, 0xc002516840?, {0x2e7a968, 0xc0020f1dc0})
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/runtime/serializer/streaming/streaming.go:77 +0xa3
k8s.io/client-go/rest/watch.(*Decoder).Decode(0xc002b16840)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/rest/watch/decoder.go:49 +0x4b
k8s.io/apimachinery/pkg/watch.(*StreamWatcher).receive(0xc00250c300)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/watch/streamwatcher.go:105 +0xdb
created by k8s.io/apimachinery/pkg/watch.NewStreamWatcher in goroutine 129
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/watch/streamwatcher.go:76 +0x125

goroutine 134 [sync.Cond.Wait, 1 minutes]:
sync.runtime_notifyListWait(0xc00370e4f8, 0x1)
        /usr/local/go/src/runtime/sema.go:527 +0x159
sync.(*Cond).Wait(0xc002a9ed40?)
        /usr/local/go/src/sync/cond.go:70 +0x85
k8s.io/client-go/tools/cache.(*DeltaFIFO).Pop(0xc00370e4d0, 0xc002504d90)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/delta_fifo.go:575 +0x236
k8s.io/client-go/tools/cache.(*controller).processLoop(0xc003174500)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/controller.go:188 +0x30
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x30?)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:226 +0x33
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0x4466c0?, {0x2e66c80, 0xc0025096b0}, 0x1, 0xc0000d6420)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:227 +0xaf
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc0025024e0?, 0x3b9aca00, 0x0, 0x0?, 0xdf8475800?)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:204 +0x7f
k8s.io/apimachinery/pkg/util/wait.Until(...)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:161
k8s.io/client-go/tools/cache.(*controller).Run(0xc003174500, 0xc0000d6420)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/controller.go:159 +0x393
k8s.io/client-go/tools/cache.(*sharedIndexInformer).Run(0xc00370e160, 0xc002524f20?)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/shared_informer.go:504 +0x2c8
k8s.io/client-go/informers.(*sharedInformerFactory).Start.func1()
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/informers/factory.go:150 +0x5c
created by k8s.io/client-go/informers.(*sharedInformerFactory).Start in goroutine 1
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/informers/factory.go:148 +0x205

goroutine 142 [sync.Cond.Wait, 1 minutes]:
sync.runtime_notifyListWait(0xc002c016c8, 0x0)
        /usr/local/go/src/runtime/sema.go:527 +0x159
sync.(*Cond).Wait(0x58?)
        /usr/local/go/src/sync/cond.go:70 +0x85
golang.org/x/net/http2.(*pipe).Read(0xc002c016b0, {0xc003cea600, 0x200, 0x200})
        /go/src/github.com/cilium/tetragon/vendor/golang.org/x/net/http2/pipe.go:76 +0xdf
golang.org/x/net/http2.transportResponseBody.Read({0xc002b94d48?}, {0xc003cea600?, 0x2?, 0x100000009?})
        /go/src/github.com/cilium/tetragon/vendor/golang.org/x/net/http2/transport.go:2558 +0x65
encoding/json.(*Decoder).refill(0xc002548140)
        /usr/local/go/src/encoding/json/stream.go:165 +0x188
encoding/json.(*Decoder).readValue(0xc002548140)
        /usr/local/go/src/encoding/json/stream.go:140 +0x9d
encoding/json.(*Decoder).Decode(0xc002548140, {0x25ff1c0, 0xc002c84cc0})
        /usr/local/go/src/encoding/json/stream.go:63 +0x75
k8s.io/apimachinery/pkg/util/framer.(*jsonFrameReader).Read(0xc002c827e0, {0xc00253c800, 0x400, 0x400})
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/framer/framer.go:152 +0x19c
k8s.io/apimachinery/pkg/runtime/serializer/streaming.(*decoder).Decode(0xc002d7e780, 0x8?, {0x2e7a968, 0xc00250c540})
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/runtime/serializer/streaming/streaming.go:77 +0xa3
k8s.io/client-go/rest/watch.(*Decoder).Decode(0xc002b16d80)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/rest/watch/decoder.go:49 +0x4b
k8s.io/apimachinery/pkg/watch.(*StreamWatcher).receive(0xc00250c500)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/watch/streamwatcher.go:105 +0xdb
created by k8s.io/apimachinery/pkg/watch.NewStreamWatcher in goroutine 139
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/watch/streamwatcher.go:76 +0x125

goroutine 94 [chan receive, 1 minutes]:
k8s.io/client-go/tools/cache.(*processorListener).run.func1()
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/shared_informer.go:969 +0x4b
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x30?)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:226 +0x33
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc000944f38?, {0x2e66c80, 0xc002bf9140}, 0x1, 0xc002ab0de0)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:227 +0xaf
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc002bf90b0?, 0x3b9aca00, 0x0, 0x20?, 0xc000944f88?)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:204 +0x7f
k8s.io/apimachinery/pkg/util/wait.Until(...)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:161
k8s.io/client-go/tools/cache.(*processorListener).run(0xc002586b40)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/shared_informer.go:968 +0x69
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1()
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:72 +0x4f
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start in goroutine 122
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:70 +0x73

goroutine 89 [chan receive, 1 minutes]:
k8s.io/client-go/tools/cache.(*sharedProcessor).run(0xc002d7e1e0, 0xc0006847a8?)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/shared_informer.go:803 +0x47
k8s.io/client-go/tools/cache.(*sharedIndexInformer).Run.(*Group).StartWithChannel.func4()
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:55 +0x1e
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1()
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:72 +0x4f
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start in goroutine 86
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:70 +0x73

goroutine 90 [chan receive, 1 minutes]:
k8s.io/client-go/tools/cache.(*controller).Run.func1()
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/controller.go:132 +0x25
created by k8s.io/client-go/tools/cache.(*controller).Run in goroutine 86
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/controller.go:131 +0xab

goroutine 91 [select, 1 minutes]:
k8s.io/client-go/tools/cache.watchHandler({0x0?, 0x0?, 0x4704d60?}, {0x2e7df60, 0xc002c282a0}, {0x7c842f8c6a20, 0xc002c2c000}, {0x2ec3af0?, 0x29ef460}, 0x0, ...)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/reflector.go:714 +0x17b
k8s.io/client-go/tools/cache.(*Reflector).watch(0xc003a1cb60, {0x0?, 0x0?}, 0xc0000d6420, 0xc000800000?)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/reflector.go:433 +0x545
k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch(0xc003a1cb60, 0xc0000d6420)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/reflector.go:358 +0x36c
k8s.io/client-go/tools/cache.(*Reflector).Run.func1()
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/reflector.go:291 +0x25
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x10?)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:226 +0x33
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc002ab0c60?, {0x2e66ca0, 0xc002d0c320}, 0x1, 0xc0000d6420)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:227 +0xaf
k8s.io/client-go/tools/cache.(*Reflector).Run(0xc003a1cb60, 0xc0000d6420)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/reflector.go:290 +0x1cd
k8s.io/client-go/tools/cache.(*controller).Run.(*Group).StartWithChannel.func2()
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:55 +0x1e
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1()
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:72 +0x4f
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start in goroutine 86
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:70 +0x73

goroutine 93 [select]:
k8s.io/client-go/tools/cache.(*Reflector).startResync(0xc003a1cb60, 0xc0000d6420, 0xc002ab0d20, 0xc002540f10?)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/reflector.go:370 +0x10f
created by k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch in goroutine 91
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/reflector.go:357 +0x348

goroutine 144 [select]:
github.com/cilium/tetragon/pkg/eventcache.(*Cache).loop(0xc00250c5c0)
        /go/src/github.com/cilium/tetragon/pkg/eventcache/eventcache.go:169 +0x117
created by github.com/cilium/tetragon/pkg/eventcache.NewWithTimer in goroutine 1
        /go/src/github.com/cilium/tetragon/pkg/eventcache/eventcache.go:238 +0x178

goroutine 119 [sync.Cond.Wait, 1 minutes]:
sync.runtime_notifyListWait(0xc0028d6028, 0x0)
        /usr/local/go/src/runtime/sema.go:527 +0x159
sync.(*Cond).Wait(0xc000948be0?)
        /usr/local/go/src/sync/cond.go:70 +0x85
k8s.io/client-go/tools/cache.(*DeltaFIFO).Pop(0xc0028d6000, 0xc002879bd0)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/delta_fifo.go:575 +0x236
k8s.io/client-go/tools/cache.(*controller).processLoop(0xc00286d9a0)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/controller.go:188 +0x30
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x30?)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:226 +0x33
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0x4466c0?, {0x2e66c80, 0xc0028d41e0}, 0x1, 0xc0000d6420)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:227 +0xaf
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc002895b10?, 0x3b9aca00, 0x0, 0x0?, 0x0?)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:204 +0x7f
k8s.io/apimachinery/pkg/util/wait.Until(...)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:161
k8s.io/client-go/tools/cache.(*controller).Run(0xc00286d9a0, 0xc0000d6420)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/controller.go:159 +0x393
k8s.io/client-go/tools/cache.(*sharedIndexInformer).Run(0xc003190160, 0x22a17f4?)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/shared_informer.go:504 +0x2c8
github.com/cilium/tetragon/pkg/k8s/client/informers/externalversions.(*sharedInformerFactory).Start.func1()
        /go/src/github.com/cilium/tetragon/vendor/github.com/cilium/tetragon/pkg/k8s/client/informers/externalversions/factory.go:119 +0x5c
created by github.com/cilium/tetragon/pkg/k8s/client/informers/externalversions.(*sharedInformerFactory).Start in goroutine 118
        /go/src/github.com/cilium/tetragon/vendor/github.com/cilium/tetragon/pkg/k8s/client/informers/externalversions/factory.go:117 +0x205

goroutine 122 [chan receive, 1 minutes]:
k8s.io/client-go/tools/cache.(*sharedProcessor).run(0xc003148280, 0x2ea2f70?)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/shared_informer.go:803 +0x47
k8s.io/client-go/tools/cache.(*sharedIndexInformer).Run.(*Group).StartWithChannel.func4()
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:55 +0x1e
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1()
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:72 +0x4f
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start in goroutine 120
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:70 +0x73

goroutine 123 [chan receive, 1 minutes]:
k8s.io/client-go/tools/cache.(*controller).Run.func1()
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/controller.go:132 +0x25
created by k8s.io/client-go/tools/cache.(*controller).Run in goroutine 120
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/controller.go:131 +0xab

goroutine 124 [select, 1 minutes]:
k8s.io/client-go/tools/cache.watchHandler({0x0?, 0x0?, 0x4704d60?}, {0x2e7ae40, 0xc002899280}, {0x7c842f8c6a20, 0xc0031902c0}, {0x2ec3af0?, 0x2a01200}, 0x0, ...)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/reflector.go:714 +0x17b
k8s.io/client-go/tools/cache.(*Reflector).watch(0xc002a300e0, {0x0?, 0x0?}, 0xc0000d6420, 0xc003c4ff80?)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/reflector.go:433 +0x545
k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch(0xc002a300e0, 0xc0000d6420)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/reflector.go:358 +0x36c
k8s.io/client-go/tools/cache.(*Reflector).Run.func1()
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/reflector.go:291 +0x25
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x10?)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:226 +0x33
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc00298cf00?, {0x2e66ca0, 0xc0031482d0}, 0x1, 0xc0000d6420)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:227 +0xaf
k8s.io/client-go/tools/cache.(*Reflector).Run(0xc002a300e0, 0xc0000d6420)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/reflector.go:290 +0x1cd
k8s.io/client-go/tools/cache.(*controller).Run.(*Group).StartWithChannel.func2()
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:55 +0x1e
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1()
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:72 +0x4f
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start in goroutine 120
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:70 +0x73

goroutine 180 [select, 1 minutes]:
k8s.io/client-go/tools/cache.(*Reflector).startResync(0xc002a300e0, 0xc0000d6420, 0xc00305ae40, 0x25cbf40?)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/reflector.go:370 +0x10f
created by k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch in goroutine 124
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/reflector.go:357 +0x348

goroutine 111 [chan receive, 1 minutes]:
k8s.io/client-go/tools/cache.(*sharedProcessor).run(0xc003148230, 0x0?)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/shared_informer.go:803 +0x47
k8s.io/client-go/tools/cache.(*sharedIndexInformer).Run.(*Group).StartWithChannel.func4()
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:55 +0x1e
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1()
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:72 +0x4f
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start in goroutine 119
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:70 +0x73

goroutine 112 [chan receive, 1 minutes]:
k8s.io/client-go/tools/cache.(*controller).Run.func1()
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/controller.go:132 +0x25
created by k8s.io/client-go/tools/cache.(*controller).Run in goroutine 119
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/controller.go:131 +0xab

goroutine 95 [select, 1 minutes]:
k8s.io/client-go/tools/cache.(*processorListener).pop(0xc002586b40)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/shared_informer.go:939 +0x10d
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1()
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:72 +0x4f
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start in goroutine 122
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:70 +0x73

goroutine 161 [select, 1 minutes]:
k8s.io/client-go/tools/cache.(*processorListener).pop(0xc002586ab0)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/shared_informer.go:939 +0x10d
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1()
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:72 +0x4f
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start in goroutine 111
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:70 +0x73

goroutine 177 [select, 1 minutes]:
k8s.io/client-go/tools/cache.watchHandler({0x0?, 0x0?, 0x4704d60?}, {0x2e7ae40, 0xc002a1cec0}, {0x7c842f8c6a20, 0xc0028d6000}, {0x2ec3af0?, 0x29fc4a0}, 0x0, ...)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/reflector.go:714 +0x17b
k8s.io/client-go/tools/cache.(*Reflector).watch(0xc0028d8000, {0x0?, 0x0?}, 0xc0000d6420, 0x0?)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/reflector.go:433 +0x545
k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch(0xc0028d8000, 0xc0000d6420)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/reflector.go:358 +0x36c
k8s.io/client-go/tools/cache.(*Reflector).Run.func1()
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/reflector.go:291 +0x25
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x10?)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:226 +0x33
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc00305ac00?, {0x2e66ca0, 0xc0029fe0a0}, 0x1, 0xc0000d6420)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:227 +0xaf
k8s.io/client-go/tools/cache.(*Reflector).Run(0xc0028d8000, 0xc0000d6420)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/reflector.go:290 +0x1cd
k8s.io/client-go/tools/cache.(*controller).Run.(*Group).StartWithChannel.func2()
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:55 +0x1e
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1()
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:72 +0x4f
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start in goroutine 119
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:70 +0x73

goroutine 182 [select, 1 minutes]:
k8s.io/client-go/tools/cache.(*Reflector).startResync(0xc0028d8000, 0xc0000d6420, 0xc00305b140, 0xc002525fb8?)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/reflector.go:370 +0x10f
created by k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch in goroutine 177
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/reflector.go:357 +0x348

goroutine 181 [select, 1 minutes]:
golang.org/x/net/http2.(*clientStream).writeRequest(0xc0028aca80, 0xc0027f4d00)
        /go/src/github.com/cilium/tetragon/vendor/golang.org/x/net/http2/transport.go:1464 +0xac7
golang.org/x/net/http2.(*clientStream).doRequest(0x0?, 0xc003c4ff10?)
        /go/src/github.com/cilium/tetragon/vendor/golang.org/x/net/http2/transport.go:1326 +0x18
created by golang.org/x/net/http2.(*ClientConn).RoundTrip in goroutine 124
        /go/src/github.com/cilium/tetragon/vendor/golang.org/x/net/http2/transport.go:1232 +0x308

goroutine 127 [sync.Cond.Wait, 1 minutes]:
sync.runtime_notifyListWait(0xc0028acc48, 0x0)
        /usr/local/go/src/runtime/sema.go:527 +0x159
sync.(*Cond).Wait(0x58?)
        /usr/local/go/src/sync/cond.go:70 +0x85
golang.org/x/net/http2.(*pipe).Read(0xc0028acc30, {0xc00319cc00, 0x200, 0x200})
        /go/src/github.com/cilium/tetragon/vendor/golang.org/x/net/http2/pipe.go:76 +0xdf
golang.org/x/net/http2.transportResponseBody.Read({0xc000947d48?}, {0xc00319cc00?, 0x2?, 0x10000000f?})
        /go/src/github.com/cilium/tetragon/vendor/golang.org/x/net/http2/transport.go:2558 +0x65
encoding/json.(*Decoder).refill(0xc002a223c0)
        /usr/local/go/src/encoding/json/stream.go:165 +0x188
encoding/json.(*Decoder).readValue(0xc002a223c0)
        /usr/local/go/src/encoding/json/stream.go:140 +0x9d
encoding/json.(*Decoder).Decode(0xc002a223c0, {0x25ff1c0, 0xc0029beaf8})
        /usr/local/go/src/encoding/json/stream.go:63 +0x75
k8s.io/apimachinery/pkg/util/framer.(*jsonFrameReader).Read(0xc00297bdd0, {0xc002a2dc00, 0x400, 0x400})
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/framer/framer.go:152 +0x19c
k8s.io/apimachinery/pkg/runtime/serializer/streaming.(*decoder).Decode(0xc0031484b0, 0x8?, {0x2e7a968, 0xc002a1cf00})
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/runtime/serializer/streaming/streaming.go:77 +0xa3
k8s.io/client-go/rest/watch.(*Decoder).Decode(0xc002597b20)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/rest/watch/decoder.go:49 +0x4b
k8s.io/apimachinery/pkg/watch.(*StreamWatcher).receive(0xc002a1cec0)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/watch/streamwatcher.go:105 +0xdb
created by k8s.io/apimachinery/pkg/watch.NewStreamWatcher in goroutine 177
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/watch/streamwatcher.go:76 +0x125

goroutine 183 [select, 1 minutes]:
golang.org/x/net/http2.(*clientStream).writeRequest(0xc0028acc00, 0xc0027f5000)
        /go/src/github.com/cilium/tetragon/vendor/golang.org/x/net/http2/transport.go:1464 +0xac7
golang.org/x/net/http2.(*clientStream).doRequest(0x0?, 0xc003c4df10?)
        /go/src/github.com/cilium/tetragon/vendor/golang.org/x/net/http2/transport.go:1326 +0x18
created by golang.org/x/net/http2.(*ClientConn).RoundTrip in goroutine 177
        /go/src/github.com/cilium/tetragon/vendor/golang.org/x/net/http2/transport.go:1232 +0x308

goroutine 184 [sync.Cond.Wait, 1 minutes]:
sync.runtime_notifyListWait(0xc0028acac8, 0x0)
        /usr/local/go/src/runtime/sema.go:527 +0x159
sync.(*Cond).Wait(0xc0028acab8?)
        /usr/local/go/src/sync/cond.go:70 +0x85
golang.org/x/net/http2.(*pipe).Read(0xc0028acab0, {0xc0039aa800, 0x200, 0x200})
        /go/src/github.com/cilium/tetragon/vendor/golang.org/x/net/http2/pipe.go:76 +0xdf
golang.org/x/net/http2.transportResponseBody.Read({0xc003c4dd48?}, {0xc0039aa800?, 0xc003c4dd10?, 0xc003c4dcac?})
        /go/src/github.com/cilium/tetragon/vendor/golang.org/x/net/http2/transport.go:2558 +0x65
encoding/json.(*Decoder).refill(0xc002889900)
        /usr/local/go/src/encoding/json/stream.go:165 +0x188
encoding/json.(*Decoder).readValue(0xc002889900)
        /usr/local/go/src/encoding/json/stream.go:140 +0x9d
encoding/json.(*Decoder).Decode(0xc002889900, {0x25ff1c0, 0xc0028e2210})
        /usr/local/go/src/encoding/json/stream.go:63 +0x75
k8s.io/apimachinery/pkg/util/framer.(*jsonFrameReader).Read(0xc0028d5ef0, {0xc0028cf400, 0x400, 0x400})
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/framer/framer.go:152 +0x19c
k8s.io/apimachinery/pkg/runtime/serializer/streaming.(*decoder).Decode(0xc0029fe5f0, 0xc003192050?, {0x2e7a968, 0xc0028992c0})
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/runtime/serializer/streaming/streaming.go:77 +0xa3
k8s.io/client-go/rest/watch.(*Decoder).Decode(0xc0028f65c0)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/rest/watch/decoder.go:49 +0x4b
k8s.io/apimachinery/pkg/watch.(*StreamWatcher).receive(0xc002899280)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/watch/streamwatcher.go:105 +0xdb
created by k8s.io/apimachinery/pkg/watch.NewStreamWatcher in goroutine 124
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/watch/streamwatcher.go:76 +0x125

Here is gops 1 output in our case, it's clear that gRPC server is not run:

# gops 1
threads:        16
memory usage:   0.540%
cpu usage:      1.152%
username:       root
cmd+args:       /usr/bin/tetragon --config-dir=/etc/tetragon/tetragon.conf.d/
elapsed time:   01:21
local/remote:   127.0.0.1:8118 <-> 0.0.0.0:0 (LISTEN)
local/remote:   0.0.0.0:2112 <-> 0.0.0.0:0 (LISTEN)
local/remote:   10.80.122.156:42272 <-> 10.43.0.1:443 (ESTABLISHED)

After upgrade to latest CI image with help of these command:

helm install --namespace kube-system \    
        --set tetragonOperator.image.override=quay.io/cilium/tetragon-operator-ci:latest \
        --set tetragon.image.override=quay.io/cilium/tetragon-ci:latest  \
        tetragon cilium/tetragon

Tetragon start failing to start with this message:

time="2024-03-26T12:26:53Z" level=info msg="tetragon, map loaded." map=tg_execve_joined_info_map_stats path=/sys/fs/bpf/tetragon/tg_execve_joined_info_map_stats sensor=__base__
time="2024-03-26T12:26:53Z" level=info msg="tetragon, map loaded." map=execve_calls path=/sys/fs/bpf/tetragon/execve_calls sensor=__base__
time="2024-03-26T12:26:53Z" level=info msg="tetragon, map loaded." map=tcpmon_map path=/sys/fs/bpf/tetragon/tcpmon_map sensor=__base__
time="2024-03-26T12:26:53Z" level=info msg="tetragon, map loaded." map=tg_conf_map path=/sys/fs/bpf/tetragon/tg_conf_map sensor=__base__
time="2024-03-26T12:26:53Z" level=info msg="tetragon, map loaded." map=tg_stats_map path=/sys/fs/bpf/tetragon/tg_stats_map sensor=__base__
time="2024-03-26T12:26:53Z" level=info msg="Loading BPF program" Attach=acct_process Program=/var/lib/tetragon/bpf_exit.o Type=kprobe
time="2024-03-26T12:26:53Z" level=info msg="BPF events statistics: 0 received, 0% events loss"
time="2024-03-26T12:26:53Z" level=info msg="Observer events statistics" errors=0 filterDrop=0 filterPass=0 lost=0 received=0
time="2024-03-26T12:26:53Z" level=info msg="sensor controller starts"
time="2024-03-26T12:26:53Z" level=fatal msg="Failed to start tetragon" error="failed prog /var/lib/tetragon/bpf_exit.o kern_version 328721 loadInstance: attaching 'event_exit_acct_process' failed: creating perf_kprobe PMU (arch-specific fallback for \"acct_process\"): token acct_process: opening perf event: operation not permitted"

OS is Oracle Linux 8.5 with CONFIG_BSD_PROCESS_ACCT enabled

# grep CONFIG_BSD_PROCESS_ACCT  /boot/config-$(uname -r)
CONFIG_BSD_PROCESS_ACCT=y
CONFIG_BSD_PROCESS_ACCT_V3=y

Also:

# grep acct_process < /proc/kallsyms
ffffffffb7d5fc20 t do_acct_process
ffffffffb7d60810 T acct_process

Tetragon Version

v1.0.0, v1.0.2, latest CI

Kernel Version

5.4.17-2136.304.4.1.el8uek.x86_64

Kubernetes Version

No response

Bugtool

No response

Relevant log output

No response

Anything else?

No response

Kornilios Kourtis · Answer 1 · Thu Mar 28 2024 16:09:16 GMT+0800 (China Standard Time)

Thanks @inliquid,

Is Oracle Linux using lockdown by any chance? What's the output of:

 sudo cat /sys/kernel/security/lockdown

inliquid · Answer 2 · Thu Mar 28 2024 17:07:30 GMT+0800 (China Standard Time)

Yes it seems so:

# cat /sys/kernel/security/lockdown
none integrity [confidentiality]

Kornilios Kourtis · Answer 3 · Thu Mar 28 2024 18:58:19 GMT+0800 (China Standard Time)

This mode prohibits loading BPF programs, which means that Tetragon cannot really work. We suggest to switch to "integrity" instead. See iovisor/bcc#2565 for more discussion on this.

There is a PR from @tixxdz to issue an early warning about this: #2149.

inliquid · Answer 4 · Thu Mar 28 2024 21:42:54 GMT+0800 (China Standard Time)

@kkourt @tixxdz thank you, makes sense!

I'll try with "integrity" mode and get back with result.

Mahé · Answer 5 · Thu Apr 18 2024 19:05:25 GMT+0800 (China Standard Time)

Given that we now have an issue to add the documentation for lockdown-related issues:

#2265

I'll close this issue, feel free to reopen this if you still have issues with this and this will not be addressed by #2265 being resolved.