Problems running gRPC server, misteriously led to failure attaching `event_exit_acct_process` on latest version
inliquid opened this issue · comments
What happened?
This issue stated as a discussion with several participants who had mostly the same problem with Tetragon v1.0.0-v1.0.2.
Symptoms are:
- Tetragon can't run due to liveness probe failure, which is based on
tetra
being connected to Tetragon over gRPC - When liveness probe is disabled Tetragon starts, but gRPC server not run despite logs say that it should:
time="2024-03-23T15:19:59Z" level=info msg="Starting gRPC server" address="localhost:54321" protocol=tcp
When this line is present there should be goroutine started with gRPC server instance.
However, what we observed is that for some mysterious reason it's not run, and stack trace shows no such goroutine - try search for "grpc" in the stack trace output:
# gops stack 1
goroutine 14 [running]:
runtime/pprof.writeGoroutineStacks({0x2e63600, 0xc00086a310})
/usr/local/go/src/runtime/pprof/pprof.go:703 +0x6a
runtime/pprof.writeGoroutine({0x2e63600?, 0xc00086a310?}, 0xc0000a9400?)
/usr/local/go/src/runtime/pprof/pprof.go:692 +0x25
runtime/pprof.(*Profile).WriteTo(0x2a6f74e?, {0x2e63600?, 0xc00086a310?}, 0x231f?)
/usr/local/go/src/runtime/pprof/pprof.go:329 +0x146
github.com/google/gops/agent.handle({0x7c842f4de5c8?, 0xc00086a310}, {0xc0007a6000?, 0x1?, 0x1?})
/go/src/github.com/cilium/tetragon/vendor/github.com/google/gops/agent/agent.go:200 +0xd2
github.com/google/gops/agent.listen({0x2e83de8, 0xc0002b9520})
/go/src/github.com/cilium/tetragon/vendor/github.com/google/gops/agent/agent.go:144 +0x1a5
created by github.com/google/gops/agent.Listen in goroutine 1
/go/src/github.com/cilium/tetragon/vendor/github.com/google/gops/agent/agent.go:122 +0x379
goroutine 1 [chan send]:
github.com/cilium/tetragon/pkg/sensors.(*Manager).RemoveAllSensors(...)
/go/src/github.com/cilium/tetragon/pkg/sensors/manager.go:278
github.com/cilium/tetragon/pkg/observer.RemoveSensors({0x2e98248?, 0xc002d7e910})
/go/src/github.com/cilium/tetragon/pkg/observer/observer.go:420 +0xc7
main.tetragonExecute.func5()
/go/src/github.com/cilium/tetragon/cmd/tetragon/main.go:303 +0x1e
main.tetragonExecute()
/go/src/github.com/cilium/tetragon/cmd/tetragon/main.go:440 +0x1c65
main.execute.func1(0xc000670700?, {0x2a63272?, 0x4?, 0x2a63276?})
/go/src/github.com/cilium/tetragon/cmd/tetragon/main.go:733 +0xcb
github.com/spf13/cobra.(*Command).execute(0xc00056b200, {0xc000072050, 0x1, 0x1})
/go/src/github.com/cilium/tetragon/vendor/github.com/spf13/cobra/command.go:944 +0x863
github.com/spf13/cobra.(*Command).ExecuteC(0xc00056b200)
/go/src/github.com/cilium/tetragon/vendor/github.com/spf13/cobra/command.go:1068 +0x3a5
github.com/spf13/cobra.(*Command).Execute(...)
/go/src/github.com/cilium/tetragon/vendor/github.com/spf13/cobra/command.go:992
main.execute()
/go/src/github.com/cilium/tetragon/cmd/tetragon/main.go:746 +0x16f
main.main()
/go/src/github.com/cilium/tetragon/cmd/tetragon/tetragon.go:11 +0x13
goroutine 138 [chan receive, 1 minutes]:
k8s.io/client-go/tools/cache.(*controller).Run.func1()
/go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/controller.go:132 +0x25
created by k8s.io/client-go/tools/cache.(*controller).Run in goroutine 134
/go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/controller.go:131 +0xab
goroutine 16 [syscall, 1 minutes]:
os/signal.signal_recv()
/usr/local/go/src/runtime/sigqueue.go:152 +0x29
os/signal.loop()
/usr/local/go/src/os/signal/signal_unix.go:23 +0x13
created by os/signal.Notify.func1.1 in goroutine 1
/usr/local/go/src/os/signal/signal.go:151 +0x1f
goroutine 49 [chan receive, 1 minutes]:
main.tetragonExecute.func3(0x0?)
/go/src/github.com/cilium/tetragon/cmd/tetragon/main.go:255 +0x50
created by main.tetragonExecute in goroutine 1
/go/src/github.com/cilium/tetragon/cmd/tetragon/main.go:253 +0xed7
goroutine 50 [chan receive, 1 minutes]:
github.com/cilium/tetragon/pkg/sensors.startSensorManager.func1()
/go/src/github.com/cilium/tetragon/pkg/sensors/manager.go:65 +0x65
created by github.com/cilium/tetragon/pkg/sensors.startSensorManager in goroutine 1
/go/src/github.com/cilium/tetragon/pkg/sensors/manager.go:60 +0xc5
goroutine 51 [IO wait, 1 minutes]:
internal/poll.runtime_pollWait(0x7c842fb4cea8, 0x72)
/usr/local/go/src/runtime/netpoll.go:343 +0x85
internal/poll.(*pollDesc).wait(0xc002b18000?, 0xc002b13c50?, 0x0)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x27
internal/poll.(*pollDesc).waitRead(...)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Accept(0xc002b18000)
/usr/local/go/src/internal/poll/fd_unix.go:611 +0x2ac
net.(*netFD).accept(0xc002b18000)
/usr/local/go/src/net/fd_unix.go:172 +0x29
net.(*TCPListener).accept(0xc002b16080)
/usr/local/go/src/net/tcpsock_posix.go:152 +0x1e
net.(*TCPListener).Accept(0xc002b16080)
/usr/local/go/src/net/tcpsock.go:315 +0x30
net/http.(*Server).Serve(0xc002d9c000, {0x2e83de8, 0xc002b16080})
/usr/local/go/src/net/http/server.go:3056 +0x364
net/http.(*Server).ListenAndServe(0xc002d9c000)
/usr/local/go/src/net/http/server.go:2985 +0x71
net/http.ListenAndServe(...)
/usr/local/go/src/net/http/server.go:3239
github.com/cilium/tetragon/pkg/metrics.EnableMetrics({0xc000589144, 0x5})
/go/src/github.com/cilium/tetragon/pkg/metrics/metrics.go:25 +0x1cf
created by main.tetragonExecute in goroutine 1
/go/src/github.com/cilium/tetragon/cmd/tetragon/main.go:323 +0x109e
goroutine 81 [select]:
k8s.io/client-go/util/workqueue.(*delayingType).waitingLoop(0xc002b98060)
/go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/util/workqueue/delaying_queue.go:276 +0x305
created by k8s.io/client-go/util/workqueue.newDelayingQueue in goroutine 31
/go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/util/workqueue/delaying_queue.go:113 +0x21f
goroutine 31 [sync.Cond.Wait, 1 minutes]:
sync.runtime_notifyListWait(0xc0038ecb50, 0x0)
/usr/local/go/src/runtime/sema.go:527 +0x159
sync.(*Cond).Wait(0xc000b808a0?)
/usr/local/go/src/sync/cond.go:70 +0x85
k8s.io/client-go/util/workqueue.(*Type).Get(0xc000b808a0)
/go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/util/workqueue/queue.go:200 +0x99
github.com/cilium/tetragon/pkg/metrics.StartPodDeleteHandler()
/go/src/github.com/cilium/tetragon/pkg/metrics/metricwithpod.go:126 +0x44
created by main.tetragonExecute in goroutine 1
/go/src/github.com/cilium/tetragon/cmd/tetragon/main.go:325 +0x10b6
goroutine 143 [select]:
github.com/cilium/tetragon/pkg/process.(*Cache).cacheGarbageCollector.func1()
/go/src/github.com/cilium/tetragon/pkg/process/cache.go:47 +0xf1
created by github.com/cilium/tetragon/pkg/process.(*Cache).cacheGarbageCollector in goroutine 1
/go/src/github.com/cilium/tetragon/pkg/process/cache.go:43 +0xe7
goroutine 137 [chan receive, 1 minutes]:
k8s.io/client-go/tools/cache.(*sharedProcessor).run(0xc002d7e190, 0xc001215dd0?)
/go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/shared_informer.go:803 +0x47
k8s.io/client-go/tools/cache.(*sharedIndexInformer).Run.(*Group).StartWithChannel.func4()
/go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:55 +0x1e
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1()
/go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:72 +0x4f
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start in goroutine 134
/go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:70 +0x73
goroutine 96 [chan receive, 1 minutes]:
k8s.io/client-go/tools/cache.(*processorListener).run.func1()
/go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/shared_informer.go:969 +0x4b
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x30?)
/go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:226 +0x33
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc002b90f38?, {0x2e66c80, 0xc002bf9170}, 0x1, 0xc002ab0e40)
/go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:227 +0xaf
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0x0?, 0x3b9aca00, 0x0, 0xc0?, 0xc002b90f88?)
/go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:204 +0x7f
k8s.io/apimachinery/pkg/util/wait.Until(...)
/go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:161
k8s.io/client-go/tools/cache.(*processorListener).run(0xc002586ab0)
/go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/shared_informer.go:968 +0x69
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1()
/go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:72 +0x4f
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start in goroutine 111
/go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:70 +0x73
goroutine 86 [sync.Cond.Wait, 1 minutes]:
sync.runtime_notifyListWait(0xc002c2c028, 0x0)
/usr/local/go/src/runtime/sema.go:527 +0x159
sync.(*Cond).Wait(0xc000943be0?)
/usr/local/go/src/sync/cond.go:70 +0x85
k8s.io/client-go/tools/cache.(*DeltaFIFO).Pop(0xc002c2c000, 0xc002bcf9f0)
/go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/delta_fifo.go:575 +0x236
k8s.io/client-go/tools/cache.(*controller).processLoop(0xc002c2a140)
/go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/controller.go:188 +0x30
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x30?)
/go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:226 +0x33
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0x4466c0?, {0x2e66c80, 0xc002bf9080}, 0x1, 0xc0000d6420)
/go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:227 +0xaf
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc002bfb6f0?, 0x3b9aca00, 0x0, 0x0?, 0xdf8475800?)
/go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:204 +0x7f
k8s.io/apimachinery/pkg/util/wait.Until(...)
/go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:161
k8s.io/client-go/tools/cache.(*controller).Run(0xc002c2a140, 0xc0000d6420)
/go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/controller.go:159 +0x393
k8s.io/client-go/tools/cache.(*sharedIndexInformer).Run(0xc00370e210, 0x0?)
/go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/shared_informer.go:504 +0x2c8
github.com/cilium/tetragon/pkg/k8s/client/informers/externalversions.(*sharedInformerFactory).Start.func1()
/go/src/github.com/cilium/tetragon/vendor/github.com/cilium/tetragon/pkg/k8s/client/informers/externalversions/factory.go:119 +0x5c
created by github.com/cilium/tetragon/pkg/k8s/client/informers/externalversions.(*sharedInformerFactory).Start in goroutine 1
/go/src/github.com/cilium/tetragon/vendor/github.com/cilium/tetragon/pkg/k8s/client/informers/externalversions/factory.go:117 +0x205
goroutine 139 [select, 1 minutes]:
k8s.io/client-go/tools/cache.watchHandler({0x0?, 0x0?, 0x4704d60?}, {0x2e7ae40, 0xc00250c500}, {0x7c842f8c6a20, 0xc00370e4d0}, {0x2ec3af0?, 0x2a1c760}, 0x0, ...)
/go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/reflector.go:714 +0x17b
k8s.io/client-go/tools/cache.(*Reflector).watch(0xc002ba4540, {0x0?, 0x0?}, 0xc0000d6420, 0x61bf?)
/go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/reflector.go:433 +0x545
k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch(0xc002ba4540, 0xc0000d6420)
/go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/reflector.go:358 +0x36c
k8s.io/client-go/tools/cache.(*Reflector).Run.func1()
/go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/reflector.go:291 +0x25
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x10?)
/go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:226 +0x33
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc002514420?, {0x2e66ca0, 0xc002d7e640}, 0x1, 0xc0000d6420)
/go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:227 +0xaf
k8s.io/client-go/tools/cache.(*Reflector).Run(0xc002ba4540, 0xc0000d6420)
/go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/reflector.go:290 +0x1cd
k8s.io/client-go/tools/cache.(*controller).Run.(*Group).StartWithChannel.func2()
/go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:55 +0x1e
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1()
/go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:72 +0x4f
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start in goroutine 134
/go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:70 +0x73
goroutine 104 [chan receive]:
k8s.io/client-go/tools/cache.(*processorListener).run.func1()
/go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/shared_informer.go:969 +0x4b
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x30?)
/go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:226 +0x33
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc000948f38?, {0x2e66c80, 0xc003056090}, 0x1, 0xc00305a000)
/go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:227 +0xaf
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0x0?, 0x3b9aca00, 0x0, 0x30?, 0xc0023da8a0?)
/go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:204 +0x7f
k8s.io/apimachinery/pkg/util/wait.Until(...)
/go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:161
k8s.io/client-go/tools/cache.(*processorListener).run(0xc002507950)
/go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/shared_informer.go:968 +0x69
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1()
/go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:72 +0x4f
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start in goroutine 47
/go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:70 +0x73
goroutine 105 [select]:
k8s.io/client-go/tools/cache.(*processorListener).pop(0xc002507950)
/go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/shared_informer.go:939 +0x10d
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1()
/go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:72 +0x4f
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start in goroutine 47
/go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:70 +0x73
goroutine 120 [sync.Cond.Wait, 1 minutes]:
sync.runtime_notifyListWait(0xc0031902e8, 0x0)
/usr/local/go/src/runtime/sema.go:527 +0x159
sync.(*Cond).Wait(0xc002528be0?)
/usr/local/go/src/sync/cond.go:70 +0x85
k8s.io/client-go/tools/cache.(*DeltaFIFO).Pop(0xc0031902c0, 0xc002585f30)
/go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/delta_fifo.go:575 +0x236
k8s.io/client-go/tools/cache.(*controller).processLoop(0xc001762e60)
/go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/controller.go:188 +0x30
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x30?)
/go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:226 +0x33
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0x4466c0?, {0x2e66c80, 0xc00297b8f0}, 0x1, 0xc0000d6420)
/go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:227 +0xaf
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc002994f90?, 0x3b9aca00, 0x0, 0x0?, 0x0?)
/go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:204 +0x7f
k8s.io/apimachinery/pkg/util/wait.Until(...)
/go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:161
k8s.io/client-go/tools/cache.(*controller).Run(0xc001762e60, 0xc0000d6420)
/go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/controller.go:159 +0x393
k8s.io/client-go/tools/cache.(*sharedIndexInformer).Run(0xc003190210, 0x0?)
/go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/shared_informer.go:504 +0x2c8
github.com/cilium/tetragon/pkg/k8s/client/informers/externalversions.(*sharedInformerFactory).Start.func1()
/go/src/github.com/cilium/tetragon/vendor/github.com/cilium/tetragon/pkg/k8s/client/informers/externalversions/factory.go:119 +0x5c
created by github.com/cilium/tetragon/pkg/k8s/client/informers/externalversions.(*sharedInformerFactory).Start in goroutine 118
/go/src/github.com/cilium/tetragon/vendor/github.com/cilium/tetragon/pkg/k8s/client/informers/externalversions/factory.go:117 +0x205
goroutine 115 [IO wait]:
internal/poll.runtime_pollWait(0x7c842fb4cdb0, 0x72)
/usr/local/go/src/runtime/netpoll.go:343 +0x85
internal/poll.(*pollDesc).wait(0xc002cddd00?, 0xc002562000?, 0x0)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x27
internal/poll.(*pollDesc).waitRead(...)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc002cddd00, {0xc002562000, 0xa000, 0xa000})
/usr/local/go/src/internal/poll/fd_unix.go:164 +0x27a
net.(*netFD).Read(0xc002cddd00, {0xc002562000?, 0xc00256603a?, 0x1a5?})
/usr/local/go/src/net/fd_posix.go:55 +0x25
net.(*conn).Read(0xc0004d6008, {0xc002562000?, 0x6f33b7?, 0xc0031421b8?})
/usr/local/go/src/net/net.go:179 +0x45
crypto/tls.(*atLeastReader).Read(0xc0005e9698, {0xc002562000?, 0xc0005e9698?, 0x0?})
/usr/local/go/src/crypto/tls/conn.go:805 +0x3b
bytes.(*Buffer).ReadFrom(0xc0031422a8, {0x2e65a40, 0xc0005e9698})
/usr/local/go/src/bytes/buffer.go:211 +0x98
crypto/tls.(*Conn).readFromUntil(0xc003142000, {0x2e63520?, 0xc0004d6008}, 0x5fcb?)
/usr/local/go/src/crypto/tls/conn.go:827 +0xde
crypto/tls.(*Conn).readRecordOrCCS(0xc003142000, 0x0)
/usr/local/go/src/crypto/tls/conn.go:625 +0x250
crypto/tls.(*Conn).readRecord(...)
/usr/local/go/src/crypto/tls/conn.go:587
crypto/tls.(*Conn).Read(0xc003142000, {0xc003195000, 0x1000, 0x98cbc9?})
/usr/local/go/src/crypto/tls/conn.go:1369 +0x158
bufio.(*Reader).Read(0xc003196000, {0xc003180200, 0x9, 0x25aecc0?})
/usr/local/go/src/bufio/bufio.go:244 +0x197
io.ReadAtLeast({0x2e63320, 0xc003196000}, {0xc003180200, 0x9, 0x9}, 0x9)
/usr/local/go/src/io/io.go:335 +0x90
io.ReadFull(...)
/usr/local/go/src/io/io.go:354
golang.org/x/net/http2.readFrameHeader({0xc003180200, 0x9, 0xa8400?}, {0x2e63320?, 0xc003196000?})
/go/src/github.com/cilium/tetragon/vendor/golang.org/x/net/http2/frame.go:237 +0x65
golang.org/x/net/http2.(*Framer).ReadFrame(0xc0031801c0)
/go/src/github.com/cilium/tetragon/vendor/golang.org/x/net/http2/frame.go:498 +0x85
golang.org/x/net/http2.(*clientConnReadLoop).run(0xc00315ff98)
/go/src/github.com/cilium/tetragon/vendor/golang.org/x/net/http2/transport.go:2275 +0x11f
golang.org/x/net/http2.(*ClientConn).readLoop(0xc003192000)
/go/src/github.com/cilium/tetragon/vendor/golang.org/x/net/http2/transport.go:2170 +0x65
created by golang.org/x/net/http2.(*Transport).newClientConn in goroutine 114
/go/src/github.com/cilium/tetragon/vendor/golang.org/x/net/http2/transport.go:821 +0xcbe
goroutine 84 [select]:
k8s.io/client-go/tools/cache.(*Reflector).startResync(0xc002ba4540, 0xc0000d6420, 0xc002ab0840, 0x209?)
/go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/reflector.go:370 +0x10f
created by k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch in goroutine 139
/go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/reflector.go:357 +0x348
goroutine 44 [sync.Cond.Wait]:
sync.runtime_notifyListWait(0xc00370e448, 0x19)
/usr/local/go/src/runtime/sema.go:527 +0x159
sync.(*Cond).Wait(0xc0036f92c0?)
/usr/local/go/src/sync/cond.go:70 +0x85
k8s.io/client-go/tools/cache.(*DeltaFIFO).Pop(0xc00370e420, 0xc0025047b0)
/go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/delta_fifo.go:575 +0x236
k8s.io/client-go/tools/cache.(*controller).processLoop(0xc0031741e0)
/go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/controller.go:188 +0x30
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x30?)
/go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:226 +0x33
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0x4466c0?, {0x2e66c80, 0xc002508ae0}, 0x1, 0xc0000d6420)
/go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:227 +0xaf
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc0025021d0?, 0x3b9aca00, 0x0, 0x0?, 0xdf8475800?)
/go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:204 +0x7f
k8s.io/apimachinery/pkg/util/wait.Until(...)
/go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:161
k8s.io/client-go/tools/cache.(*controller).Run(0xc0031741e0, 0xc0000d6420)
/go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/controller.go:159 +0x393
k8s.io/client-go/tools/cache.(*sharedIndexInformer).Run(0xc00370e0b0, 0x0?)
/go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/shared_informer.go:504 +0x2c8
k8s.io/client-go/informers.(*sharedInformerFactory).Start.func1()
/go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/informers/factory.go:150 +0x5c
created by k8s.io/client-go/informers.(*sharedInformerFactory).Start in goroutine 1
/go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/informers/factory.go:148 +0x205
goroutine 85 [select, 1 minutes]:
golang.org/x/net/http2.(*clientStream).writeRequest(0xc002c01680, 0xc002aa7f00)
/go/src/github.com/cilium/tetragon/vendor/golang.org/x/net/http2/transport.go:1464 +0xac7
golang.org/x/net/http2.(*clientStream).doRequest(0xc0025024e0?, 0xc002522fb8?)
/go/src/github.com/cilium/tetragon/vendor/golang.org/x/net/http2/transport.go:1326 +0x18
created by golang.org/x/net/http2.(*ClientConn).RoundTrip in goroutine 139
/go/src/github.com/cilium/tetragon/vendor/golang.org/x/net/http2/transport.go:1232 +0x308
goroutine 109 [select, 1 minutes]:
golang.org/x/net/http2.(*clientStream).writeRequest(0xc0028ac780, 0xc0027f4700)
/go/src/github.com/cilium/tetragon/vendor/golang.org/x/net/http2/transport.go:1464 +0xac7
golang.org/x/net/http2.(*clientStream).doRequest(0xc0025021d0?, 0xc002529fb8?)
/go/src/github.com/cilium/tetragon/vendor/golang.org/x/net/http2/transport.go:1326 +0x18
created by golang.org/x/net/http2.(*ClientConn).RoundTrip in goroutine 129
/go/src/github.com/cilium/tetragon/vendor/golang.org/x/net/http2/transport.go:1232 +0x308
goroutine 47 [chan receive, 1 minutes]:
k8s.io/client-go/tools/cache.(*sharedProcessor).run(0xc002d7e140, 0x0?)
/go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/shared_informer.go:803 +0x47
k8s.io/client-go/tools/cache.(*sharedIndexInformer).Run.(*Group).StartWithChannel.func4()
/go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:55 +0x1e
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1()
/go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:72 +0x4f
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start in goroutine 44
/go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:70 +0x73
goroutine 48 [chan receive, 1 minutes]:
k8s.io/client-go/tools/cache.(*controller).Run.func1()
/go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/controller.go:132 +0x25
created by k8s.io/client-go/tools/cache.(*controller).Run in goroutine 44
/go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/controller.go:131 +0xab
goroutine 129 [select]:
k8s.io/client-go/tools/cache.watchHandler({0x0?, 0x0?, 0x4704d60?}, {0x2e7ae40, 0xc00250c300}, {0x7c842f8c6a20, 0xc00370e420}, {0x2ec3af0?, 0x2a1b8e0}, 0x0, ...)
/go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/reflector.go:714 +0x17b
k8s.io/client-go/tools/cache.(*Reflector).watch(0xc002ba40e0, {0x0?, 0x0?}, 0xc0000d6420, 0xc003c4fd80?)
/go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/reflector.go:433 +0x545
k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch(0xc002ba40e0, 0xc0000d6420)
/go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/reflector.go:358 +0x36c
k8s.io/client-go/tools/cache.(*Reflector).Run.func1()
/go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/reflector.go:291 +0x25
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x10?)
/go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:226 +0x33
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc002514090?, {0x2e66ca0, 0xc002d7e2d0}, 0x1, 0xc0000d6420)
/go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:227 +0xaf
k8s.io/client-go/tools/cache.(*Reflector).Run(0xc002ba40e0, 0xc0000d6420)
/go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/reflector.go:290 +0x1cd
k8s.io/client-go/tools/cache.(*controller).Run.(*Group).StartWithChannel.func2()
/go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:55 +0x1e
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1()
/go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:72 +0x4f
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start in goroutine 44
/go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:70 +0x73
goroutine 108 [select]:
k8s.io/client-go/tools/cache.(*Reflector).startResync(0xc002ba40e0, 0xc0000d6420, 0xc00305a8a0, 0xf?)
/go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/reflector.go:370 +0x10f
created by k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch in goroutine 129
/go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/reflector.go:357 +0x348
goroutine 106 [chan receive]:
k8s.io/client-go/tools/cache.(*processorListener).run.func1()
/go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/shared_informer.go:969 +0x4b
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x30?)
/go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:226 +0x33
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc003c53738?, {0x2e66c80, 0xc0030560c0}, 0x1, 0xc00305a060)
/go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:227 +0xaf
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0x0?, 0x3b9aca00, 0x0, 0x0?, 0xc003c53788?)
/go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:204 +0x7f
k8s.io/apimachinery/pkg/util/wait.Until(...)
/go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:161
k8s.io/client-go/tools/cache.(*processorListener).run(0xc0025079e0)
/go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/shared_informer.go:968 +0x69
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1()
/go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:72 +0x4f
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start in goroutine 47
/go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:70 +0x73
goroutine 107 [select]:
k8s.io/client-go/tools/cache.(*processorListener).pop(0xc0025079e0)
/go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/shared_informer.go:939 +0x10d
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1()
/go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:72 +0x4f
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start in goroutine 47
/go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:70 +0x73
goroutine 133 [sync.Cond.Wait]:
sync.runtime_notifyListWait(0xc0028ac7c8, 0x18)
/usr/local/go/src/runtime/sema.go:527 +0x159
sync.(*Cond).Wait(0x2e79fe0?)
/usr/local/go/src/sync/cond.go:70 +0x85
golang.org/x/net/http2.(*pipe).Read(0xc0028ac7b0, {0xc00216a001, 0x7dff, 0x7dff})
/go/src/github.com/cilium/tetragon/vendor/golang.org/x/net/http2/pipe.go:76 +0xdf
golang.org/x/net/http2.transportResponseBody.Read({0x2a62177?}, {0xc00216a001?, 0x2a73ede?, 0xa?})
/go/src/github.com/cilium/tetragon/vendor/golang.org/x/net/http2/transport.go:2558 +0x65
encoding/json.(*Decoder).refill(0xc001717e00)
/usr/local/go/src/encoding/json/stream.go:165 +0x188
encoding/json.(*Decoder).readValue(0xc001717e00)
/usr/local/go/src/encoding/json/stream.go:140 +0x9d
encoding/json.(*Decoder).Decode(0xc001717e00, {0x25ff1c0, 0xc0005e9dd0})
/usr/local/go/src/encoding/json/stream.go:63 +0x75
k8s.io/apimachinery/pkg/util/framer.(*jsonFrameReader).Read(0xc002509560, {0xc002192000, 0x8000, 0xa000})
/go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/framer/framer.go:152 +0x19c
k8s.io/apimachinery/pkg/runtime/serializer/streaming.(*decoder).Decode(0xc002d7e550, 0xc002516840?, {0x2e7a968, 0xc0020f1dc0})
/go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/runtime/serializer/streaming/streaming.go:77 +0xa3
k8s.io/client-go/rest/watch.(*Decoder).Decode(0xc002b16840)
/go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/rest/watch/decoder.go:49 +0x4b
k8s.io/apimachinery/pkg/watch.(*StreamWatcher).receive(0xc00250c300)
/go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/watch/streamwatcher.go:105 +0xdb
created by k8s.io/apimachinery/pkg/watch.NewStreamWatcher in goroutine 129
/go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/watch/streamwatcher.go:76 +0x125
goroutine 134 [sync.Cond.Wait, 1 minutes]:
sync.runtime_notifyListWait(0xc00370e4f8, 0x1)
/usr/local/go/src/runtime/sema.go:527 +0x159
sync.(*Cond).Wait(0xc002a9ed40?)
/usr/local/go/src/sync/cond.go:70 +0x85
k8s.io/client-go/tools/cache.(*DeltaFIFO).Pop(0xc00370e4d0, 0xc002504d90)
/go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/delta_fifo.go:575 +0x236
k8s.io/client-go/tools/cache.(*controller).processLoop(0xc003174500)
/go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/controller.go:188 +0x30
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x30?)
/go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:226 +0x33
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0x4466c0?, {0x2e66c80, 0xc0025096b0}, 0x1, 0xc0000d6420)
/go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:227 +0xaf
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc0025024e0?, 0x3b9aca00, 0x0, 0x0?, 0xdf8475800?)
/go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:204 +0x7f
k8s.io/apimachinery/pkg/util/wait.Until(...)
/go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:161
k8s.io/client-go/tools/cache.(*controller).Run(0xc003174500, 0xc0000d6420)
/go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/controller.go:159 +0x393
k8s.io/client-go/tools/cache.(*sharedIndexInformer).Run(0xc00370e160, 0xc002524f20?)
/go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/shared_informer.go:504 +0x2c8
k8s.io/client-go/informers.(*sharedInformerFactory).Start.func1()
/go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/informers/factory.go:150 +0x5c
created by k8s.io/client-go/informers.(*sharedInformerFactory).Start in goroutine 1
/go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/informers/factory.go:148 +0x205
goroutine 142 [sync.Cond.Wait, 1 minutes]:
sync.runtime_notifyListWait(0xc002c016c8, 0x0)
/usr/local/go/src/runtime/sema.go:527 +0x159
sync.(*Cond).Wait(0x58?)
/usr/local/go/src/sync/cond.go:70 +0x85
golang.org/x/net/http2.(*pipe).Read(0xc002c016b0, {0xc003cea600, 0x200, 0x200})
/go/src/github.com/cilium/tetragon/vendor/golang.org/x/net/http2/pipe.go:76 +0xdf
golang.org/x/net/http2.transportResponseBody.Read({0xc002b94d48?}, {0xc003cea600?, 0x2?, 0x100000009?})
/go/src/github.com/cilium/tetragon/vendor/golang.org/x/net/http2/transport.go:2558 +0x65
encoding/json.(*Decoder).refill(0xc002548140)
/usr/local/go/src/encoding/json/stream.go:165 +0x188
encoding/json.(*Decoder).readValue(0xc002548140)
/usr/local/go/src/encoding/json/stream.go:140 +0x9d
encoding/json.(*Decoder).Decode(0xc002548140, {0x25ff1c0, 0xc002c84cc0})
/usr/local/go/src/encoding/json/stream.go:63 +0x75
k8s.io/apimachinery/pkg/util/framer.(*jsonFrameReader).Read(0xc002c827e0, {0xc00253c800, 0x400, 0x400})
/go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/framer/framer.go:152 +0x19c
k8s.io/apimachinery/pkg/runtime/serializer/streaming.(*decoder).Decode(0xc002d7e780, 0x8?, {0x2e7a968, 0xc00250c540})
/go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/runtime/serializer/streaming/streaming.go:77 +0xa3
k8s.io/client-go/rest/watch.(*Decoder).Decode(0xc002b16d80)
/go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/rest/watch/decoder.go:49 +0x4b
k8s.io/apimachinery/pkg/watch.(*StreamWatcher).receive(0xc00250c500)
/go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/watch/streamwatcher.go:105 +0xdb
created by k8s.io/apimachinery/pkg/watch.NewStreamWatcher in goroutine 139
/go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/watch/streamwatcher.go:76 +0x125
goroutine 94 [chan receive, 1 minutes]:
k8s.io/client-go/tools/cache.(*processorListener).run.func1()
/go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/shared_informer.go:969 +0x4b
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x30?)
/go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:226 +0x33
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc000944f38?, {0x2e66c80, 0xc002bf9140}, 0x1, 0xc002ab0de0)
/go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:227 +0xaf
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc002bf90b0?, 0x3b9aca00, 0x0, 0x20?, 0xc000944f88?)
/go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:204 +0x7f
k8s.io/apimachinery/pkg/util/wait.Until(...)
/go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:161
k8s.io/client-go/tools/cache.(*processorListener).run(0xc002586b40)
/go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/shared_informer.go:968 +0x69
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1()
/go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:72 +0x4f
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start in goroutine 122
/go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:70 +0x73
goroutine 89 [chan receive, 1 minutes]:
k8s.io/client-go/tools/cache.(*sharedProcessor).run(0xc002d7e1e0, 0xc0006847a8?)
/go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/shared_informer.go:803 +0x47
k8s.io/client-go/tools/cache.(*sharedIndexInformer).Run.(*Group).StartWithChannel.func4()
/go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:55 +0x1e
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1()
/go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:72 +0x4f
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start in goroutine 86
/go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:70 +0x73
goroutine 90 [chan receive, 1 minutes]:
k8s.io/client-go/tools/cache.(*controller).Run.func1()
/go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/controller.go:132 +0x25
created by k8s.io/client-go/tools/cache.(*controller).Run in goroutine 86
/go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/controller.go:131 +0xab
goroutine 91 [select, 1 minutes]:
k8s.io/client-go/tools/cache.watchHandler({0x0?, 0x0?, 0x4704d60?}, {0x2e7df60, 0xc002c282a0}, {0x7c842f8c6a20, 0xc002c2c000}, {0x2ec3af0?, 0x29ef460}, 0x0, ...)
/go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/reflector.go:714 +0x17b
k8s.io/client-go/tools/cache.(*Reflector).watch(0xc003a1cb60, {0x0?, 0x0?}, 0xc0000d6420, 0xc000800000?)
/go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/reflector.go:433 +0x545
k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch(0xc003a1cb60, 0xc0000d6420)
/go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/reflector.go:358 +0x36c
k8s.io/client-go/tools/cache.(*Reflector).Run.func1()
/go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/reflector.go:291 +0x25
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x10?)
/go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:226 +0x33
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc002ab0c60?, {0x2e66ca0, 0xc002d0c320}, 0x1, 0xc0000d6420)
/go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:227 +0xaf
k8s.io/client-go/tools/cache.(*Reflector).Run(0xc003a1cb60, 0xc0000d6420)
/go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/reflector.go:290 +0x1cd
k8s.io/client-go/tools/cache.(*controller).Run.(*Group).StartWithChannel.func2()
/go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:55 +0x1e
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1()
/go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:72 +0x4f
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start in goroutine 86
/go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:70 +0x73
goroutine 93 [select]:
k8s.io/client-go/tools/cache.(*Reflector).startResync(0xc003a1cb60, 0xc0000d6420, 0xc002ab0d20, 0xc002540f10?)
/go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/reflector.go:370 +0x10f
created by k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch in goroutine 91
/go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/reflector.go:357 +0x348
goroutine 144 [select]:
github.com/cilium/tetragon/pkg/eventcache.(*Cache).loop(0xc00250c5c0)
/go/src/github.com/cilium/tetragon/pkg/eventcache/eventcache.go:169 +0x117
created by github.com/cilium/tetragon/pkg/eventcache.NewWithTimer in goroutine 1
/go/src/github.com/cilium/tetragon/pkg/eventcache/eventcache.go:238 +0x178
goroutine 119 [sync.Cond.Wait, 1 minutes]:
sync.runtime_notifyListWait(0xc0028d6028, 0x0)
/usr/local/go/src/runtime/sema.go:527 +0x159
sync.(*Cond).Wait(0xc000948be0?)
/usr/local/go/src/sync/cond.go:70 +0x85
k8s.io/client-go/tools/cache.(*DeltaFIFO).Pop(0xc0028d6000, 0xc002879bd0)
/go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/delta_fifo.go:575 +0x236
k8s.io/client-go/tools/cache.(*controller).processLoop(0xc00286d9a0)
/go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/controller.go:188 +0x30
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x30?)
/go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:226 +0x33
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0x4466c0?, {0x2e66c80, 0xc0028d41e0}, 0x1, 0xc0000d6420)
/go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:227 +0xaf
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc002895b10?, 0x3b9aca00, 0x0, 0x0?, 0x0?)
/go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:204 +0x7f
k8s.io/apimachinery/pkg/util/wait.Until(...)
/go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:161
k8s.io/client-go/tools/cache.(*controller).Run(0xc00286d9a0, 0xc0000d6420)
/go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/controller.go:159 +0x393
k8s.io/client-go/tools/cache.(*sharedIndexInformer).Run(0xc003190160, 0x22a17f4?)
/go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/shared_informer.go:504 +0x2c8
github.com/cilium/tetragon/pkg/k8s/client/informers/externalversions.(*sharedInformerFactory).Start.func1()
/go/src/github.com/cilium/tetragon/vendor/github.com/cilium/tetragon/pkg/k8s/client/informers/externalversions/factory.go:119 +0x5c
created by github.com/cilium/tetragon/pkg/k8s/client/informers/externalversions.(*sharedInformerFactory).Start in goroutine 118
/go/src/github.com/cilium/tetragon/vendor/github.com/cilium/tetragon/pkg/k8s/client/informers/externalversions/factory.go:117 +0x205
goroutine 122 [chan receive, 1 minutes]:
k8s.io/client-go/tools/cache.(*sharedProcessor).run(0xc003148280, 0x2ea2f70?)
/go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/shared_informer.go:803 +0x47
k8s.io/client-go/tools/cache.(*sharedIndexInformer).Run.(*Group).StartWithChannel.func4()
/go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:55 +0x1e
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1()
/go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:72 +0x4f
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start in goroutine 120
/go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:70 +0x73
goroutine 123 [chan receive, 1 minutes]:
k8s.io/client-go/tools/cache.(*controller).Run.func1()
/go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/controller.go:132 +0x25
created by k8s.io/client-go/tools/cache.(*controller).Run in goroutine 120
/go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/controller.go:131 +0xab
goroutine 124 [select, 1 minutes]:
k8s.io/client-go/tools/cache.watchHandler({0x0?, 0x0?, 0x4704d60?}, {0x2e7ae40, 0xc002899280}, {0x7c842f8c6a20, 0xc0031902c0}, {0x2ec3af0?, 0x2a01200}, 0x0, ...)
/go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/reflector.go:714 +0x17b
k8s.io/client-go/tools/cache.(*Reflector).watch(0xc002a300e0, {0x0?, 0x0?}, 0xc0000d6420, 0xc003c4ff80?)
/go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/reflector.go:433 +0x545
k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch(0xc002a300e0, 0xc0000d6420)
/go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/reflector.go:358 +0x36c
k8s.io/client-go/tools/cache.(*Reflector).Run.func1()
/go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/reflector.go:291 +0x25
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x10?)
/go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:226 +0x33
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc00298cf00?, {0x2e66ca0, 0xc0031482d0}, 0x1, 0xc0000d6420)
/go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:227 +0xaf
k8s.io/client-go/tools/cache.(*Reflector).Run(0xc002a300e0, 0xc0000d6420)
/go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/reflector.go:290 +0x1cd
k8s.io/client-go/tools/cache.(*controller).Run.(*Group).StartWithChannel.func2()
/go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:55 +0x1e
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1()
/go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:72 +0x4f
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start in goroutine 120
/go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:70 +0x73
goroutine 180 [select, 1 minutes]:
k8s.io/client-go/tools/cache.(*Reflector).startResync(0xc002a300e0, 0xc0000d6420, 0xc00305ae40, 0x25cbf40?)
/go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/reflector.go:370 +0x10f
created by k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch in goroutine 124
/go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/reflector.go:357 +0x348
goroutine 111 [chan receive, 1 minutes]:
k8s.io/client-go/tools/cache.(*sharedProcessor).run(0xc003148230, 0x0?)
/go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/shared_informer.go:803 +0x47
k8s.io/client-go/tools/cache.(*sharedIndexInformer).Run.(*Group).StartWithChannel.func4()
/go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:55 +0x1e
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1()
/go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:72 +0x4f
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start in goroutine 119
/go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:70 +0x73
goroutine 112 [chan receive, 1 minutes]:
k8s.io/client-go/tools/cache.(*controller).Run.func1()
/go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/controller.go:132 +0x25
created by k8s.io/client-go/tools/cache.(*controller).Run in goroutine 119
/go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/controller.go:131 +0xab
goroutine 95 [select, 1 minutes]:
k8s.io/client-go/tools/cache.(*processorListener).pop(0xc002586b40)
/go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/shared_informer.go:939 +0x10d
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1()
/go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:72 +0x4f
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start in goroutine 122
/go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:70 +0x73
goroutine 161 [select, 1 minutes]:
k8s.io/client-go/tools/cache.(*processorListener).pop(0xc002586ab0)
/go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/shared_informer.go:939 +0x10d
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1()
/go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:72 +0x4f
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start in goroutine 111
/go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:70 +0x73
goroutine 177 [select, 1 minutes]:
k8s.io/client-go/tools/cache.watchHandler({0x0?, 0x0?, 0x4704d60?}, {0x2e7ae40, 0xc002a1cec0}, {0x7c842f8c6a20, 0xc0028d6000}, {0x2ec3af0?, 0x29fc4a0}, 0x0, ...)
/go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/reflector.go:714 +0x17b
k8s.io/client-go/tools/cache.(*Reflector).watch(0xc0028d8000, {0x0?, 0x0?}, 0xc0000d6420, 0x0?)
/go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/reflector.go:433 +0x545
k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch(0xc0028d8000, 0xc0000d6420)
/go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/reflector.go:358 +0x36c
k8s.io/client-go/tools/cache.(*Reflector).Run.func1()
/go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/reflector.go:291 +0x25
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x10?)
/go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:226 +0x33
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc00305ac00?, {0x2e66ca0, 0xc0029fe0a0}, 0x1, 0xc0000d6420)
/go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:227 +0xaf
k8s.io/client-go/tools/cache.(*Reflector).Run(0xc0028d8000, 0xc0000d6420)
/go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/reflector.go:290 +0x1cd
k8s.io/client-go/tools/cache.(*controller).Run.(*Group).StartWithChannel.func2()
/go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:55 +0x1e
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1()
/go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:72 +0x4f
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start in goroutine 119
/go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:70 +0x73
goroutine 182 [select, 1 minutes]:
k8s.io/client-go/tools/cache.(*Reflector).startResync(0xc0028d8000, 0xc0000d6420, 0xc00305b140, 0xc002525fb8?)
/go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/reflector.go:370 +0x10f
created by k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch in goroutine 177
/go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/reflector.go:357 +0x348
goroutine 181 [select, 1 minutes]:
golang.org/x/net/http2.(*clientStream).writeRequest(0xc0028aca80, 0xc0027f4d00)
/go/src/github.com/cilium/tetragon/vendor/golang.org/x/net/http2/transport.go:1464 +0xac7
golang.org/x/net/http2.(*clientStream).doRequest(0x0?, 0xc003c4ff10?)
/go/src/github.com/cilium/tetragon/vendor/golang.org/x/net/http2/transport.go:1326 +0x18
created by golang.org/x/net/http2.(*ClientConn).RoundTrip in goroutine 124
/go/src/github.com/cilium/tetragon/vendor/golang.org/x/net/http2/transport.go:1232 +0x308
goroutine 127 [sync.Cond.Wait, 1 minutes]:
sync.runtime_notifyListWait(0xc0028acc48, 0x0)
/usr/local/go/src/runtime/sema.go:527 +0x159
sync.(*Cond).Wait(0x58?)
/usr/local/go/src/sync/cond.go:70 +0x85
golang.org/x/net/http2.(*pipe).Read(0xc0028acc30, {0xc00319cc00, 0x200, 0x200})
/go/src/github.com/cilium/tetragon/vendor/golang.org/x/net/http2/pipe.go:76 +0xdf
golang.org/x/net/http2.transportResponseBody.Read({0xc000947d48?}, {0xc00319cc00?, 0x2?, 0x10000000f?})
/go/src/github.com/cilium/tetragon/vendor/golang.org/x/net/http2/transport.go:2558 +0x65
encoding/json.(*Decoder).refill(0xc002a223c0)
/usr/local/go/src/encoding/json/stream.go:165 +0x188
encoding/json.(*Decoder).readValue(0xc002a223c0)
/usr/local/go/src/encoding/json/stream.go:140 +0x9d
encoding/json.(*Decoder).Decode(0xc002a223c0, {0x25ff1c0, 0xc0029beaf8})
/usr/local/go/src/encoding/json/stream.go:63 +0x75
k8s.io/apimachinery/pkg/util/framer.(*jsonFrameReader).Read(0xc00297bdd0, {0xc002a2dc00, 0x400, 0x400})
/go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/framer/framer.go:152 +0x19c
k8s.io/apimachinery/pkg/runtime/serializer/streaming.(*decoder).Decode(0xc0031484b0, 0x8?, {0x2e7a968, 0xc002a1cf00})
/go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/runtime/serializer/streaming/streaming.go:77 +0xa3
k8s.io/client-go/rest/watch.(*Decoder).Decode(0xc002597b20)
/go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/rest/watch/decoder.go:49 +0x4b
k8s.io/apimachinery/pkg/watch.(*StreamWatcher).receive(0xc002a1cec0)
/go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/watch/streamwatcher.go:105 +0xdb
created by k8s.io/apimachinery/pkg/watch.NewStreamWatcher in goroutine 177
/go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/watch/streamwatcher.go:76 +0x125
goroutine 183 [select, 1 minutes]:
golang.org/x/net/http2.(*clientStream).writeRequest(0xc0028acc00, 0xc0027f5000)
/go/src/github.com/cilium/tetragon/vendor/golang.org/x/net/http2/transport.go:1464 +0xac7
golang.org/x/net/http2.(*clientStream).doRequest(0x0?, 0xc003c4df10?)
/go/src/github.com/cilium/tetragon/vendor/golang.org/x/net/http2/transport.go:1326 +0x18
created by golang.org/x/net/http2.(*ClientConn).RoundTrip in goroutine 177
/go/src/github.com/cilium/tetragon/vendor/golang.org/x/net/http2/transport.go:1232 +0x308
goroutine 184 [sync.Cond.Wait, 1 minutes]:
sync.runtime_notifyListWait(0xc0028acac8, 0x0)
/usr/local/go/src/runtime/sema.go:527 +0x159
sync.(*Cond).Wait(0xc0028acab8?)
/usr/local/go/src/sync/cond.go:70 +0x85
golang.org/x/net/http2.(*pipe).Read(0xc0028acab0, {0xc0039aa800, 0x200, 0x200})
/go/src/github.com/cilium/tetragon/vendor/golang.org/x/net/http2/pipe.go:76 +0xdf
golang.org/x/net/http2.transportResponseBody.Read({0xc003c4dd48?}, {0xc0039aa800?, 0xc003c4dd10?, 0xc003c4dcac?})
/go/src/github.com/cilium/tetragon/vendor/golang.org/x/net/http2/transport.go:2558 +0x65
encoding/json.(*Decoder).refill(0xc002889900)
/usr/local/go/src/encoding/json/stream.go:165 +0x188
encoding/json.(*Decoder).readValue(0xc002889900)
/usr/local/go/src/encoding/json/stream.go:140 +0x9d
encoding/json.(*Decoder).Decode(0xc002889900, {0x25ff1c0, 0xc0028e2210})
/usr/local/go/src/encoding/json/stream.go:63 +0x75
k8s.io/apimachinery/pkg/util/framer.(*jsonFrameReader).Read(0xc0028d5ef0, {0xc0028cf400, 0x400, 0x400})
/go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/framer/framer.go:152 +0x19c
k8s.io/apimachinery/pkg/runtime/serializer/streaming.(*decoder).Decode(0xc0029fe5f0, 0xc003192050?, {0x2e7a968, 0xc0028992c0})
/go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/runtime/serializer/streaming/streaming.go:77 +0xa3
k8s.io/client-go/rest/watch.(*Decoder).Decode(0xc0028f65c0)
/go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/rest/watch/decoder.go:49 +0x4b
k8s.io/apimachinery/pkg/watch.(*StreamWatcher).receive(0xc002899280)
/go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/watch/streamwatcher.go:105 +0xdb
created by k8s.io/apimachinery/pkg/watch.NewStreamWatcher in goroutine 124
/go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/watch/streamwatcher.go:76 +0x125
Here is gops 1
output in our case, it's clear that gRPC server is not run:
# gops 1
threads: 16
memory usage: 0.540%
cpu usage: 1.152%
username: root
cmd+args: /usr/bin/tetragon --config-dir=/etc/tetragon/tetragon.conf.d/
elapsed time: 01:21
local/remote: 127.0.0.1:8118 <-> 0.0.0.0:0 (LISTEN)
local/remote: 0.0.0.0:2112 <-> 0.0.0.0:0 (LISTEN)
local/remote: 10.80.122.156:42272 <-> 10.43.0.1:443 (ESTABLISHED)
After upgrade to latest CI image with help of these command:
helm install --namespace kube-system \
--set tetragonOperator.image.override=quay.io/cilium/tetragon-operator-ci:latest \
--set tetragon.image.override=quay.io/cilium/tetragon-ci:latest \
tetragon cilium/tetragon
Tetragon start failing to start with this message:
time="2024-03-26T12:26:53Z" level=info msg="tetragon, map loaded." map=tg_execve_joined_info_map_stats path=/sys/fs/bpf/tetragon/tg_execve_joined_info_map_stats sensor=__base__
time="2024-03-26T12:26:53Z" level=info msg="tetragon, map loaded." map=execve_calls path=/sys/fs/bpf/tetragon/execve_calls sensor=__base__
time="2024-03-26T12:26:53Z" level=info msg="tetragon, map loaded." map=tcpmon_map path=/sys/fs/bpf/tetragon/tcpmon_map sensor=__base__
time="2024-03-26T12:26:53Z" level=info msg="tetragon, map loaded." map=tg_conf_map path=/sys/fs/bpf/tetragon/tg_conf_map sensor=__base__
time="2024-03-26T12:26:53Z" level=info msg="tetragon, map loaded." map=tg_stats_map path=/sys/fs/bpf/tetragon/tg_stats_map sensor=__base__
time="2024-03-26T12:26:53Z" level=info msg="Loading BPF program" Attach=acct_process Program=/var/lib/tetragon/bpf_exit.o Type=kprobe
time="2024-03-26T12:26:53Z" level=info msg="BPF events statistics: 0 received, 0% events loss"
time="2024-03-26T12:26:53Z" level=info msg="Observer events statistics" errors=0 filterDrop=0 filterPass=0 lost=0 received=0
time="2024-03-26T12:26:53Z" level=info msg="sensor controller starts"
time="2024-03-26T12:26:53Z" level=fatal msg="Failed to start tetragon" error="failed prog /var/lib/tetragon/bpf_exit.o kern_version 328721 loadInstance: attaching 'event_exit_acct_process' failed: creating perf_kprobe PMU (arch-specific fallback for \"acct_process\"): token acct_process: opening perf event: operation not permitted"
OS is Oracle Linux 8.5 with CONFIG_BSD_PROCESS_ACCT enabled
# grep CONFIG_BSD_PROCESS_ACCT /boot/config-$(uname -r)
CONFIG_BSD_PROCESS_ACCT=y
CONFIG_BSD_PROCESS_ACCT_V3=y
Also:
# grep acct_process < /proc/kallsyms
ffffffffb7d5fc20 t do_acct_process
ffffffffb7d60810 T acct_process
Tetragon Version
v1.0.0, v1.0.2, latest CI
Kernel Version
5.4.17-2136.304.4.1.el8uek.x86_64
Kubernetes Version
No response
Bugtool
No response
Relevant log output
No response
Anything else?
No response
Thanks @inliquid,
Is Oracle Linux using lockdown by any chance? What's the output of:
sudo cat /sys/kernel/security/lockdown
Yes it seems so:
# cat /sys/kernel/security/lockdown
none integrity [confidentiality]
This mode prohibits loading BPF programs, which means that Tetragon cannot really work. We suggest to switch to "integrity" instead. See iovisor/bcc#2565 for more discussion on this.
There is a PR from @tixxdz to issue an early warning about this: #2149.