Local built image does not work as expected on MacOS arm64

[f1ko]

What happened?

Built a clean local image with make images on commit 505ce7f861efd13dbe49e48a730857b7b6736679.
When running that image on a kind cluster (orbstack, arm64) the tetragon container shuts down with an error.

Tetragon Version

Local build:

# tetra version
CLI version: 505ce7f8

Kernel Version

From the node Tetragon is running on:

# uname -a
Linux kind-control-plane 6.4.16-orbstack-00105-g14094bfeec09 #1 SMP Mon Sep 18 21:45:38 UTC 2023 aarch64 GNU/Linux

Kubernetes Version

$ kubectl version
Client Version: v1.29.1
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.27.3

Bugtool

No response

Relevant log output

$ kubectl logs -f tetragon-nhrxh -c tetragon
time="2024-03-25T13:37:51Z" level=info msg="Loaded config from directory" config-dir=/etc/tetragon/tetragon.conf.d/
time="2024-03-25T13:37:51Z" level=info msg="Starting gops server" addr="localhost:8118"
time="2024-03-25T13:37:51Z" level=info msg="Starting tetragon" version=505ce7f8
time="2024-03-25T13:37:51Z" level=info msg="config settings" config="map[bpf-lib:/var/lib/tetragon/ btf: config-dir:/etc/tetragon/tetragon.conf.d/ cpuprofile: data-cache-size:1024 debug:true disable-kprobe-multi:false enable-export-aggregation:false enable-k8s-api:true enable-msg-handling-latency:false enable-pid-set-filter:false enable-pod-info:false enable-policy-filter:true enable-policy-filter-debug:false enable-process-ancestors:true enable-process-cred:false enable-process-ns:false enable-tracing-policy-crd:true event-queue-size:10000 export-aggregation-buffer-size:10000 export-aggregation-window-size:15s export-allowlist:{\"event_set\":[\"PROCESS_EXEC\", \"PROCESS_EXIT\", \"PROCESS_KPROBE\", \"PROCESS_UPROBE\", \"PROCESS_TRACEPOINT\"]} export-denylist:{\"health_check\":true}\n{\"namespace\":[\"\", \"cilium\", \"kube-system\"]} export-file-compress:false export-file-max-backups:5 export-file-max-size-mb:10 export-file-perm:600 export-file-rotation-interval:0s export-filename:/var/run/cilium/tetragon/tetragon.log export-rate-limit:-1 expose-kernel-addresses:false field-filters:{} force-large-progs:false force-small-progs:false generate-docs:false gops-address:localhost:8118 k8s-kubeconfig-path: kernel: kmods:[] log-format:text log-level:info memprofile: metrics-label-filter:namespace,workload,pod,binary metrics-server::2112 netns-dir:/var/run/docker/netns/ pprof-addr: process-cache-size:65536 procfs:/procRoot rb-queue-size:65535 rb-size:0 rb-size-total:0 release-pinned-bpf:true server-address:localhost:54321 tracing-policy: tracing-policy-dir:/etc/tetragon/tetragon.tp.d verbose:0]"
time="2024-03-25T13:37:51Z" level=info msg="Tetragon pid file creation succeeded" pid=6434 pidfile=/var/run/tetragon/tetragon.pid
time="2024-03-25T13:37:51Z" level=debug msg="probing detectModifyReturnSyscall using __x64_sys_getcpu"
time="2024-03-25T13:37:51Z" level=error msg="detect modify return syscall" error="failed to load: attach Tracing/ModifyReturn: find target for fmod_ret __x64_sys_getcpu: find target in modules: iterate modules: get next BTF ID: function not implemented"
time="2024-03-25T13:37:51Z" level=info msg="BPF detected features: override_return: false, buildid: false, kprobe_multi: false, fmodret: false, fmodret_syscall: false, signal: false, large: false"
time="2024-03-25T13:37:51Z" level=warning msg="inode mismatch: procfs does not appear to be host procfs" expected inode=4026531836 inode=4026532516 path=/procRoot/1/ns/pid
time="2024-03-25T13:37:51Z" level=warning msg="BPF filesystem is going to be mounted automatically in /run/cilium/bpffs. However, it probably means that Cilium is running inside container and BPFFS is not mounted on the host. for more information, see: https://cilium.link/err-bpf-mount"
time="2024-03-25T13:37:51Z" level=debug msg="Detected mounted BPF filesystem at /run/cilium/bpffs"
time="2024-03-25T13:37:51Z" level=info msg="BPF: successfully released pinned BPF programs and maps" bpf-dir=/run/cilium/bpffs/tetragon
time="2024-03-25T13:37:51Z" level=info msg="Enabling policy filtering"
time="2024-03-25T13:37:51Z" level=info msg="BPF events statistics: 0 received, 0% events loss"
time="2024-03-25T13:37:51Z" level=info msg="Observer events statistics" errors=0 filterDrop=0 filterPass=0 lost=0 received=0
time="2024-03-25T13:37:51Z" level=fatal msg="Failed to start tetragon" error="failed to initialize policy filter state: creating map: inner map: load BTF: detect support for Map BTF (Var/Datasec): detect support for BTF: function not implemented"

### Anything else?

Environment:

Running on MacBook M2 Pro (arm64).
Using kind on top of orbstack.

[Mahé]

it seems your problem is that your kernel misses a function needed by Tetragon, I would suggest using a more usual distro that supports all our kernel feature needs like Debian or Ubuntu for example. I don't really know how orbstack is configuring their kernel but it seems BTF is missing "detect support for BTF: function not implemented"