kprobe:vfs_fstat and kretprobe:vfs_fstat unavailable on Ubuntu 20.04

[kimono-koans]

With bpftrace installed:

`$ sudo bpftrace -l | grep fstat

tracepoint:syscalls:sys_enter_fstatfs
tracepoint:syscalls:sys_exit_fstatfs
tracepoint:syscalls:sys_enter_newfstatat
tracepoint:syscalls:sys_exit_newfstatat
tracepoint:syscalls:sys_enter_newfstat
tracepoint:syscalls:sys_exit_newfstat
kprobe:__ia32_compat_sys_x86_fstatat
kprobe:__x32_compat_sys_x86_fstatat
kprobe:__ia32_compat_sys_x86_fstat64
kprobe:__x32_compat_sys_x86_fstat64
kprobe:__x64_sys_fstat
kprobe:__ia32_sys_fstat
kprobe:__x64_sys_newfstatat
kprobe:__ia32_sys_newfstatat
kprobe:__x64_sys_newfstat
kprobe:__ia32_sys_newfstat
kprobe:__ia32_compat_sys_newfstatat
kprobe:__x32_compat_sys_newfstatat
kprobe:__ia32_compat_sys_newfstat
kprobe:__x32_compat_sys_newfstat
kprobe:__x64_sys_fstatfs
kprobe:__ia32_sys_fstatfs
kprobe:__x64_sys_fstatfs64
kprobe:__ia32_sys_fstatfs64
kprobe:__ia32_compat_sys_fstatfs
kprobe:__x32_compat_sys_fstatfs
kprobe:kcompat_sys_fstatfs64
kprobe:__ia32_compat_sys_fstatfs64
kprobe:__x32_compat_sys_fstatfs64
kprobe:ifstat`

Was able to get the following to work:

`tracepoint:syscalls:sys_enter_fstatfs
/@trace[pid, args->fd] != ""/
{
$fd = args->fd;

            // Pass args to kretprobe
            @fstat_exit[tid] = 1;
            @fstat_fd[tid] = \$fd;
    }

    tracepoint:syscalls:sys_exit_fstatfs
    /@fstat_exit[pid]/
    {
            \$fd = @fstat_fd[tid];
            \$fstype = @trace[pid, \$fd];

            // Args from kprobe
            delete(@fstat_exit[tid]);
            delete(@fstat_fd[tid]);

            ${TIME_FMT:+time($RAW_TIME_FMT);
            }printf("|fstat|%d.%d|%s|%d|%s|",
                    uid, gid, comm, pid,
                    \$fstype);
            printf("%d\n", \$fd);
    }`

Thanks for the hard work!

[freebsdfrau]

Thank you for work necessary to make it work on Ubuntu 20.0.

Do you know the best way to detect Ubuntu 20+ so that I can write the necessary shell code to generate different bpftrace code on that OS? Preferably some way that doesn't involve forking a command -- for example, on RedHat and CentOS I can read /etc/redhat-release ... do you suggest reading /etc/os-release on Ubuntu or is there a better (more easily machine-parseable) source?

[kimono-koans]

You could check /etc/os-release on Ubuntu but it might be better to check for the kprobes like below:

NAME="Ubuntu" VERSION="20.04.1 LTS (Focal Fossa)" ID=ubuntu ID_LIKE=debian PRETTY_NAME="Ubuntu 20.04.1 LTS" VERSION_ID="20.04" HOME_URL="https://www.ubuntu.com/" SUPPORT_URL="https://help.ubuntu.com/" BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" VERSION_CODENAME=focal UBUNTU_CODENAME=focal

`#!/bin/sh
if [ $( bpftrace -l | grep -c -e "kprobe:vfs_fstat" -e "kprobe:vfs_fstat and kretprobe:vfs_fstat" ) -eq 0 ]; then
tracepoint:syscalls:sys_enter_fstatfs
/@trace[pid, args->fd] != ""/
{
$fd = args->fd;

            // Pass args to kretprobe
            @fstat_exit[tid] = 1;
            @fstat_fd[tid] = \$fd;
    }

    tracepoint:syscalls:sys_exit_fstatfs
    /@fstat_exit[pid]/
    {
            \$fd = @fstat_fd[tid];
            \$fstype = @trace[pid, \$fd];

            // Args from kprobe
            delete(@fstat_exit[tid]);
            delete(@fstat_fd[tid]);

            ${TIME_FMT:+time($RAW_TIME_FMT);
            }printf("|fstat|%d.%d|%s|%d|%s|",
                    uid, gid, comm, pid,
                    \$fstype);
            printf("%d\n", \$fd);
    }

else
kprobe:vfs_fstat
/@trace[pid, arg0] != ""/
{
$fd = arg0;
// Pass args to kretprobe
@fstat_exit[tid] = 1;
@fstat_fd[tid] = $fd;
}
kretprobe:vfs_fstat
/@fstat_exit[tid]/
{
$fd = @fstat_fd[tid];
$ret = retval;
$fstype = @trace[pid, $fd];
// Args from kprobe
delete(@fstat_exit[tid]);
delete(@fstat_fd[tid]);
${TIME_FMT:+time($RAW_TIME_FMT);
}printf("|fstat|%d.%d|%s|%d|%s|",
uid, gid, comm, pid,
$fstype);
printf("%d|%d\n", $fd, $ret);
}
fi`

[kimono-koans]

if [ $( bpftrace -l | grep -c -e "kprobe:vfs_fstat" -e kretprobe:vfs_fstat" ) -eq 0 ]; then ... fi

[freebsdfrau]

After investigation, it turns out that the probe we want for kernels that are lacking vfs_fstat is instead vfs_statx_fd as seen by this change in the Linux kernel torvalds/linux@da9aa5d

I have seen 3.x kernels with vfs_fstat and 4.x kernels with vfs_statx_fd and it seems to be that the appropriate thing to do is to test for the particular probe and use the proper one -- irrespective of other probe decisions (implying that we should probably do a single "bpftrace -l" and run that through an evaluated awk expression to set probe decisions in the calling namespace, avoiding a multiplying factor)

[freebsdfrau]

@electricboogie should be fixed now. Will wait for verification before closing.

[kimono-koans]

Seems to work!