Giters

ros-realtime / linux-real-time-kernel-builder

build and setup RT kernel for the ROS 2 testing

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

unable to install lttng on rt linux kernel

flynneva opened this issue · comments

commented

so I recently ran into this problem while trying to setup ros2_tracing to track system performance. essentially I was unable to sudo apt install lttng-modules-dkms as it said the RT kernel was not officially supported.

I did some digging online and came across this thread that looks to be exactly the same issue.

It goes on to suggest a solution of compiling lttng from source....which ideally i would like to avoid.

As lttng is a pretty universal tracer tool I was wondering if an alternative proposal could happen where we add it to the Dockerfile here so that we can compile it right into the kernel for use later on.

I found this README for lttng-modules that looks like it has instructions to add it directly to a kernel build.

lttng-modules-README.md

### Kernel built-in support

It is also possible to build these modules as part of a kernel image. Simply
run the [`scripts/built-in.sh`](scripts/built-in.sh) script with the path to
your kernel source directory as an argument.  It will symlink the
lttng-modules directory in the kernel sources and add an include in the kernel
Makefile.

Then configure your kernel as usual and enable the `CONFIG_LTTNG` option.


### Required kernel config options

Make sure your target kernel has the following config options enabled:

  - `CONFIG_MODULES`: loadable module support (not strictly required
     when built into the kernel),
  - `CONFIG_KALLSYMS`: see files in [`wrapper`](wrapper); this is
     necessary until the few required missing symbols are exported to GPL
     modules from mainline,
  - `CONFIG_HIGH_RES_TIMERS`: needed for LTTng 2.x clock source,
  - `CONFIG_TRACEPOINTS`: kernel tracepoint instrumentation
     (enabled as a side-effect of any of the perf/ftrace/blktrace
     instrumentation features).
  - `CONFIG_KPROBES` (5.7+): use kallsyms for kernel 5.7 and newer.
commented

would adding lttng to the kernel build process be an acceptable enhancement here? or is it not acceptable to add it to the default kernel build?

commented

@flynneva yes, we should do it. It was not clear to me whether RT and lttng work together, but apparently they do.
Do you want to give it a try and provide a PR?

commented

I was planning on trying do this soon, but if you can figure out how to do it it'd be great!

commented

so I think I got this working but I wont know for sure until the build fully completes.

you can see what I did in #18 in the Dockerfile to get it added to the kernel. the trickiest bit was figuring out that we also needed the scripts/rt-patch-version.sh which I only guessed at since it wasnt anywhere in the docs that I could find.

what makes me think I have it right is during the build, right at the beginning it prompted me a few questions specifically about LTTng and its configuration...so fingers crossed.

commented

would love to have someone more familiar with lttng and ros2_tracing check my work over on #18, however I think I've got it working following the normal kernel build instructions from the README.

the only change to the process that i did was i skipped sudo apt install lttng-dkms-modules since we built it into the kernel. everything seems to be working OK...however I dont fully understand if what im seeing in the tracing metadata is "right" or not.

maybe @christophebedard @razr or @LanderU knows more and can understand if things look OK?

commented

I think it looks good! The metadata file always looks a bit weird (it's not 100% text I think?). You can check the CTF spec if you want (https://diamon.org/ctf/), but it's probably not worth it 😝

You could simply take a look at the userspace and kernel traces separately to make sure stuff was recorded:

$ babeltrace /home/ubuntu/.ros/tracing/my-tracing-session/kernel
$ babeltrace /home/ubuntu/.ros/tracing/my-tracing-session/ust

With the default setup, the kernel trace will probably have a lot more events, which you can count by piping the babeltrace output to wc -l.

commented

awesome to hear @christophebedard, thanks! yes I removed a bunch of gibberish at the beginning (non-text stuff) when copying to github so thats probably what you were talking about.

commented

looking good:

babeltrace /home/ubuntu/.ros/tracing/my-tracing-session/kernel | wc -l
63546

babeltrace /home/ubuntu/.ros/tracing/my-tracing-session/ust | wc -l
130

thanks for the tips!