Perfcollect data from ubuntu not showing up cpu stacks in Perfview

[Manish Joshi]

I ran perfcollect from ubuntu VM for asp.net core 2.1 application by specifying pid of the dotnet process. The perf trace file was generated successfully and I was able to view the data on ubuntu with ./perfcollect view session4.trace.zip command.
However, when I copied the trace file on windows and opened it in Perfview, it could not load the CPU Stacks.
PFB snapshot for the same.

The events data is visible though.
I checked the underlying 'perf.data.txt' file and found it empty, but the 'perf.data' file is not empty.
Can you please help here to see what's going wrong?

[Michael Logutov]

Got the same trying to use perfcollect to profile running container.
My scenario was like this:

1. find container (docker ps | grep ...)
2. find it's pid (docker top)
3. profile it (./perfcollect collect test -pid 12345)

And after Ctrl+C I've got a warning libcoreclr.so not found in perf data. Please verify that your .NET Core process is running and consuming CPU.

And empty trace file.

[Sung Yoon Whang]

Hi @codeSonic,

Could you please provide us with a bit more information? Specifically:

• Which version of lttng are you using? (you can get this via the command lttng --version)
• Would it be possible for you to share that trace with us? If not, a minimal repro app that we can trace on our machines would be helpful for us to take actions on this issue faster.

@MichaelLogutov What you are seeing seems to be a different issue... Just to clarify, are you trying to run the perfcollect script inside your Docker container, or are you running it from the host machine? If you're running it from the host machine that's expected because perfcollect won't be able to see that PID inside its own process map. You'd want to run perfcollect from Docker itself. You can check out an example of how to do that in https://github.com/dotnet/corefx-tools/blob/master/src/performance/perfcollect/docker-demo/Dockerfile. If that doesn't solve your issue, could you file a different issue and we can take it there so that we don't have two separate issues going on in the same thread?

Thanks!

[Paras Patidar]

Hi ,
Similer issue here.
I have collected trace form Kubernetes container.
when viewing in to container itself by unziping trace and using perf report , it shows all call stack.

however , when i copy same trace zip on windows , it shows events and eventstats but CPU stacks are empty (same as shown by @codeSonic .

here are sample traces :
https://drive.google.com/uc?id=1ZM35Cf4WfkckUMigSo0qhwbbdhSEXPBm&export=download

and

https://drive.google.com/file/d/1cg0TnD6VjKcgm89upmZ8Pvn8QW8vIksN/view?usp=sharing

lttng version : lttng (LTTng Trace Control) 2.9.3 - Joannès
container OS :
PRETTY_NAME="Debian GNU/Linux 9 (stretch)"
NAME="Debian GNU/Linux"
VERSION_ID="9"
VERSION="9 (stretch)"

perf version : perf version 5.0.rc7.g8a61716
perview version : 2.0.26

PS : for dotnet core container , perfcollect install is unable to install perf , thus i manually build perf executable using :

git clone https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git

cd linux/tools/perf

make

cp perf /usr/bin

[Sung Yoon Whang]

Thanks @paraspatidar!

I took a quick look and it seems like this issue may be a duplicate of #84. In perfcollect.log, I can see:

Running /usr/bin/perf script -i perf.data.merged -F comm,pid,tid,cpu,time,period,event,ip,sym,dso,trace > perf.data.txt
'trace' not valid for hardware events. Ignoring.
'trace' not valid for software events. Ignoring.
'trace' not valid for hardware-cache events. Ignoring.
'trace' not valid for unknown events. Ignoring.
'trace' not valid for unknown events. Ignoring.
'trace' not valid for unknown events. Ignoring.
Samples for 'cpu-clock' event do not have CPU attribute set. Cannot print 'cpu' field.

Running /usr/bin/perf script -i perf.data.merged -f comm,pid,tid,cpu,time,event,ip,sym,dso,trace > perf.data.txt
  Error: Couldn't find script `comm,pid,tid,cpu,time,event,ip,sym,dso,trace'

 See perf script -l for available scripts.

So in the end perf.data.txt file is not created due to this error and that file is what PerfView reads when it tries to print the stack so you're seeing the empty cpu stacks.

I'll investigate further and get back with a fix soon but just wanted to share this so that your concerns aren't left unanswered :-)