"Failed to inject profiler into <pid>"
jakewins opened this issue · comments
I'm running the following command:
$ ./profiler.sh list 5015
And am getting the following output:
Failed to inject profiler into 5015
linux-vdso.so.1 (0x00007ffd4ddcb000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f62c2080000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f62c1e63000)
libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f62c1ae1000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f62c17dd000)
libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f62c15c6000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f62c1227000)
/lib64/ld-linux-x86-64.so.2 (0x00007f62c291a000)
The process I'm trying to attach to is running:
$ java -version
openjdk version "1.8.0_151"
OpenJDK Runtime Environment (build 1.8.0_151-8u151-b12-1~deb9u1-b12)
OpenJDK 64-Bit Server VM (build 25.151-b12, mixed mode)
The OS is:
$ uname -a
Linux ip-<snip> 4.9.0-5-amd64 #1 SMP Debian 4.9.65-3+deb9u2 (2018-01-04) x86_64 GNU/Linux
Please, remove > /dev/null at profiler.sh:48 and rerun the script to see if jattach prints an error message.
Well.. I did that, retried, and the attach worked just fine. Tried adding > /dev/null back, for sanity, but it seems to work now. I'm not entirely sure what I changed. I'll come back with updated output if I'm able to reproduce it again.
I've got the same issue
After removing all >/dev/nulls the output is:
Connected to remote JVM
Response code = -1
Failed to inject profiler into 32698
linux-vdso.so.1 => (0x00007fffa0d2d000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f27e9042000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f27e8e26000)
libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f27e8b1f000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f27e889d000)
libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f27e8687000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f27e82fa000)
/lib64/ld-linux-x86-64.so.2 (0x00007f27e98e5000)
Couldn't find any solution out there.
@apangin , please advise
@krems Are you building async-profiler from master without changes?
Please check what the following command prints.
java -agentpath:/path/to/libasyncProfiler.so=start,summary,flat -version
Yes
$ java -agentpath:./build/libasyncProfiler.so=start,summary,flat -version
Started [cpu] profiling
java version "1.8.0_101"
Java(TM) SE Runtime Environment (build 1.8.0_101-b13)
Java HotSpot(TM) 64-Bit Server VM (build 25.101-b13, mixed mode)
--- Execution profile ---
Total samples: 10
Unknown (native): 3 (30.00%)
Frame buffer usage: 0.0079%
2091384 (20.60%) 2 Method::make_jmethod_id(ClassLoaderData*, Method*)
1136257 (11.19%) 1 java.io.BufferedWriter.write
1015855 (10.00%) 1 java.lang.Thread.<init>
1012302 ( 9.97%) 1 java.util.LinkedHashMap.linkNodeLast
1007078 ( 9.92%) 1 ClassFileParser::parseClassFile(Symbol*, ClassLoaderData*, Handle, KlassHandle, GrowableArray<Handle>*, TempNewSymbol&, bool, Thread*)
1006586 ( 9.91%) 1 SignatureStream::SignatureStream(Symbol*, bool)
998909 ( 9.84%) 1 SymbolTable::lookup_only(char const*, int, unsigned int&)
995460 ( 9.80%) 1 clock_gettime
I should mention that I have to run profiler via sudo -u user ./profiler.sh
OK, the library seems to work.
Is anything printed in stderr of the target JVM when you run profiler.sh?
No output, but I found out what was the problem.
I'd assume this is our administration settings issue.
Even though I was running profiler.sh under user, the fact that async-profiler/* files were not in that user's home dir prevented profiler from attaching to the JVM process.
Moving async-profiler dir to user's home fixed the problem
I hit a similar error when I tried to attach the async profiler to something that was running inside a LXC container. Running it from inside the container did the trick for me.
Thank you for the details. The clue is that profiler library is loaded by the target JVM on behalf of the user of the JVM process. This implies that async-profiler binary must be accessible by that user by exactly the same absolute path. I'll update troubleshooting sections with this information.
Dear @apangin,
I encounter this issue when I run async profiler in host, my application is running as container in this host.
[root@ip-1-1-1-1 async-profiler-2.9-linux-x64]# java -agentpath:/tmp/async-profiler-2.9-linux-x64/build/libasyncProfiler.so=start,flat -version
Profiling started
openjdk version "11.0.18" 2023-01-17 LTS
OpenJDK Runtime Environment Corretto-11.0.18.10.1 (build 11.0.18+10-LTS)
OpenJDK 64-Bit Server VM Corretto-11.0.18.10.1 (build 11.0.18+10-LTS, mixed mode)
Execution profile
Total samples : 0
ns percent samples top
[root@ip-1-1-1-1 async-profiler-2.9-linux-x64]# ./profiler.sh -d 30 -e mprotect -f /tmp/a.html 13751
Failed to inject profiler into 13751
./profiler.sh: line 116: /bin/true: No such file or directory
[root@ip-1-1-1-1 async-profiler-2.9-linux-x64]# ./profiler.sh -v
Async-profiler 2.9 built on Nov 26 2022
Copyright 2016-2021 Andrei Pangin
[oot@ip-1-1-1-1 async-profiler-2.9-linux-x64]#
I give all the permission(-rwxrwxrwx) to libasyncProfiler.so.
[root@ip-1-1-1-1 build]# ll
total 460
-rwxrwxrwx 1 root root 5015 Nov 26 22:40 async-profiler.jar
-rwxrwxrwx 1 root root 46172 Nov 26 22:40 converter.jar
-rwxrwxrwx 1 root root 18218 Nov 26 22:40 fdtransfer
-rwxrwxrwx 1 root root 24850 Nov 26 22:40 jattach
-rwxrwxrwx 1 root root 361312 Nov 26 22:40 libasyncProfiler.so
[root@ip-1-1-1-1 build]#
Could u please kindly help to check?
@roy-20230331 libasyncProfiler.so should be available in the container at the same path as in the host. Please see the linked questions for details.
I see, thx @apangin !