NCoz
chadbrewbaker opened this issue · comments
I'm interested in taking a stab at extracting bits of JCoz to support the Python virtual machine and the CLR runtime.
Any tips?
Unfortunately, I don't have anything helpful for you on the technical side. You should join the Gitter, though, if you have questions about JCoz: https://gitter.im/JCoz-profiler/community
Which bits are you interested in extracting, exactly? JCoz has two components:
- The native agent where a given JVM process is causally profiled and on which experiments are run.
- The service layer where a user can configure and initiate a profiling session.
(1) might be useful to you by giving some optics into how causal profiling is implemented for an interpreted language. A lot of it is very JVM specific given that we're using JVMTI to hook into the events we require to causally profile a JVM process. I'd imagine that the Python runtime has some similar native agent interface. I'd be interested in hearing about it if so, and I wonder if we could abstract any logic in JCoz into a separate shared library that we could share as a dependency.
(2) Should might you some intuition about usability. One of the problems with causal profilers is usability. Though it's a brilliant and surprisingly simple technique, its implementation is generally complex and requires a bit of expertise to set up. The service layer is JCoz' attempt to somewhat mitigate that. It's possible that we could share some code there as well, but that depends on whether Python has a JNI type interface for communicating between a Python process and its native agent. If that's true, it might more valuable for us to just abstract JCoz into supporting the profiling of both languages and/or having a general interpreted language causal profiler.
I'd imagine that the Python runtime has some similar native agent interface.
Disclaimer: Assuming Python does have such an interface, there will likely be a lot of subtle technical problems that will need to be solved for the profiler to actually be useful. For example, JVMTI exposes a hook for getting stack traces for threads, but it waits until "safe points" which significantly bias the profiler. To get around this, we had to link against an undocumented JVM function called _ AsyncGetCallTrace_ and run it on some timed cadence. Such technical problems are the especially challenging aspect of writing a causal profiler: despite being a sampling profiler, it's difficult to write such an "invasive" profiler and still get accurate and unbiased results.
The Python interpreter may or may not have such barriers to entry, but I would be very surprised if it didn't.
One more thing I should mention: I don't think a causal profiler will buy you much for Python. Python threads don't actually execute in parallel, so you won't really gain any information when running experiments. Causally profiling a multi-process Python program is a different story, but AFAIK multi-process causal profiling is still in its very early stages.