Add support for OpenTelemetry tracing
saschagrunert opened this issue · comments
We're already using the tracing crate, which has support for Open Telemetry (OTEL) as well: https://github.com/tokio-rs/tracing/tree/master/tracing-opentelemetry
Using the crate would probably incorporate adding those, too:
Independently how we wanna use it from a configuration perspective, having OTEL support would increase the observability of the runtime stack (together with CRI-O) by gaining much more insights about what is going on in conmon-rs.
This seems very interesting. If you don't mind I want to tackle this issue.
Quick search into repo with keyword 'debug_span' revealed 6 files currently being traced. Is this issue about turning those tracing points into OTEL compatible tracepoints?
Awesome, great to read @doehyunbaek! 🙏
Yes, I think having defined spans for all available RPCs would be a great first step, for example:
conmon-rs/conmon-rs/server/src/rpc.rs
Lines 71 to 72 in 1c4766f
Later on we can decide if we want to make them smaller, add metrics on top of that or provide additional profiling data.
To make it optional, I would assume that we add a configuration flag to the server which allows us to enable/disable telemetry.
Thanks for reply. Really appreciate it.
Unfortunately, I think I would be occupied with day job / IRL minimum 2 weeks, I would like to tackle it after that. Would that be ok? If anyone tries to take this issue in the meantime I would not mind him taking the issue.
@doehyunbaek yep that's totally fine, no need to rush on this topic! 👍
@doehyunbaek do you still have interest in working on it? Just wanted to check-in if priorities have been changed.
Hi. I quit my previous job in the meantime and am planning on switching to other fields which do not involve container technologies :(. I do not have interest in working on it. Sorry for late update.
I can give this a shot. I can pick at it here and there, but can really sit down and work on it next week when my internship ends (assuming I don't get it done before then).
Hi, I just started working on this issue, and I wanted to try and run the tests to see if that would provide any data on the Jaeger dashboard. However, I keep getting this error when I try to do make integration
, and I can't seem to nail down what could be going wrong here. I have a feeling its an environment issue, but I seem to have the correct $PATH
exec: "go": executable file not found in $PATH
$ echo $PATH
/usr/local/opt/gnu-getopt/bin:/usr/local/opt/gnu-getopt/bin:/home/jakecorrenti/.cargo/bin:/home/jakecorrenti/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/local/go/bin
make integration
runs the tests with sudo
, what is sudo echo $PATH
? I 'm guessing it's missing /usr/local/go/bin
This appears to be fine as well
$ sudo echo $PATH
/usr/local/opt/gnu-getopt/bin:/home/jakecorrenti/.cargo/bin:/home/jakecorrenti/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/local/go/bin:/usr/local/go/bin
This has been resolved.
what about sudo -E echo $PATH
?
I figured it out, I needed to add /usr/local/go/bin
to /etc/sudoers