containers / conmon-rs

We're already using the tracing crate, which has support for Open Telemetry (OTEL) as well: https://github.com/tokio-rs/tracing/tree/master/tracing-opentelemetry

Using the crate would probably incorporate adding those, too:

Independently how we wanna use it from a configuration perspective, having OTEL support would increase the observability of the runtime stack (together with CRI-O) by gaining much more insights about what is going on in conmon-rs.

This seems very interesting. If you don't mind I want to tackle this issue.

Quick search into repo with keyword 'debug_span' revealed 6 files currently being traced. Is this issue about turning those tracing points into OTEL compatible tracepoints?

Awesome, great to read @doehyunbaek! 🙏

Yes, I think having defined spans for all available RPCs would be a great first step, for example:

conmon-rs/conmon-rs/server/src/rpc.rs

Lines 71 to 72 in 1c4766f

    
           let span = new_root_span!("create_container", id.as_str()); 
        
           let _enter = span.enter();

Later on we can decide if we want to make them smaller, add metrics on top of that or provide additional profiling data.

To make it optional, I would assume that we add a configuration flag to the server which allows us to enable/disable telemetry.

Thanks for reply. Really appreciate it.

Unfortunately, I think I would be occupied with day job / IRL minimum 2 weeks, I would like to tackle it after that. Would that be ok? If anyone tries to take this issue in the meantime I would not mind him taking the issue.

@doehyunbaek yep that's totally fine, no need to rush on this topic! 👍

@doehyunbaek do you still have interest in working on it? Just wanted to check-in if priorities have been changed.

Hi. I quit my previous job in the meantime and am planning on switching to other fields which do not involve container technologies :(. I do not have interest in working on it. Sorry for late update.

I can give this a shot. I can pick at it here and there, but can really sit down and work on it next week when my internship ends (assuming I don't get it done before then).

Hi, I just started working on this issue, and I wanted to try and run the tests to see if that would provide any data on the Jaeger dashboard. However, I keep getting this error when I try to do make integration, and I can't seem to nail down what could be going wrong here. I have a feeling its an environment issue, but I seem to have the correct $PATH

exec: "go": executable file not found in $PATH

$ echo $PATH
/usr/local/opt/gnu-getopt/bin:/usr/local/opt/gnu-getopt/bin:/home/jakecorrenti/.cargo/bin:/home/jakecorrenti/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/local/go/bin

make integration runs the tests with sudo, what is sudo echo $PATH? I 'm guessing it's missing /usr/local/go/bin

This appears to be fine as well

$ sudo echo $PATH
/usr/local/opt/gnu-getopt/bin:/home/jakecorrenti/.cargo/bin:/home/jakecorrenti/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/local/go/bin:/usr/local/go/bin

This has been resolved.

what about sudo -E echo $PATH?

I figured it out, I needed to add /usr/local/go/bin to /etc/sudoers

	let span = new_root_span!("create_container", id.as_str());
	let _enter = span.enter();

Add support for OpenTelemetry tracing