[Feature Request] OpenTelemetry support

Question

[Feature Request] OpenTelemetry support

cretz opened this issue a year ago · comments

Describe the solution you'd like

Need OpenTelemetry interceptor as separately published project

John Kattenhorn · Answer 1 · Fri May 26 2023 23:45:43 GMT+0800 (China Standard Time)

Sounds like a good idea. Do you have any design thoughts on this yet, I'm thinking about this too, I think it should be possible to do Spans, Tracing and Metrics.

Any ideas of what Metrics would be useful to track? Could we contribute on this one ?

Chad Retz · Answer 2 · Sat May 27 2023 00:24:07 GMT+0800 (China Standard Time)

This is only for tracing. OTel is already supported for metrics (you can create your own TemporalRuntime w/ metrics telemetry options set to an OTel gRPC URL and use it throughout the life of your application). https://docs.temporal.io/references/sdk-metrics are the metrics we track.

I have been working on the tracing part just this morning and I'll jot some notes down here. So it will be interceptor powered (like Python's, see https://github.com/temporalio/sdk-python/blob/main/temporalio/contrib/opentelemetry.py and https://github.com/temporalio/samples-python/tree/main/open_telemetry), but I am struggling with how best to split the project up.

So I was first thinking of a separate project for Temporalio.Extensions.OpenTelemetry, but since OTel support is basically just using https://learn.microsoft.com/en-us/dotnet/core/diagnostics/distributed-tracing, a more generic Temporalio.Extensions.Tracing project may make more sense (didn't want it part of main project because our oldest supported .NET versions would require System.Diagnostics.DiagnosticSource as an explicit dependency which I was wanting to avoid).

The problem I am at now is that I have to use https://github.com/open-telemetry/opentelemetry-dotnet/blob/main/src/OpenTelemetry.Api/Context/Propagation/BaggagePropagator.cs and https://github.com/open-telemetry/opentelemetry-dotnet/blob/main/src/OpenTelemetry.Api/Context/Propagation/CompositeTextMapPropagator.cs to serialize spans to Temporal headers that are compatible with other languages who have OTel too. I am digging into if I can use .NET's DistributedContextPropagator easily without making the user do too much work or if I am going to need to make OTel helpers for them and therefore an OTel specific project again.

Active research is ongoing.

John Kattenhorn · Answer 3 · Sat May 27 2023 00:39:21 GMT+0800 (China Standard Time)

On the Span tracking I was thinking about the same earlier today too. The only other solution I’ve used before was specific to AppInsights Spans so not going to work with OTel. Let me know where you land, and I can ask our team to get involved and share some opinion / insight. Thanks JK From: Chad Retz ***@***.***> Sent: Friday, May 26, 2023 9:24 AM To: temporalio/sdk-dotnet ***@***.***> Cc: John Kattenhorn ***@***.***>; Comment ***@***.***> Subject: [EXTERNAL]:Re: [temporalio/sdk-dotnet] [Feature Request] OpenTelemetry support (Issue #56) CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe. This is only for tracing. OTel is already supported for metrics (you can create your own TemporalRuntime w/ metrics telemetry options set to an OTel gRPC URL and use it throughout the life of your application). https://docs.temporal.io/references/sdk-metrics are the metrics we track. I have been working on the tracing part just this morning and I'll jot some notes down here. So it will be interceptor powered (like Python's, see https://github.com/temporalio/sdk-python/blob/main/temporalio/contrib/opentelemetry.py and https://github.com/temporalio/samples-python/tree/main/open_telemetry), but I am struggling with how best to split the project up. So I was first thinking of a separate project for Temporalio.Extensions.OpenTelemetry, but since OTel support is basically just using https://learn.microsoft.com/en-us/dotnet/core/diagnostics/distributed-tracing, a more generic Temporalio.Extensions.Tracing project may make more sense (didn't want it part of main project because our oldest supported .NET versions would require System.Diagnostics.DiagnosticSource as an explicit dependency which I was wanting to avoid). The problem I am at now is that I have to use https://github.com/open-telemetry/opentelemetry-dotnet/blob/main/src/OpenTelemetry.Api/Context/Propagation/BaggagePropagator.cs and https://github.com/open-telemetry/opentelemetry-dotnet/blob/main/src/OpenTelemetry.Api/Context/Propagation/CompositeTextMapPropagator.cs to serialize spans to Temporal headers that are compatible with other languages who have OTel too. I am digging into if I can use .NET's DistributedContextPropagator easily without making the user do too much work or if I am going to need to make OTel helpers for them and therefore an OTel specific project again. Active research is ongoing. — Reply to this email directly, view it on GitHub<#56 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AAEWOOJVLRCQPGEILTCT5ATXIDKLFANCNFSM6AAAAAAXLGDOCM>. You are receiving this because you commented.Message ID: ***@***.******@***.***>>

Chad Retz · Answer 4 · Sat May 27 2023 04:55:38 GMT+0800 (China Standard Time)

@johnkattenhorn - Maybe you can help me here. I have restored an activity context and baggage via propagation similar to this. This has the ActivityContext but there's no way I see to attach current context in OpenTelemetry terms given the ActivityContext only. Other OTel SDKs let me set the "current" context, but .NET only allows the "current" activity and I don't want to create a new activity with the parent, I just want to restore the parent to "current". Any ideas?

John Kattenhorn · Answer 5 · Sat May 27 2023 05:05:29 GMT+0800 (China Standard Time)

Oh man sound like it works like AppInsights, inthink we had to recreate the activity as you could only set the parentID in the constructor and copy everything across.

I'm going to ask someone from our team but just boarding a flight back to the UK so might be a delay.

Chad Retz · Answer 6 · Sat May 27 2023 05:13:50 GMT+0800 (China Standard Time)

Thanks! Definitely no rush (can be days/weeks, don't waste weekend time). I also have a question out on #otel-dotnet on CNCF Slack. I am sure I'm just missing something. Maybe that's what I have to do is create the activity and just set all the fields I can and know about and set it as Activity.Current.