Activity.Kind improperly set and Activity.ActivitySourceName missing when using Auto-Instrumentation for incoming requests

Question

Activity.Kind improperly set and Activity.ActivitySourceName missing when using Auto-Instrumentation for incoming requests

watkinsmike opened this issue 5 months ago · comments

Bug Report

Symptom

Describe the bug
I believe the Activity.Kind being produced from my traces are not properly set to Server as expected and and Activity.ActivitySourceName is empty when using auto-instrumentation on a self-contained simple .NET 8.0 web app.

Because the Kind is not Server I believe this is affecting my service mapping in both AWS X-RAY/Grafana.

Expected behavior
I expect Activity.Kind to be Server for incoming HTTP requests and to have a non-empty ActivitySourceName. Looking at the getting started docs I expect something like:

Activity.ActivitySourceName: OpenTelemetry.Instrumentation.AspNetCore
Activity.DisplayName:        /rolldice
Activity.Kind:               Server

Actual

Activity.TraceId:            8a412ea70c2454ea35c60848131c7a91
Activity.SpanId:             d6409440d1c1b04e
Activity.TraceFlags:         Recorded
Activity.ActivitySourceName:
Activity.DisplayName:        GET /rolldice/{player?}
Activity.Kind:               Internal
Activity.StartTime:          2024-01-03T18:49:32.9035202Z
Activity.Duration:           00:00:00.0388405
Activity.Tags:
    net.host.name: localhost
    net.host.port: 8080
    http.method: GET
    http.scheme: http
    http.target: /rolldice
    http.url: http://localhost:8080/rolldice
    http.flavor: 1.1
    http.user_agent: curl/8.1.2
    http.route: /rolldice/{player?}
    http.status_code: 200
Resource associated with Activity:
    container.id: 3f41cd49cfffe4008f6e4a58fff32adf7f7bd4275a7a1d6dc902a9fd2718f5fb
    telemetry.distro.name: opentelemetry-dotnet-instrumentation
    telemetry.distro.version: 1.2.0
    telemetry.sdk.name: opentelemetry
    telemetry.sdk.language: dotnet
    telemetry.sdk.version: 1.6.0
    service.name: hello-world

Runtime environment (please complete the following information):

OpenTelemetry Automatic Instrumentation version: 1.2.0
OS: Ubuntu 22.04
.NET version: 8

Reproduce

Steps to reproduce the behavior:

Create a simple app
dotnet new webapi
dotnet add package OpenTelemetry.AutoInstrumentation --prerelease
Modify code from the example in the Getting Started docs: https://opentelemetry.io/docs/instrumentation/net/automatic/getting-started/
Build/Run the application using hardcoded environment variables derived from the instrument.sh that would be exported from instrument.sh

FROM mcr.microsoft.com/dotnet/sdk:8.0-jammy AS build
WORKDIR /source

COPY src/hello-world/ .

RUN dotnet publish --runtime linux-x64 -v q -c Release -o /app --self-contained

FROM mcr.microsoft.com/dotnet/runtime:8.0-jammy

ENV OTEL_TRACES_EXPORTER=none
ENV OTEL_METRICS_EXPORTER=none
ENV OTEL_LOGS_EXPORTER=none
ENV OTEL_DOTNET_AUTO_TRACES_CONSOLE_EXPORTER_ENABLED=true
ENV OTEL_SERVICE_NAME=hello-world

ENV ASPNETCORE_HOSTINGSTARTUPASSEMBLIES=OpenTelemetry.AutoInstrumentation.AspNetCoreBootstrapper
ENV CORECLR_ENABLE_PROFILING=1
ENV CORECLR_PROFILER={918728DD-259F-4A6A-AC2B-B85E1B658318}
ENV CORECLR_PROFILER_PATH=/app/OpenTelemetry.AutoInstrumentation.Native.so
ENV DOTNET_STARTUP_HOOKS=/app/OpenTelemetry.AutoInstrumentation.StartupHook.dll
ENV OTEL_DOTNET_AUTO_HOME=/app
ENV OTEL_DOTNET_AUTO_RULE_ENGINE_ENABLED=false

WORKDIR /app
COPY --from=build /app .
ENTRYPOINT ["./hello-world"]

Hit /rolldice route and observe metric telemetry

otel-dotnet-auto-hello-world-1-20240103.log
[2024-01-03T19:08:41.8672642Z] [Information] OpenTelemetry tracer initialized.
[2024-01-03T19:08:41.8849735Z] [Information] EventSource=OpenTelemetry-AutoInstrumentation, Message=OpenTelemetry tracer initialized.
[2024-01-03T19:08:41.9219648Z] [Information] OpenTelemetry meter initialized.
[2024-01-03T19:08:41.9241148Z] [Information] EventSource=OpenTelemetry-AutoInstrumentation, Message=OpenTelemetry meter initialized.
[2024-01-03T19:08:41.9858280Z] [Information] The profiler has been initialized with 17 definitions for FFAFA5168C4F4718B40CA8788875C2DA.
[2024-01-03T19:08:41.9878884Z] [Information] EventSource=OpenTelemetry-AutoInstrumentation, Message=The profiler has been initialized with 17 definitions for FFAFA5168C4F4718B40CA8788875C2DA.
[2024-01-03T19:08:41.9913448Z] [Information] The profiler has been initialized with 0 derived definitions.
[2024-01-03T19:08:41.9933661Z] [Information] EventSource=OpenTelemetry-AutoInstrumentation, Message=The profiler has been initialized with 0 derived definitions.
[2024-01-03T19:08:41.9972930Z] [Information] EventSource=OpenTelemetry-AutoInstrumentation-StartupHook, Message=StartupHook initialized successfully!
[2024-01-03T19:08:42.0341182Z] [Information] EventSource=OpenTelemetry-AutoInstrumentation, Message=Logs: Loaded AddOpenTelemetry from LoggingBuilder.
[2024-01-03T19:08:42.1282065Z] [Information] EventSource=OpenTelemetry-AutoInstrumentation, Message=BootstrapperHostingStartup loaded for application with name hello-world.

otel-dotnet-auto-hello-world-1-AspNetCoreBootstrapper-20240103.log
[2024-01-03T19:08:42.1210927Z] [Information] BootstrapperHostingStartup loaded for application with name hello-world.

otel-dotnet-auto-hello-world-1-Loader-20240103.log
[2024-01-03T19:08:41.4389461Z] [Information] Managed Loader TryLoadManagedAssembly()

otel-dotnet-auto-hello-world-1-StartupHook-20240103.log
[2024-01-03T19:08:41.2306398Z] [Information] Rule Engine: MinSupportedFrameworkRule evaluation success.
[2024-01-03T19:08:41.3866140Z] [Information] OTEL_DOTNET_AUTO_RULE_ENGINE_ENABLED is set to false, skipping rule engine validation.
[2024-01-03T19:08:41.3886278Z] [Information] Initialization.
[2024-01-03T19:08:41.9952490Z] [Information] StartupHook initialized successfully!

otel-dotnet-auto-native-hello-world-1.log
[2024-01-03T19:08:40.846585637Z] [1|1] [info] OpenTelemetry CLR Profiler 1.2.0 on Linux (amd64)
[2024-01-03T19:08:40.846876842Z] [1|1] [info] JIT Inlining is enabled.
[2024-01-03T19:08:40.846891198Z] [1|1] [info] NGEN is disabled.
[2024-01-03T19:08:40.847020103Z] [1|13] [info] Initializing ReJIT request thread.
[2024-01-03T19:08:40.847348038Z] [1|1] [info] Profiler filepath: /app/OpenTelemetry.AutoInstrumentation.Native.so
[2024-01-03T19:08:40.847362824Z] [1|1] [info] Profiler attached.
[2024-01-03T19:08:40.856924455Z] [1|1] [info] COR library: System.Private.CoreLib 8.0.0
[2024-01-03T19:08:41.391872308Z] [1|1] [info] ModuleLoadFinished: OpenTelemetry.AutoInstrumentation.Loader loaded into AppDomain 93981679129920 [clrhost]
[2024-01-03T19:08:41.458056240Z] [1|1] [info] Rewriting PInvokes to native: /app/OpenTelemetry.AutoInstrumentation.Native.so
[2024-01-03T19:08:41.458201997Z] [1|1] [info] Rewriting pinvoke for: AddInstrumentations
[2024-01-03T19:08:41.458220634Z] [1|1] [info] Rewriting pinvoke for: AddDerivedInstrumentations
[2024-01-03T19:08:41.984834060Z] [1|1] [info] AddInstrumentations: received id: FFAFA5168C4F4718B40CA8788875C2DA from managed side with 17 integrations.
[2024-01-03T19:08:41.984945482Z] [1|1] [info] Total number of modules to analyze: 23
[2024-01-03T19:08:41.985218818Z] [1|1] [info] InternalAddInstrumentation: Total integrations in profiler: 17
[2024-01-03T19:08:42.016708318Z] [1|13] [info] ReJIT handler stored metadata for 139748277831896 Microsoft.Extensions.Logging AppDomain 93981679129920 clrhost
[2024-01-03T19:08:42.016797738Z] [1|13] [info] Request ReJIT done for 1 methods
[2024-01-03T19:08:42.020218403Z] [1|1] [info] *** CallTarget_RewriterCallback() Finished: Microsoft.Extensions.Logging.LoggingBuilder..ctor() [IsVoid=1, IsStatic=0, IntegrationType=OpenTelemetry.AutoInstrumentation.Instrumentations.Logger.LoggingBuilderIntegration, Arguments=1]

watkinsmike · Answer 1 · Thu Jan 04 2024 05:44:17 GMT+0800 (China Standard Time)

Adding a simple route that makes an outbound HTTP call seems to work properly, but the inbound still doesn't work as expected.

app.MapGet("/outgoing", () =>
{
    HttpClient httpClient = new HttpClient();
    using HttpResponseMessage response = httpClient.GetAsync("https://aws.amazon.com").Result;
});

Activity.TraceId:            4afef166a4ef9496e7895d950356925c
Activity.SpanId:             acb128bcf8cc718a
Activity.TraceFlags:         Recorded
Activity.ParentSpanId:       bec3fc31dc7fef5c
Activity.ActivitySourceName: System.Net.Http
Activity.DisplayName:        HTTP GET
Activity.Kind:               Client
Activity.Tags:
    http.method: GET
    http.url: https://aws.amazon.com
    http.status_code: 200
    ...


Activity.TraceId:            4afef166a4ef9496e7895d950356925c
Activity.SpanId:             bec3fc31dc7fef5c
Activity.TraceFlags:         Recorded
Activity.ActivitySourceName:
Activity.DisplayName:        GET /outgoing
Activity.Kind:               Internal

Rasmus Kuusmann · Answer 2 · Thu Jan 04 2024 18:31:27 GMT+0800 (China Standard Time)

Possibly the issue in OpenTelemetry.Instrumentation.AspNetCore in open-telemetry repo.

HttpInListener.cs

#if !NET7_0_OR_GREATER
            ActivityInstrumentationHelper.SetActivitySourceProperty(activity, ActivitySource);
            ActivityInstrumentationHelper.SetKindProperty(activity, ActivityKind.Server);
#endif

I guess they expect framework level assignment, but somehow it doesn't work.

watkinsmike · Answer 3 · Mon Jan 08 2024 22:30:46 GMT+0800 (China Standard Time)

Should I open an issue there? This only affects the auto-instrumentation of self-contained apps when using Nuget package OpenTelemetry.AutoInstrumentation so I thought here was appropriate.

Confirmed that .NET 6 does not have this issue.
Current workaround (which is unideal for us because it requires updating a lot of Dockerfile and the potential for dependency conflicts) is to not use package installation and build a base layer that installs the otel components manually

FROM mcr.microsoft.com/dotnet/sdk:8.0-jammy AS otel-dotnet-auto

ARG OTEL_VERSION=1.2.0
ADD https://github.com/open-telemetry/opentelemetry-dotnet-instrumentation/releases/download/v${OTEL_VERSION}/otel-dotnet-auto-install.sh otel-dotnet-auto-install.sh
RUN apt-get update && apt-get install -y unzip && \
    OTEL_DOTNET_AUTO_HOME="/otel-dotnet-auto" sh otel-dotnet-auto-install.sh

# final stage/image
FROM mcr.microsoft.com/dotnet/runtime:8.0-jammy

ENV ASPNETCORE_HOSTINGSTARTUPASSEMBLIES=OpenTelemetry.AutoInstrumentation.AspNetCoreBootstrapper
ENV CORECLR_ENABLE_PROFILING=1
ENV CORECLR_PROFILER={918728DD-259F-4A6A-AC2B-B85E1B658318}
ENV CORECLR_PROFILER_PATH=/otel-dotnet-auto/linux-x64/OpenTelemetry.AutoInstrumentation.Native.so
ENV DOTNET_STARTUP_HOOKS=/otel-dotnet-auto/net/OpenTelemetry.AutoInstrumentation.StartupHook.dll
ENV DOTNET_ADDITIONAL_DEPS=/otel-dotnet-auto/AdditionalDeps
ENV DOTNET_SHARED_STORE=/otel-dotnet-auto/store
ENV OTEL_DOTNET_AUTO_HOME=/otel-dotnet-auto
ENV OTEL_DOTNET_AUTO_RULE_ENGINE_ENABLED=false

COPY --from=otel-dotnet-auto /otel-dotnet-auto /otel-dotnet-auto
...

Rasmus Kuusmann · Answer 4 · Mon Jan 08 2024 22:58:23 GMT+0800 (China Standard Time)

@watkinsmike thanks for feedback. If only Auto Instrumentation is causing this issue, it's definitely something to do with Auto Instrumentation. Can be verified with a sample app that has direct SDK and OpenTelemetry.Instrumentation.AspNetCore references, same versions and manual setup (by the SDK docs).

Rasmus Kuusmann · Answer 5 · Mon Jan 15 2024 20:22:46 GMT+0800 (China Standard Time)

Executed some tests using OpenTelemetry.AutoIstrumentation 1.2 and OpenTelemetry SDK 1.6.
Interestingly when application has direct reference to OpenTelemetry.Instrumentation.AspNetCore it does not work. When removed and OpenTelemetry.Instrumentation.AspNetCore was loaded from the shared store then everything worked.

Direct OpenTelemetry.Instrumentation.AspNetCore reference comes via dotnet add package OpenTelemetry.AutoInstrumentation --prerelease

For OpenTelemetry package it did not matter, if there was a direct reference.
System.Diagnostics.DiagnosticSource, OpenTelemetry and OpenTelemetry.Instrumentation.AspNetCore file versions were exactly the same just the location matters where OpenTelemetry.Instrumentation.AspNetCore is loaded.

Currently no ideas, why that location matters.

CC: @Kielek

Rasmus Kuusmann · Answer 6 · Mon Jan 15 2024 20:36:57 GMT+0800 (China Standard Time)

Upgrading to OpenTelemetry.AutoInstrumentation main branch and OpenTelemetry 1.7 does not resolve the issue.

Rasmus Kuusmann · Answer 7 · Mon Jan 15 2024 20:52:34 GMT+0800 (China Standard Time)

Interestingly decompiled dlls with the same version differ by:
(which is exactly the same source as commented - #3212 (comment))

HttpInListener.cs

ActivityInstrumentationHelper.SetActivitySourceProperty(activity, ActivitySource);
ActivityInstrumentationHelper.SetKindProperty(activity, ActivityKind.Server);

AutoInstrumentation seems to load correct compiled version while direct reference doesn't 🤔

Rasmus Kuusmann · Answer 8 · Mon Jan 15 2024 21:40:36 GMT+0800 (China Standard Time)

Synced with @Kielek, AutoInstrumentation actually loads wrong target framework (currently fixed to .NET 6), so pre-compilator flags do not execute corretly.