dotnet / aspnetcore

ASP.NET Core is a cross-platform .NET framework for building modern cloud-based web applications on Windows, Mac, or Linux.

Home Page:https://asp.net

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Member-level trimming breaks ASP.NET tracing instrumentation

kerams opened this issue · comments

Description

Traces produced by OpenTelemetry instrumentation for ASP.NET (AddOpenTelemetryTracing) lack almost all expected data when running an application with member-level trimming enabled. AddHttpClientInstrumentation does not seem to be affected.

Reproduction Steps

Create a Kestrel server serving favicon.png. Set the following in the project file:

<TargetFramework>net7.0</TargetFramework>
<PublishTrimmed>true</PublishTrimmed>
<TrimMode>link</TrimMode>

Set up OpenTelemtry instrumentation for ASP.NET, send it to Jaeger (or use the console exporter).

Publish as self-contained for win-x64 (also reproducible for linux-musl-x64), run.

Send a GET request for /favicon.png.

Expected behavior

Span has the right display name and attributes.

vivaldi_ha6f7DeNXm

Actual behavior

Span is for all intents and purposes empty.

vivaldi_EqtZBT6RaV

Regression?

No response

Known Workarounds

Use <TrimMode>copyused</TrimMode>.

Configuration

Built with SDK 7.0.100-preview.5.22307.18.
On Windows 10 running the self-contained executable.
On Linux running in Docker with base mcr.microsoft.com/dotnet/runtime-deps:7.0.0-preview.5-alpine3.16.

Other information

No response

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

TrimMode=link is the default since .NET 6.0. TrimMode only sets trimming policies for assemblies that are manifested as trimmable. It doesn't affect other assemblies. TrimMode=link is the optimal value and there's usually no need to change the default.

If TrimMode=copyused works, it likely means some assembly that is manifested trimmable is not actually trimmable.

@davidfowl should we move this to aspnet repo?

Yep. ASP.NET doesn't support member level trimming.

Weird, I've never heard that before. Is it mentioned anywhere in the docs?

How come member-level is the default in ASP.NET projects if it's not supported? And how come there are no trim warnings (I could have missed those).

I don't suppose it's possible to use member-level on everything apart from ASP.NET assemblies?

Tagging subscribers to 'linkable-framework': @eerhardt, @vitek-karas, @LakshanF, @sbomer, @joperezr
See info in area-owners.md if you want to be subscribed.

Issue Details

Description

Traces produced by OpenTelemetry instrumentation for ASP.NET (AddOpenTelemetryTracing) lack almost all expected data when running an application with member-level trimming enabled. AddHttpClientInstrumentation does not seem to be affected.

Reproduction Steps

Create a Kestrel server serving favicon.png. Set the following in the project file:

<TargetFramework>net7.0</TargetFramework>
<PublishTrimmed>true</PublishTrimmed>
<TrimMode>link</TrimMode>

Set up OpenTelemtry instrumentation for ASP.NET, send it to Jaeger (or use the console exporter).

Publish as self-contained for win-x64 (also reproducible for linux-musl-x64), run.

Send a GET request for /favicon.png.

Expected behavior

Span has the right display name and attributes.

vivaldi_ha6f7DeNXm

Actual behavior

Span is for all intents and purposes empty.

vivaldi_EqtZBT6RaV

Regression?

No response

Known Workarounds

Use <TrimMode>copyused</TrimMode>.

Configuration

Built with SDK 7.0.100-preview.5.22307.18.
On Windows 10 running the self-contained executable.
On Linux running in Docker with base mcr.microsoft.com/dotnet/runtime-deps:7.0.0-preview.5-alpine3.16.

Other information

No response

Author: kerams
Assignees: -
Labels:

untriaged, linkable-framework

Milestone: -

@kerams You should assume nothing works with member level trimming by default in .NET. Anything that uses reflection that wasn't explicitly designed to be trimmed will break. People that have been successful with trimming in general have been either lucky of very determined. .NET 7 is the first version where we're officially trying to mark more of the framework with "this will break when you trim" or describe any reflection we do where possible.

Oh wait, I just re-read this. This is a .NET 7 application and you're losing members when using Otel. This looks like a diagnostic source issue. We hard code a couple of fields to preserve here but only in some cases.

cc @JamesNK

@kerams you mention that you didn't get any warnings - that is very weird. Can you please make sure that you didn't disable them, typically via SuppressTrimAnalysisWarnings property?

@vitek-karas, I've tried creating a minimal C# repro (the solution where I noticed the problem is F#) and I do see the warnings. A similar minimal F# project does not seem to surface them. @vzarytovskii, could it be possible or am I doing something wrong?

The original issue still stands though, right? The trimming metadata is not 100% correct in some assembly, otherwise the latter would not be touched during trimming?

The promise is only: If there are no warnings, the app should work when trimmed. If F# (or something there) disables the warnings, it probably shouldn't do that.

I vaguely remember that open telemetry specifically is relatively problematic for trimming, but I don't remember the details and if there were some potential solutions.

Ok, my misunderstanding lay in thinking that trimming only affected assemblies that include relevant metadata, and assemblies that are explicitly enumerated in project files with TrimmableAssembly.

Thanks, feel free to close this if there's nothing actionable here.

"Trim safety" is a global problem. Currently we enable trimming on assemblies which themselves don't cause problems (or are correctly annotated), but it's a heuristic. If you have code in your app which uses reflection to reach into some private field of the framework type, it is likely to break - we would trim the framework type, potentially removing the field. That's where the warnings come in - those should work globally and should tell you that the reflection code which does this is problematic (if we can't figure it out).

I vaguely remember that open telemetry specifically is relatively problematic for trimming, but I don't remember the details and if there were some potential solutions.

Diagnostic source is fundamentally not trim safe since it basically (https://github.com/dotnet/runtime/blob/475858d14c580a765c3e32cbb3f90cd5825e8b5c/src/libraries/System.Diagnostics.DiagnosticSource/src/System/Diagnostics/DiagnosticSource.cs#L19).

Most consumers of these events use reflection to consume the data to avoid versioning problems (e.g. https://github.com/open-telemetry/opentelemetry-dotnet/blob/f234829964a2e59078d3d02c1a881ff35ffd65b8/src/OpenTelemetry/DiagnosticSourceInstrumentation/PropertyFetcher.cs)

This type is doing reflection on the framework (which is marked trim safe) and it doesn't declare that it's doing that so as a result, these members disappear.

@davidfowl should this go back to dotnet/runtime?

No, this should probably be in open telemetry but we should investigate and understand the repro first.

I repoed this with gRPC + OpenTelemetry + Zipkin

HttpClient is generating correct info with OT. I looked at what they're doing to still work and DynamicDependency is added to HttpClient diagnostic source APIs while we aren't (at least with these ones).

Updated: #42512

Verified as fixed with #42591.

image