Loss of custom Span details since 2.27

[Dave Shaw]

Describe the bug
We recently upgraded to the 2.31 (from 2.22) Datadog.Trace package and noticed we were no longer getting any data for our custom span in the metrics in the UI.
After checking a few versions this works in 2.26 and is broken in 2.27, .30 and .31.

We have also noticed that our custom span no longer appears as a "Service Entry" span or "All span" in 2.27, but does appear in 2.26, when looking in APM.

To Reproduce

The code we use to create the span is similar to:

using (var ddScope = Datadog.Trace.Tracer.Instance.StartActive($"XX > YY"))
{
	ddScope.Span.ResourceName = request.MessageNamek__BackingField;
	ddScope.Span.SetTag("aa.bb.one", one);
	ddScope.Span.SetTag("aa.bb.two", two);
	ddScope.Span.SetTag("aa.bb.three", three);
	ddScope.Span.SetTag("aa.bb.four", four);

        // Some WCF API Call.
}

Using the above code then checking the Datadog. If you look for the metric trace.XX_YY.hits (or duration) it only shows values when we are running 2.26.

Expected behavior
Should have values in trace.XX_YY
Should should Service Entry spans / All spans for our custom Span.

Screenshots
In both cases if you open the Root Span, you can still see the custom span:

But when you look at Service Entry / All you don't see it:
Working:

Not working:

Runtime environment (please complete the following information):

• MSI Installer and Datadog.Trace
• Both MSI and Package are 2.31
• OS: Windows Server 2019 (1809)
• CLR: .NET 6 / ASP.NET Core 6, Hosted inprocess in IIS.

[Andrew Lock]

Hi @Dave-EMIS, sorry to hear you're having issues. I don't quite understand what you're seeing though - if the custom span is shown in the trace (as it appears to be when you open the root span), then by definition it should appear in the "All spans" list 🤔

One thing to bear-in-mind is that if you have a "version-mismatch" scenario, your custom spans are technically part of a "distributed" trace, which complicates things a bit. Can you confirm that you kept your MSI and Datadog.Trace packages in-sync when testing the upgrade?

If you can reproduce the issue when the versions are the same, it would probably be best to create a new support ticket here: https://help.datadoghq.com/hc/en-us/requests/new so we can capture the extra details to figure this out quickly 🙂 Thanks!

[Dave Shaw]

Hi @andrewlock - yes we did test with 2.31 of the MSI and the Package, even after a full reboot of the VM just to clear out and caching issues and the same problem happened.

One of my team will log a ticket on website for you

[Andrew Lock]

Hope you resolved the issue with support @Dave-EMIS - closing the issue here