traceparent http header field is not extracted correctly when datadog agent also inserts it
wondernet opened this issue · comments
What happened?
Steps to Reproduce
Datadog agent is needed in order to reproduce this issue, or use the custom http header that will be provided as an example.
We have a system that has datadog agent and we added open telemetry instrumentation, we noticed that traces are not properly propagated for open telemetry (related to traceparent http header field). If datadog instrumentation is disabled, then it works as expected.
After more investigation that took days, we found out that the http headers info for a request looks like this:
GET /url_here
HTTP/1.1
Host: host_here
X-Request-ID: 35960605cb59c8a78e5382c64b527d31
X-Real-IP: 127.0.0.1
X-Forwarded-For: 127.0.0.1
X-Forwarded-Host: host_here
X-Forwarded-Port: 443
X-Forwarded-Proto: https
X-Forwarded-Scheme: https
X-Scheme: https
x-datadog-trace-id: 192969926701373593
x-datadog-parent-id: 3667278417456668462
x-datadog-sampling-priority: 0
x-datadog-tags: _dd.p.tid=66844c8200000000
traceparent: 00-66844c820000000002ad91203d5e5899-32e4ca4a64f61f2e-00
tracestate: dd=t.tid:66844c8200000000;t.dm:-1;s:0;p:32e4ca4a64f61f2e
accept: */*
accept-language: *
sec-fetch-mode: cors
user-agent: node
accept-encoding: br, gzip, deflate
traceparent: 00-3229ee39e7d4886d979ce4843ea4b89d-d98a9da85188de98-01
Here we can see how traceparent http header is present twice, once from datadog and once from open telemetry, first one is from datadog and second one from open telemetry. (We can check that easily by converting the value of x-datadog-parent-id in hex, hex(3667278417456668462) = 32e4ca4a64f61f2e)
And based on the code that open telemetry has (hopefully I got it right):
extract(context: Context, carrier: unknown, getter: TextMapGetter): Context {
const traceParentHeader = getter.get(carrier, TRACE_PARENT_HEADER);
if (!traceParentHeader) return context;
const traceParent = Array.isArray(traceParentHeader)
? traceParentHeader[0]
: traceParentHeader;
It will not check traceParentHeader[1] in this particular case, where the header value added by open telemetry is.
In the same headers we can also notice how datadog also sends separately in specific http headers the trace id information
x-datadog-trace-id: 192969926701373593
x-datadog-parent-id: 3667278417456668462
Expected Result
The expected result would be for open telemetry to get the right traceparent value that was set by opentelemetry.
Actual Result
Open telemetry in this particular case seems to get the traceparent http header value that was set by datadog.
Additional Details
This issue is important for those that want to transition from datadog to opentelemetry and want to run both tools at the same time while the transition takes place.
OpenTelemetry Setup Code
No response
package.json
No response
Relevant log output
No response
It seems like there are two monitoring tools (opentelemetry and dd agent) in the same process. This is not supported by OpenTelemetry, nor is two traceparent headers (which is explicitly against the trace context spec). Please contact your vendor support for this issue.
This is the issue that I have, because there are 2 vendors, any solution for it? What vendor to contact?