Making Distributed Tracing configurable
cmillar-salesloft opened this issue · comments
Is your feature request related to a problem? Please describe.
There appears to be no way to disable Distributed Tracing via the agent's configuration. Not all of our applications need Distributed Tracing so we'd like to disable it where it is not used, given the additional costs.
Describe the solution you'd like
Other New Relic language agents allow for configuring whether Distributed Tracing is on or off. For example the Ruby agent allows clients to disable Distributed Tracing via an environment variable NEW_RELIC_DISTRIBUTED_TRACING_ENABLED, docs here. It would be great if the Elixir agent provided a way to configure this as well.
Describe alternatives you've considered
N/A
Have you tried just commenting out this line? You can fork and try it in a test enviroment and see if that does the trick for you. Please let us know.
defmodule MyExternalService do
use NewRelic.Tracer
@trace {:request, category: :external}
def request(method, url, headers) do
NewRelic.set_span(:http, url: url, method: method, component: "HttpClient")
headers = headers ++ NewRelic.distributed_trace_headers(:http)
HttpClient.request(method, url, headers)
end
end
or try this option,
in the New Relic Elixir Agent, you can update your application's configuration file. In your config.exs or the appropriate environment-specific configuration file, add the following configuration:
elixir
config :new_relic_agent,
distributed_tracing_enabled: false
This will disable distributed tracing for the New Relic Elixir Agent. Make sure to restart your application after making the changes for them to take effect.
@nvira08 I'm not seeing that config as an option.
@cmillar-salesloft Are you seeing distributed trace data coming to NewRelic? I'm not seeing anything automatic about it (except in the NewRelic.Instrumented.HTTPoison module). As @nvira08 mentions, if you don't add the distributed_trace_headers to your requests, I'm not seeing it happen automatically.
regarding existing configuration options:
I concur that we don't see
config :new_relic_agent,
distributed_tracing_enabled: false
as an existing config option.
Though there are 2 other options related to Distributed Tracing/Spans that I saw
- One for choosing to use infinite_tracing vs sampling
- I also found a
span_event_per_minute, I tried setting that to 0 to see if I could effectively disable sample based span reporting but it didn't work (and wouldn't be the right solution anyways)
regarding data we're seeing recorded in NewRelic
Are you seeing distributed trace data coming to NewRelic?
Yes, we're seeing non-http categorySpans reported for elixir applications. For example, we see generic and datastore category Spans reported with only harvest_enabled, app_name, and license_key agent configuration.
- generic -> Transaction Root Process
- generic -> Process
- datastore -> Datastore/statement/Postgres/:table/:action
I don't think datastore and generic/process are distributed traces (distributed traces correlate across services). That's transactions you're seeing AFAIK. Without that … what are you using NR's agent for?
If you don't want datastore you can disable it with https://hexdocs.pm/new_relic_agent/NewRelic.Config.html#feature?/1-instrumentation
config :new_relic_agent, ecto_instrumentation_enabled: false
Distributed tracing tracks and observes service requests as they flow through distributed systems.
https://docs.newrelic.com/docs/distributed-tracing/concepts/introduction-distributed-tracing/
Thanks Tony,
I'm one of Chris's coworkers. It's possible we're getting the terminology wrong.
Our "data ingested" chart has two series, one for "APM events" and another for "Tracing".
We were able to reduce the "Tracing" component for our Ruby services by setting NEW_RELIC_DISTRIBUTED_TRACING_ENABLED="false" in the environment. When we did so, we still got performance data in the Transaction table in New Relic but no longer saw anything in the Span table.
Currently in Elixir, we're seeing both Span and Transaction data and still have "Tracing" data ingest. Our goal is to stop ingesting the "Tracing" data since we don't use it. We've been assuming it was distributed tracing based on the Ruby libraries variable name.
No such config to disable is available.
But, as I've said, I'm not seeing much if anything that is automatically creating spans.
If you've removed @trace, that's the most likely thing creating spans.
But that's about the limit of my knowledge at this time.
Feel free to look at the code. PRs welcome.