auto-configuration not working
cwrau opened this issue · comments
Describe the bug
I have no OTEL_EXPORTER_OTLP*_ENDPOINT
variables set, yet after starting up an exporter is created and logs errors;
Failed to export spans. Server responded with gRPC status code 2. Error message: Failed to connect to localhost/[0:0:0:0:0:0:0:1]:4317
Steps to reproduce
Use the AutoConfiguredOpenTelemetrySdk
What did you expect to see?
Opentelemetry should not do anything when not configured to do so.
What did you see instead?
Opentelemetry tried to send spans somewhere
What version and what artifacts are you using?
implementation(platform("io.opentelemetry:opentelemetry-bom:1.37.0"))
implementation(platform("io.opentelemetry:opentelemetry-bom-alpha:1.37.0-alpha"))
implementation("io.opentelemetry", "opentelemetry-api")
implementation("io.opentelemetry", "opentelemetry-extension-kotlin")
implementation("io.opentelemetry", "opentelemetry-sdk-extension-autoconfigure")
implementation("io.opentelemetry.instrumentation", "opentelemetry-okhttp-3.0", "2.3.0-alpha")
implementation("io.opentelemetry.instrumentation", "opentelemetry-jdbc", "2.3.0-alpha")
Environment
Compiler: openjdk version "17.0.10" 2024-01-16
Additional context
Current workaround is to check various ENV vars for their existence and set "otel.traces.exporter" to "none"
if not set, but that defeats the purpose of auto-configuration
To give a deeper explanation, by default, if the property "otel.traces.exporter" is not set, otlp is being used
But if I set "otel.traces.exporter" to "none" in code, as a sane default, setting "otel.exporter" to "otlp" doesn't work during deployment.
Same goes for metrics, setting it to "prometheus" doesn't allow "otel.exporter" to be used for this.
I understand that the more specific settings are for granular overrides, but how should I configure metrics to use prometheus by default, traces to not be sent by default and then use the same backend, via "otel.exporter", for both during deployment?
This behavior is as specified in the specification, and hence is not a bug. (I'll remove the label).
I think the way to achieve what you want is to just make sure that you are setting your properties to what you want in all cases, rather than trying to change the specified auto-configuration default to one that gets you into this jam.
I think the way to achieve what you want is to just make sure that you are setting your properties to what you want in all cases, rather than trying to change the specified auto-configuration default to one that gets you into this jam.
That's what I thought I did, setting the default to none so the deployer can override it if they want, but I don't see a good way to do it?
My end goal would be the following;
- By default nothing will be sent (to nowhere obviously) and especially no logs will be produced by this.
- This specifically needs
otel.traces.exporter
set tonone
.
- This specifically needs
- The deployer can set
OTEL_EXPORTER
tootlp
andOTEL_EXPORTER_ENDPOINT
to the specific endpoint and everything works.- This won't override the specific setting.
This looks like it's impossible to do, but if you have a hint as to how I can do this in a way I missed, I'd love to hear it.
I think the way to achieve what you want is to just make sure that you are setting your properties to what you want in all cases, rather than trying to change the specified auto-configuration default to one that gets you into this jam.
That's what I thought I did, setting the default to none so the deployer can override it if they want, but I don't see a good way to do it?
No, what I'm saying is to not try to change the "default", but make sure you set what you want in all cases. By changing the default, you're changing the behavior in ways that won't support what you want.
I think the way to achieve what you want is to just make sure that you are setting your properties to what you want in all cases, rather than trying to change the specified auto-configuration default to one that gets you into this jam.
That's what I thought I did, setting the default to none so the deployer can override it if they want, but I don't see a good way to do it?
No, what I'm saying is to not try to change the "default", but make sure you set what you want in all cases. By changing the default, you're changing the behavior in ways that won't support what you want.
But the default is not working and logging errors.
I may be stupid, hopefully!, but could you explain to me how to achieve my desired setup;
- by default, don't send anything, it's not going to work anyways
- open prometheus port for metrics
- don't do anything with logs
- allow the ops person to set a single, 3 at max, env vars to configure the whole tracing, logging and metrics stuff
- maybe also just for one of those
does this approach work for you? open-telemetry/opentelemetry-java-contrib#1276
does this approach work for you? open-telemetry/opentelemetry-java-contrib#1276
Not really, by setting the default to none
the setting OTEL_EXPORTER
and OTEL_EXPORTER_OTLP_ENDPOINT
don't work anymore, the ops person has to set OTEL_{TRACES,METRICS,LOGGING}_EXPORTER
and OTEL_EXPORTER_OTLP_{TRACES,METRICS,LOGGING}_ENDPOINT
, 6 env vars in total.
The only thing I can think of to achieve the thing I want would be to check a couple env vars in our code, to decide if we want to set the default to none
or not, which of course is stupid when using a "autoconfigure" library.
To be even more precise, I want the following setup to work;
- by default don't to anything
- ops person can set
OTEL_EXPORTER=otlp
andOTEL_EXPORTER_OTLP_ENDPOINT=...
, that's it
Currently, we're lucky that we only have traces enabled for otlp, as we do the following;
private val tracingConfigured = System.getenv("OTEL_EXPORTER_OTLP_TRACES_ENDPOINT").isNullOrBlank().not()
[...]
val properties = mapOf(
"otel.metrics.exporter" to "prometheus",
"otel.exporter.prometheus.port" to "9090",
"otel.logs.exporter" to "none",
)
if (!tracingConfigured) {
logger.info { "tracingEndpoint not set, not sending traces." }
properties + ("otel.traces.exporter" to "none")
} else {
properties
}
but as I said, this defeats a part of "autoconfigure"
I was saying that you could just set all 3 signal-specific env vars always, and not try to coerce a "default" value if some particular env var isn't set.
I was saying that you could just set all 3 signal-specific env vars always, and not try to coerce a "default" value if some particular env var isn't set.
So you're saying what I want is not possible, I understand.
If that is the desired situation, then I guess this can be closed?
I believe that what you want is not in the semantics of the auto-configure module, no. If providing default properties won't work for you, then I think you might be out of luck. It doesn't seem (to me) that setting a few extra env vars should be too big a burden for an SRE running operations.
It doesn't seem (to me) that setting a few extra env vars should be too big a burden for an SRE running operations.
Of course, it's nothing big, it's just annoying to have to set half a dozen env vars, if 2 could suffice (and the docs make it seem that they should)
And, everything that is written has to be maintained, checked in the future, can have typos, ...
I just want to make it as easy as possible for everyone involved