Giters

open-telemetry / opentelemetry-java

OpenTelemetry Java SDK

Home Page:https://opentelemetry.io

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

auto-configuration not working

cwrau opened this issue · comments

commented

Describe the bug
I have no OTEL_EXPORTER_OTLP*_ENDPOINT variables set, yet after starting up an exporter is created and logs errors;

Failed to export spans. Server responded with gRPC status code 2. Error message: Failed to connect to localhost/[0:0:0:0:0:0:0:1]:4317

Steps to reproduce
Use the AutoConfiguredOpenTelemetrySdk

What did you expect to see?
Opentelemetry should not do anything when not configured to do so.

What did you see instead?
Opentelemetry tried to send spans somewhere

What version and what artifacts are you using?

    implementation(platform("io.opentelemetry:opentelemetry-bom:1.37.0"))
    implementation(platform("io.opentelemetry:opentelemetry-bom-alpha:1.37.0-alpha"))
    implementation("io.opentelemetry", "opentelemetry-api")
    implementation("io.opentelemetry", "opentelemetry-extension-kotlin")
    implementation("io.opentelemetry", "opentelemetry-sdk-extension-autoconfigure")
    implementation("io.opentelemetry.instrumentation", "opentelemetry-okhttp-3.0", "2.3.0-alpha")
    implementation("io.opentelemetry.instrumentation", "opentelemetry-jdbc", "2.3.0-alpha")

Environment
Compiler: openjdk version "17.0.10" 2024-01-16

Additional context
Current workaround is to check various ENV vars for their existence and set "otel.traces.exporter" to "none" if not set, but that defeats the purpose of auto-configuration

commented

To give a deeper explanation, by default, if the property "otel.traces.exporter" is not set, otlp is being used
But if I set "otel.traces.exporter" to "none" in code, as a sane default, setting "otel.exporter" to "otlp" doesn't work during deployment.
Same goes for metrics, setting it to "prometheus" doesn't allow "otel.exporter" to be used for this.

I understand that the more specific settings are for granular overrides, but how should I configure metrics to use prometheus by default, traces to not be sent by default and then use the same backend, via "otel.exporter", for both during deployment?

commented

This behavior is as specified in the specification, and hence is not a bug. (I'll remove the label).

I think the way to achieve what you want is to just make sure that you are setting your properties to what you want in all cases, rather than trying to change the specified auto-configuration default to one that gets you into this jam.

commented

I think the way to achieve what you want is to just make sure that you are setting your properties to what you want in all cases, rather than trying to change the specified auto-configuration default to one that gets you into this jam.

That's what I thought I did, setting the default to none so the deployer can override it if they want, but I don't see a good way to do it?

My end goal would be the following;

  • By default nothing will be sent (to nowhere obviously) and especially no logs will be produced by this.
    • This specifically needs otel.traces.exporter set to none.
  • The deployer can set OTEL_EXPORTER to otlp and OTEL_EXPORTER_ENDPOINT to the specific endpoint and everything works.
    • This won't override the specific setting.

This looks like it's impossible to do, but if you have a hint as to how I can do this in a way I missed, I'd love to hear it.

commented

I think the way to achieve what you want is to just make sure that you are setting your properties to what you want in all cases, rather than trying to change the specified auto-configuration default to one that gets you into this jam.

That's what I thought I did, setting the default to none so the deployer can override it if they want, but I don't see a good way to do it?

No, what I'm saying is to not try to change the "default", but make sure you set what you want in all cases. By changing the default, you're changing the behavior in ways that won't support what you want.

commented

I think the way to achieve what you want is to just make sure that you are setting your properties to what you want in all cases, rather than trying to change the specified auto-configuration default to one that gets you into this jam.

That's what I thought I did, setting the default to none so the deployer can override it if they want, but I don't see a good way to do it?

No, what I'm saying is to not try to change the "default", but make sure you set what you want in all cases. By changing the default, you're changing the behavior in ways that won't support what you want.

But the default is not working and logging errors.

I may be stupid, hopefully!, but could you explain to me how to achieve my desired setup;

  • by default, don't send anything, it's not going to work anyways
    • open prometheus port for metrics
    • don't do anything with logs
  • allow the ops person to set a single, 3 at max, env vars to configure the whole tracing, logging and metrics stuff
    • maybe also just for one of those
commented

does this approach work for you? open-telemetry/opentelemetry-java-contrib#1276

Not really, by setting the default to none the setting OTEL_EXPORTER and OTEL_EXPORTER_OTLP_ENDPOINT don't work anymore, the ops person has to set OTEL_{TRACES,METRICS,LOGGING}_EXPORTER and OTEL_EXPORTER_OTLP_{TRACES,METRICS,LOGGING}_ENDPOINT, 6 env vars in total.

The only thing I can think of to achieve the thing I want would be to check a couple env vars in our code, to decide if we want to set the default to none or not, which of course is stupid when using a "autoconfigure" library.

To be even more precise, I want the following setup to work;

  • by default don't to anything
  • ops person can set OTEL_EXPORTER=otlp and OTEL_EXPORTER_OTLP_ENDPOINT=..., that's it

Currently, we're lucky that we only have traces enabled for otlp, as we do the following;

    private val tracingConfigured = System.getenv("OTEL_EXPORTER_OTLP_TRACES_ENDPOINT").isNullOrBlank().not()
    [...]
                val properties = mapOf(
                    "otel.metrics.exporter" to "prometheus",
                    "otel.exporter.prometheus.port" to "9090",
                    "otel.logs.exporter" to "none",
                )

                if (!tracingConfigured) {
                    logger.info { "tracingEndpoint not set, not sending traces." }
                    properties + ("otel.traces.exporter" to "none")
                } else {
                    properties
                }

but as I said, this defeats a part of "autoconfigure"

commented

I was saying that you could just set all 3 signal-specific env vars always, and not try to coerce a "default" value if some particular env var isn't set.

commented

I was saying that you could just set all 3 signal-specific env vars always, and not try to coerce a "default" value if some particular env var isn't set.

So you're saying what I want is not possible, I understand.

If that is the desired situation, then I guess this can be closed?

commented

I believe that what you want is not in the semantics of the auto-configure module, no. If providing default properties won't work for you, then I think you might be out of luck. It doesn't seem (to me) that setting a few extra env vars should be too big a burden for an SRE running operations.

commented

It doesn't seem (to me) that setting a few extra env vars should be too big a burden for an SRE running operations.

Of course, it's nothing big, it's just annoying to have to set half a dozen env vars, if 2 could suffice (and the docs make it seem that they should)

And, everything that is written has to be maintained, checked in the future, can have typos, ...

I just want to make it as easy as possible for everyone involved