Exemplar with data from header?
backbord opened this issue · comments
Hi,
I would like to add a trace id from a request as an exemplar, if that makes sense.
The exemplar callback however seems to be called w/o arguments in
Would it be possible, to pass the request to the exemplar callback, allowing for
def my_exemplars(request: Request) -> dict[str, str]:
return {"trace_id": request.headers.get("Trace-Id", "")}
...
exemplars=my_exemplars
...or even have individual fields be callbacks like it is done with labels in
exemplars=lambda: {"trace_id": from_header("Trace-Id")}?
Any help would be greatly appreciated.
PS: I'm new to exemplars and might be misinterpreting something. :)
Are you using OpenTelemetry for tracing? If so, see this example app (most of this is just otel boilerplate, but see the get_trace_id function and PrometheusMiddleware config):
from fastapi import FastAPI
from opentelemetry import trace
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.instrumentation.fastapi import FastAPIInstrumentor
from starlette_exporter import PrometheusMiddleware, handle_openmetrics
trace.set_tracer_provider(TracerProvider(resource=Resource.create({"service.name": "exemplar-test"})))
tracer = trace.get_tracer_provider().get_tracer(__name__)
app = FastAPI()
def get_trace_id():
trace_id = trace.get_current_span().get_span_context().trace_id
return hex(trace_id)[2:]
app.add_middleware(
PrometheusMiddleware,
exemplars=lambda: {"trace_id": get_trace_id()} # the function above
)
@app.get("/")
async def index():
with tracer.start_as_current_span("index"):
return {"message": get_trace_id()}
app.add_route("/metrics", handle_openmetrics, ["GET"])
FastAPIInstrumentor.instrument_app(app)However since you mention a Trace-Id header I'm wondering if maybe you're using something different. Let me know if that's the case and we can work out a way to pass the request object into the exemplar callback.
Thanks for the quick reply and aplogies for my late reply!
I intended to use a HTTP header that the client (or the app) can set for tracing a request (over multiple services in metrics and/or logs), and I was under the impression, that it might be a good fit for exemplars.
So my use case would be:
- Client sends a request with a "Trace-Id" header to a service (sometimes called "X-Request-ID" or "X-Trace-ID" or "X-Correlation-ID")
- The trace id may be an arbitrary string but should be unique, often a uuid 4 string.
- The service handles the request:
- When logging, the trace id is added to the log.
- => When collecting metrics, the trace id should be attached to the metrics for that request, if possible
- If the service issues requests to other services itself, it might choose to send the same trace id to trace the requests context over multiple services. (As far as I understand, this would be a new trace id within the same span id in OpenTelemetry.)
- The trace id is repeated in the response header.
It would be great if I could use exemplars with such a header value.
However, I'm not using OpenTelemetry so far and might confuse OpenTelemetry specific traces (or the new HTTP trace context headers?) with those non-standard correlation ID HTTP headers (see also: The Value of Correlation IDs).
That makes sense. It sounds like access to the request object (like you originally said 😄 ) is needed to pull the value from headers. My biggest concern would be adding the request to callbacks might break some existing users' callback functions, but I will see if I can come up with. Any ideas welcome.
As for an alternative solution, I don't have experience with this library so this is not a recommendation but I have come across this https://github.com/tomwojcik/starlette-context