ruanbekker / grafana-tempo-loki-tracing

Grafana Distributed Tracing Example with: Tempo, Prometheus, Loki, Grafana and Python Flask

grafana-tempo-loki-tracing

Grafana Distributed Tracing Example with: Tempo, Prometheus, Loki, Grafana and Python Flask

Getting Started

Boot the stack:

docker compose up --build -d || docker-compose up -d

The datasources will be auto-configured defined in configs/grafana/datasources.yaml.

Example Requests

Requests:

$ curl http://localhost:5002/process_payment
Payment processed 

$ curl http://localhost:5001/create_order
Order created with id 1 

$ curl http://localhost:5001/list_payments
Payments listed 

Logs:

order-service  | [2023-11-30 05:33:07,334] INFO in app: Trace ID: 1660e64b3807719aa4898445766895b8
order-service  | 172.18.0.1 - - [30/Nov/2023 05:33:07] "GET /create_order HTTP/1.1" 200 -
payment-service  | [2023-11-30 05:33:18,883] INFO in app: Trace ID: 335c0cd1cd947c3de92b7cc9a06386e9
payment-service  | 172.18.0.1 - - [30/Nov/2023 05:33:18] "GET /process_payment HTTP/1.1" 200 -

Screenshots

Explore traces:

Query:

Node Graph:

When we use the payment-service container logs in Loki:

Extras

With span.set_attribute we can enrich some of the visuals:

        with tracer.start_as_current_span("database_operation") as span:
            sql_query = "INSERT INTO order (description) VALUES ('New Order')"
            span.set_attribute("db.statement", sql_query)  # Add SQL query to span

Produces: