[Bug]: Cannot send log to jaeger from a python application
FirelightFlagboy opened this issue · comments
What happened?
I'm trying to send logs from my python application to Jaeger
but it seems to be stuck when it send the first log line.
Steps to reproduce
I've used the following python code (inspired by otel logs collection yoda example)
import logging
import time
from opentelemetry import trace
from opentelemetry.sdk.resources import SERVICE_NAME, Resource
OTLP_GRPC_ENDPOINT = "localhost:4317"
resource = Resource.create({SERVICE_NAME: "testing-jaeger"})
def configure_logger(endpoint: str, resource: Resource) -> None:
import logging
from opentelemetry import _logs
from opentelemetry.exporter.otlp.proto.grpc._log_exporter import (
OTLPLogExporter,
)
from opentelemetry.sdk._logs import LoggerProvider, LoggingHandler
from opentelemetry.sdk._logs.export import SimpleLogRecordProcessor
provider = LoggerProvider(resource=resource)
_logs.set_logger_provider(provider)
processor = SimpleLogRecordProcessor(OTLPLogExporter(endpoint=endpoint, insecure=True))
provider.add_log_record_processor(processor)
handler = LoggingHandler(level=logging.DEBUG, logger_provider=provider)
logging.getLogger().addHandler(handler)
configure_logger(OTLP_GRPC_ENDPOINT, resource)
logger = logging.getLogger(__name__)
logger.setLevel(logging.DEBUG)
i = 0
try:
while True:
print(f"Logging {i=}")
logger.info(f"[{i=}] This is a test log message.")
print(f"Done logging {i=}")
i += 1
time.sleep(0.5)
except KeyboardInterrupt:
raise SystemExit(0)
When I execute the script with python
python test.py
I only have the first Logging i=0
printed and nothing, the program seems to be stuck (as Done logging i=0
is never printed)
Expected behavior
I expect jaeger to process the log events.
If I replace jaeger All in One
with otel/opentelemetry-collector
the script work but I would prefer to directly send the log events to jaeger
.
Relevant log output
Jaeger All in One log from docker compose
jaeger-1 | 2024-02-07T11:09:08.714988198Z 2024/02/07 11:09:08 maxprocs: Leaving GOMAXPROCS=16: CPU quota undefined
jaeger-1 | 2024-02-07T11:09:08.718266956Z 2024/02/07 11:09:08 application version: git-commit=b620f0eaefa5f17ec4bd01195149f158023a4e8e, git-version=v1.53.0, build-date=2024-01-08T18:05:40Z
jaeger-1 | 2024-02-07T11:09:08.718688076Z {"level":"info","ts":1707304148.718641,"caller":"flags/service.go:119","msg":"Mounting metrics handler on admin server","route":"/metrics"}
jaeger-1 | 2024-02-07T11:09:08.718697654Z {"level":"info","ts":1707304148.718671,"caller":"flags/service.go:125","msg":"Mounting expvar handler on admin server","route":"/debug/vars"}
jaeger-1 | 2024-02-07T11:09:08.718790508Z {"level":"info","ts":1707304148.7187724,"caller":"flags/admin.go:130","msg":"Mounting health check on admin server","route":"/"}
jaeger-1 | 2024-02-07T11:09:08.718815745Z {"level":"info","ts":1707304148.7187939,"caller":"flags/admin.go:144","msg":"Starting admin HTTP server","http-addr":":14269"}
jaeger-1 | 2024-02-07T11:09:08.718820595Z {"level":"info","ts":1707304148.7188053,"caller":"flags/admin.go:122","msg":"Admin server started","http.host-port":"[::]:14269","health-status":"unavailable"}
jaeger-1 | 2024-02-07T11:09:08.718901717Z {"level":"info","ts":1707304148.7188568,"caller":"grpc@v1.60.0/clientconn.go:429","msg":"[core][Channel #1] Channel created","system":"grpc","grpc_log":true}
jaeger-1 | 2024-02-07T11:09:08.718907638Z {"level":"info","ts":1707304148.7188892,"caller":"grpc@v1.60.0/clientconn.go:1724","msg":"[core][Channel #1] original dial target is: \"localhost:4317\"","system":"grpc","grpc_log":true}
jaeger-1 | 2024-02-07T11:09:08.718921834Z {"level":"info","ts":1707304148.7189007,"caller":"grpc@v1.60.0/clientconn.go:1731","msg":"[core][Channel #1] parsed dial target is: resolver.Target{URL:url.URL{Scheme:\"localhost\", Opaque:\"4317\", User:(*url.Userinfo)(nil), Host:\"\", Path:\"\", RawPath:\"\", OmitHost:false, ForceQuery:false, RawQuery:\"\", Fragment:\"\", RawFragment:\"\"}}","system":"grpc","grpc_log":true}
jaeger-1 | 2024-02-07T11:09:08.718929379Z {"level":"info","ts":1707304148.7189078,"caller":"grpc@v1.60.0/clientconn.go:1745","msg":"[core][Channel #1] fallback to scheme \"passthrough\"","system":"grpc","grpc_log":true}
jaeger-1 | 2024-02-07T11:09:08.718938215Z {"level":"info","ts":1707304148.7189176,"caller":"grpc@v1.60.0/clientconn.go:1753","msg":"[core][Channel #1] parsed dial target is: passthrough:///localhost:4317","system":"grpc","grpc_log":true}
jaeger-1 | 2024-02-07T11:09:08.718943365Z {"level":"info","ts":1707304148.7189252,"caller":"grpc@v1.60.0/clientconn.go:1886","msg":"[core][Channel #1] Channel authority set to \"localhost:4317\"","system":"grpc","grpc_log":true}
jaeger-1 | 2024-02-07T11:09:08.719062909Z {"level":"info","ts":1707304148.7190316,"caller":"grpc@v1.60.0/resolver_wrapper.go:196","msg":"[core][Channel #1] Resolver state updated: {\n \"Addresses\": [\n {\n \"Addr\": \"localhost:4317\",\n \"ServerName\": \"\",\n \"Attributes\": null,\n \"BalancerAttributes\": null,\n \"Metadata\": null\n }\n ],\n \"Endpoints\": [\n {\n \"Addresses\": [\n {\n \"Addr\": \"localhost:4317\",\n \"ServerName\": \"\",\n \"Attributes\": null,\n \"BalancerAttributes\": null,\n \"Metadata\": null\n }\n ],\n \"Attributes\": null\n }\n ],\n \"ServiceConfig\": null,\n \"Attributes\": null\n} (resolver returned new addresses)","system":"grpc","grpc_log":true}
jaeger-1 | 2024-02-07T11:09:08.719088517Z {"level":"info","ts":1707304148.7190528,"caller":"grpc@v1.60.0/balancer_wrapper.go:161","msg":"[core][Channel #1] Channel switches to new LB policy \"pick_first\"","system":"grpc","grpc_log":true}
jaeger-1 | 2024-02-07T11:09:08.719098275Z {"level":"info","ts":1707304148.7190785,"caller":"grpc@v1.60.0/balancer_wrapper.go:213","msg":"[core][Channel #1 SubChannel #2] Subchannel created","system":"grpc","grpc_log":true}
jaeger-1 | 2024-02-07T11:09:08.719108164Z {"level":"info","ts":1707304148.719087,"caller":"grpc@v1.60.0/clientconn.go:532","msg":"[core][Channel #1] Channel Connectivity change to CONNECTING","system":"grpc","grpc_log":true}
jaeger-1 | 2024-02-07T11:09:08.719113474Z {"level":"info","ts":1707304148.7190943,"caller":"grpc@v1.60.0/clientconn.go:335","msg":"[core][Channel #1] Channel exiting idle mode","system":"grpc","grpc_log":true}
jaeger-1 | 2024-02-07T11:09:08.719118193Z {"level":"info","ts":1707304148.7191036,"caller":"grpc@v1.60.0/clientconn.go:1223","msg":"[core][Channel #1 SubChannel #2] Subchannel Connectivity change to CONNECTING","system":"grpc","grpc_log":true}
jaeger-1 | 2024-02-07T11:09:08.719147748Z {"level":"info","ts":1707304148.7191117,"caller":"grpc@v1.60.0/clientconn.go:1338","msg":"[core][Channel #1 SubChannel #2] Subchannel picks a new address \"localhost:4317\" to connect","system":"grpc","grpc_log":true}
jaeger-1 | 2024-02-07T11:09:08.719186621Z {"level":"info","ts":1707304148.7191627,"caller":"memory/factory.go:85","msg":"Memory storage initialized","configuration":{"MaxTraces":0}}
jaeger-1 | 2024-02-07T11:09:08.719279625Z {"level":"info","ts":1707304148.7192621,"caller":"static/strategy_store.go:145","msg":"Loading sampling strategies","filename":"/etc/jaeger/sampling_strategies.json"}
jaeger-1 | 2024-02-07T11:09:08.719451107Z {"level":"warn","ts":1707304148.7194314,"caller":"grpc@v1.60.0/clientconn.go:1400","msg":"[core][Channel #1 SubChannel #2] grpc: addrConn.createTransport failed to connect to {Addr: \"localhost:4317\", ServerName: \"localhost:4317\", }. Err: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:4317: connect: connection refused\"","system":"grpc","grpc_log":true}
jaeger-1 | 2024-02-07T11:09:08.719462348Z {"level":"info","ts":1707304148.7194438,"caller":"grpc@v1.60.0/clientconn.go:1225","msg":"[core][Channel #1 SubChannel #2] Subchannel Connectivity change to TRANSIENT_FAILURE, last error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:4317: connect: connection refused\"","system":"grpc","grpc_log":true}
jaeger-1 | 2024-02-07T11:09:08.719472878Z {"level":"info","ts":1707304148.7194526,"caller":"grpc@v1.60.0/clientconn.go:532","msg":"[core][Channel #1] Channel Connectivity change to TRANSIENT_FAILURE","system":"grpc","grpc_log":true}
jaeger-1 | 2024-02-07T11:09:08.722421626Z {"level":"info","ts":1707304148.722374,"caller":"grpc@v1.60.0/server.go:667","msg":"[core][Server #3] Server created","system":"grpc","grpc_log":true}
jaeger-1 | 2024-02-07T11:09:08.722440822Z {"level":"info","ts":1707304148.7224193,"caller":"server/grpc.go:104","msg":"Starting jaeger-collector gRPC server","grpc.host-port":"[::]:14250"}
jaeger-1 | 2024-02-07T11:09:08.722452023Z {"level":"info","ts":1707304148.722425,"caller":"server/http.go:56","msg":"Starting jaeger-collector HTTP server","http host-port":":14268"}
jaeger-1 | 2024-02-07T11:09:08.722528006Z {"level":"info","ts":1707304148.7224956,"caller":"grpc@v1.60.0/server.go:867","msg":"[core][Server #3 ListenSocket #4] ListenSocket created","system":"grpc","grpc_log":true}
jaeger-1 | 2024-02-07T11:09:08.722538485Z {"level":"info","ts":1707304148.7225058,"caller":"app/collector.go:146","msg":"Not listening for Zipkin HTTP traffic, port not configured"}
jaeger-1 | 2024-02-07T11:09:08.722548885Z {"level":"info","ts":1707304148.722528,"caller":"handler/otlp_receiver.go:82","msg":"OTLP receiver status change","status":"StatusStarting"}
jaeger-1 | 2024-02-07T11:09:08.722554666Z {"level":"warn","ts":1707304148.7225347,"caller":"internal@v0.91.0/warning.go:40","msg":"Using the 0.0.0.0 address exposes this server to every network interface, which may facilitate Denial of Service attacks","documentation":"https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/security-best-practices.md#safeguards-against-denial-of-service-attacks"}
jaeger-1 | 2024-02-07T11:09:08.722583440Z {"level":"info","ts":1707304148.7225623,"caller":"grpc@v1.60.0/server.go:667","msg":"[core][Server #5] Server created","system":"grpc","grpc_log":true}
jaeger-1 | 2024-02-07T11:09:08.722590693Z {"level":"info","ts":1707304148.7225714,"caller":"otlpreceiver@v0.91.0/otlp.go:83","msg":"Starting GRPC server","endpoint":"0.0.0.0:4317"}
jaeger-1 | 2024-02-07T11:09:08.722623825Z {"level":"warn","ts":1707304148.7225924,"caller":"internal@v0.91.0/warning.go:40","msg":"Using the 0.0.0.0 address exposes this server to every network interface, which may facilitate Denial of Service attacks","documentation":"https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/security-best-practices.md#safeguards-against-denial-of-service-attacks"}
jaeger-1 | 2024-02-07T11:09:08.722634756Z {"level":"info","ts":1707304148.722605,"caller":"otlpreceiver@v0.91.0/otlp.go:101","msg":"Starting HTTP server","endpoint":"0.0.0.0:4318"}
jaeger-1 | 2024-02-07T11:09:08.722641679Z {"level":"info","ts":1707304148.7226117,"caller":"grpc@v1.60.0/server.go:867","msg":"[core][Server #5 ListenSocket #6] ListenSocket created","system":"grpc","grpc_log":true}
jaeger-1 | 2024-02-07T11:09:08.722649253Z {"level":"info","ts":1707304148.7226303,"caller":"grpc/builder.go:74","msg":"Agent requested insecure grpc connection to collector(s)"}
jaeger-1 | 2024-02-07T11:09:08.722665433Z {"level":"info","ts":1707304148.7226481,"caller":"grpc@v1.60.0/clientconn.go:429","msg":"[core][Channel #7] Channel created","system":"grpc","grpc_log":true}
jaeger-1 | 2024-02-07T11:09:08.722671234Z {"level":"info","ts":1707304148.7226553,"caller":"grpc@v1.60.0/clientconn.go:1724","msg":"[core][Channel #7] original dial target is: \"localhost:14250\"","system":"grpc","grpc_log":true}
jaeger-1 | 2024-02-07T11:09:08.722682816Z {"level":"info","ts":1707304148.722667,"caller":"grpc@v1.60.0/clientconn.go:1731","msg":"[core][Channel #7] parsed dial target is: resolver.Target{URL:url.URL{Scheme:\"localhost\", Opaque:\"14250\", User:(*url.Userinfo)(nil), Host:\"\", Path:\"\", RawPath:\"\", OmitHost:false, ForceQuery:false, RawQuery:\"\", Fragment:\"\", RawFragment:\"\"}}","system":"grpc","grpc_log":true}
jaeger-1 | 2024-02-07T11:09:08.722694939Z {"level":"info","ts":1707304148.7226744,"caller":"grpc@v1.60.0/clientconn.go:1745","msg":"[core][Channel #7] fallback to scheme \"passthrough\"","system":"grpc","grpc_log":true}
jaeger-1 | 2024-02-07T11:09:08.722709256Z {"level":"info","ts":1707304148.722681,"caller":"grpc@v1.60.0/clientconn.go:1753","msg":"[core][Channel #7] parsed dial target is: passthrough:///localhost:14250","system":"grpc","grpc_log":true}
jaeger-1 | 2024-02-07T11:09:08.722714105Z {"level":"info","ts":1707304148.7226853,"caller":"grpc@v1.60.0/clientconn.go:1886","msg":"[core][Channel #7] Channel authority set to \"localhost:14250\"","system":"grpc","grpc_log":true}
jaeger-1 | 2024-02-07T11:09:08.722740795Z {"level":"info","ts":1707304148.7227242,"caller":"grpc@v1.60.0/resolver_wrapper.go:196","msg":"[core][Channel #7] Resolver state updated: {\n \"Addresses\": [\n {\n \"Addr\": \"localhost:14250\",\n \"ServerName\": \"\",\n \"Attributes\": null,\n \"BalancerAttributes\": null,\n \"Metadata\": null\n }\n ],\n \"Endpoints\": [\n {\n \"Addresses\": [\n {\n \"Addr\": \"localhost:14250\",\n \"ServerName\": \"\",\n \"Attributes\": null,\n \"BalancerAttributes\": null,\n \"Metadata\": null\n }\n ],\n \"Attributes\": null\n }\n ],\n \"ServiceConfig\": null,\n \"Attributes\": null\n} (resolver returned new addresses)","system":"grpc","grpc_log":true}
jaeger-1 | 2024-02-07T11:09:08.722755101Z {"level":"info","ts":1707304148.7227395,"caller":"grpc@v1.60.0/balancer_wrapper.go:161","msg":"[core][Channel #7] Channel switches to new LB policy \"round_robin\"","system":"grpc","grpc_log":true}
jaeger-1 | 2024-02-07T11:09:08.722777443Z {"level":"info","ts":1707304148.7227626,"caller":"grpc@v1.60.0/balancer_wrapper.go:213","msg":"[core][Channel #7 SubChannel #8] Subchannel created","system":"grpc","grpc_log":true}
jaeger-1 | 2024-02-07T11:09:08.722799154Z {"level":"info","ts":1707304148.7227843,"caller":"base/balancer.go:182","msg":"[roundrobin]roundrobinPicker: Build called with info: {map[]}","system":"grpc","grpc_log":true}
jaeger-1 | 2024-02-07T11:09:08.722804875Z {"level":"info","ts":1707304148.7227921,"caller":"grpc@v1.60.0/clientconn.go:532","msg":"[core][Channel #7] Channel Connectivity change to CONNECTING","system":"grpc","grpc_log":true}
jaeger-1 | 2024-02-07T11:09:08.722815465Z {"level":"info","ts":1707304148.7227998,"caller":"grpc@v1.60.0/clientconn.go:335","msg":"[core][Channel #7] Channel exiting idle mode","system":"grpc","grpc_log":true}
jaeger-1 | 2024-02-07T11:09:08.722849839Z {"level":"info","ts":1707304148.7228215,"caller":"grpc@v1.60.0/clientconn.go:1223","msg":"[core][Channel #7 SubChannel #8] Subchannel Connectivity change to CONNECTING","system":"grpc","grpc_log":true}
jaeger-1 | 2024-02-07T11:09:08.722856982Z {"level":"info","ts":1707304148.722836,"caller":"grpc@v1.60.0/clientconn.go:1338","msg":"[core][Channel #7 SubChannel #8] Subchannel picks a new address \"localhost:14250\" to connect","system":"grpc","grpc_log":true}
jaeger-1 | 2024-02-07T11:09:08.722911525Z {"level":"info","ts":1707304148.7228804,"caller":"grpc/builder.go:115","msg":"Checking connection to collector"}
jaeger-1 | 2024-02-07T11:09:08.722919279Z {"level":"info","ts":1707304148.7228978,"caller":"grpc/builder.go:131","msg":"Agent collector connection state change","dialTarget":"localhost:14250","status":"CONNECTING"}
jaeger-1 | 2024-02-07T11:09:08.723028584Z {"level":"info","ts":1707304148.7230043,"caller":"grpc@v1.60.0/clientconn.go:1223","msg":"[core][Channel #7 SubChannel #8] Subchannel Connectivity change to READY","system":"grpc","grpc_log":true}
jaeger-1 | 2024-02-07T11:09:08.723048472Z {"level":"info","ts":1707304148.7230287,"caller":"base/balancer.go:182","msg":"[roundrobin]roundrobinPicker: Build called with info: {map[SubConn(id:8):{{Addr: \"localhost:14250\", ServerName: \"\", }}]}","system":"grpc","grpc_log":true}
jaeger-1 | 2024-02-07T11:09:08.723055004Z {"level":"info","ts":1707304148.7230377,"caller":"grpc@v1.60.0/clientconn.go:532","msg":"[core][Channel #7] Channel Connectivity change to READY","system":"grpc","grpc_log":true}
jaeger-1 | 2024-02-07T11:09:08.723064642Z {"level":"info","ts":1707304148.723044,"caller":"grpc/builder.go:131","msg":"Agent collector connection state change","dialTarget":"localhost:14250","status":"READY"}
jaeger-1 | 2024-02-07T11:09:08.723425058Z {"level":"info","ts":1707304148.7233608,"caller":"all-in-one/main.go:261","msg":"Starting agent"}
jaeger-1 | 2024-02-07T11:09:08.723506620Z {"level":"info","ts":1707304148.7234795,"caller":"app/agent.go:69","msg":"Starting jaeger-agent HTTP server","http-port":5778}
jaeger-1 | 2024-02-07T11:09:08.723565401Z {"level":"info","ts":1707304148.7235441,"caller":"grpc@v1.60.0/server.go:667","msg":"[core][Server #11] Server created","system":"grpc","grpc_log":true}
jaeger-1 | 2024-02-07T11:09:08.723769063Z {"level":"info","ts":1707304148.7237506,"caller":"app/static_handler.go:107","msg":"Using UI configuration","path":""}
jaeger-1 | 2024-02-07T11:09:08.723824186Z {"level":"info","ts":1707304148.7238073,"caller":"app/server.go:220","msg":"Query server started","http_addr":"[::]:16686","grpc_addr":"[::]:16685"}
jaeger-1 | 2024-02-07T11:09:08.723830578Z {"level":"info","ts":1707304148.7238145,"caller":"healthcheck/handler.go:129","msg":"Health Check state change","status":"ready"}
jaeger-1 | 2024-02-07T11:09:08.723835707Z {"level":"info","ts":1707304148.7238212,"caller":"app/server.go:300","msg":"Starting GRPC server","port":16685,"addr":":16685"}
jaeger-1 | 2024-02-07T11:09:08.723841969Z {"level":"info","ts":1707304148.723827,"caller":"grpc@v1.60.0/server.go:867","msg":"[core][Server #11 ListenSocket #12] ListenSocket created","system":"grpc","grpc_log":true}
jaeger-1 | 2024-02-07T11:09:08.723854643Z {"level":"info","ts":1707304148.7238393,"caller":"app/server.go:284","msg":"Starting HTTP server","port":16686,"addr":":16686"}
jaeger-1 | 2024-02-07T11:09:09.719619671Z {"level":"info","ts":1707304149.7195423,"caller":"grpc@v1.60.0/clientconn.go:1225","msg":"[core][Channel #1 SubChannel #2] Subchannel Connectivity change to IDLE, last error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:4317: connect: connection refused\"","system":"grpc","grpc_log":true}
jaeger-1 | 2024-02-07T11:09:09.719643065Z {"level":"info","ts":1707304149.719587,"caller":"grpc@v1.60.0/clientconn.go:1223","msg":"[core][Channel #1 SubChannel #2] Subchannel Connectivity change to CONNECTING","system":"grpc","grpc_log":true}
jaeger-1 | 2024-02-07T11:09:09.719650319Z {"level":"info","ts":1707304149.719595,"caller":"grpc@v1.60.0/clientconn.go:1338","msg":"[core][Channel #1 SubChannel #2] Subchannel picks a new address \"localhost:4317\" to connect","system":"grpc","grpc_log":true}
jaeger-1 | 2024-02-07T11:09:09.719933710Z {"level":"info","ts":1707304149.7198985,"caller":"grpc@v1.60.0/clientconn.go:1223","msg":"[core][Channel #1 SubChannel #2] Subchannel Connectivity change to READY","system":"grpc","grpc_log":true}
jaeger-1 | 2024-02-07T11:09:09.719946935Z {"level":"info","ts":1707304149.7199137,"caller":"grpc@v1.60.0/clientconn.go:532","msg":"[core][Channel #1] Channel Connectivity change to READY","system":"grpc","grpc_log":true}
Screenshot
No response
Additional context
No response
Jaeger backend version
1.53.0
SDK
Here the python dependencies I'm using:
python==3.9.18
opentelemetry-api==1.22.0
opentelemetry-exporter-otlp==1.22.0
opentelemetry-exporter-otlp-proto-common==1.22.0
opentelemetry-exporter-otlp-proto-grpc==1.22.0
opentelemetry-instrumentation==0.43b0
opentelemetry-instrumentation-logging==0.43b0
opentelemetry-proto==1.22.0
opentelemetry-sdk==1.22.0
Pipeline
OTEL SDK -> [OTEL] -> Jaeger All in One
Stogage backend
Memory
Operating system
Linux
Deployment model
Docker compose
Deployment configs
Docker compose:
services:
jaeger:
image: jaegertracing/all-in-one@sha256:1147036b9becc1eca6006f1af7eb2f0c2fe039a8d313ce15c6aa52a6a877f9f7 # jaegertracing/all-in-one:1.53.0
restart: on-failure
ports:
- 16686:16686
- 4317:4317
- 4318:4318
environment:
LOG_LEVEL: debug
COLLECTOR_OTLP_ENABLED: "true"
Jaeger is a tracing backend, it does not collect logs.