Logfmt
ad-m opened this issue · comments
Hello,
Thank for you project. It is very helpful quickstarter!
I notice that logs created by NodeJS application is not properly parsed by promtail, so derrived fields is not work. What do you think about use of logfmt which is supported by logfmt? Loki support that format. Despite of structure it's easy to read.
Kind regards
Thanks for your suggestion!
Appreciate if you could describe with an example what is expected and what is currently happening.
I am discovering how logs might match and correlate. It's to early to provide implementation. I notice that format of traceId is incosistent, so promtail does not parse it and Grafana does not link it.
Runtime logger ( https://github.com/mnadeem/nodejs-opentelemetry-tempo/blob/main/src/server.js#L65 ):
logger.info('doSomeWorkInNewNested2Span traceId %s:%s', childSpan.context().traceId, childSpan.context().spanId);
Access logger ( https://github.com/mnadeem/nodejs-opentelemetry-tempo/blob/main/src/log_config.js#L45 ):
format: (req, res, format) => format(`:remote-addr - traceId: ${req.traceId} - ":method :url HTTP/:http-version" :status :content-length ":referrer" ":user-agent"`)
Promtail ( https://github.com/mnadeem/nodejs-opentelemetry-tempo/blob/main/etc/promtail-local.yaml#L26 ):
expression: '^(?P<timestamp>\d{4}-\d{2}-\d{2}\s\d{1,2}\:\d{2}\:\d{2}\.\d{3})\s+(?P<level>[A-Z]{4,5})\s[(?P<serviceName>.*),(?P<traceId>.*),(?P<spanId>.*)]\s(?P<pid>\d)\s---\s[\s*(?P<thread>.*)]\s(?P<logger>.*)\s+\:\s(?P<message>.*)$'
Grafana datasource ( https://github.com/mnadeem/nodejs-opentelemetry-tempo/blob/main/data/grafana-data/datasources/datasource.yml#L58 ) :
matcherRegex: (?:traceID|trace_id)=(\w+)
(Notice, traceID used in that expession does not match to traceId used in logs)
My idea is to unify format of logs to allows jump from logs to traces and back.
Thanks! very good suggestion to unify logging, I will change things!
FYI: I have unified things for Java implementation https://reachmnadeem.wordpress.com/2021/03/22/integrating-loki-with-tempo-using-grafana/
Done!
Will try to implement it in even better way, later