Parsing JSON formatted field on OCP 4.6
gnnivlek opened this issue · comments
Hi, is it possible to split a JSON formatted field into separated fields?
For instance, I have a field called message with value like this:
{"level": "ACCESS", "response_code":401, url:"/login"}
I would like to have a separated fields, such as message.level, message.response_code and message.url instead of just one message field. Can I achieve this using cluster logging in OCP 4.6? Thank you.
The config is close, but no cigar:
<source>
@type tail
@id container-input
path "/var/log/containers/*.log"
exclude_path ["/var/log/containers/fluentd-*_openshift-logging_*.log", "/var/log/containers/elasticsearch-*_openshift-logging_*.log", "/var/log/containers/kibana-*_openshift-logging_*.log"]
pos_file "/var/log/es-containers.log.pos"
refresh_interval 5
rotate_wait 5
tag kubernetes.*
read_from_head "true"
@label @MEASURE
<parse>
@type multi_format
<pattern>
format json
time_format '%Y-%m-%dT%H:%M:%S.%N%Z'
keep_time_key true
</pattern>
<pattern>
format regexp
expression /^(?<time>.+) (?<stream>stdout|stderr)( (?<logtag>.))? (?<log>.*)$/
time_format '%Y-%m-%dT%H:%M:%S.%N%:z'
keep_time_key true
</pattern>
</parse>
</source>
but since output will look like:
2021-05-03T18:22:04.809311738+02:00 stdout F {"@timestamp":"2021-05-03T18:22:04.807+02:00","@version":"1","message":"Post-processing PropertySource instances","logger_name":"com.ulisesbocchio.jasyptspringboot.configuration.EnableEncryptablePropertiesBeanFactoryPostProcessor","thread_name":"main","level":"INFO","level_value":20000}
it doesn't match the json pattern :(
https://access.redhat.com/solutions/5045721 would solve it AFAIK.
This requires you to set the operator into unmanaged, edit the config to parse the message. The functionality to support JSON log messages is being addressed with the release of logging 5.1 in epic https://issues.redhat.com/browse/LOG-785 This version of logging is only available on OCP 4.7 or greater