google / flogger

Hi Floggers :)

I have some questions in relation to extending Flogger, while using Logback as the logging backend, and the resulting JSON log statement output.
This seemed like the right place since the Google Group does appear to be equally active.

I have looked through the issues here and I think my question somewhat overlaps with :

Question about log output format:

I have implemented an extension of the AbstractLogger which is working as it should - it has log APIs specific to my domain which is one of main drivers for using Flogger.
However, when I am logging in JSON format, I am getting log statements similar to this:

{
    "@timestamp":"2022-01-24T08:25:27.303+01:00",
    "@version":"1",
    "message":"This is the log statement with the argument value: '42' end of log statement. 
    [CONTEXT mySingleMetaKeyA=\"valueA\" myrepeatedKeyB=\"valueX\" myrepeatedKeyB=\"valueZ\"  myTimestampMS=1643009127300  ]",
    "logger_name":"mypackage.SpringBootApplication",
    "thread_name":"restartedMain","level":"ERROR","level_value":40000
}

Instead I would like to get the following JSON output for the log object:

{
    "@timestamp":"2022-01-24T08:25:27.303+01:00",
    "@version":"1",
    "message":"This is the log statement with the argument value: '42' end of log statement." ,
    "logger_name":"mypackage.SpringBootApplication",
    "thread_name":"restartedMain","level":"ERROR","level_value":40000,
    "metadata":{
        "mySingleMetaKeyA": "valueA",
        "myRepeatedKeyB":["valueX", "valueZ"],
        "myTimestampMS":1643009127300
    }
}

My goal is to have the metadata key/values and tags, which I have used in my custom log API with the with() method, extracted into either a metadata field containing the actual value pairs as shown above or to simply have all the entries placed in the root level of the log object.

The motivation being that this log statement is to be picked up by an Elastic Filebeat module and via Logstash end up in an ElasticSeach index - a very common scenario I would think.
And so, being able to build on top of the metadata key-values pairs for searching and filtering etc. makes it extremely useful to have these values not be a part of the actual log statement.

Any suggestions or hints as to I could accomplish this?

Question about the tags concept:

I have read the explanations about Metadata and Tags but I am however a bit confused about the use of the word Tag to describe a key/value pair.
I would have assumed a tag would be only a single value such as SystemYTag and could be included in the output like this:

{
    "logger_name":"mypackage.SpringBootApplication",
    "metadata":{
        "myTimestampMS":1643009127300
    },
    "tags": ["tagA", "tagB"]
}

I would think it could be possbile then, to use the term labels to describe those values but the label term is used as a name for MetadataKey names and as well in LoggingScope and ValueQueue to name a few.
Maybe LogLabels could be a candidate. :)

Environment info:

I am running the test using JDK 8, a Spring Boot service based on 2.2.9 and the Logback configuration files used in this Spring Boot version can be found here:
Spring Boot Logback config files.

I am using the following logging relevant libs/versions:

com.google.flogger:flogger:jar:0.7.4
com.google.flogger:flogger-system-backend:jar:0.7.4
com.google.flogger:flogger-slf4j-backend:jar:0.7.4
ch.qos.logback:logback-classic:jar:1.2.3:compile
net.logstash.logback:logstash-logback-encoder:jar:5.3:compile

I hope I am getting my use case and my point across clearly - otherwise please ask me to clarify further.

With regards to building the Flogger project, I also have a small question: How do I build it? :)
I do not have experience with Bazel but installed it and got it running.
However, I could no immediately see how I am supposed to build the entire project as I would normally do with a multi-module project in Maven.
Any hints would be much appreciated.
I would like to try out making some changes locally and so building the project and installing the jars locally will make that a bit easier. ;)

Also, if I were to post questions related to Flogger on StackOverflow, which tag(s) should I use?

Thank you in advance
Henrik

I forgot to mention that this blog entry provides some context, examples and motivation for applying what is referred to as Structured Logging:
https://www.innoq.com/en/blog/structured-logging/

First things first:
To build the entire project, "bazel build //..." should work from the top level directory.

As for "Tags", perhaps the name isn't perfect, but that's not changing anytime soon. A single "tag" is a key with a set of values. The idea is to record the sort of simple (not really structured) data people often want in logs (e.g. request IDs, task labels).

The trouble is that these are all in a global namespace. In library code you don't know if someone else already used the "id" tag you were thinking of using, and you don't want one user to override another by accident.

So the design is such that adding a key-value pair is an append only operation. If you add "id" -> X and someone else adds "id" -> Y, then "id" maps to both X and Y. So the way to think of tags is to be able to answer the question "what are the total set of unique key-value pairs added?", not "what is the single value for this key?".

Now, to the main event ...

Flogger has always been designed to support structured logging, and any lack of support in the default backends is purely because we've not seen people requesting that support to be built into the default behaviour.

It's really cool that you've got Flogger mostly working for your use case, and I'm glad to say that everything you need is already available if you're will to do a little bit of work on your end.

Firstly, the "[CONTEXT xxx ]" stuff is just what the default formatter does with a log statement. If you've implemented your own backend (yay!) then you just need to make it handle the LogData instances in a structured way. Sadly I've not had time to write a proper doc for this in open-source yet, but I think there's enough JavaDoc in the relevant classes to get you through.

I'm assuming from here that you don't intend to leave any metadata on the end of the message at all and you want to handle all of it in a structured (or at least semi structured) way. Basically you'll need to avoid using SimpleMessageFormatter (that's just a helper that get used for default backends). Your backend will need to handle the LogData directly (but it's not that hard).

The classes you are after are:
https://github.com/google/flogger/blob/master/api/src/main/java/com/google/common/flogger/backend/MetadataProcessor.java
https://github.com/google/flogger/blob/master/api/src/main/java/com/google/common/flogger/backend/MetadataHandler.java

You need a subclass of MetadataHandler<Context> (that's the code which knows what to do with metadata and it must be thread safe and stateless). This is where the logic for handling metadata and tags lives.

You need a Context type (your choice) which gets passed into the MetadateProcessor "process()" method. This is where per-log statement state lives and is tightly bound to your Handler implementation. This will typically be able to access things like a buffer in which to output its results.

Then get the LogData from the backend, obtain the MetadataProcessor instance from that (this is a view into the metadata for the log statement) and do something like.

    MyMetadataContext ctx = new MyMetadataContext(myStructuredOutputMap, ...);
    // This enumerates all the metadata and calls back the given handler methods with the given context.
    metadataProcessor.process(metadataHandler, ctx);
    ctx.myPostProcessingIfNecessary();  // optional

So your handler could, for example, put the metadata key/values into a JSON proxy instance, where the context passed in just the JSON proxy or a class which has access to it.

See similar code in:

flogger/api/src/main/java/com/google/common/flogger/backend/SimpleMessageFormatter.java

Line 184 in 5e41099

private final MetadataHandler<KeyValueHandler> handler =

For the raw message itself, just do:

BaseMessageFormatter.appendFormattedMessage(logData, buffer);

as that ignores all metadata.

Hope this helps (and feel free to add links to partially finished code if you want a 2nd look).

Cheers,
David

@hagbard - thank you for the prompt response.
Based on your input, I will look into it again once I have had time, and then I will get back to you.

Actually it's even easier that I suggest above since I wrote a helper class for builder metadata handlers.

https://github.com/google/flogger/blob/master/api/src/main/java/com/google/common/flogger/backend/MetadataKeyValueHandlers.java#L82

An example of what you might implement:

/** A handler (created once per log statement) to record metadata in a structured way. */
final class MyStructuredOutput implements KeyValueHandler {
  // Add whatever fields are needed here to store the key/value pairs in a structured way.

  // Setting the formatted log message.
  void setLogMessage(String message) { /* your code here */ }

  // Adding Metadata and key/value strings.
  void addMetadataLabelOnly(String label) { /* your code here */ }
  void addMetadataKeyValuePair(String label, String value) { /* your code here */ }

  // ... Other methods for recording values from the LogData instance.

  // Handles metadata and unwrapped tags as simple key/value pairs.
  @Override
  public void handle(String label, @NullableDecl Object value) {
    // A null value is possible and means we just record the label. Typically this only comes
    // from Tags (which are unwrapped by the emit() method of the LogContext#TAGS key).
    if (value == null) {
      addMetadataLabelOnly(label);
    } else {
      // For values of unknown types, the only really safe thing is to turn then into strings.
      // Values with known types (e.g. protocol buffers) could be handled specially here.
      // Known *keys* should be handled by an entirely separate MetadataHandler if they
      // are not being added to myMetadata (e.g. metadata which changes logging backend
      // behavior).
      addMetadataKeyValuePair(label, value.toString());
    }
  }
}

And then, somewhere in your LoggerBackend:

  // The set passed in is a set of key which we want to ignore. If there are keys we need to handle differently we can either
  // use MetadataHandler.Builder to add specific handlers or ignore keys here and make a 2nd custom handler.
  private static final MetadataHandler MY_HANDLER = MetadataKeyValueHandlers.getDefaultHandler(ImmutableSet.of());

  ...
  // Created once per log statement (or maybe reused in the ThreadLocal if you're careful to account for reentrant logging).
  MyStructuredOutput out = new MyStructuredOutput(...);

  // Set log message ...
  // How you treat the log message depends on whether you are storing a string in a data structure
  // or appending directly to a StringBuilder. I am assuming you are storing it in a field of some data structure.
  // If appending to an existing StringBuilder, only use appendFormattedMessage() and avoid a String copy.
  out.setLogMessage(logData.getTemplateContext() == null
      ? MessageUtils.safeToString(logData.getLiteralArgument())  // does nothing if the literal was a String.
      : BaseMessageFormatter.appendFormattedMessage(logData, new StringBuilder()).toString());

  // Add metadata ...
  // Created once per log statement as a combined view of log-site and context scoped metadata.
  // Can be reused in this log statement if you have multiple handlers. Do NOT store it elsewhere.
  MetadataProcessor metadata =
      MetadataProcessor.forScopeAndLogSite(Platform.getInjectedMetadata(), logData.getMetadata());
  MY_HANDLER.process(metadata, out);

Mapping MetadataKey and Tags to separate log objects fields and out of the log statement/message