nigelsim / mdc4spring

Declarative Logging Mapping Diagnostic Context for Spring.

MDC4Spring

Simple and secure management of Mapped Diagnostic Contexts (MDCs) for multiple logging systems with Spring AOP. Supported logging systems: Slf4J/Logback, Log4J, Log4J2.

Reasoning

Mapped Diagnostic Context (MDC) provides a way to enrich log traces with contextual information about the execution scope. Instead of including business parameters manually in each log message, we may consider as a good practice to provide them inside the MDC associated with the trace. This could be especially useful when you are using ELK stack, DataDog or any other log aggregation system to track the execution of your business flows.

Example of Logback logging trace that contains MDC with business data formatted by JSON-based appender:

{
  "timestamp": "2022-07-26 09:01:51.482",
  "level": "INFO",
  "thread": "main",
  "mdc": {
    "requestUuid": "b8e0cd40-0cc6-11ed-861d-0242ac120002",
    "sourceIp": "127.0.0.1",
    "instance": "instance1.mybusinessdomain:8080",
    "action" : "createOrder",
    "order.transactionId": "184325928574329523",
    "order.clientId": "web-57961e5e0242ac120002",
    "order.customerId": "A123456789",
    "order.assigneeUserId": "192385",
    "order.id": "2349682"
  },
  "logger": "com.github.throwable.mdc4springdemo.ShippingOrderController",
  "message": "Order created",
  "context": "default"
}

Different logging libraries provide a way of setting MDC parameters using thread-local context that require a user to manually control the proper cleanup of them when the execution flow leaves the scope. So if one forgets to clean them properly it may later pollute log messages with outdated information or even provoke memory leaks in thread-pooled environments.

The idea is to use annotations that in a declarative way define method’s arguments to be included as MDC parameters when the method is executed, and automatically manage MDC scopes performing their correct cleanup.

Example

class OrderProcessor {
    // define a new MDC that will be cleared automatically when the method returns
    @WithMDC
    public void assignToUser(
            // add method arguments with their values to current MDC
            @MDCParam String orderId, @MDCParam String userId
    ) {
        // programmatically add a new parameter to current MDC
        MDC.param("transactionId", this.getCurrentTransactionId());

        // business logic...

        // The MDC of the log message will contain parameters: orderId, userId, transactionId 
        log.info("User was successfully assigned to order");
        // All these parameters defined within the scope of the method
        //  will automatically be removed when the method returns
    }
}

(back to top)

Getting Started

Prerequisites

The library works with Java 8+ and Spring Boot or Spring Framework environments (does not rely on any specific version). Currently, it supports Slf4J/Logback, Log4j2 and Log4j logging systems. By default, the logging system is detected using classpath library resolution, but you can change this behavior setting com.github.throwable.mdc4spring.loggers.LoggerMDCAdapter system property to LoggerMDCAdapter implementation class.

Installation

Add the following dependency to your project's build file.

Maven

<dependency>
    <groupId>com.github.throwable.mdc4spring</groupId>
    <artifactId>mdc4spring</artifactId>
    <version>1.0</version>
</dependency>

Gradle

dependencies {
    compile("com.github.throwable.mdc4spring:mdc4spring:1.0")
}

If you are using Spring Boot the configuration will be applied automatically by autoconfiguration mechanism. In case if you are not using Spring Boot you have to import com.github.throwable.mdc4spring.spring.MDCAutoConfiguration manually.

Probably you also need to configure your log subsystem to see MDC parameters in your log traces. See an example of JSON-based appender.

(back to top)

Usage

Method argument parameters

Simple usage: log messages will contain MDC with orderId and userId parameters that will automatically be removed after the method returns.

class OrderProcessor {
    @WithMDC
    public void assignToUser(@MDCParam String orderId, @MDCParam String userId) {
        // business logic...
        log.info("User assigned to order");
    }
}

By default, argument names will be used as parameter names, see considerations, but you can also define custom names for them.

class OrderProcessor {
    @WithMDC
    public void assignToUser(@MDCParam("order.id") String orderId, @MDCParam("user.id") String userId) {
        log.info("User assigned to order");
    }
}

The value of any argument can be converted inline using Spring expression language. In this case order.id and user.id parameters will be set to order.getId() and user.getId() respectively. Note that if any error occurs during the expression evaluation the execution will not be interrupted and a parameter value will be set to a string like #EVALUATION ERROR#: exception message.

class OrderProcessor {
    @WithMDC
    public void assignToUser(
            @MDCParam(name = "order.id", eval = "id") Order order,
            @MDCParam(name = "user.id", eval = "id") User user)
    {
        log.info("User assigned to order");
    }
}

Additional parameters

A set of additional MDC parameters can be defined for any method, whose values are evaluated and added to current MDC during each method execution. The eval attribute defines a SpEL expression that can contain references to the evaluation context.

class RequestProcessor {
    @WithMDC
    @MDCParam(name = "request.uid",   eval = "T(java.util.UUID).randomUUID()")                // (1)
    @MDCParam(name = "app.id",        eval = "#environment['spring.application.name']")       // (2)
    @MDCParam(name = "jvm.rt",        eval = "#systemProperties['java.runtime.version']")     // (3)
    @MDCParam(name = "tx.id",         eval = "transactionId")                                 // (4)
    @MDCParam(name = "source.ip",     eval = "#request.remoteIpAddr")                         // (5)
    @MDCParam(name = "operation",     eval = "#className + '/' + #methodName")                // (6)
    @MDCParam(name = "client.id",     eval = "@authenticationService.getCurrentClientId()")   // (7)
    public void processRequest(Request request) {
      // ...
    }
    
    private String getTransactionId() {
    // ...
    }
}

The example above contains:

1. Providing parameter's value calling a static method of some class.
2. Accessing Spring configuration property using #environment variable.
3. Obtaining JVM system property using #systemProperties variable.
4. Accessing a property of the local bean. The whole bean object is available in expression evaluation as #root variable. The property may be a getter or a local field of any visibility level.
5. All arguments are available within the expression using #argumentName variables.
6. Variables #className and #methodName contain the fully-qualified class name and the method name respectively.
7. Using @beanName notation you can reference any named bean within the Spring Application Context.

MDC and the method scope

@WithMDC and @MDCParam annotations may also be defined at class level. In this case they provoke the same effect as being applied to each method.

@WithMDC
@MDCParam(name = "request.uid",   eval = "T(java.util.UUID).randomUUID()")
@MDCParam(name = "app.id",        eval = "#environment['spring.application.name']")
@MDCParam(name = "operation",     eval = "#className + '/' + #methodName")
class OrderProcessor { 
    public void createOrder(@MDCParam(name = "order.id", eval = "id") Order order) {}
    public void updateOrder(@MDCParam(name = "order.id", eval = "id") Order order) {}
    public void removeOrder(@MDCParam(name = "order.id") String orderId) {}
    public Order getOrder(@MDCParam(name = "order.id") String orderId) {}
}

MDC may also be named. This adds a correspondent prefix to any parameter defined within its scope.

@WithMDC("order")
class OrderProcessor {
    // The name of MDC parameter will be "order.id"
    public void createOrder(@MDCParam(name = "id", eval = "id") Order order) {
    }
}

In a cascade invocations of two methods that both annotated with WithMDC, a new 'nested' MDC will be created for the nested method invocation. This MDC will contain parameters defined inside the nested method and it will be closed after the method returns (see considerations). All parameters contained in the 'outer' MDC will remain as is in log messages.

class OrderProcessor {
    @WithMDC
    public void createOrder(@MDCParam(name = "orderId", eval = "id") Order order) {
        // Call a method that defines a 'nested' MDC
        Customer customer = customerRepository.findCustomerByName(order.getCustomerId());
        log.info("after the 'nested' MDC call returns the customerId parameter will no longer exist in log messages");
    }
}
class CustomerRepository {
    // Defines a 'nested' MDC. The parameter customerId belongs to this new MDC
    // and will be cleared after the method returns
    @WithMDC
    public Customer findCustomerById(@MDCParam String customerId) {
        log.info("this log message will have orderId and customerId parameters defined");
    }
}

Any call to a 'nested' method that defines @MDCParam but is not annotated with @WithMDC annotation will simply add a new parameter to the current MDC that will remain after the method returns.

class OrderProcessor {
    @WithMDC
    public void createOrder(@MDCParam(name = "orderId", eval = "id") Order order) {
        // Call a method current MDC
        Customer customer = customerRepository.findCustomerByName(order.getCustomerId());
        log.info("after the method call we still have a customerId parameter in our MDC");
    }
}
class CustomerRepository {
    // The parameter customerId will be added to current MDC, and will be kept after the method returns.
    public Customer findCustomerById(@MDCParam String customerId) {
        log.info("this log message will have orderId and customerId parameters defined");
    }
}

Output parameters

With @MDCOutParam annotation you can define an output parameter that will be added to the current MDC after the method returns. Its value is evaluated using the value returned by this method.

class OrderProcessor {
    @WithMDC
    public void createOrder(Order order) {
        User user = userRepository.findUserById(order.getUserId());
        log.info("this log message will have userId, userName and userGroup MDC parameters");
    }
}
class UserRepository {
    @MDCOutParam(name = "userName", eval = "name")
    @MDCOutParam(name = "userGroup", eval = "group.name")
    public User findUserById(@MDCParam String userId) {}
}

Defining MDC programmatically

That gives you a full control over MDC scopes and parameter definitions. Use try-with-resources block to ensure a proper cleanup of all defined parameters.

class OrderProcessor {
    public void createOrder(Order order) {
        try (CloseableMDC rootMdc = MDC.create()) {
            // Add a param to current MDC
            MDC.param("order.id", order.getId());
            log.info("order.id is added to MDC");
            
            try (CloseableMDC nestedMdc = MDC.create()) {
                // Add a param to nested MDC (nearest for current execution scope)
                MDC.param("customer.id", order.getCustomerId());
                log.info("Both order.id and customer.id appear in log messages");
            }
            log.info("order.id is still remains in messages but customer.id is removed with its MDC");
        }
    }
}

Alternatively you may use a lambda-based API to define MDC scopes.

class OrderProcessor {
    public void createOrder(Order order) {
        MDC.with().param("order.id", order.getId()).run(() -> {
            log.info("order.id is added to MDC");
            Customer customer = MDC.with().param("customer.id", order.getCustomerId()).apply(() -> {
                log.info("Both order.id and customer.id appear in log messages");
                return customerRepository.findCustomerById(order.getCustomerId());
            });
            log.info("order.id is still remains in messages but customer.id is removes with its MDC");
        });
    }
}

(back to top)

Considerations and limitations

Method invocations

The library uses Spring AOP to intercept annotated method invocations, so these considerations must be taken into account:

• The annotated method must be invoked from outside the bean scope. Local calls are not intercepted by Spring AOP, thus any method annotation will be ignored in this case.

  class MyBean {
    @Lazy @Autowired MyBean self;
  
    public void publicMethod(@MDCParam String someParam) {
        anotherPublicMethod("this call is local, so 'anotherParam' will not be included in MDC");
        self.anotherPublicMethod("this call is proxied, so 'anotherParam' will be included hin MDC");
    }
    public void anotherPublicMethod(@MDCParam String anotherParam) {
        log.info("Some log trace");
    } 
  } 

• Spring AOP does not intercept private methods, so if you invoke an inner bean's private method, it will have no effect on it.

In both of the cases above you should define your parameters in an imperative way using MDC.param().

Method argument names

By default, Java compiler does not keep method argument names in generated bytecode, so it may cause possible problems with parameter name resolutions when using @MDCParam for method's arguments. There are three ways to avoid this problem:

• If you are using Spring Boot and Spring Boot Maven or Gradle plugin, the generated bytecode will already contain all method arguments with their names, and no additional action is required.
• If you are not using Spring Boot plugin you may tell your compiler to preserve method argument names by adding -parameters argument to javac invocation.
• You may also provide parameter names explicitly in your code:

  public User findUserById(@MDCParam("userId") String userId)
  

Acknowledgements

Configuring Logback for JSON output

Add these dependencies to your pom.xml:

<dependencies>
    <dependency>
        <groupId>ch.qos.logback.contrib</groupId>
        <artifactId>logback-json-classic</artifactId>
        <version>0.1.5</version>
    </dependency>
    <dependency>
        <groupId>ch.qos.logback.contrib</groupId>
        <artifactId>logback-jackson</artifactId>
        <version>0.1.5</version>
    </dependency>
    <dependency>
        <groupId>com.fasterxml.jackson.core</groupId>
        <artifactId>jackson-databind</artifactId>
        <version>2.13.3</version>
    </dependency>
</dependencies>

Create a new appender or modify an existing one setting JsonLayout:

<configuration>
    <appender name="jsonConsole" class="ch.qos.logback.core.ConsoleAppender">
        <layout class="ch.qos.logback.contrib.json.classic.JsonLayout">
            <jsonFormatter
                    class="ch.qos.logback.contrib.jackson.JacksonJsonFormatter">
                <prettyPrint>true</prettyPrint>
            </jsonFormatter>
            <timestampFormat>yyyy-MM-dd' 'HH:mm:ss.SSS</timestampFormat>
        </layout>
    </appender>

    <root level="DEBUG">
        <appender-ref ref="jsonConsole" />
    </root>
</configuration>

Now log messages will be written in JSON format that will include all MDC parameters.

Configuring output for Elasticsearch

Please refer to these resources:

• Configuring output for ELK
• Adding APM Agent for log correlation

Roadmap

• Make library working with annotated interfaces
• Save and restore current MDC parameters to raw Map
• Intercept @Async calls maintaining the same MDC
• Spring WebFlux support?
• CDI & JakartaEE support?
  • Add jboss-log-manager support.
• Future research:
  • Annotation-processor based compile-time code enhancement
  • Agent-based runtime class transformations

(back to top)

License

Distributed under the Apache Version 2.0 License. See the license file LICENSE.md for more information.

(back to top)