SpringBoot Webflux reactive non blocking application. Performance drop using Logstash when sending logs to logstash server

[patpatpat123]

Hello team,

This is my first post in this repo.
If not anything else, wanted to say thanks for the project.
It is easy to use, clear and simple.

Wanted to reach out to report an issue:

Setup: I have a basic Spring Webflux reactive non blocking web application.
Upon invoking the rest endpoint, I have application logs.

We forward those logs to a Logstash endpoint (which later will forward to our ElasticSearch)
Therefore, this is the push model (it is not like logstash or filebeat will come and "poll" the logs)

While the logs are indeed being sent, we are unfortunately observing performance drop.
Periodically, we can see our app "freeze". After investigation with actuator, it seems the app "freezes" when trying to send the logs.
The freezes are disappearing if we do not enable logstash forwarding (when we just print the logs in console without forwarding, the app is fine).

We are running our reactive application using 1CPU only. Therefore, would it be possible Logstash performs a blocking operation when sending the logs, thus blocking the reactor core?

If yes, can we have a non blocking version when sending the logs please?
Thank you

       <dependency>
            <groupId>net.logstash.logback</groupId>
            <artifactId>logstash-logback-encoder</artifactId>
            <version>7.2</version>
        </dependency>

<?xml version="1.0" encoding="UTF-8"?>
<configuration>

    <property scope="context" name="log.fileExtension" value="log"/>
    <property scope="context" name="log.directory" value="/logs"/>
    <property scope="context" name="log.fileName" value="reactiveapp"/>

    <appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
        <layout class="ch.qos.logback.classic.PatternLayout">
            <pattern>[%d{yyyy-MM-dd HH:mm:ss.SSS}] [${HOSTNAME}] [%thread] %level %logger{36}@%method:%line - %msg%n</pattern>
        </layout>
    </appender>

    <appender name="FILE" class="ch.qos.logback.core.rolling.RollingFileAppender">
        <rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
            <fileNamePattern>${log.directory}/${log.fileName}.%d{yyyy-MM-dd}.${log.fileExtension}</fileNamePattern>
        </rollingPolicy>
        <encoder>
            <pattern>[%d{yyyy-MM-dd HH:mm:ss.SSS}] [${HOSTNAME}] [%thread] %level %logger{36}@%method:%line - %msg%n</pattern>
        </encoder>
    </appender>

    <appender name="STASH" class="net.logstash.logback.appender.LogstashTcpSocketAppender">
        <destination>mylogstash:5000</destination>
        <!-- encoder is required -->
        <encoder class="net.logstash.logback.encoder.LogstashEncoder"/>
        <keepAliveDuration>5 minutes</keepAliveDuration>
    </appender>

    <logger name="org.hibernate" level="TRACE">
        <appender-ref ref="STASH"/>
    </logger>

    <root level="INFO">
        <appender-ref ref="STDOUT"/>
        <appender-ref ref="STASH"/>
        <appender-ref ref="FILE"/>
    </root>
</configuration>

[Phil Clay]

Hi @patpatpat123,

The LogstashTcpSocketAppender is an asynchronous appender, in that it does not block the thread that calls the logger. The log event is put onto a ring buffer. Another thread retrieves the event from the ring buffer, encodes it in JSON, and sends the JSON over the TCP connection. If the TCP connection is slow, and the ringbuffer backs up, then when new events are attempted to be added to the ring buffer, those events will be dropped by default (so as to not prevent blocking the calling thread).

The sending thread uses blocking I/O. However, this should not block the thread calling the logger, since it is a different thread. A 1CPU machine should be able to schedule these threads appropriately and not freeze the app, however there will be some context switching overhead.

Perhaps you could use a profiler to dig in a little more? I use yourkit quite a bit when profiling, and it can capture a CPU profiling snapshot that might help.

[patpatpat123]

Hello @philsttr,

Very clear answer. Indeed, from your writing, the root cause should not be the appender.
Running some tests with yourkit asap and will update or close this.

Again, many thanks

[patpatpat123]

Hello team, it seems indeed the LogstashTcpSocketAppender is not the root cause of the issue.

Apologies for the inconveniences caused. Closing