Azure Service Bus memory leak issue with ServiceBusReceiverClient, where Reactor core threads continuously accumulate, leading to a heap dump problem.

[JeevanjyotKaur]

Describe the bug
We are correctly creating and closing ServiceBusReceiverClient, yet encountering a memory leak issue.
Thread dumps reveal that multiple Reactor core threads are accumulating, indicating they are not being disposed of properly. This has led to an increased thread count, resulting in higher heap usage and pos getting restarted.

Exception or Stack Trace
Heap Dump with Multiple threads

Stack track of a single thread

To Reproduce
Azure Service Bus Library : azure-messaging-servicebus:jar:7.17.9

Code Snippet
public void consumeMessage() {
ClientSecretCredential credential = new ClientSecretCredentialBuilder()
.clientId(ServiceBusConstants.OACC_clientID)
.clientSecret(ServiceBusConstants.OACC_clientSecret)
.tenantId(ServiceBusConstants.OACC_tenantID)
.build();

    ServiceBusReceiverClient receiver = new ServiceBusClientBuilder()
            .credential("namespace.servicebus.windows.net", credential)
            .receiver()
            .receiveMode(ServiceBusReceiveMode.PEEK_LOCK)
            .queueName("queue1")
            .buildClient();

    List<ServiceBusReceivedMessage> messageList = new ArrayList();

    try {
        ServiceBusReceivedMessage peekMessage = receiver.peekMessage();
        if (Objects.nonNull(peekMessage)) {
            IterableStream<ServiceBusReceivedMessage> receivedMessages = receiver.receiveMessages(1);
            receivedMessages.stream().forEach(message -> {
                messageList.add(message);                  
            });
        }
    } catch (Exception e) {
        System.out.println("EXCEPTION " + e);
    }
    receiver.close();
} 

Expected behavior
Threads should not get accumulated and get closed immediately after closing the client thus it will not cause pos restarts.

Setup (please complete the following information):

• IDE: IntelliJ
• Library/Libraries: Azure Service Bus Library : azure-messaging-servicebus:jar:7.17.9
• Java version: 17
• Application Behavior : Our application follows a polling mechanism, where a ServiceBusReceiverClient is created every second, retrieves a message, and is then closed.

[github-actions]

@anuchandy @conniey @lmolkova

[github-actions]

Thank you for your feedback. Tagging and routing to the team member best able to assist.

[JeevanjyotKaur]

@anuchandy @conniey @lmolkova Can you please look into this issue? Its causing multiple pod restarts for us.

Thank you!

[Connie Yau]

One thing to note is that receiverClients should be long lived objects. Every time you create a new object, it has to create a new socket and renegotiate with the service.

[Anu Thomas]

To add to what Connie mentioned, given that connections are frequently established and disposed of (once every second), the pattern generates heap objects at a rate higher than the garbage collector can handle efficiently, although it may seem like a leak, it is not in such scenario. Each connection disposal requires the reactor (IO) thread to shut down other internal resources, which could otherwise result in a heap leak. The recommendation here is to use the client for some time once created, you may also explore using ServiceBusProcessorClient (with global scope) which can process each message as it arrives without needing app to poll.

[Anu Thomas]

Closing this. Explained the potential issue with the current pattern and provided guidance to address it.

[JeevanjyotKaur]

Thanks @anuchandy @conniey
As per your suggestion we are working on re-using the client.

@anuchandy @conniey Can you please let me know For how long a ServiceBusRecieverClient can be reused if its not closed?
Any default closing time for it?