Azure / azure-storage-java

Microsoft Azure Storage Library for Java

Home Page:https://docs.microsoft.com/en-us/java/api/overview/azure/storage

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Continuously call of TransferManager methods makes not to work blob accesses

tsbttk opened this issue · comments

Which service(blob, file, queue, table) does this issue concern?

blob

Which version of the SDK was used?

10.1.0, 10.5.0

What problem was encountered?

In our application, we use azure-storage-blob-10.1.0.jar to access to the Azure blob service.

We use "TransferManager.downloadBlobToFile(downloadFileChannel, blob, null, null).blockingGet()" method
and "TransferManager.uploadFileToBlockBlob(fileChannel, blob, 8 * 1024 * 1024, null).blockingGet()" method.

After we call these methods continuously (for about 13 days or call these methods thousands times), we can not access to the blob service via these methods. After certain moment these methods become to return timeout.

We could get a heapdump while the phenomenon was happening. From the heapdump we notice SharedChannelPool#requests instance that is instance of ConcurrentLinkedDeque has many requests that is not transacted.
we also notice it is because size of SharedChannelPool#leased instance is 64, and SharedChannelPool#poolSize is also 64.
So SharedChannelPool thread stops at L.109

                        while (leased.size() >= poolSize && !closed) {
                            sync.wait();	// L.109
                        }

(We know now 64 is the number of core (2) * 32 from source code of libraries.)

As a trial, while the application is running, we stop calling thse download and upload methods.
After we stop calling these download and upload methods, the size of SharedChannelPool#leased map stays 64.
So we think that something like leak may exist in SharedChannelPool#leased map.

We want to know how we can call these 2 methods continuously.

Have you found a mitigation/solution?

No, not yet.