Giters

Azure / Azurite

A lightweight server clone of Azure Storage that simulates most of the commands supported by it with minimal dependencies

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Timeout on blocking read for 500000000 NANOSECONDS

snalli opened this issue · comments

commented

I would really like to know why this happens because we have seen this error in production against real Azure Storage as well as our tests against Azurite.

Which service(blob, file, queue, table) does this issue concern?

Blob

Which version of the Azurite was used?

Where do you get Azurite? (npm, DockerHub, NuGet, Visual Studio Code Extension)

npm

What's the Node.js version?

What problem was encountered?

Our tests are crashing, they were perfectly fine until last night.

com.github.ambry.cloud.RecoveryNetworkClientTest > testMetadataRecoveryClientFailure FAILED
    com.github.ambry.cloud.CloudStorageException: Failed to upload blob (localhost-0/AAYQQgQACAAAAQAAAAAAAAAAQuodutHmRcCvZfo0Js8h9A) to Azure blob storage because Timeout on blocking read for 500000000 NANOSECONDS
        at com.github.ambry.cloud.azure.AzureCloudDestination.toCloudStorageException(AzureCloudDestination.java:191)
        at com.github.ambry.cloud.azure.AzureCloudDestinationSync.uploadBlob(AzureCloudDestinationSync.java:433)
        at com.github.ambry.cloud.azure.AzureCloudDestinationSync.uploadBlobs(AzureCloudDestinationSync.java:352)
        at com.github.ambry.cloud.RecoveryNetworkClientTest.before(RecoveryNetworkClientTest.java:196)

        Caused by:
        java.lang.IllegalStateException: Timeout on blocking read for 500000000 NANOSECONDS
            at reactor.core.publisher.BlockingSingleSubscriber.blockingGet(BlockingSingleSubscriber.java:124)
            at reactor.core.publisher.Mono.block(Mono.java:1766)
            at com.azure.storage.common.implementation.StorageImplUtils.blockWithOptionalTimeout(StorageImplUtils.java:149)
            at com.azure.storage.blob.BlobClient.uploadWithResponse(BlobClient.java:337)
            at com.github.ambry.cloud.azure.AzureCloudDestinationSync.uploadBlob(AzureCloudDestinationSync.java:407)
            ... 2 more

Steps to reproduce the issue?

IDK, happened without any changes on our side

If possible, please provide the debug log using the -d parameter, replacing <pathtodebuglog> with an appropriate path for your OS, or review the instructions for docker containers:

-d "<pathtodebuglog>"

Please be sure to remove any PII or sensitive information before sharing!
The debug log will log raw request headers and bodies, so that we can replay these against Azurite using REST and create tests to validate resolution.

Have you found a mitigation/solution?

No

commented

Hi @snalli ,

Seems you are using Java SDK to try to upload a blob to Azure Storage/Azurite? The timeout exception itself seems thrown out by the Java SDK instead of from Azure Storage/Azurite. Maybe you can ask Java SDK first to see in what condition it would throw out this kind of exception?

It'd be helpful if you can share:

  1. Java SDK version
  2. Azurite version
  3. Code you use

Thanks
Emma

commented

Will talk to the azure sdk team, thanks