The service request was not made within 10 seconds of doBlockingWrite being invoked

[luxhoon1216]

Describe the bug

I'm not sure if it's a bug or incorrect settings in my side. I'm occasionally getting the error while uploading files to S3. My application is based on SpringBoot 2.3.4.RELEASE with jetty.

implementation "software.amazon.awssdk:s3-transfer-manager:2.21.9"
implementation "software.amazon.awssdk.crt:aws-crt:0.28.0"

Expected Behavior

Files are uploaded without any error.

Current Behavior

It occasionally fails due to this error but not always.

message:  The service request was not made within 10 seconds of doBlockingWrite being invoked. Make sure to invoke the service request BEFORE invoking doBlockingWrite if your caller is single-threaded. 
    stack_trace:  java.lang.IllegalStateException: The service request was not made within 10 seconds of doBlockingWrite being invoked. Make sure to invoke the service request BEFORE invoking doBlockingWrite if your caller is single-threaded.
	at software.amazon.awssdk.core.async.BlockingInputStreamAsyncRequestBody.waitForSubscriptionIfNeeded(BlockingInputStreamAsyncRequestBody.java:110)
	at software.amazon.awssdk.core.async.BlockingInputStreamAsyncRequestBody.writeInputStream(BlockingInputStreamAsyncRequestBody.java:74)

Reproduction Steps

I'm sharing my code.

S3ClientConfig

@Bean
    public S3AsyncClient getS3AsyncClient(@Value("${s3ClientConfig.region}") String region,
                                          @Value("${s3ClientConfig.proxyHost}") String proxyHost,
                                          @Value("${s3ClientConfig.proxyPort}") Integer proxyPort,
                                          AwsCredentialsProviderChain awsCredentialsProviderChain) {
        Region awsRegion = Region.of(region);

        S3CrtAsyncClientBuilder s3CrtAsyncClientBuilder =
            S3AsyncClient.crtBuilder()
                .credentialsProvider(awsCredentialsProviderChain)
                .region(awsRegion);

        if (isProxyDefined(proxyHost, proxyPort)) {
            s3CrtAsyncClientBuilder.httpConfiguration(getS3CrtConfigWithProxy(proxyHost, proxyPort));
        }

        return s3CrtAsyncClientBuilder.build();
    }

    @Bean
    public S3TransferManager getS3TransferManager(S3AsyncClient s3AsyncClient) {
        return S3TransferManager.builder()
            .s3Client(s3AsyncClient)
            .build();
    }

    private S3CrtHttpConfiguration getS3CrtConfigWithProxy(String proxyHost, Integer proxyPort) {
        S3CrtProxyConfiguration proxyConfiguration = S3CrtProxyConfiguration.builder()
            .host(proxyHost).port(proxyPort).build();
        return S3CrtHttpConfiguration.builder().proxyConfiguration(proxyConfiguration).build();
    }

FileUpload

public long uploadFileStream(String bucketName, String fileName, InputStream fileStream, long assetSize) {
        BlockingInputStreamAsyncRequestBody body =
            AsyncRequestBody.forBlockingInputStream(assetSize);

        Upload upload = s3TransferManager.upload(builder -> builder
            .requestBody(body)
            .putObjectRequest(req -> req.bucket(bucketName).key(fileName))
            .build());

        long uploadedFileSize = body.writeInputStream(fileStream);

        upload.completionFuture().join();

        LOGGER.info("uploaded stream successfully: filename {}, assetSize {}, actualUploadedSize {}", fileName, assetSize, uploadedFileSize);
        return uploadedFileSize;
    }

Possible Solution

No response

Additional Information/Context

When it happens, it happens in only one instance out of 8 instances even though traffic is distributed evenly across the cluster.

aws-crt-java version used

0.28.0

Java version used

11

Operating System and version

Amazon Linux, version 2 (EC2)

[Joseph Klix]

Can you provide us with the crt logs from when this is happening?

[github-actions]

Greetings! It looks like this issue hasn’t been active in longer than a week. We encourage you to check if this is still an issue in the latest release. Because it has been longer than a week since the last update on this, and in the absence of more information, we will be closing this issue soon. If you find that this is still a problem, please feel free to provide a comment or add an upvote to prevent automatic closure, or if the issue is already closed, please feel free to open a new one.