Timeout waiting for connection from pool while calling S3client.getObject

Question

Timeout waiting for connection from pool while calling S3client.getObject

sushilkumarchaudhary opened this issue 7 years ago · comments

sushilkumarchaudhary commented 7 years ago

All,
I see issue while trying to read a file from S3. The behavior is totally random and some time it does appear and sometime it does not. What could I missing here. its using all the default connection property.

code which is calling get method :

S3Object object = amazonS3Client.getObject(new GetObjectRequest(s3Bucket, fileName));
try {
  bytes = IOUtils.toByteArray(object. getObjectContent());
} catch (IOException e) {
  AmazonS3TemplateException exception = new AmazonS3TemplateException(
      "Exception while downloading object from S3 bucket '" + s3Bucket + "'", e);
  LOGGER.error("AmazonS3Template.downloadObject", exception.getMessage(), exception);
  throw exception;
}finally{
	if(object!=null){
		object.close();
	}
}

Here are the Exception stack track :

Unable to execute HTTP request: Timeout waiting for connection from pool
com.amazonaws.SdkClientException: Unable to execute HTTP request: Timeout waiting for connection from pool
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleRetryableException(AmazonHttpClient.java:1069) ~[aws-java-sdk-core-1.11.125.jar!/:?]
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1035) ~[aws-java-sdk-core-1.11.125.jar!/:?]
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:742) ~[aws-java-sdk-core-1.11.125.jar!/:?]
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:716) ~[aws-java-sdk-core-1.11.125.jar!/:?]
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:699) ~[aws-java-sdk-core-1.11.125.jar!/:?]
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:667) ~[aws-java-sdk-core-1.11.125.jar!/:?]
at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:649) ~[aws-java-sdk-core-1.11.125.jar!/:?]
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:513) ~[aws-java-sdk-core-1.11.125.jar!/:?]
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4169) ~[aws-java-sdk-s3-1.11.125.jar!/:?]
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4116) ~[aws-java-sdk-s3-1.11.125.jar!/:?]
at com.amazonaws.services.s3.AmazonS3Client.getObject(AmazonS3Client.java:1365) ~[aws-java-sdk-s3-1.11.125.jar!/:?]
at com.capitalone.cerberus.amazon.s3.template.AmazonS3Template.downloadObject(AmazonS3Template.java:82) ~[classes!/:01.00.00.22-SNAPSHOT]
at
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_151]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_151]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_151]
Caused by: org.apache.http.conn.ConnectionPoolTimeoutException: Timeout waiting for connection from pool
at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.leaseConnection(PoolingHttpClientConnectionManager.java:292) ~[httpclient-4.5.3.jar!/:4.5.3]
at org.apache.http.impl.conn.PoolingHttpClientConnectionManager$1.get(PoolingHttpClientConnectionManager.java:269) ~[httpclient-4.5.3.jar!/:4.5.3]
at sun.reflect.GeneratedMethodAccessor39.invoke(Unknown Source) ~[?:?]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_151]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_151]
at com.amazonaws.http.conn.ClientConnectionRequestFactory$Handler.invoke(ClientConnectionRequestFactory.java:70) ~[aws-java-sdk-core-1.11.125.jar!/:?]
at com.amazonaws.http.conn.$Proxy98.get(Unknown Source) ~[?:?]
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:191) ~[httpclient-4.5.3.jar!/:4.5.3]
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) ~[httpclient-4.5.3.jar!/:4.5.3]
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) ~[httpclient-4.5.3.jar!/:4.5.3]
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) ~[httpclient-4.5.3.jar!/:4.5.3]
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56) ~[httpclient-4.5.3.jar!/:4.5.3]
at com.amazonaws.http.apache.client.impl.SdkHttpClient.execute(SdkHttpClient.java:72) ~[aws-java-sdk-core-1.11.125.jar!/:?]
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1190) ~[aws-java-sdk-core-1.11.125.jar!/:?]
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1030) ~[aws-java-sdk-core-1.11.125.jar!/:?]
... 15 more
2017-11-30 21:29:48:

Dongie Agnir · Answer 1 · Sat Dec 02 2017 02:54:08 GMT+0800 (China Standard Time)

Normally this error is caused by the object content stream being left open but it looks like you're closing them here.

Is there any chance you're doing many of these downloads concurrently in a threadpool?

sushilkumarchaudhary · Answer 2 · Sat Dec 02 2017 04:11:26 GMT+0800 (China Standard Time)

Yes, I am using a thread pool of 50 threads but each of threads are trying to download different file.

Do we need separate s3 connection for each thread?

Dongie Agnir · Answer 3 · Sat Dec 02 2017 04:46:43 GMT+0800 (China Standard Time)

@sushilkumarchaudhary Yes, the number of concurrent requests should be <= the number of max connections in the connection pool; otherwise the requests can get blocked trying to lease a connection from the pool and then eventually timeout.

sushilkumarchaudhary · Answer 4 · Sat Dec 02 2017 04:54:03 GMT+0800 (China Standard Time)

I see the default max connection is also 50, which is mostly less or equal to number of threads running. I will increase max connection and will test it.Any other things which could be causing this.

Dongie Agnir · Answer 5 · Sat Dec 02 2017 05:31:39 GMT+0800 (China Standard Time)

@sushilkumarchaudhary Hmm those are the two situations I'm aware of. Please let me know what the results are.

sushilkumarchaudhary · Answer 6 · Tue Dec 05 2017 11:21:23 GMT+0800 (China Standard Time)

@dagnir it looks like it working for me so far. I will report if found any issue.

Varun Nandi · Answer 7 · Wed Dec 06 2017 13:01:20 GMT+0800 (China Standard Time)

Happy to hear the code is working fine. Feel free to reopen if you face the issue again.

nidhiwalkover · Answer 8 · Tue Jan 30 2018 14:53:43 GMT+0800 (China Standard Time)

I am also facing the same issue. I am facing issues while resolving this.

Can you help me how to check no of connections in pool and How can i limit them and how to close them.

Rohith Nandakumar · Answer 9 · Wed Jan 31 2018 20:40:35 GMT+0800 (China Standard Time)

@nidhiwalkover I was having the same issue. It turned out I hadn't closed all the S3Objects properly across the application. There was one piece of code that had a few objects open, which eventually lead to this error. Scan through your code and make sure all instances of S3Objects are closed.

Sample code below that I use to get an InputStream of the file.

public InputStream getObject(String path) throws IOException {
  
  InputStream is = null;
  S3Object s3Obj = null;
  
  try {
    s3Obj = s3.getObject(bucket, path);
    // There might be a better way to do this.
    try(S3ObjectInputStream stream = s3Obj.getObjectContent()) {
      ByteArrayOutputStream temp = new ByteArrayOutputStream();
      IOUtils.copy(stream, temp);
      is = new ByteArrayInputStream(temp.toByteArray());
    }
  } catch (AmazonServiceException ase) {
    logger.error(ase.getMessage(), ase);
  } catch (AmazonClientException ace) {
    logger.error(ace.getMessage(), ace);
  } finally {
    if (s3Obj != null) {
      try {
        // Close the object
        s3Obj.close(); 
      } catch (IOException e) {
        logger.error("Unable to close S3 object: {}", e.getMessage(), e);
      }
    }
  }
  return is;
}

nidhiwalkover · Answer 10 · Wed Jan 31 2018 21:04:19 GMT+0800 (China Standard Time)

Hello, Thank You so much for updating me. In my case, I am using AwsS3Class private static final AmazonS3 *S3* = new AmazonS3Client(); And I am performing all operations with this object. For Ex: *S3Object o = S3.getObject(BUCKET, filePath);* *S3.deleteObject(new DeleteObjectRequest(BUCKET, filePath));* And with this s3 object, I am unable to find any method to release the object. can you send me any sample code to delete this s3 object??? Regards, Nidhi

Rohith Nandakumar · Answer 11 · Thu Feb 01 2018 16:05:52 GMT+0800 (China Standard Time)

You need to close only the "S3Object" not the "AmazonS3" object.
So add o.close(); after you finish using o.

nidhiwalkover · Answer 12 · Thu Feb 01 2018 16:22:34 GMT+0800 (China Standard Time)

Okay. Whenever i do S3Object o = S3.getObject(BUCKET, filePath); I have to close o.close. My problem is resolved with this. *Thank You* Regards, Nidhi

…

On Thu, Feb 1, 2018 at 1:36 PM, Rohith Nandakumar ***@***.***> wrote: You need to close only the "S3Object" not the "AmazonS3" object. So add o.close(); after you finish using o. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1405 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AHZPQkxwJLb5vyAsyHb-y_bY12DEYHQ2ks5tQXBrgaJpZM4QyoLQ> .

Maroun Maroun · Answer 13 · Mon Sep 16 2019 20:40:40 GMT+0800 (China Standard Time)

How exactly does this create a timeout? Does the thread keep waiting until its input stream gets closed in order to release the connection?

Vishal Chavda · Answer 14 · Thu Mar 25 2021 13:59:30 GMT+0800 (China Standard Time)

Can anyone help me for same connection timeout from pool error?

Diego Moreira · Answer 15 · Sat Apr 06 2024 01:17:31 GMT+0800 (China Standard Time)

@nidhiwalkover I was having the same issue. It turned out I hadn't closed all the S3Objects properly across the application. There was one piece of code that had a few objects open, which eventually lead to this error. Scan through your code and make sure all instances of S3Objects are closed.

Sample code below that I use to get an InputStream of the file.

public InputStream getObject(String path) throws IOException {
  
  InputStream is = null;
  S3Object s3Obj = null;
  
  try {
    s3Obj = s3.getObject(bucket, path);
    // There might be a better way to do this.
    try(S3ObjectInputStream stream = s3Obj.getObjectContent()) {
      ByteArrayOutputStream temp = new ByteArrayOutputStream();
      IOUtils.copy(stream, temp);
      is = new ByteArrayInputStream(temp.toByteArray());
    }
  } catch (AmazonServiceException ase) {
    logger.error(ase.getMessage(), ase);
  } catch (AmazonClientException ace) {
    logger.error(ace.getMessage(), ace);
  } finally {
    if (s3Obj != null) {
      try {
        // Close the object
        s3Obj.close(); 
      } catch (IOException e) {
        logger.error("Unable to close S3 object: {}", e.getMessage(), e);
      }
    }
  }
  return is;
}

Boy, this save my life, thank you.

Jeyhun Mammadsaidov · Answer 16 · Wed Jul 03 2024 03:54:24 GMT+0800 (China Standard Time)

public S3Object downloadFileFromS3bucket(String bucketName, String fileName) {
    S3Object s3Object = null;
    try {
        log.info("Downloading {} from bucket: {}", fileName, bucketName);
        s3Object = amazonS3Client.getObject(bucketName, fileName);
        return s3Object;
    } catch (Exception e) {
        log.error("Error downloading file {} from bucket {}", fileName, bucketName, e);
        throw new RuntimeException("Error downloading file from S3", e);
    } finally {
        if (s3Object != null) {
            try {
                s3Object.close();
            } catch (IOException e) {
                log.warn("Error closing S3Object", e);
            }
        }
    }
}