listBuckets fails after a failed createBucket

Question

listBuckets fails after a failed createBucket

Hendrik-H opened this issue 6 years ago · comments

When a createBucket fails due to a conflict with an existing bucket a following listBuckets call fails with this exception:

Exception in thread "main" com.ibm.cloud.objectstorage.services.s3.model.AmazonS3Exception: Unknown Version (Service: Amazon S3; Status Code: 400; Error Code: 400 Unknown Version; Request ID: null), S3 Extended Request ID: null
	at com.ibm.cloud.objectstorage.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1588)
	at com.ibm.cloud.objectstorage.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1258)
	at com.ibm.cloud.objectstorage.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1030)
	at com.ibm.cloud.objectstorage.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:742)
	at com.ibm.cloud.objectstorage.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:716)
	at com.ibm.cloud.objectstorage.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:699)
	at com.ibm.cloud.objectstorage.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:667)
	at com.ibm.cloud.objectstorage.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:649)
	at com.ibm.cloud.objectstorage.http.AmazonHttpClient.execute(AmazonHttpClient.java:513)
	at com.ibm.cloud.objectstorage.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3864)
	at com.ibm.cloud.objectstorage.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3811)
	at com.ibm.cloud.objectstorage.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3805)
	at com.ibm.cloud.objectstorage.services.s3.AmazonS3Client.listBuckets(AmazonS3Client.java:879)
	at com.ibm.cloud.objectstorage.services.s3.AmazonS3Client.listBuckets(AmazonS3Client.java:885)
	at scratch.cos.Main.main(Main.java:51)

The extra strange thing is that the wire trace shows no difference between a failing listBuckets call and working one, exception from the signature.

smcgrath · Answer 1 · Thu Jan 10 2019 21:23:19 GMT+0800 (China Standard Time)

@Hendrik-H I will look into this issue and get back to you with my findings.

smcgrath · Answer 2 · Fri Jan 11 2019 19:58:07 GMT+0800 (China Standard Time)

@Hendrik-H would you be able to paste your code here, ensuring to remove all credentials info, Im having issues replicating the error.

Hendrik-H · Answer 3 · Fri Jan 11 2019 20:07:00 GMT+0800 (China Standard Time)

@smcgrath-IBM my test code looks like this:

package scratch.cos;

import com.ibm.cloud.objectstorage.AmazonServiceException;
import com.ibm.cloud.objectstorage.ClientConfiguration;
import com.ibm.cloud.objectstorage.auth.AWSStaticCredentialsProvider;
import com.ibm.cloud.objectstorage.auth.BasicAWSCredentials;
import com.ibm.cloud.objectstorage.client.builder.AwsClientBuilder.EndpointConfiguration;
import com.ibm.cloud.objectstorage.services.s3.AmazonS3;
import com.ibm.cloud.objectstorage.services.s3.AmazonS3ClientBuilder;
import com.ibm.cloud.objectstorage.services.s3.model.Bucket;
import com.ibm.cloud.objectstorage.services.s3.model.CreateBucketRequest;

public class Main {
	
	public static void main(String[] args) throws Exception {
        String endpointURL = "https://s3.us.cloud-object-storage.appdomain.cloud";
        
        AmazonS3 cos = AmazonS3ClientBuilder.standard()
        		.withCredentials(new AWSStaticCredentialsProvider(new BasicAWSCredentials("...", "...")))
                .withEndpointConfiguration(new EndpointConfiguration(endpointURL, null))
                .withPathStyleAccessEnabled(true)
                .withClientConfiguration(new ClientConfiguration().withRequestTimeout(5000))
                .build();
        
        //cos.createBucket("test-bucket", "us-standard"); // all the none deprecated options don't seem to work correctly
        try {
        	System.out.println("exists: " + cos.doesBucketExist("test-bucket"));
//        	cos.deleteBucket("test-bucket");
        	cos.createBucket("test-bucket", "us-standard"); // all the none deprecated options don't seem to work correctly
        } catch (AmazonServiceException e) {
        	System.err.println(e.getErrorCode());
        	System.err.println(e.getErrorMessage());
        }
        
        for (Bucket bucket: cos.listBuckets()) {
        	System.out.println(bucket.getName() + " (created: " + bucket.getCreationDate() + " / owner: " + bucket.getOwner() + ")");
        }
	}
}

the output is:

exists: true
BucketAlreadyExists
The requested bucket name is not available. The bucket namespace is shared by all users of the system. Please select a different name and try again.
Exception in thread "main" com.ibm.cloud.objectstorage.services.s3.model.AmazonS3Exception: Unknown Version (Service: Amazon S3; Status Code: 400; Error Code: 400 Unknown Version; Request ID: null), S3 Extended Request ID: null
	at com.ibm.cloud.objectstorage.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1588)
	at com.ibm.cloud.objectstorage.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1258)
	at com.ibm.cloud.objectstorage.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1030)
	at com.ibm.cloud.objectstorage.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:742)
	at com.ibm.cloud.objectstorage.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:716)
	at com.ibm.cloud.objectstorage.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:699)
	at com.ibm.cloud.objectstorage.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:667)
	at com.ibm.cloud.objectstorage.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:649)
	at com.ibm.cloud.objectstorage.http.AmazonHttpClient.execute(AmazonHttpClient.java:513)
	at com.ibm.cloud.objectstorage.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3864)
	at com.ibm.cloud.objectstorage.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3811)
	at com.ibm.cloud.objectstorage.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3805)
	at com.ibm.cloud.objectstorage.services.s3.AmazonS3Client.listBuckets(AmazonS3Client.java:879)
	at com.ibm.cloud.objectstorage.services.s3.AmazonS3Client.listBuckets(AmazonS3Client.java:885)
	at scratch.cos.Main.main(Main.java:35)

Without the failing createBucket call the listBucket call works without any error.
I'm using the latest SDK version: 2.4.1

smcgrath · Answer 4 · Fri Jan 11 2019 20:31:45 GMT+0800 (China Standard Time)

Thanks @Hendrik-H I can see the same issue now, although the listbuckets is throwing a 'No URI' message in the exception for myself. I have raised an internal JIRA to track this issue for a future patch release - CSAFE-49102. This git issue will be updated when we have a solution ready.

smcgrath · Answer 5 · Thu Mar 28 2019 22:00:36 GMT+0800 (China Standard Time)

@Hendrik-H can you set this value on your ClientConfiguration object .withUseExpectContinue(false) this has resolved the issue for me

smcgrath · Answer 6 · Fri Mar 29 2019 22:29:52 GMT+0800 (China Standard Time)

Hi @Hendrik-H did you get a chance to try this configuration?

Hendrik-H · Answer 7 · Fri Mar 29 2019 22:31:52 GMT+0800 (China Standard Time)

hi @smcgrath-IBM , I saw your update but did not get to it yet. I have it on my todo list and will let you know as soon as I had a chance. I hope to get to it in the next days.

smcgrath · Answer 8 · Wed Apr 03 2019 16:59:15 GMT+0800 (China Standard Time)

Hi @Hendrik-H any update on the solution I provided?

Hendrik-H · Answer 9 · Wed Apr 03 2019 17:19:32 GMT+0800 (China Standard Time)

@smcgrath-IBM yes, I can confirm that my problem gets resolved that way. Is that the final resolution or just a work around? What's the exact semantics of that options? I could not really find any documentation explaining what effects it has.

smcgrath · Answer 10 · Wed Apr 03 2019 17:31:09 GMT+0800 (China Standard Time)

Thats good news. Setting ExpectContinue to false ensures a new tcp connection with the server for each request. If this solution works for you, let me know if we can close out this issue.

Hendrik-H · Answer 11 · Wed Apr 03 2019 17:35:01 GMT+0800 (China Standard Time)

So that will have a negative performance impact. As a work around that's ok but I don't consider that to be a fix. Clearly there is something wrong in the state handling of client.

smcgrath · Answer 12 · Wed Apr 03 2019 23:04:17 GMT+0800 (China Standard Time)

@Hendrik-H this is been investigated from the server side. I will leave this issue open and advise you when a solution is in place instead of the work around.

smcgrath · Answer 13 · Wed Apr 10 2019 23:17:42 GMT+0800 (China Standard Time)

@Hendrik-H, just an update, an issue has been found on the load balancer with how it interacts with the application server at a packet level. Work is in progress to resolve.

Hendrik-H · Answer 14 · Wed Apr 10 2019 23:23:04 GMT+0800 (China Standard Time)

@smcgrath-IBM awesome! guess it will make testing a bit hard for me but as you have the full test code you should be able to verify that once you have a version with the fix setup.

barry-hueston · Answer 15 · Wed Jul 24 2019 07:15:08 GMT+0800 (China Standard Time)

Hi @Hendrik-H,
This ticket has been open quite some time and as the underlying issue is not directly related to the code within the SDK, I would like to close this issue.
The issue has been root caused to a component outside of the SDK, and discussions are underway with the vendor on the issue and possible fix. At this time we don't foresee any SDK needs to address this issue and there is no value in keeping this issue open on the SDK.