Missing Bucket in path when using "forcePathStyle"
clementguillot opened this issue · comments
Describe the bug
Using a local develoment environment with any S3-comptatible backend (e.g. minio
), I try to configure a S3 Client with forcePathStyle = true
.
When I call .presignGetObject
, the result URL does not contain the bucket in the path. However, it works as expected when this parameter is false
or null
.
Expected behavior
When forcePathStyle
is false
or null
, presigned URL has pattern http://{bucket}.{endpoint}/{key}
.
But, once this parameter is true
, presigned URL has pattern http://{endpoint}/{key}
and bucket is missing, expected URL is http://{endpoint}/{bucket}/{key}
.
Current behavior
Bucket is missing from built-URL. When accessing the generated URL, server throws a 403 error.
Steps to Reproduce
Let's consider the following sippnet:
val localS3 =
S3Client {
endpointUrl = Url.parse("http://localhost:9000")
region = "us-east-1"
forcePathStyle = true
credentialsProvider =
StaticCredentialsProvider {
accessKeyId = "access-key"
secretAccessKey = "secret-key"
}
}
val getRequest =
GetObjectRequest {
bucket = "my-bucket"
key = "my-file-path"
}
val url = localS3.presignGetObject(getRequest, 1.hours).url.toString()
url
looks like this: http://localhost:9000/my-file-path?x-id=GetObject&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=...
Expected: http://localhost:9000/my-bucket/my-file-path?x-id=GetObject&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=...
Note that with the same local S3 backend, but using the following NodeJS snippet, it works as expected:
const s3client = new S3Client({
endpoint: "http://localhost:9000",
region: "us-east-1",
credentials: {
accessKeyId: "access-key",
secretAccessKey: "secret-key",
},
forcePathStyle: true,
})
const getCommand = new GetObjectCommand({
Bucket: "my-bucket",
Key: "my-file-path"
})
getSignedUrl(s3client, getCommand, {expiresIn: 3600}).then((result) => {
console.log("pre-signed URL" + result)
})
Print: http://localhost:9000/my-bucket/my-file-path?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Content-Sha256=...
Possible Solution
No response
Context
No response
AWS Kotlin SDK version used
1.0.30
Platform (JVM/JS/Native)
OpenJDK Runtime Environment GraalVM CE 17.0.7+7.1 and Kotlin 1.9.22
Operating System and version
Linux Ubuntu 22.04.3
Hi @clementguillot, thanks for the bug report. I've been able to reproduce the issue locally. It appears to be related to presigning and not necessarily to using a custom endpoint (e.g., for MinIO). I'll root cause the issue and get a fix posted shortly.
The fix for this issue has been merged to main and should be included in tomorrow's release. Closing this for now but please re-open if the issue still occurs on 1.0.32 or later.
⚠️ COMMENT VISIBILITY WARNING⚠️
Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.
Thank you @ianbotsf for the fix!
@ianbotsf unfortunatelly, using v1.0.32
, bucket's name is now part of the presigned URL by the link doesn't work and Minio throws a 403:
<?xml version="1.0" encoding="UTF-8"?>
<Error>
<Code>SignatureDoesNotMatch</Code>
<Message>The request signature we calculated does not match the signature you provided. Check your key and signing method.</Message>
<Key>65983dbe7ee2fa0039df4ff1/1689923702657342873</Key>
<BucketName>my-bucket</BucketName>
<Resource>/my-bucket/my-file/path</Resource>
<RequestId>17A93FCF1CFEE650</RequestId>
<HostId>dd9025bab4ad464b049177c95eb6ebf374d3b3fd1af9251148b658df7ac2e3e8</HostId>
</Error>
Hmm, that's unexpected. We added a new presigning E2E which uses forcePathStyle = true
and it passes when run against S3. Can you reproduce the failure on S3 directly?
Hi @ianbotsf,
I tried a couple of scenarios as below:
- Using S3, it works as expected with path style. URL looks like this:
https://s3.eu-west-3.amazonaws.com/test-bucket-cguillot/my-file......
and both GET and PUT are ok. - Using Scaleway (French provider), also works as expected.
- Using local Minio in Docker:
- Works as expected with NodeJS, server returns the file or 404 if the key doesn't exist
- Works as expected with Rush, same behavior as Node
- Doesn't work with Kotlin SDK, backend returns 403 error with
SignatureDoesNotMatch
Unfortunately, I can't say if the error is on Minio's side, but it seems that Node and Rust SDKs can tackle this. It would be definitely great if Kotlin SDK could too ;)
Please let me know if there is anything I can do to help with this!
EDIT: I remember that we had a "hack" in our Node codebase for a while due to that issue, when using a specific port (:9000
) in the endpoint: aws/aws-sdk-js-v3#2121 maybe could it be related?
@ianbotsf I confirm that Kotlin SDK works as expected when local Minio is on port 80 (or 443). Issue is no longer related to path style but is about using a port in the endpoint URL. Maybe should I open another issue?
Yes this should be a new issue. I've opened #1177 and I'll re-close this one.
⚠️ COMMENT VISIBILITY WARNING⚠️
Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.