AWSSDK.S3 3.7.307.4 escapes colon in bucket name in GetPreSignedURL
desdoades opened this issue · comments
Describe the bug
We use ceph S3 where bucket names are not a subdomain but after the domain, e.g. https://example.com/11213ef55:mybucket/path/to/file.jpg.
When using AWSSDK.S3 3.7.307.4 it percent-encodes a colon in the bucket name (which is a valid character in an URL path) when calling GetPreSignedURL(request).
This breaks the upload to our S3 file host since it can not match to the correct bucket.
Can this be fixed on your end or would you say thats a problem of the ceph S3?
Expected Behavior
We are currently on AWSSDK 3.7.103.13 where it does not percent-encode a colon in the bucket name.
In this version the file upload to S3 works.
Current Behavior
Because of the colon percent-encode the s3 ceph file server returns a
Forbidden
AccessDenied
...
when trying to upload a file to the pre signed URL.
Reproduction Steps
That should not be neccessary
Possible Solution
No response
Additional Information/Context
No response
AWS .NET SDK and/or Package version used
AWSSDK.S3 3.7.307.4
Targeted .NET Platform
.NET 8
Operating System and version
Windows 10
@desdoades Good morning. Thanks for reporting the issue. Could you please share if you are using AWS S3 service or simulated service that mimics S3?
Thanks,
Ashish
EDIT: Looks like you are using CEPH S3 service.
Need reproduction with actual AWS S3 service.
@desdoades Good afternoon. Please confirm if your bucket is named 11213ef55:mybucket
. AWS S3 console doesn't allow :
(colon) character in bucket name. I'm unsure how CEPH S3 service is allowing it. So it appears be an issue with CEPH S3 side, not SDK side since :
is not a valid character in bucket name.
Thanks,
Ashish
Dear @ashishdhingra,
we are not using AWS S3 but netways S3 which uses ceph. As I said it does not have the bucketname in the subdomain but in the path.
Our actual bucket is not named 11213ef55:mybucket
but very similar, so it does contain a colon.
Dear @ashishdhingra,
we are not using AWS S3 but netways S3 which uses ceph. As I said it does not have the bucketname in the subdomain but in the path. Our actual bucket is not named
11213ef55:mybucket
but very similar, so it does contain a colon.
@desdoades Thanks for the reply. As mentioned in my previous comment, colon :
character is not supported in AWS bucket names. The colon :
is a reserved character in a URL and it has a special meaning in identifying the protocol used in the URL (http:, https:, ftp: etc.). However, you can still add a colon in other parts of a URL by using URL encoding. URL encoding is a process of converting reserved characters into their corresponding hexadecimal code so that they can be safely transmitted over the internet.
Please contact Ceph S3 support to handle this character at their end (this may be as simple as decoding the URL before processing various segments).
Thanks,
Ashish
Dear @ashishdhingra,
While AWS might not support them many site allow colons in the URL.
Still thank you, I understand you wont do anything about it. I will try to reach our ceph S3 support.
Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.