Communication with MinIO not working when forcePathStyle = false

Question

Communication with MinIO not working when forcePathStyle = false

RodrasSilva opened this issue 3 months ago · comments

Describe the bug

When configuring S3Client, if I set the forcePathStyle config to false I get the following error when calling SDK methods:

Caused by: software.amazon.awssdk.crt.http.HttpException: Host name was invalid for dns resolution.
	at software.amazon.awssdk.crt.http.HttpClientConnection.onConnectionAcquired(HttpClientConnection.java:112)

The S3Client is configured as follows:

 return S3Client {
            this.httpClient(CrtHttpEngine) {
                connectTimeout = 1500.milliseconds
                socketWriteTimeout = 1500.milliseconds
                socketReadTimeout = 1500.milliseconds
                maxConcurrency = 150.toUInt()
                connectionAcquireTimeout = 30.minutes
            }
            this.region = "us-east-1"
            this.credentialsProvider = StaticCredentialsProvider {
                accessKeyId = "user"
                secretAccessKey = "password"
            }
            this.retryStrategy {
                delayProvider {
                    initialDelay = 100.milliseconds
                    maxBackoff = 1000.milliseconds
                }
                maxAttempts = 3
            }
            this.forcePathStyle = false
            this.endpointUrl = Url.parse("http://localhost:9000")
        }

The communication with the SDK is working as expected when forcePathStyle is equal to true.

Expected behavior

The SDK should behave the same when the forcePathStyle is false or true.

Current behavior

When forcePathStyle is configured as false, the SDK loses its ability to communicate with the object storage system.

Steps to Reproduce

Create an SDK client as described above, call any SDK method, and change the forcePathStyle between true and false to verify the issue.
MinIO is running in a container with the following environment variables:

  MINIO_ROOT_USER: "user"
  MINIO_ROOT_PASSWORD: "password"
  MINIO_DOMAIN: "minio:9000" # this tells MinIO to accept virtual host-style URLs, minio is also the container name

Possible Solution

No response

Context

This discovery surfaced during the transition from the Java AWS SDK to the Kotlin SDK. The configurations employed in the Kotlin SDK mirror those utilized in the Java SDK, including MinIO settings, which remain consistent. Notably, while utilizing the Java SDK, interaction with MinIO was achieved by adjusting the forcePathStyle parameter between true and false states.

AWS Kotlin SDK version used

1.0.54

Platform (JVM/JS/Native)

JVM

Operating System and version

macOS 14.3.1 (Sonoma)

Aaron Todd · Answer 1 · Mon Feb 19 2024 20:42:47 GMT+0800 (China Standard Time)

Thanks for the issue.

The SDK should behave the same when the forcePathStyle is false or true.

This is not going to hold true for this config setting. When forcePathStyle is false the SDK will default to using "virtual" hosted bucket endpoints which look like https://bucketname.s3.amazonaws.com (there are some exceptions to this e.g. the bucket name isn't a valid DNS host name). When it's set to true it will force the endpoint to something like: https://s3.amazonaws.com/bucketname

You can read more about how this setting works by looking at the addressing_style section of the S3 specific settings here.

You say the Java SDK works like this. I'd be interested in seeing what the actual HTTP request URI and host are. If you can provide some more logs for each that would be helpful. See logging in the Kotlin SDK developer guide for more info if needed.

Rodrigo Silva · Answer 2 · Mon Feb 19 2024 21:48:04 GMT+0800 (China Standard Time)

Hello @aajtodd
I understand that url used to make requests when forcePathStyle is false is different since it will use the default "virtual" hosted bucket endpoints.
I've added the following property to the S3Client:

 return S3Client {
            this.logMode =  LogRequestWithBody + LogResponse
            ...
      }

When forcePathStyle=true the logs are the following:

[xxxx] 2024-02-19T13:38:08,898 munch-imageservice-api [cb-io-kv-5-3] DEBUG aws.smithy.kotlin.runtime.http.operation.AuthHandler - resolved endpoint: Endpoint(uri=http://localhost:9000/images, headers=null, attributes={AttributeKey(aws.smithy.kotlin#endpointAuthSchemes)=[AuthOptionImpl(schemeId=AuthSchemeId(id=aws.auth#sigv4), attributes={AttributeKey(aws.smithy.kotlin.signing#AwsSigningRegion)=us-east-1, AttributeKey(aws.smithy.kotlin.signing#AwsSigningService)=s3, AttributeKey(aws.smithy.kotlin.signing#UseDoubleUriEncode)=false})]})
[xxxx] 2024-02-19T13:38:08,914 munch-imageservice-api [cb-io-kv-5-3] DEBUG aws.smithy.kotlin.runtime.auth.awssigning.DefaultAwsSignerImpl - Calculated signature: a7198217064fdd5973c97658dcaa1f19dd04f3f16725152af36465fdc876fea7
[UKDH2C0W2TL4] 2024-02-19T13:38:08,924 munch-imageservice-api [cb-io-kv-5-3] DEBUG httpTraceMiddleware - HttpRequest:
PUT /images/078dd317-b51d-4ea6-b2be-00716d6d77fc?x-id=PutObject
Host: localhost
Content-Length: 227963
Content-Type: image/png
User-Agent: aws-sdk-kotlin/1.0.54 ua/2.0 api/s3#1.0.54 os/macos#14.3.1 lang/kotlin#1.9.22 md/javaVersion#17.0.10 md/jvmName#OpenJDK_64-Bit_Server_VM md/jvmVersion#17.0.10+7-LTS 
x-amz-user-agent: aws-sdk-kotlin/1.0.54
amz-sdk-invocation-id: 22f4ae0a-93dd-413e-adae-fefdb4e38770
amz-sdk-request: attempt=1; max=3
X-Amz-Content-Sha256: 3967b4fc85eca8a835cc5c69800362a7c4c5050abe3e36260251edc63eba518f
X-Amz-Date: 20240219T133808Z
Authorization: AWS4-HMAC-SHA256 Credential=admin/20240219/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;content-length;content-type;host;x-amz-content-sha256;x-amz-date;x-amz-user-agent, Signature=a7198217064fdd5973c97658dcaa1f19dd04f3f16725152af36465fdc876fea7
... 
[xxxx] 2024-02-19T13:38:08,969 munch-imageservice-api [DefaultDispatcher-worker-1] DEBUG httpTraceMiddleware - HttpResponse:
HTTP 200: OK
Accept-Ranges: bytes
Content-Length: 0
ETag: "3942b3b4bed2e098ea28a0a81fd38e91"
Server: MinIO
Strict-Transport-Security: max-age=31536000; includeSubDomains
Vary: Origin;Accept-Encoding
X-Amz-Id-2: dd9025bab4ad464b049177c95eb6ebf374d3b3fd1af9251148b658df7ac2e3e8
X-Amz-Request-Id: 17B5472C435208EA
X-Content-Type-Options: nosniff
X-Xss-Protection: 1; mode=block
x-amz-version-id: ce6dfc55-1b20-416e-95f5-4ebfdb9e8900
Date: Mon, 19 Feb 2024 13:38:09 GMT

When forcePathStyle=false the logs are the following:

[xxxx] 2024-02-19T13:44:55,021 [main @coroutine#3] DEBUG aws.smithy.kotlin.runtime.http.operation.AuthHandler - resolved endpoint: Endpoint(uri=http://images.localhost:9000, headers=null, attributes={AttributeKey(aws.smithy.kotlin#endpointAuthSchemes)=[AuthOptionImpl(schemeId=AuthSchemeId(id=aws.auth#sigv4), attributes={AttributeKey(aws.smithy.kotlin.signing#AwsSigningRegion)=us-east-1, AttributeKey(aws.smithy.kotlin.signing#AwsSigningService)=s3, AttributeKey(aws.smithy.kotlin.signing#UseDoubleUriEncode)=false})]})
[UKDH2C0W2TL4] 2024-02-19T13:44:55,033 [main @coroutine#3] DEBUG aws.smithy.kotlin.runtime.auth.awssigning.DefaultAwsSignerImpl - Calculated signature: 93b143465ac1b42166d99f9490bf39205eec688d88be852910eb97a215b810e5
[UKDH2C0W2TL4] 2024-02-19T13:44:55,041 [main @coroutine#3] DEBUG httpTraceMiddleware - HttpRequest:
GET /?versions=
Host: images.localhost
User-Agent: aws-sdk-kotlin/1.0.54 ua/2.0 api/s3#1.0.54 os/macos#14.3.1 lang/kotlin#1.9.22 md/javaVersion#17.0.10 md/jvmName#OpenJDK_64-Bit_Server_VM md/jvmVersion#17.0.10+7-LTS 
x-amz-user-agent: aws-sdk-kotlin/1.0.54
amz-sdk-invocation-id: e8aeb02b-7f97-4568-8f6b-0dda0cd5b14f
amz-sdk-request: attempt=1; max=3
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20240219T134455Z
Authorization: AWS4-HMAC-SHA256 Credential=admin/20240219/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;host;x-amz-content-sha256;x-amz-date;x-amz-user-agent, Signature=93b143465ac1b42166d99f9490bf39205eec688d88be852910eb97a215b810e5


[UKDH2C0W2TL4] 2024-02-19T13:44:55,061 [main @coroutine#3] DEBUG aws.smithy.kotlin.runtime.http.middleware.RetryMiddleware - request failed with non-retryable error
[UKDH2C0W2TL4] 2024-02-19T13:44:55,062 [main @coroutine#3] DEBUG io.github.resilience4j.circuitbreaker.internal.CircuitBreakerStateMachine - CircuitBreaker 'munch-s3-image-circuit-breaker-tests' recorded an exception as failure:
aws.smithy.kotlin.runtime.http.HttpException: software.amazon.awssdk.crt.http.HttpException: Host name was invalid for dns resolution.
	at aws.smithy.kotlin.runtime.http.engine.crt.ConnectionManager.acquire(ConnectionManager.kt:81) ~[http-client-engine-crt-jvm-1.0.13.jar:?]
	at aws.smithy.kotlin.runtime.http.engine.crt.ConnectionManager$acquire$1.invokeSuspend(ConnectionManager.kt) ~[http-client-engine-crt-jvm-1.0.13.jar:?]
	at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33) ~[kotlin-stdlib-1.9.22.jar:1.9.22-release-704]
	at kotlinx.coroutines.internal.ScopeCoroutine.afterResume(Scopes.kt:32) ~[kotlinx-coroutines-core-jvm-1.7.3.jar:?]
	at kotlinx.coroutines.AbstractCoroutine.resumeWith(AbstractCoroutine.kt:102) ~[kotlinx-coroutines-core-jvm-1.7.3.jar:?]
	at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:46) ~[kotlin-stdlib-1.9.22.jar:1.9.22-release-704]
	at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:106) ~[kotlinx-coroutines-core-jvm-1.7.3.jar:?]
	at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:584) ~[kotlinx-coroutines-core-jvm-1.7.3.jar:?]
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:793) ~[kotlinx-coroutines-core-jvm-1.7.3.jar:?]
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:697) ~[kotlinx-coroutines-core-jvm-1.7.3.jar:?]
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:684) ~[kotlinx-coroutines-core-jvm-1.7.3.jar:?]
Caused by: software.amazon.awssdk.crt.http.HttpException: Host name was invalid for dns resolution.

Aaron Todd · Answer 3 · Mon Feb 19 2024 21:58:45 GMT+0800 (China Standard Time)

Thanks for the debug output.

In the first output the Host header is set to localhost. In the second the Host header is set to images.localhost which corresponds to a virtual hosted bucket with bucket name images.

This would appear correct to me. I'm not sure what the expected behavior you're looking for is here.

Notably, while utilizing the Java SDK, interaction with MinIO was achieved by adjusting the forcePathStyle parameter between true and false states.

Are you saying the Java SDK works with both settings against minio or are you saying you had to toggle it to get it to work? I'd be curious to see the actual code and the debug output that shows what URI/host is being used for the Java SDK. As far as I can tell the Kotlin SDK is behaving correctly here for the configuration given.

github-actions · Answer 4 · Tue Feb 20 2024 21:54:45 GMT+0800 (China Standard Time)

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.