Unable to mock S3's presignGetObject call

Question

Unable to mock S3's presignGetObject call

hugoncosta opened this issue 3 months ago · comments

Hugo Costa commented 3 months ago

Describe the bug

I am unable to mock a call to the S3Client's presignGetObject function as part of a unit test.

Expected behavior

The test should run successfully.

Current behavior

The test fails with the following stacktrace

java.lang.IllegalArgumentException:  is not a valid inet host

	at aws.smithy.kotlin.runtime.net.HostKt.hostParseImpl(Host.kt:35)
	at aws.smithy.kotlin.runtime.net.HostKt.access$hostParseImpl(Host.kt:1)
	at aws.smithy.kotlin.runtime.net.Host$Companion.parse(Host.kt:14)
	at aws.smithy.kotlin.runtime.http.operation.OperationEndpointKt$setResolvedEndpoint$1.invoke(OperationEndpoint.kt:70)
	at aws.smithy.kotlin.runtime.http.operation.OperationEndpointKt$setResolvedEndpoint$1.invoke(OperationEndpoint.kt:66)
	at aws.smithy.kotlin.runtime.http.request.HttpRequestBuilderKt.url(HttpRequestBuilder.kt:71)
	at aws.smithy.kotlin.runtime.http.operation.OperationEndpointKt.setResolvedEndpoint(OperationEndpoint.kt:66)
	at aws.smithy.kotlin.runtime.auth.awssigning.PresignerKt.presignRequest(Presigner.kt:35)
	at aws.sdk.kotlin.services.s3.presigners.PresignersKt.presignGetObject(Presigners.kt:52)
	at aws.sdk.kotlin.services.s3.presigners.PresignersKt.presignGetObject$default(Presigners.kt:40)
	at aws.sdk.kotlin.services.s3.presigners.PresignersKt.presignGetObject-exY8QGI(Presigners.kt:30)
	at com.amazon.networkvalidator.s3.S3AccessorTest$getObjectPresigned$1$1.invokeSuspend(S3AccessorTest.kt:74)
	at com.amazon.networkvalidator.s3.S3AccessorTest$getObjectPresigned$1$1.invoke(S3AccessorTest.kt)
	at com.amazon.networkvalidator.s3.S3AccessorTest$getObjectPresigned$1$1.invoke(S3AccessorTest.kt)
	at io.mockk.impl.eval.RecordedBlockEvaluator$record$block$2$1.invokeSuspend(RecordedBlockEvaluator.kt:27)
	at io.mockk.impl.eval.RecordedBlockEvaluator$record$block$2$1.invoke(RecordedBlockEvaluator.kt)
	at io.mockk.impl.eval.RecordedBlockEvaluator$record$block$2$1.invoke(RecordedBlockEvaluator.kt)
	at io.mockk.InternalPlatformDsl$runCoroutine$1.invokeSuspend(InternalPlatformDsl.kt:23)
	at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
	at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:104)
	at kotlinx.coroutines.EventLoopImplBase.processNextEvent(EventLoop.common.kt:277)
	at kotlinx.coroutines.BlockingCoroutine.joinBlocking(Builders.kt:95)
	at kotlinx.coroutines.BuildersKt__BuildersKt.runBlocking(Builders.kt:69)
	at kotlinx.coroutines.BuildersKt.runBlocking(Unknown Source)
	at kotlinx.coroutines.BuildersKt__BuildersKt.runBlocking$default(Builders.kt:48)
	at kotlinx.coroutines.BuildersKt.runBlocking$default(Unknown Source)
	at io.mockk.InternalPlatformDsl.runCoroutine(InternalPlatformDsl.kt:22)
	at io.mockk.impl.eval.RecordedBlockEvaluator$record$block$2.invoke(RecordedBlockEvaluator.kt:27)
	at io.mockk.impl.eval.RecordedBlockEvaluator$enhanceWithRethrow$1.invoke(RecordedBlockEvaluator.kt:76)
	at io.mockk.impl.recording.JvmAutoHinter.autoHint(JvmAutoHinter.kt:23)
	at io.mockk.impl.eval.RecordedBlockEvaluator.record(RecordedBlockEvaluator.kt:39)
	at io.mockk.impl.eval.EveryBlockEvaluator.every(EveryBlockEvaluator.kt:30)
	at io.mockk.MockKDsl.internalCoEvery(API.kt:100)
	at io.mockk.MockKKt.coEvery(MockK.kt:169)
	at com.amazon.networkvalidator.s3.S3AccessorTest$getObjectPresigned$1.invokeSuspend(S3AccessorTest.kt:74)
	at com.amazon.networkvalidator.s3.S3AccessorTest$getObjectPresigned$1.invoke(S3AccessorTest.kt)
	at com.amazon.networkvalidator.s3.S3AccessorTest$getObjectPresigned$1.invoke(S3AccessorTest.kt)
	at kotlinx.coroutines.test.TestBuildersKt__TestBuildersKt$runTest$2$1$1.invokeSuspend(TestBuilders.kt:316)
	at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
	at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:104)
	at kotlinx.coroutines.test.TestDispatcher.processEvent$kotlinx_coroutines_test(TestDispatcher.kt:24)
	at kotlinx.coroutines.test.TestCoroutineScheduler.tryRunNextTaskUnless$kotlinx_coroutines_test(TestCoroutineScheduler.kt:99)
	at kotlinx.coroutines.test.TestBuildersKt__TestBuildersKt$runTest$2$1$workRunner$1.invokeSuspend(TestBuilders.kt:322)
	at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
	at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:104)
	at kotlinx.coroutines.EventLoopImplBase.processNextEvent(EventLoop.common.kt:277)
	at kotlinx.coroutines.BlockingCoroutine.joinBlocking(Builders.kt:95)
	at kotlinx.coroutines.BuildersKt__BuildersKt.runBlocking(Builders.kt:69)
	at kotlinx.coroutines.BuildersKt.runBlocking(Unknown Source)
	at kotlinx.coroutines.BuildersKt__BuildersKt.runBlocking$default(Builders.kt:48)
	at kotlinx.coroutines.BuildersKt.runBlocking$default(Unknown Source)
	at kotlinx.coroutines.test.TestBuildersJvmKt.createTestResult(TestBuildersJvm.kt:10)
	at kotlinx.coroutines.test.TestBuildersKt__TestBuildersKt.runTest-8Mi8wO0(TestBuilders.kt:310)
	at kotlinx.coroutines.test.TestBuildersKt.runTest-8Mi8wO0(Unknown Source)
	at kotlinx.coroutines.test.TestBuildersKt__TestBuildersKt.runTest-8Mi8wO0(TestBuilders.kt:168)
	at kotlinx.coroutines.test.TestBuildersKt.runTest-8Mi8wO0(Unknown Source)
	at kotlinx.coroutines.test.TestBuildersKt__TestBuildersKt.runTest-8Mi8wO0$default(TestBuilders.kt:160)
	at kotlinx.coroutines.test.TestBuildersKt.runTest-8Mi8wO0$default(Unknown Source)
	at com.amazon.networkvalidator.s3.S3AccessorTest.getObjectPresigned(S3AccessorTest.kt:73)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
	at org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:727)
	at org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60)
	at org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:131)
	at org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:156)
	at org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestableMethod(TimeoutExtension.java:147)
	at org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestMethod(TimeoutExtension.java:86)
	at org.junit.jupiter.engine.execution.InterceptingExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(InterceptingExecutableInvoker.java:103)
	at org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.lambda$invoke$0(InterceptingExecutableInvoker.java:93)
	at org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:106)
	at org.junit.jupiter.engine.execution.InvocationInterceptorChain.proceed(InvocationInterceptorChain.java:64)
	at org.junit.jupiter.engine.execution.InvocationInterceptorChain.chainAndInvoke(InvocationInterceptorChain.java:45)
	at org.junit.jupiter.engine.execution.InvocationInterceptorChain.invoke(InvocationInterceptorChain.java:37)
	at org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.invoke(InterceptingExecutableInvoker.java:92)
	at org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.invoke(InterceptingExecutableInvoker.java:86)
	at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeTestMethod$7(TestMethodTestDescriptor.java:217)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.invokeTestMethod(TestMethodTestDescriptor.java:213)
	at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:138)
	at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:68)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:151)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
	at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
	at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:41)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:155)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
	at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
	at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:41)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:155)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
	at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95)
	at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.submit(SameThreadHierarchicalTestExecutorService.java:35)
	at org.junit.platform.engine.support.hierarchical.HierarchicalTestExecutor.execute(HierarchicalTestExecutor.java:57)
	at org.junit.platform.engine.support.hierarchical.HierarchicalTestEngine.execute(HierarchicalTestEngine.java:54)
	at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:147)
	at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:127)
	at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:90)
	at org.junit.platform.launcher.core.EngineExecutionOrchestrator.lambda$execute$0(EngineExecutionOrchestrator.java:55)
	at org.junit.platform.launcher.core.EngineExecutionOrchestrator.withInterceptedStreams(EngineExecutionOrchestrator.java:102)
	at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:54)
	at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:114)
	at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:86)
	at org.junit.platform.launcher.core.DefaultLauncherSession$DelegatingLauncher.execute(DefaultLauncherSession.java:86)
	at org.junit.platform.launcher.core.SessionPerRequestLauncher.execute(SessionPerRequestLauncher.java:53)
	at com.intellij.junit5.JUnit5IdeaTestRunner.startRunnerWithArgs(JUnit5IdeaTestRunner.java:57)
	at com.intellij.rt.junit.IdeaTestRunner$Repeater$1.execute(IdeaTestRunner.java:38)
	at com.intellij.rt.execution.junit.TestsRepeater.repeat(TestsRepeater.java:11)
	at com.intellij.rt.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:35)
	at com.intellij.rt.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:232)
	at com.intellij.rt.junit.JUnitStarter.main(JUnitStarter.java:55)

Steps to Reproduce

@Disabled
@Test
fun getObjectPresigned_exception() = runTest {
    coEvery { client.presignGetObject(GET_OBJECT_REQUEST, any()) } throws S3Exception("Error")
    assertThrows<IllegalStateException> { s3Accessor.getObjectPresigned(S3BUCKET, KEY) }
}

the function it calls is the following

class S3Accessor(
    private val client: S3Client = S3Client { region = "us-east-1" }
) {

suspend fun getObjectPresigned(
        s3Bucket: String,
        keyName: String
    ): String {
        val request = GetObjectRequest {
            bucket = s3Bucket
            key = keyName
        }

        return try {
            client.presignGetObject(request, 6.hours).url.toString()
        } catch (e: S3Exception) {
            log.error(e)
            error("There was an issue creating the presigned link for $keyName from ${s3Bucket}: ${e.message}")
        }
    }    
}

Possible Solution

No response

Context

I am trying to create a unit test to ensure that if an error is thrown, it is thrown captured as I expect it. I am also facing the issue if I want to mock that the function returns some ByteStream.

AWS Kotlin SDK version used

1.0.67 (?)

Platform (JVM/JS/Native)

JVM (JDK17)

Operating System and version

macOS Sonoma 14.3.1

Matas · Answer 1 · Fri Mar 15 2024 04:29:53 GMT+0800 (China Standard Time)

Thanks for the report! I am able to replicate the failure. The reason it throws this exception is because MockK tries to instantiate an empty Host for mocking which then causes a parse failure in the SDK.

...
2024-03-14 16:24:17 TRACE io.mockk.impl.instantiation.AbstractMockFactory:17 - Building proxy for Host hashcode=dde6f87
2024-03-14 16:24:17 TRACE io.mockk.proxy.jvm.ProxyMaker:17 - Class class aws.smithy.kotlin.runtime.net.Host is sealed, will use its subclass class aws.smithy.kotlin.runtime.net.Host$Domain to build proxy
2024-03-14 16:24:17 TRACE io.mockk.proxy.jvm.transformation.JvmInlineInstrumentation:17 - Retransforming classes aws.smithy.kotlin.runtime.net.Host$Domain, aws.smithy.kotlin.runtime.net.Host
2024-03-14 16:24:17 TRACE io.mockk.proxy.jvm.ProxyMaker:17 - Taking instance of class aws.smithy.kotlin.runtime.net.Host$Domain itself because it is final.
2024-03-14 16:24:17 TRACE io.mockk.proxy.jvm.ProxyMaker:17 - Instantiating proxy for class aws.smithy.kotlin.runtime.net.Host$Domain via instantiator
2024-03-14 16:24:17 TRACE io.mockk.proxy.jvm.ObjenesisInstantiator:17 - Creating new empty instance of class aws.smithy.kotlin.runtime.net.Host$Domain
hostname:

Note: hostname is an empty string at this point , so parsing can never succeed.

This seems to be an issue with MockK / your configuration of it rather than the SDK. I'm not too familiar with using MockK, is this enough extra information to help resolve your issue?

Also, do you need to use MockK? If not, you could consider writing an interceptor to throw this exception rather than have MockK do it.

Hugo Costa · Answer 2 · Wed Mar 20 2024 22:33:50 GMT+0800 (China Standard Time)

I didn't include any configuration, and for mocking purposes, I wouldn't expect any needed. Is it not possible to include a default hostname so that MockK doesn't fail? From all the s3 methods, this is the only one that fails.

As to why use MockK, this function fails as well

@Disabled
@Test
fun getObjectPresigned() = runTest {
    coEvery { client.presignGetObject(GET_OBJECT_REQUEST, 6.hours) } returns HTTP_REQUEST
    val output = s3Accessor.getObjectPresigned(S3BUCKET, KEY)
    assertEquals(Url.toString(), output)
}