Support test methods with Kotlin suspend modifier
IRus opened this issue · comments
Goals
Support running suspend test in JUnit Jupiter:
class Kit {
@Test
suspend fun foo() {
delay(1000) // suspend call
assertEquals(1, 1)
}
}Currently, such test can be written this way:
class Kit {
@Test
fun foo() = runBlocking {
delay(1000) // suspend call
assertEquals(1, 1)
assertThrows { /* ... */ }
Unit // test should return void, but `assertThrows` returns `Throwable`, so `foo` returns `Throwable` too
}
}Also, will be nice to provide CoroutineScope through params, or as receiver in extension:
class Kit {
suspend fun foo(scope: CoroutinesScope) { /* ... */ } // (1)
suspend fun CoroutinesScope.foo() { /* ... */ } // (2)
}1 and 2 actually the same on bytecode level. suspend is optional.
And finally, support for runBlockingTest:
class Kit {
suspend fun foo(scope: TestCoroutinesScope) { /* ... */ }
suspend fun TestCoroutinesScope.foo() { /* ... */ }
}What can be done currently
ParameterResolver can be used to provide stubs for Continuation, CoroutineScope and TestCoroutineScope. These stub arguments can be replaced with real arguments in invocation.
Problems
Current extensions points not enough to implement this feature as extensions, since:
- Discovery. Jupiter discovers tests that returns
void, butsuspend funreturnsObject; - Invocation.
InvocationInterceptorin 5.5-M1(SNAPSHOT) don't providing mechanism to override actual invocation, only to decoration of existing invocation. Conversion ofmethodtokotlinFunction, and then executing usingcallSuspendis necessary to executesuspend fun.
Also, my slides about this topic.
Related to #1851
@marcphilipp Yes, not resolved, this is different issues
Please resolve this! My tests looked all fine until I discovered that they were not run at all anymore after adding suspending functions :/
An alternate work around is to create a wrapper around runBlocking() that returns Unit
fun suspendingTest(context: CoroutineContext = EmptyCoroutineContext, block: suspend CoroutineScope.() -> Any): Unit {
runBlocking(context, block)
Unit
}That will allow tests to use suspending methods:
@Test
fun `a suspending test should compile and be found`() = suspendingTest {
assertThat(thing()).isEqualTo("a string")
}
suspend fun thing() = "a string"Would adding that suspendingTest method be useful for JUnit or would supporting tests with suspend be preferred?
@IRus Thanks for pointing that out! At least now this issue documents it as an option.
I would be careful with using runBlockingTest. It does a lot of things, one of them is overriding the behavior of delay(), to allow the test to control the passage of time across coroutines. By default it makes all calls to delay() a no-op.
Anyone who needs to wait in a test will get unexpected behavior if they use delay (which would be the recommended way of doing it).
Might be interesting to know that I implemented @IRus' suggestion in jqwik.net. With the extension point (aka lifecycle hook) for invokeMethods present, the implementation is rather simple: https://github.com/jlink/jqwik/blob/main/kotlin/src/main/kotlin/net/jqwik/kotlin/internal/SuspendedPropertyMethodsHook.kt
@IRus I did not implement parameter resolution for CoroutineScope and TestCoroutineScope. In which cases would these be necessary?
@jlink I wonder how you work around discovery issue: suspend methods returns Object
Scope is nice to have to set test dispatcher by default to TestCoroutineDispatcher (which supports "time" manipulations) for example.
I wonder how you work around discovery issue:
suspendmethods returnsObject
jqwik doesn't have that problem since it allows any return type in test/property methods.
I wonder how you work around discovery issue:
suspendmethods returnsObjectjqwik doesn't have that problem since it allows any return type in test/property methods.
So it's an engine on its own, ok
This is a recurrent source of bugs for us.
A correct test that test a buggy piece of production code get ignored because of this.
Would be nice to see junit supports this.
In the meantime, I opeend a ticket to prevent IntelliJ users against that mistake
I have discovered a way to make suspend work with JUnit Jupiter tests by bridging them through a @TestTemplate.
https://gist.github.com/ephemient/01d6e5766e6f8ea02839b4d7c3f94e55
However, this is not ideal as it doesn't play well with other extensions (e.g. #378).
There really needs to be either better extension points or built-in support for this.
I also ran into this because I had the idea of trying to write a TestMethodInvoker that worked with the Quarkus test extensions that would handle setting up Vertx & Panache session context for suspend functions similar to the existing @RunOnVertxContext annotation that works with Mutiny.Uni. This would appear to be perfectly possible if it were not for JUnit ignoring non-void methods - because of the logic in IsTestableMethod - and there being no way for an extension to override this.
(I suspect this limitation also explains the cumbersome UniAsserter mechanism that @RunOnVertxContext uses, rather than having test methods that return Uni.)
For anyone coming from a search engine, this is still not supported, but it looks possible using runTest, which replaces the deprecated runBlockingTest.
Has anyone tried using the new runTest?
I am using it.
Adding suspend support is not straightforward as runTest accepts parameters and add a receiver to the scope, which would be missing by only adding suspend to the test.
Yes runTest works fine. It’s just rather confusing that it’s not supported out of the box at this point.
runTest is great...
... until you need to integrate with Lincheck and would like to run suspending set up code in @BeforeAll.