Irregularities found when extending a test class with annotations - Bug

[harshitAtlassian]

Describe the bug

Flank behaviour differs from Gcloud.

Gcloud takes into account the Parameterized annotation associated with the base class while the flank requires that the Parameterized annotation to be associated with the extended class as well.

To Reproduce

Create any test class and mark it with annotation @RunWith(value = Parameterized::class). Now, extend this class to make another test class. The test cases of the extended class won't be executed unless we mark it with annotation @RunWith(value = Parameterized::class) which was not the case with Gcloud.

Example ->

@RunWith(value = Parameterized::class)
open class SampleScreenshotTest(private val asset: String)

class SampleScreenshotDarkTest(asset: String) : SampleScreenshotTest(asset) 

GCloud executes tests for SampleScreenshotDarkTest where as flank doesn't.

Expected behavior

It should not be mandatory to annotate extended class with @RunWith(value = Parameterized::class) to run its test cases.

Details (please complete the following information):

Have you tested on the latest Flank snapshot?
Yes, tested on Flank v23.10.1

[Yoann Goular]

Similar issue on my side regarding the support of the @Suite.SuiteClasses annotation :

package com.example.test_app.bar

import com.example.test_app.BaseInstrumentedTest
import org.junit.runner.RunWith
import org.junit.runners.Suite
import com.example.test_app.bar.BarInstrumentedTest
import com.example.test_app.foo.FooInstrumentedTest

@Suite.SuiteClasses(
    BarInstrumentedTest::class,
    FooInstrumentedTest::class
)
@RunWith(Suite::class)
class FooBarTestSuite : BaseInstrumentedTest()

It is properly supported using an adb command adb shell am instrumentation -e "class com.example.test_app.bar.FooBarTestSuite" com.test.package/com.test.runner as well as using Gcloud CLI directly (--test-targets "class com.example.test_app.bar.FooBarTestSuite").

I was able to run my test suite in both cases but it fails using Flank with the following output:

RunTests
  No tests for app-debug-androidTest.apk

  Saved 0 shards to /path/to/the/project/app/build/fladle/virtual/results/2024-01-10_16-41-38.544741_vfbO/android_shards.json
  Uploading [android_shards.json] to https://console.developers.google.com/storage/browser/project/path...

There are no Android tests to run.

Total run duration: 0m  4s
        - Preparation: 0m 2s

> Task :app:execFlankVirtual FAILED

At first sight, it seems to be either a filter or shard calculation issue, maybe in CreateAndroidTestContext 🤷‍♂️:

private suspend fun List<AndroidTestContext>.setupShards(): List<AndroidTestContext> = coroutineScope {
    map { testContext ->
        async {
            val newArgs = testContext.args
            val filters = TestFilters.fromTestTargets(newArgs.testTargets, newArgs.testTargetsForShard)
            when {
                testContext !is InstrumentationTestContext -> testContext
                newArgs.useCustomSharding -> testContext.userShards(newArgs.customSharding)
                newArgs.useTestTargetsForShard -> testContext.downloadApks().calculateDummyShards(newArgs, filters)
                else -> testContext.downloadApks().calculateShards(newArgs, filters)
            }
        }
    }.awaitAll().dropEmptyInstrumentationTest()
}

The dropEmptyInstrumentationTest is suspicious to me as well as the shard calculation. It seems that it fails to compute the shards (0 shards detectedfrom the logs) and it considers there are no tests to run.

Is there any workaround I could apply to run my test suites on Firebase Test Lab using Flank? I tried to deactivate the sharding using disableSharding.set(true) but it was not successful either. 🙏

Thanks

EDIT: I was able to find a workaround -> Adding an empty fake test succeeds in executing the test suite.

package com.example.test_app.bar

import com.example.test_app.BaseInstrumentedTest
import org.junit.runner.RunWith
import org.junit.runners.Suite
import com.example.test_app.bar.BarInstrumentedTest
import com.example.test_app.foo.FooInstrumentedTest

@Suite.SuiteClasses(
    BarInstrumentedTest::class,
    FooInstrumentedTest::class
)
@RunWith(Suite::class)
class FooBarTestSuite : BaseInstrumentedTest(){
    @Test
    fun workaroundTestToWorkWithFlank() {
    }
}

That being said, it seems that if any of the tests inside the suite is failing and I set flakyTestAttempts.set(1), it will rerun the full suite again instead of just the failing test... The JUnit 4 @Suite.SuiteClasses is not well supported in the library.

Also, the sharding is not efficient either in this case as it takes longer to execute 62 tests using @Suite.SuiteClasses (Total run duration: 18m 52s) than to execute 1599 regular tests (Total run duration: 11m 33s) 🙈.

Do you think this is something that could be improved?