Giters

MarathonLabs / marathon

Cross-platform test runner

Home Page:https://docs.marathonlabs.io

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Android multiple test apks installation issue

msd117c opened this issue · comments

commented

Describe the bug
I have a multimodule app and I want to run test cases on several library modules. I specify them as described in the documentation, one entry for each library's test application (and an entry with application + test application in case of the android app module).

It runs the tests as expected but I noticed that the installation is doing something weird. If I specify one test application it works as expected. If I specify two, it installs them twice, if I specify three it installs each of them three times and so on. It consumes a lot of time and depending on the amount of test apks I need to increase the timeout. In any case I don't think it is the expected behavior (correct me if I'm wrong).

To Reproduce
Steps to reproduce the behaviour:
Configuration:

  ...
  testParserConfiguration:
    type: "remote"
    instrumentationArgs:
      listener: "com.malinskiy.adam.junit4.android.listener.TestAnnotationProducer"
  instrumentationArgs:
    debug: "false"
  disableWindowAnimation: true
  autoGrantPermission: true
  screenRecordConfiguration:
    preferableRecorderType: "screenshot"
  applicationPmClear: true
  testApplicationPmClear: true
  waitForDevicesTimeoutMillis: 60000
  outputs:
    - testApplication: "..."
    - testApplication: "..."
    - testApplication: "..."

More details:
I am using only one emulator and all other configuration has been disabled (i.e batchingStrategy, retryStrategy, shardingStrategy...)

Expected behavior
Marathon installs each apk only once per apk entry.

Logs and reports

[AndroidDevice - execution - 127.0.0.1:5037:emulator-5554-1 @coroutine#13] <AndroidDevice> Device 127.0.0.1:5037:emulator-5554 booted!
[main @coroutine#2] <AndroidAppInstaller> Installing application output to 127.0.0.1:5037:emulator-5554
[main @coroutine#2] <AndroidAppInstaller> Installing instrumentation package to 127.0.0.1:5037:emulator-5554
[main @coroutine#2] <AndroidAppInstaller> Uninstalling com.example1 from 127.0.0.1:5037:emulator-5554
[main @coroutine#2] <AndroidAppInstaller> Success
[main @coroutine#2] <AndroidAppInstaller> Installing com.example1, /Users/***/androidTest/debug/example1-debug-androidTest.apk to 127.0.0.1:5037:emulator-5554
[main @coroutine#2] <AndroidDevice> Transferred example1-debug-androidTest.apk to/from 127.0.0.1:5037:emulator-5554. 158100,75 KB/s (42371118 bytes in 0,2680)
[main @coroutine#2] <AndroidDevice> /data/local/tmp/example1-debug-androidTest.apk synced in 121ms
[main @coroutine#2] <AndroidAppInstaller> Success
[main @coroutine#2] <AndroidAppInstaller> Prepare installation finished for 127.0.0.1:5037:emulator-5554
[main @coroutine#2] <AndroidAppInstaller> Installing application output to 127.0.0.1:5037:emulator-5554
[main @coroutine#2] <AndroidAppInstaller> Installing instrumentation package to 127.0.0.1:5037:emulator-5554
[main @coroutine#2] <AndroidAppInstaller> Uninstalling com.example2 from 127.0.0.1:5037:emulator-5554
[main @coroutine#2] <AndroidAppInstaller> Success
[main @coroutine#2] <AndroidAppInstaller> Installing com.example2, /Users/***/androidTest/debug/example2-debug-androidTest.apk to 127.0.0.1:5037:emulator-5554
[main @coroutine#2] <AndroidDevice> Transferred example2-debug-androidTest.apk to/from 127.0.0.1:5037:emulator-5554. 204397,06 KB/s (41697904 bytes in 0,2040)
[main @coroutine#2] <AndroidDevice> /data/local/tmp/example2-debug-androidTest.apk synced in 115ms
[main @coroutine#2] <AndroidAppInstaller> Success
[main @coroutine#2] <AndroidAppInstaller> Prepare installation finished for 127.0.0.1:5037:emulator-5554
[main @coroutine#2] <AndroidAppInstaller> Installing application output to 127.0.0.1:5037:emulator-5554
[main @coroutine#2] <AndroidAppInstaller> Installing instrumentation package to 127.0.0.1:5037:emulator-5554
[main @coroutine#2] <AndroidAppInstaller> Uninstalling com.example3 from 127.0.0.1:5037:emulator-5554
[main @coroutine#2] <AndroidAppInstaller> Success
[main @coroutine#2] <AndroidAppInstaller> Installing com.example3, /Users/***/androidTest/debug/example3-debug-androidTest.apk to 127.0.0.1:5037:emulator-5554
[main @coroutine#2] <AndroidDevice> Transferred example3-debug-androidTest.apk to/from 127.0.0.1:5037:emulator-5554. 200837,32 KB/s (41975145 bytes in 0,2090)
[main @coroutine#2] <AndroidDevice> /data/local/tmp/example3-debug-androidTest.apk synced in 114ms
[main @coroutine#2] <AndroidAppInstaller> Success
[main @coroutine#2] <AndroidAppInstaller> Prepare installation finished for 127.0.0.1:5037:emulator-5554
[main @coroutine#2] <AndroidAppInstaller> Installing application output to 127.0.0.1:5037:emulator-5554
[main @coroutine#2] <AndroidAppInstaller> Installing instrumentation package to 127.0.0.1:5037:emulator-5554
[main @coroutine#2] <AndroidAppInstaller> Uninstalling com.example1 from 127.0.0.1:5037:emulator-5554
[main @coroutine#2] <AndroidAppInstaller> Success
[main @coroutine#2] <AndroidAppInstaller> Installing com.example1, /Users/***/androidTest/debug/example1-debug-androidTest.apk to 127.0.0.1:5037:emulator-5554
[main @coroutine#2] <AndroidDevice> Transferred example1-debug-androidTest.apk to/from 127.0.0.1:5037:emulator-5554. 195258,06 KB/s (42371118 bytes in 0,2170)
[main @coroutine#2] <AndroidDevice> /data/local/tmp/example1-debug-androidTest.apk synced in 120ms
[main @coroutine#2] <AndroidAppInstaller> Success
[main @coroutine#2] <AndroidAppInstaller> Prepare installation finished for 127.0.0.1:5037:emulator-5554
[main @coroutine#2] <AndroidAppInstaller> Installing application output to 127.0.0.1:5037:emulator-5554
[main @coroutine#2] <AndroidAppInstaller> Installing instrumentation package to 127.0.0.1:5037:emulator-5554
[main @coroutine#2] <AndroidAppInstaller> Uninstalling com.example2 from 127.0.0.1:5037:emulator-5554
[main @coroutine#2] <AndroidAppInstaller> Success
[main @coroutine#2] <AndroidAppInstaller> Installing com.example2, /Users/***/androidTest/debug/example2-debug-androidTest.apk to 127.0.0.1:5037:emulator-5554
[main @coroutine#2] <AndroidDevice> Transferred example2-debug-androidTest.apk to/from 127.0.0.1:5037:emulator-5554. 230370,17 KB/s (41697904 bytes in 0,1810)
[main @coroutine#2] <AndroidDevice> /data/local/tmp/example2-debug-androidTest.apk synced in 117ms
[main @coroutine#2] <AndroidAppInstaller> Success
[main @coroutine#2] <AndroidAppInstaller> Prepare installation finished for 127.0.0.1:5037:emulator-5554
[main @coroutine#2] <AndroidAppInstaller> Installing application output to 127.0.0.1:5037:emulator-5554
[main @coroutine#2] <AndroidAppInstaller> Installing instrumentation package to 127.0.0.1:5037:emulator-5554
[main @coroutine#2] <AndroidAppInstaller> Uninstalling com.example3 from 127.0.0.1:5037:emulator-5554
[main @coroutine#2] <AndroidAppInstaller> Success
[main @coroutine#2] <AndroidAppInstaller> Installing com.example3, /Users/***/androidTest/debug/example3-debug-androidTest.apk to 127.0.0.1:5037:emulator-5554
[main @coroutine#2] <AndroidDevice> Transferred example3-debug-androidTest.apk to/from 127.0.0.1:5037:emulator-5554. 217487,05 KB/s (41975145 bytes in 0,1930)
[main @coroutine#2] <AndroidDevice> /data/local/tmp/example3-debug-androidTest.apk synced in 112ms
[main @coroutine#2] <AndroidAppInstaller> Success
[main @coroutine#2] <AndroidAppInstaller> Prepare installation finished for 127.0.0.1:5037:emulator-5554
[main @coroutine#2] <AndroidAppInstaller> Installing application output to 127.0.0.1:5037:emulator-5554
[main @coroutine#2] <AndroidAppInstaller> Installing instrumentation package to 127.0.0.1:5037:emulator-5554
[main @coroutine#2] <AndroidAppInstaller> Uninstalling com.example1 from 127.0.0.1:5037:emulator-5554
[main @coroutine#2] <AndroidAppInstaller> Success
[main @coroutine#2] <AndroidAppInstaller> Installing com.example1, /Users/***/androidTest/debug/example1-debug-androidTest.apk to 127.0.0.1:5037:emulator-5554
[main @coroutine#2] <AndroidDevice> Transferred example1-debug-androidTest.apk to/from 127.0.0.1:5037:emulator-5554. 165511,72 KB/s (42371118 bytes in 0,2560)
[main @coroutine#2] <AndroidDevice> /data/local/tmp/example1-debug-androidTest.apk synced in 115ms
[main @coroutine#2] <AndroidAppInstaller> Success
[main @coroutine#2] <AndroidAppInstaller> Prepare installation finished for 127.0.0.1:5037:emulator-5554
[main @coroutine#2] <AndroidAppInstaller> Installing application output to 127.0.0.1:5037:emulator-5554
[main @coroutine#2] <AndroidAppInstaller> Installing instrumentation package to 127.0.0.1:5037:emulator-5554
[main @coroutine#2] <AndroidAppInstaller> Uninstalling com.example2 from 127.0.0.1:5037:emulator-5554
[main @coroutine#2] <AndroidAppInstaller> Success
[main @coroutine#2] <AndroidAppInstaller> Installing com.example2, /Users/***/androidTest/debug/example2-debug-androidTest.apk to 127.0.0.1:5037:emulator-5554
[main @coroutine#2] <AndroidDevice> Transferred example2-debug-androidTest.apk to/from 127.0.0.1:5037:emulator-5554. 227852,46 KB/s (41697904 bytes in 0,1830)
[main @coroutine#2] <AndroidDevice> /data/local/tmp/example2-debug-androidTest.apk synced in 114ms
[main @coroutine#2] <AndroidAppInstaller> Success
[main @coroutine#2] <AndroidAppInstaller> Prepare installation finished for 127.0.0.1:5037:emulator-5554
[main @coroutine#2] <AndroidAppInstaller> Installing application output to 127.0.0.1:5037:emulator-5554
[main @coroutine#2] <AndroidAppInstaller> Installing instrumentation package to 127.0.0.1:5037:emulator-5554
[main @coroutine#2] <AndroidAppInstaller> Uninstalling com.example3 from 127.0.0.1:5037:emulator-5554
[main @coroutine#2] <AndroidAppInstaller> Success
[main @coroutine#2] <AndroidAppInstaller> Installing com.example3, /Users/***/androidTest/debug/example3-debug-androidTest.apk to 127.0.0.1:5037:emulator-5554
[main @coroutine#2] <AndroidDevice> Transferred example3-debug-androidTest.apk to/from 127.0.0.1:5037:emulator-5554. 234497,21 KB/s (41975145 bytes in 0,1790)
[main @coroutine#2] <AndroidDevice> /data/local/tmp/example3-debug-androidTest.apk synced in 113ms
[main @coroutine#2] <AndroidAppInstaller> Success
[main @coroutine#2] <AndroidAppInstaller> Prepare installation finished for 127.0.0.1:5037:emulator-5554
Took 21518 ms

Devices (please complete the following information):

  • Device: Pixel 4XL emulator
  • OS: API 31

Additional context
I thought that this might be happening due to not cleaning the emulator between executions but the same behavior is happening on the CI (fresh emulator on each execution), the marathon execution takes, for instance, 21 minutes and according to the report the test cases were executed in ~6 minutes.

In case more info is needed I will add it here with the question's context.

commented

That's an interesting one. There is a duplication of installation expected from the remote test parser, but not to this extent. Device preparation involves installing all of the testing bundles, and the test parser mistakenly called prepare for each bundle, hence the behaviour you observed. Thanks for submitting this!

commented

@Malinskiy do you have a planned date for a new release with this change? If not how could I build a new release locally to use the cli with these changes? Is there anything I have to be aware of when doing that? Thank you again!

commented

No, I have a newborn and don't have any timelines for the next months to come.
You can repeat the steps on CI: https://github.com/MarathonLabs/marathon/blob/develop/.github/workflows/ci.yml#L80 without the release flags

commented

Also there is another gradle task for local installation that is documented here https://docs.marathonlabs.io/runner/intro/contribute#cli

commented

Great, thank you for all the quick and helpful support!