Inconsistent html report
MyDogTom opened this issue · comments
Describe the bug
marathon/html/pools/omni.html
page says that test failed, but when I click on failed test and go to test details, report says that test succeeded. Individual test report page is the following: marathon/html/pools/omni/127.0.0.1-5037-emulator-5554/com.example.MyTest.html
I execute single test classMyTest.html
which contains 7 different tests. 5 passes, but 2 tests fail. I noticed that on tests list page (marathon/html/pools/omni.html
) every test references the same marathon/html/pools/omni/127.0.0.1-5037-emulator-5554/com.example.MyTest.html
file and marathon/html/pools/omni/127.0.0.1-5037-emulator-5554
folder contains only one file. This file contains only one entry about the last executed test.
I suspect that it might be related to count sharding usage
shardingStrategy:
type: "count"
count: 5
To Reproduce
Configuration file
name: "UI tests"
outputDir: "../../build/reports/marathon"
screenRecordingPolicy: "ON_FAILURE"
deviceInitializationTimeoutMillis: 180000
debug: true
shardingStrategy:
type: "count"
count: 5
executionStrategy:
mode: ALL_SUCCESS
uncompletedTestRetryQuota: 3
vendorConfiguration:
type: "Android"
# Required for parameterized tests to be run correctly. More info in https://github.com/MarathonLabs/marathon/blob/develop/docs/docs/android/configure.md#test-parser
testParserConfiguration:
type: "remote"
instrumentationArgs:
listener: "com.malinskiy.adam.junit4.android.listener.TestAnnotationProducer"
notAnnotation: "com.my.package.ScreenshotTestSuite"
instrumentationArgs:
debug: "false"
disableWindowAnimation: true
autoGrantPermission: true
screenRecordConfiguration:
preferableRecorderType: "screenshot"
applicationPmClear: true
testApplicationPmClear: true
waitForDevicesTimeoutMillis: 60000
outputs:
- testApplication: "../../integration_tests/build/outputs/apk/androidTest/debug/integration_tests-debug-androidTest.apk"
outputConfiguration:
maxPath: 1024
filteringConfiguration:
allowlist:
- type: "simple-class-name"
values:
- "MyFlakyTest"
Expected behavior
Test detail report page should also report failure with all relevant information (screenshot, logs, etc).
Logs and reports
Sorry, I cannot share full report as it might contain private information. But I'm more than happy to answer any additional question.