MarathonLabs / marathon

Cross-platform test runner

Home Page:https://docs.marathonlabs.io

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Inconsistent html report

MyDogTom opened this issue · comments

Describe the bug

marathon/html/pools/omni.html page says that test failed, but when I click on failed test and go to test details, report says that test succeeded. Individual test report page is the following: marathon/html/pools/omni/127.0.0.1-5037-emulator-5554/com.example.MyTest.html

I execute single test classMyTest.html which contains 7 different tests. 5 passes, but 2 tests fail. I noticed that on tests list page (marathon/html/pools/omni.html) every test references the same marathon/html/pools/omni/127.0.0.1-5037-emulator-5554/com.example.MyTest.html file and marathon/html/pools/omni/127.0.0.1-5037-emulator-5554 folder contains only one file. This file contains only one entry about the last executed test.

I suspect that it might be related to count sharding usage

shardingStrategy:
  type: "count"
  count: 5

To Reproduce
Configuration file

name: "UI tests"
outputDir: "../../build/reports/marathon"
screenRecordingPolicy: "ON_FAILURE"
deviceInitializationTimeoutMillis: 180000
debug: true
shardingStrategy:
  type: "count"
  count: 5
executionStrategy:
  mode: ALL_SUCCESS
uncompletedTestRetryQuota: 3
vendorConfiguration:
  type: "Android"
  # Required for parameterized tests to be run correctly. More info in https://github.com/MarathonLabs/marathon/blob/develop/docs/docs/android/configure.md#test-parser
  testParserConfiguration:
    type: "remote"
    instrumentationArgs:
      listener: "com.malinskiy.adam.junit4.android.listener.TestAnnotationProducer"
      notAnnotation: "com.my.package.ScreenshotTestSuite"
  instrumentationArgs:
    debug: "false"
  disableWindowAnimation: true
  autoGrantPermission: true
  screenRecordConfiguration:
    preferableRecorderType: "screenshot"
  applicationPmClear: true
  testApplicationPmClear: true
  waitForDevicesTimeoutMillis: 60000
  outputs:
    - testApplication: "../../integration_tests/build/outputs/apk/androidTest/debug/integration_tests-debug-androidTest.apk"
outputConfiguration:
  maxPath: 1024
filteringConfiguration:
  allowlist:
    - type: "simple-class-name"
      values:
        - "MyFlakyTest"

Expected behavior

Test detail report page should also report failure with all relevant information (screenshot, logs, etc).

Logs and reports

Sorry, I cannot share full report as it might contain private information. But I'm more than happy to answer any additional question.