Report does not include retry option

[Valeriia Tymchenko]

I use xcresulttool + fastlane
Fastlane has a retry option and it means that in case of tests failure they can be re-run several times.
So, if tests failed at the first run then the report published for this step meanwhile the previous job has not been completed yet and the tests can be passed at the second step.

So, you need to wait until the previous step finishes and then publish the test report.

Thanks

[Kishikawa Katsumi]

Please provide me some additional information to help me understand the issue.

What is fastlane's retry option? Do you mean the number_of_retries here?
https://docs.fastlane.tools/actions/run_tests/

It looks like it won't proceed to the next step during the retry even if this option is enabled, is it not?

If possible, can you show me the GitHub Actions Yaml file or (if it's a public repository) the results of running it?

[Valeriia Tymchenko]

Yes, I mean number_of_retries
I see that report published at first retry run. And even if the second retry run is successful, the report is already published and marked as failed.

[Kishikawa Katsumi]

Can you share with me the YAML file of GitHub Action so I can reproduce the issue?
If you can't, I'd like to know the actual Fastlane command you're using instead.

[Valeriia Tymchenko]

this is a fastlane action:

desc "Running Test Suite"
    lane :run_tests do
          run_tests(scheme: "SomeScheme",
               testplan: "TestPlan",
               number_of_retries: 2,
               output_directory: "./build/DerivedData/test_output"
              )

github action workflow

name: RegressionTests

on:
 workflow_run:
    workflows: ["RunBuild"]
    types:
      - completed
 repository_dispatch:
    types: [trigger-tests]
 pull_request:
    branches: [ 'develop' ]

concurrency:
 group: run-tests-${{ github.head_ref }}
 cancel-in-progress: true

jobs:
  TestSuite:
    runs-on: self-hosted

    steps:
      - name: Set Global Variables
        run: |
          echo "LC_ALL=en_US.UTF-8" >> $GITHUB_ENV
          echo "LANG=en_US.UTF-8" >> $GITHUB_ENV
      - uses: actions/checkout@v2
        with:
          ref: ${{ github.event.client_payload.ref }}
      - run: echo ${{ github.event.client_payload.sha }}
    
      - name: Gem caching
        id: gem_cache
        uses: actions/cache@v2
        continue-on-error: true
        with:
          path: vendor/bundle
          key: ${{ runner.os }}-gems-${{ hashFiles('**/Gemfile.lock') }}
          restore-keys: |
            ${{ runner.os }}-gems-
            
      - name: Running Test Suite
        uses: maierj/fastlane-action@v2.0.1
        with:
          lane: "run_tests"
          bundle-install-path: "vendor/bundle"
        
      - name: Publish Test Report
        uses: kishikawakatsumi/xcresulttool@v1.0.3
        if: success() || failure()
        with:
          title: TestReport
          path: build/DerivedData/test_output/report.xcresult
  
      - name: Trigger Next Step
        if: success()
        uses: peter-evans/repository-dispatch@v1
        with:
          token: ${{ secrets.REPO_TRIGGER_PAT }}
          repository: ${{ github.repository }}
          event-type: trigger-build
          client-payload: '{"ref": "${{ github.ref }}", "sha": "${{ github.sha }}"}'

[Robin Daugherty]

This is not fastlane-specific. Xcode test plans support retries. If the first attempt fails and a successive attempt passes, this is reported by xcresulttool as a failure status.

[Duncan Cunningham]

Was there any progress on this issue, does xcresultool still report a test that was re-run and passed, as failed?

[Takuto NAKAMURA (Kyome)]

@kishikawakatsumi
I created a sample project to check this issue related to -retry-tests-on-failure option.

Check out the summary of this action.
https://github.com/Kyome22/RetryTestSample/actions/runs/9072460407

I ran a test that intentionally failed twice and then passed the third time.
I expected the xcresult result to pass, but it was treated as a failure.