pester / Pester

Pester is the ubiquitous test and mock framework for PowerShell.

Home Page:https://pester.dev/

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

SkipRemainingOnFailure reports skipped on filtered and explicitly skipped tests

fflaten opened this issue · comments

Checklist

What is the issue?

Run.SkipRemainingOnFailure processes tests that are both excluded by filter and/or skipped using -Skip. They are included in the summary

In the repro below, 3 tests were included by filter but one was skipped. The summary says Remaining tests skipped after first failure: 3 when only 1 test was actually skipped by the plugin.

Pester v5.6.0-beta1

Starting discovery in 1 files.
Discovery found 4 tests in 18ms.
Filter 'Tag' set to ('Demo').
Filters selected 3 tests to run.
Running tests.
Describing A
This is test 2
  [-] Test 1 11ms (4ms|7ms)
   Expected 2, but got 1.
   at 1 | Should -Be 2, untitled:Untitled-7:5
   at <ScriptBlock>, untitled:Untitled-7:5
 Describing B
   [!] Test 3 6ms (0ms|6ms)
   [!] Test 4 0ms (0ms|0ms)
Tests completed in 67ms
Tests Passed: 0, Failed: 1, Skipped: 2, Inconclusive: 0, NotRun: 1
Remaining tests skipped after first failure: 3

Expected Behavior

Don't mark excluded tests as Skip.
Update plugin summary to only count skipped tests that it was responsible for (exclude -Skip tests).

Steps To Reproduce

$sb = {
    Describe 'A' {
        It 'Test 1' -Tag 'Demo' {
            Write-Host 'This is test 2'
            1 | Should -Be 2
        }
        It 'Test 2' {
            Write-Host 'This is test 3'
        }
        Describe 'B' {
            It 'Test 3' -Tag 'Demo' {
                Write-Host 'This is test 2'
            }
            It 'Test 4' -Tag 'Demo' -Skip {
                Write-Host 'This is test 3'
            }
        }
    }
}

$conf = New-PesterConfiguration
$conf.Output.Verbosity = 'Detailed'
$conf.Run.ScriptBlock = $sb
$conf.Run.SkipRemainingOnFailure = 'Block'
$conf.Filter.Tag = 'Demo'

Invoke-Pester -Configuration $conf

Describe your environment

Pester version : 5.6.0-beta1 /workspaces/Pester/bin/Pester.psm1
PowerShell version : 7.4.1
OS version : Unix 5.15.133.1

Possible Solution?

Update criteria from if (-not $test.Executed) to if ($test.ShouldRun -and -not $test.Skip -and -not $test.Executed).