Block setup and teardown executed when tests are skipped

[Frode Flaten]

Checklist

• Issue has a meaningful title
• I have searched the existing issues. See all issues
• I have tested using the latest version of Pester. See Installation and update guide.

What is the issue?

BeforeAll and AfterAll are executed when all tests are skipped using -Skip.

This is a known limitation/backlog item as seen in this runtime test:

Pester/tst/Pester.Runtime.ts.ps1

Lines 1228 to 1275 in 7889a88

     t "skipping all items in a block will skip the parent block" { 
         # this is not implemented, but is a possible feature 
         # which could be implemented together with "if any test is explicitly unskipped in child 
         # then the block should run, this will be needed for running tests explicitly by path I think 
         # it also should be taken into consideration whether or not adding a lazySkip is a good idea and how it would 
         # affect implementation of this. Right now skipping the block goes from parent down, and skipping all items in a block 
         # will not prevent the parent block setups from running 
  
         #     $container = @{ 
         #         OneTimeTestSetup = 0 
         #         OneTimeTestTeardown = 0 
         #         EachTestSetup = 0 
         #         EachTestTeardown = 0 
         #         TestRun = 0 
         #     } 
  
         #     $actual = Invoke-Test -SessionState $ExecutionContext.SessionState -BlockContainer (New-BlockContainerObject -ScriptBlock { 
         #         New-Block "parent block" { 
         #             New-Block "parent block" { 
         #                 # putting this in child block because each test setup is not supported in root block 
         #                 New-OneTimeTestSetup -ScriptBlock { $container.OneTimeTestSetup++ } 
         #                 New-OneTimeTestTeardown -ScriptBlock { $container.OneTimeTestTeardown++ } 
  
         #                 New-EachTestSetup -ScriptBlock { $container.EachTestSetup++ } 
         #                 New-EachTestTeardown -ScriptBlock { $container.EachTestTeardown++ } 
  
         #                 New-Test "test1" -Skip { 
         #                     $container.TestRun++ 
         #                     "a" 
         #                 } 
  
         #                 New-Test "test2" -Skip { 
         #                     $container.TestRun++ 
         #                     "a" 
         #                 } 
         #             } 
         #         } 
         #     }) 
  
         #     # $actual.Blocks[0].Skip | Verify-True 
         #     $actual.Blocks[0].ErrorRecord.Count | Verify-Equal 0 
         #     $container.TestRun | Verify-Equal 0 
         #     $container.OneTimeTestSetup | Verify-Equal 0 
         #     $container.OneTimeTestTeardown | Verify-Equal 0 
         #     $container.EachTestSetup | Verify-Equal 0 
         #     $container.EachTestTeardown | Verify-Equal 0 
     } 
 } 

Expected Behavior

Setup and teardown are not executed when they're not needed

Note: Tests skipped using Set-ItResult doesn't count as they're skipped during Run-phase (too late)

Steps To Reproduce

$sb = {
    Describe 'A' {
        BeforeAll {
            Write-Host 'Before ALL -A'
        }
        BeforeEach {
            Write-Host 'Before EACH -A'
        }
        AfterEach {
            Write-Host 'After EACH -A'
        }

        It 'Test 1' -Skip {
            Write-Host 'This is test 1'
        }

        Describe 'B' {
            BeforeAll {
                Write-Host 'Before ALL -B'
            }
            AfterAll {
                Write-Host 'After ALL -B'
            }
            BeforeEach {
                Write-Host 'Before EACH -B'
            }
            AfterEach {
                Write-Host 'After EACH -B'
            }

            It 'Test 2' -Skip {
                Write-Host 'This is test 2'
            }
        }

        AfterAll {
            Write-Host 'After ALL -A'
        }
    }

}

$conf = New-PesterConfiguration
$conf.Output.Verbosity = 'Detailed'
$conf.Run.PassThru = $true
$conf.Run.ScriptBlock = $sb

$r = Invoke-Pester -Configuration $conf

Output:

Pester v5.6.0-beta1

Starting discovery in 1 files.
Discovery found 2 tests in 35ms.
Running tests.
Before ALL -A
Describing A
  [!] Test 1 6ms (0ms|6ms)
Before ALL -B
 Describing B
   [!] Test 2 3ms (0ms|3ms)
After ALL -B
After ALL -A
Tests completed in 77ms
Tests Passed: 0, Failed: 0, Skipped: 2, Inconclusive: 0, NotRun: 0

Describe your environment

Pester version : 5.6.0-beta1 /workspaces/Pester/bin/Pester.psm1
PowerShell version : 7.4.1
OS version : Unix 5.15.133.1

Possible Solution?

No response

[Jakub Jareš]

I thought this was implemented, but maybe I've implemented it just on a smaller scope. Should we put this into 6.0.0 milestone?

[Frode Flaten]

I think so. Just in case it affects something.

Not sure where to implement this yet. Mark block as Skip during PostProcess-DiscoveredBlock?

[Frode Flaten]

Related #2424. If the plugin skips remaining tests the Before-/AfterAll should not run on subsequent blocks. That means we might need to make the decision during Run or both.

[Jakub Jareš]

I don't remember where this is done, there was some function that is recursing down to figure out all the skipped steps, and then recursing up to mark all items as skipped, when they have all children skipped.