$PSRule.TargetType no longer matches $TargetObject.Type in 2.8.1

Question

$PSRule.TargetType no longer matches $TargetObject.Type in 2.8.1

tstooke opened this issue a year ago · comments

Description of the issue

$PSRule.TargetType and $TargetObject.Type do not match anymore. When asserting rules with PSRule 2.7.0 and PSRule.Rules.Azure 1.24.2, the values are the same. After upgrading PSRule to 2.8.1, the values no longer match. This unexpected behavior also happens with PSRule.Rules.Azure 1.26.1 with PSRule 2.8.1.

This is breaking the -Type precondition for our custom Rules that we have added for use with the PSRule.Rules.Azure Rules.

To Reproduce

Steps to reproduce the issue:

Create a custom Rule that either uses the -Type precondition or references the $PSRule.TargetType property.
Set the condition to look for a specific Azure Resource Type, such as Microsoft.Cache/redis.
Assert the rules against a Bicep file that uses a Bicep module to create an Azure Redis Resource (using the module keyword rather than directly using the resource keyword).
Include the following Write-Host commands in the Rule to debug the values.

    Write-Host "Target Object:" $TargetObject
    Write-Host "Target Object:" $PSRule.TargetObject
    Write-Host "Target Type:" $PSRule.TargetType
    Write-Host "Target Object Type:" $TargetObject.Type

Expected behaviour

In PSRule 2.7.0, the $TargetObject.Type and $PSRule.TargetType were identical.

Error output

We ran the assert command in Verbose and confirmed that our custom rules are being found and the preconditions are being evaluated. They simply resolve to $false because $PSRule.TargetType does not have the expected value with the latest version of PSRule.

With the above Write-Host commands in the rule, we get the following (abbreviated) output.

Using PSRule v2.7.0
Using PSRule.Rules.Azure v1.24.2

Target Object: @{type=Microsoft.Cache/redis; name=test-name; location=eastus; identity=; properties=; id=/subscriptions/ffffffff-ffff-ffff-ffff-ffffffffffff/resourceGroups/ps-rule-test-rg/providers/Microsoft.Cache/redis/test-name}
Target Object: @{type=Microsoft.Cache/redis; name=test-name; location=eastus; identity=; properties=; id=/subscriptions/ffffffff-ffff-ffff-ffff-ffffffffffff/resourceGroups/ps-rule-test-rg/providers/Microsoft.Cache/redis/test-name}
Target Type: Microsoft.Cache/redis
Target Object Type: Microsoft.Cache/redis

Using PSRule v2.8.1
Using PSRule.Rules.Azure v1.24.2

Target Object: @{type=Microsoft.Cache/redis; name=test-name; location=eastus; identity=; properties=; id=/subscriptions/ffffffff-ffff-ffff-ffff-ffffffffffff/resourceGroups/ps-rule-test-rg/providers/Microsoft.Cache/redis/test-name; resources=System.Object[]}
Target Object: @{type=Microsoft.Cache/redis; name=test-name; location=eastus; identity=; properties=; id=/subscriptions/ffffffff-ffff-ffff-ffff-ffffffffffff/resourceGroups/ps-rule-test-rg/providers/Microsoft.Cache/redis/test-name; resources=System.Object[]}
Target Type: System.Management.Automation.PSCustomObject
Target Object Type: Microsoft.Cache/redis

Module in use and version:

Module: PSRule
Version: 2.7.0 vs 2.8.1
Module: PSRule.Rules.Azure
Version: 1.24.2 and 1.26.1

Captured output from $PSVersionTable:

Name                           Value
----                           -----
PSVersion                      7.3.4
PSEdition                      Core
GitCommitId                    7.3.4
OS                             Microsoft Windows 10.0.19045
Platform                       Win32NT
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0…}
PSRemotingProtocolVersion      2.3
SerializationVersion           1.1.0.1
WSManStackVersion              3.0

Additional context

We just noticed that the $TargetObject with PSRule 2.8.1 has an additional resources property that is not present with 2.7.0. We did not explicitly enable any experimental features nor change anything with our rules or other configuration after updating PSRule.

This looks potentially similar to an issue that we had when we temporarily enabled the User Types experimental feature in Bicep. The resulting ARM templates had an additional layer of nesting because the Deployment was defined with a different JSON structure. Is that feature enabled by PSRule in 2.8.1, by chance?

github-actions · Answer 1 · Fri Apr 21 2023 05:27:13 GMT+0800 (China Standard Time)

Thanks for raising your first issue, the team appreciates the time you have taken 😉

Bernie White · Answer 2 · Fri Apr 21 2023 13:56:22 GMT+0800 (China Standard Time)

@tstooke Thanks for reporting the issue and providing this detail.

As a temporary workaround you should be able to set the following in your ps-rule.yaml options to override these settings:

binding:
  targetType:
  - 'type'

Bernie White · Answer 3 · Fri Apr 21 2023 20:05:46 GMT+0800 (China Standard Time)

@tstooke I believe this is caused by bug fix #1473 which was included in v2.8.0.

The bug fix, fixes an incorrect behaviour occurring with binding configurations that was causing the binding configuration of a rules module (such as PSRule.Rules.Azure) to leak out.

The unintentional impact of this was:

Local rules were automatically getting the default binding configuration overridden. Which your rules may have been relying on.
Other modules binding configurations were being squashed with the first module configuration.

Based on this are you able to confirm if the Binding.TargetType option is set?

If not, please see Binding type which includes the recommended settings for custom rules.

Please let us know if this does not fix this issue with PSRule v2.8.1 and PSRule.Rules.Azure v1.26.1.

I hope this helps.

TLS · Answer 4 · Fri Apr 21 2023 22:08:14 GMT+0800 (China Standard Time)

Thank you so much for the quick help with this! With your guidance on the binding config and the Rule.IncludeLocal setting, we have things back to working again.

We must have been relying on that bug to apply the binding config for both the module rules and the local rules in the same assertion run.

To clarify, we added this to the ps-rule.yml file:

binding:
  targetType:
    - 'resourceType'
    - 'type'

And in our PowerShell script, we added the -RuleIncludeLocal $True parameter to the PSRule Option object when we assert with the PSRule.Rules.Azure module:

$config = @{
    AZURE_BICEP_FILE_EXPANSION = $True
    AZURE_BICEP_CHECK_TOOL = $True
    AZURE_BICEP_MINIMUM_VERSION = "0.16.2"
    AZURE_BICEP_FILE_EXPANSION_TIMEOUT = 60
}

$options = New-PSRuleOption `
    -NotProcessedWarning $False `
    -InputPathIgnore @("*", "!**/*.tests.bicep") `
    -RuleIncludeLocal $True `
    -Configuration $config

Assert-PSRule -Module "PSRule.Rules.Azure" -Path "./.ps-rule/" -Format File -InputPath '.' -Option $options

We set the options this way because our script executes multiple Assert commands with different options. We're using PSRule for Bicep and other things in our pipelines.

We have now updated to PSRule 2.8.1, PSRule.Rules.Azure 1.26.1 and Bicep 0.16.2 with successful assertion runs.

One thing that caught us is that the Including Custom Rules section mentions that it's only needed when specifying a baseline, which we aren't doing in our Assert command. Does the PSRule.Rules.Azure module automatically declare a baseline if we don't set one? We're using the default GA rules, so we didn't feel that we needed to specify the baseline.

Bernie White · Answer 5 · Sat Apr 22 2023 00:02:13 GMT+0800 (China Standard Time)

@tstooke Thanks for that.

Using a baseline depends on your use case. For enterprise use, using a quarterly baseline is often helpful to avoid new rules breaking PR merges. For example, if a new rule is introduced that was not previously required, new PRs or build may break causing someone to update all the affected code unrelated to the current change they are merging.

Baselines allow you to update all affected more gradually then switch the baseline.

Using a scheduled build without a baseline and using PR builds with a baseline is a good mix.

PSRule for Azure does implement a default baseline called Azure.Default that is what filters the rules so that only GA rules are run by default (https://github.com/Azure/PSRule.Rules.Azure/blob/d6d31c3a1ce2669edd573b70b5849d6d0b0104af/src/PSRule.Rules.Azure/rules/Baseline.Rule.yaml#L9-L17).

That said, for the default baseline you shouldn't need to set Rule.IncludeLocal to true. But if you use any other PSRule for Azure baseline you will need to.

Is that the behaviour you are experiencing or something else?

TLS · Answer 6 · Sat Apr 22 2023 00:14:54 GMT+0800 (China Standard Time)

We will probably switch to a baseline in the near future for the reasons you mention.

Since we're not explicitly setting a baseline, as shown in the code snippet above, we expected to not need the Rule.IncludeLocal setting. However, when we execute the Assert without it, none of our custom rules are executed. Or, at least, none of them show up in the output. When we set Rule.IncludeLocal to true, our custom rules do show up as expected, alongside the Azure rules.

This behavior only happens when we include -Module 'PSRule.Rules.Azure' in the Assert command. With other commands that don't use a Module, we don't have to set Rule.IncludeLocal. Those other runs are not testing the Bicep files, though - they're testing supporting files like JSON settings and such.

Looking at the definition of the default Azure baseline, I don't see how it's different than the others. It doesn't explicitly set the includeLocal setting in the definition. Is that set somewhere else in the code for the default baseline?

Bernie White · Answer 7 · Sat Apr 22 2023 14:32:19 GMT+0800 (China Standard Time)

@tstooke Thanks for the additional feedback.

There is nothing special about the baseline except it is the default baseline for the PSRule.Rules.Azure module. So it should implicitly inherit Rule.IncludeLocal = true.

https://github.com/Azure/PSRule.Rules.Azure/blob/d6d31c3a1ce2669edd573b70b5849d6d0b0104af/src/PSRule.Rules.Azure/rules/Config.Rule.yaml#L51-L52

If any baseline is specified by the command-line or options then PSRule operates under the assumption (at least currently) that you only want to run rules that would be selected by that baseline unless overridden by setting Rule.IncludeLocal explicitly in options.

Since all the baselines in PSRule for Azure require you to tag your rules with a release tag if you don't tag your rules they would not be included.

In summary, you should still set Rule.IncludeLocal = true if you plan to use a specific baseline. But shouldn't have to currently since you are not targeting a baseline. So we'll log this as a separate bug since we believe the original issue is resolved.

TLS · Answer 8 · Mon Apr 24 2023 21:48:29 GMT+0800 (China Standard Time)

Ah, I see! Our custom rules are not tagged with GA, so they're not included with the default baseline. As soon as we add that tag, they start showing up and Rule.IncludeLocal is not needed. I'll post further on the other issue, as the original topic for this issue has been resolved. Thank you!