(integ-tests-alpha,custom-resources): snapshots are always outdated
kornicameister opened this issue · comments
Describe the bug
After upgrading to 2.146.0 from 2.145.0 where #30418 was introduced I am observing that running integration tests via integ-runner always fails. Snapshot is being reported as outdated, like on screenshot above. Although I can successfully rebuild/rerun tests and I can observe a change in snapshots next time I run tests I once again observe situation from screenshot
Once I had downgraded to 2.145.0 on both aws-cdk-lib, @aws-cdk/integ-runner and @aws-cdk/integ-tests-alpha to 2.145.0 tests are passing without a problem.
Behavior suggests a regression of sort around: #30418
Also the problem occurs, I believe, around custom resources I am using.
Expected Behavior
Screenshot are deterministically stored and next checks aren't failing.
Current Behavior
Reproduction Steps
N/A
Possible Solution
N/A
Additional Information/Context
N/A
CDK CLI Version
2.146.0
Framework Version
No response
Node.js Version
18.16.0
OS
MacOS sierra
Language
TypeScript
Language Version
5.4.5
Other information
No response
Hmm.. I thought I had fixed by downgrading to 2.145.0 but now I am running a test that has changed custom resource and it fails with same reason. This time going down to 2.144.0 fixes the problem consistently.
Yes I think #30418 would make your snapshot need to update as it makes logApiResponseData: true to logApiResponseData: undefined. This is a bug fix and existing snapshots would need to update.
@pahud but they are always updating and never actually doing that.
I'm updating a massive amount of our integration tests and experiencing this same thing. It looks like this is being injected at the time the test runs instead of at the build time. It's resulting in a good portion of our tests only working with the --dry-run flag being added, which we absolutely do not want. We want people running the the tests. I'm changing this to a p1.