Add precision and tolerance apis
fruitcoder opened this issue · comments
The missing tolerance was mentioned here #15 but no PR was added yet. I'll leave this here while I'm experimenting with the implementation to fix failing snapshots on my CI.
I'm in the process of adding AccessibilitySnapshot to a project I'm working on, and this is our last remaining blocker. I noticed #64 from a bit over a year ago: is there anything I can do to help get this over the line? More than happy to create a new PR if needed.
We're still using our fork. Would be great to get this merged ir any info what would contribute to it being merged
Hey! Sorry, this one fell through my GitHub queue.
We've been exploring snapshot precision/tolerance APIs internally for snapshots in general. Reducing precision is unfortunately necessary when working across multiple types of machines, but we've seen a lot of cases where it's easy to let non-trivial regressions slip through with reasonable precision levels. I'm hoping we can find some alternative comparison methods that resolve the issue while reducing the possibility of letting through regressions.
cc @jhneves
@fruitcoder Does using per-pixel tolerance alone (without overall tolerance) fix your CI issues? I think that one is a bit safer than overall tolerance.
Are you talking about the perceptualPrecision that's mentioned in the new swift-snapshit-testing 1.10.0 release or was this something I could have used before. If so, can you point me in the right direction? 😅
Ahh you're using SnapshotTesting. Per-pixel tolerance is only supported out of box in iOSSnapshotTestCase right now (see overallTolerance vs perPixelTolerance here).
Perceptual precision is very similar to per-pixel precision, just using a different definition for how "similar" two pixels are. It potentially could be more reliable than standard per-pixel precision, but I haven't had a chance to run it through our test suite yet.
Hey @fruitcoder, just following up here - did you get a chance to test whether per-pixel/perceptual precision is sufficient to fix your CI issues?
Hey @NickEntin! Is there anything I can test here? I'm using snapshot testing and still can't find a way to get an image strategy that uses any kind of tolerance 🤔
@fruitcoder Try this branch. That should give you a parameter for .accessibilityImage(perceptualPrecision: ...). Try something like 0.995 to start.