Feature request: Add option to specify acceptable percentage of different pixels
A-CGray opened this issue · comments
I am trying to use odiff in the CI pipeline of a plotting tool I maintain called NicePlots to test that the images output by our examples remain consistent. We are running into the issue that, even using the --antialiasing command line option, we still have images differing on very small numbers of pixels due to what I assume are slight differences in anti-aliasing on different systems.
Here's an example from one of our recent CI runs:
Reference image
Test image
Difference
odiff --aa --threshold=0.1 --dm doumont-light_style_demo_reference.png doumont-light_style_demo.png doumont-light_style_demo-diff.png
Failure! Images are different.
Different pixels: 386 (0.003490%)
In our case, these minor differences are not the kind of thing we're looking to catch with our tests. We tried playing around with the --threshold option but found we had to make it impractically high to get past these issues.
It would be extremely useful to be able to specify that a certain percentage of differing pixels (e.g <= 0.01%) is acceptable, by adding a command line option like --tolerance=0.01.
Second this, I was looking forward to using odiff for a while, but to my dismay discovered that even a dockerized browser generates very subtle text rendering differences in CI vs local, which odiff picks up even with antialiasing: true.
This is a classic problem of any VR which could be solved by percentage detection, but I would recommend to run this type of checks always in one docker container instead.
If you still want to use percentages you can read the output of the odiff. It returns pixels count and percentage.
Just read this line and accept test instead of panic if percentage is less than 1 for example.
Or do you use the binary directly and requesting specific option for that?
https://github.com/dmtrKovalenko/odiff/blob/main/bin/node-bindings/odiff.d.ts#L39C21-L39C21
Hey Dmitry, thanks for responding to this.
I'm using the odiff CLI and just judging whether the images match based on the exit code, see here
From @Klaster1's comment it seems that running the tests in a docker container is not guaranteed to fix this issue. I guess parsing the odiff output would work but that seems like a very complex solution.
The solution I had in mind would be to add a tolerance command line option, so I can call odiff --antialiasing --tolerance=0.1 image1.png image2.png to tell odiff to exit with a 0 code as long 0.1% or less of the pixels are different between the images.
As far as I can see, this would only require some very simple changes to the logic here to exit 0 when diffPercentage <= tolerance instead of diffCount == 0.
That is reasonable and I can do that quickly or feel free to open PR.
In the mean time there is also an option called --parsable-output which makes output a simple string separated by values which will be very easy to parse even with sh. But I'll try to add that asap
Awesome, thanks @dmtrKovalenko ! I would have tried implementing something but have no experience with reason/OCaml. I will test out that parsable output option in the meantime
For future reference, I was able to implement some logic in bash to achieve this check outside odiff: