Questions about flaky comparison results

Question

Questions about flaky comparison results

jradom opened this issue 4 years ago · comments

Hello,

I use your tool and it had worked great for me until now. Here Is my scenario.
I have an app that is going through an upgrade.

I make screenshots using Selenium for several pages and store them as baseline images (before the upgrade).
I run my app after the upgrade and make screenshots of the same pages
I compare on the fly each new and old/baseline pages and use .setAllowingPercentOfDifferentPixels(10)
Sometimes my result is MATCH or MISMATCH or SIZE_MISMATCH and its' random

What can you suggest to make it work stable?
Oh, differences in fact are minimal. I use the Chrome browser to run my app

Your help will be greatly appreciated

Thanks a lot in advance

Jeff

Roman Beekeeper · Answer 1 · Mon Aug 31 2020 15:38:57 GMT+0800 (China Standard Time)

Hi @jradom,
Firstly, setAllowingPercentOfDifferentPixels(10) is a really powerful configuration, which was added for you (as I remember).

I'd suggest starting comparing without this config, it's for different MISMATCH and MATCH

If we speak about SIZE_MISMATCH - it's harder. Before say something I need to reproduce it. From my perspective, if you have this result, it means, that the screenshots are not the same.

Hope, it was helpful for yiu.

jradom · Answer 2 · Tue Sep 01 2020 12:55:49 GMT+0800 (China Standard Time)

What does SIZE_MISMATCH actually mean? Is it different size in bytes, number of pixels, size in pixels? Please, clarify. Thanks Jeff

…

On Mon, Aug 31, 2020 at 12:39 AM Roman Beskrovnyi ***@***.***> wrote: Hi @jradom <https://github.com/jradom>, Firstly, setAllowingPercentOfDifferentPixels(10) is a really powerful configuration, which was added for you (as I remember). I'd suggest starting comparing without this config, it's for different MISMATCH and MATCH If we speak about SIZE_MISMATCH - it's harder. Before say something I need to reproduce it. From my perspective, if you have this result, it means, that the screenshots are not the same. Hope, it was helpful for yiu. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#191 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AE4ZGPKYCMZW3LW3QXOKONTSDNHR5ANCNFSM4QJDVS2A> .

Roman Beekeeper · Answer 3 · Tue Sep 01 2020 14:51:16 GMT+0800 (China Standard Time)

SIZE_MISMATCH means, that the image sizes are different. For example:
image1 has 100x1000 pixels
image2 has 101x1000 pixels
In this case, would be SIZE_MISMATCH

Best regards,
Roman

jradom · Answer 4 · Wed Sep 02 2020 03:33:38 GMT+0800 (China Standard Time)

Now, If images on the surface look identical but ill use some tool and will resize a second one like image1 has 100x1000 pixels image2 has 100x1000 pixels Will they match after resizing? Please, let me know what you think. I use Selenium screenshots programmatically to create baselines and actual images currently. Looks like it might has ome issue. Do you a good tool which allow to make reliable screenshots using Java for example to make reliable images and then comare with your tool? Thanks Jeff

…

On Mon, Aug 31, 2020 at 11:51 PM Roman Beskrovnyi ***@***.***> wrote: SIZE_MISMATCH means, that the image sizes are different. For example: image1 has 100x1000 pixels image2 has 101x1000 pixels In this case, would be SIZE_MISMATCH Best regards, Roman — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#191 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AE4ZGPPJ7AW52UDBAMQZSJLSDSKXHANCNFSM4QJDVS2A> .

Roman Beekeeper · Answer 5 · Wed Sep 02 2020 04:45:03 GMT+0800 (China Standard Time)

I don't know for sure how selenium works, that's why I can't help with it.
Also, I'm not familiar with automation testing tools and approaches. I'm a developer, not qa engineer..
I wish I can help you...

jradom · Answer 6 · Wed Sep 02 2020 04:51:13 GMT+0800 (China Standard Time)

Actually my question was about resizing of a second image and then compare if that will work or not

…

Sent from my iPhone

On Sep 1, 2020, at 1:45 PM, Roman Beskrovnyi ***@***.***> wrote: I don't know for sure how selenium works, that's why I can't help with it. Also, I'm not familiar with automation testing tools and approaches. I'm a developer, not qa engineer.. I wish I can help you... — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

jradom · Answer 7 · Wed Sep 09 2020 13:00:59 GMT+0800 (China Standard Time)

Hi Roman, It looks like we are moving away from Java to Javascript/Typescript. Can you suggest a similar to your tool (it's really great) but which can be used with Javascript/Typescript? Please, let me know Thanks Jeff

…

On Tue, Sep 1, 2020 at 12:33 PM Jeff Radom ***@***.***> wrote: Now, If images on the surface look identical but ill use some tool and will resize a second one like image1 has 100x1000 pixels image2 has 100x1000 pixels Will they match after resizing? Please, let me know what you think. I use Selenium screenshots programmatically to create baselines and actual images currently. Looks like it might has ome issue. Do you a good tool which allow to make reliable screenshots using Java for example to make reliable images and then comare with your tool? Thanks Jeff On Mon, Aug 31, 2020 at 11:51 PM Roman Beskrovnyi < ***@***.***> wrote: > SIZE_MISMATCH means, that the image sizes are different. For example: > image1 has 100x1000 pixels > image2 has 101x1000 pixels > In this case, would be SIZE_MISMATCH > > Best regards, > Roman > > — > You are receiving this because you were mentioned. > Reply to this email directly, view it on GitHub > <#191 (comment)>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/AE4ZGPPJ7AW52UDBAMQZSJLSDSKXHANCNFSM4QJDVS2A> > . >

jradom · Answer 8 · Tue Nov 10 2020 17:09:34 GMT+0800 (China Standard Time)

Roman,

I've just downloaded your Jar file but how can I run it, put parameters and output. Like I said I plan to call CLI from Typrescript

Please, let me know

Thanks a lot in advance

Jeff

Roman Beekeeper · Answer 9 · Tue Nov 10 2020 23:53:26 GMT+0800 (China Standard Time)

Jeff,
This jar of image-comparison doesn't have main method to run as a jar.
For this purpose must be created the separated project with the main method with image-comparison as a dependency.

jradom · Answer 10 · Wed Nov 11 2020 23:21:40 GMT+0800 (China Standard Time)

If this is doable to make a jar work in a short time frame it’ll be great. Then, I’ll be able to continue using your great library

…

Sent from my iPhone

On Nov 10, 2020, at 7:53 AM, Roman Beskrovnyi ***@***.***> wrote: Jeff, This jar of image-comparison doesn't have main method to run as a jar. For this purpose must be created the separated project with the main method with image-comparison as a dependency. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

jradom · Answer 11 · Sat Nov 14 2020 03:09:36 GMT+0800 (China Standard Time)

Hi Roman, Like I had said before we like your current tool and I plan to continue using it with Typescript because of great features and results. So, is it a feasible time frame you can make CLI with parameters and output exit values available. Please, let me know if I should wait or just try switching to some other tool. (still prefer yours though). Please, let me know one way or another Thanks in advance Jeff

…

On Tue, Nov 10, 2020 at 7:53 AM Roman Beskrovnyi ***@***.***> wrote: Jeff, This jar of image-comparison doesn't have *main* method to run as a jar. For this purpose must be created the separated project with the main method with image-comparison as a dependency. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#191 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AE4ZGPMF3L3ABW4VU2YECSLSPFOYRANCNFSM4QJDVS2A> .