Comparison is flaky

[Biswadeep Bose]

I have got a test running using cypress-image-snapshot

I see a lot of flakiness around . Currently my cypress tests are set to retry:1 on failures which means it will run twice if first fails

On first run it caught it correctly

On second run the test passes , i have ben trying to reolve it from past few days can't get thorugh

Here is my setup :

addMatchImageSnapshotCommand({
    e2eSpecDir: 'cypress/e2e',
    comparisonMethod: 'pixelmatch',
    failureThreshold: 0.01,
    failureThresholdType: 'percent',
    customSnapshotsDir: 'snapshots',
    //config options for Cypress.screenshot()
    capture: 'fullPage',
    // blackout: ['[data-gui="trustpilot-rating"]'],  
    disableTimersAndAnimations: true,
    allowSizeMismatch: true
})

blackout was initially there to generate the image and then i want it to purposely fail but there seems flakiness

[Simon Smith]

Hey,

I come across this often on my own projects that use this plugin. It's usually caused by something updating/rendering once the screenshot process has begun. Cypress will scroll the page, take snapshots and stitch them together to create a full page image. In that time you need to be sure nothing will change

Taking a screenshot is an asynchronous action that takes around 100ms to complete. By the time the screenshot is taken, it is possible something in your application may have changed. It is important to realize that the screenshot may not reflect what your application looked like 100% when the command was issued.

When passing fullPage to the capture option, Cypress scrolls the application under test from top to bottom, takes screenshots at each point and stitches them together

https://docs.cypress.io/api/commands/screenshot

From my experience Cypress will measure the entire document height and use that to work out how much to scroll and when to take the different screenshots before stitching them together. Once this process has started if any elements render later or update then the document height will change and you get errors where parts of the image are cut off.

In your example image it looks very much like this is happening. You could debug this by setting a temporary wait() command and see if it fixes it. If it does then you'd need to narrow down what parts of the page are updating and assert they are what you expect before starting the snapshot

[Biswadeep Bose]

I have ran my tests on local vs docker , keep having these rendering differences
Is there any way to tackle this?

[Simon Smith]

How do you mean local vs Docker? Is that snapshot taken outside of Docker and compared against one taken inside Docker?

[Biswadeep Bose]

I have the run the test locally on my machine using cypress runner & then run the cypress test via docker as well. I can see a difference in the images being generated

[Simon Smith]

That would be expected, unless your host machine is identical to the Docker environment. I always generate images within Docker to match my CI environment as I work on Mac but CI tends to be some form of Linux distro, usually Ubuntu

[Simon Smith]

I'm going to close this as I don't think this a problem with this plugin, but feel free to continue any discussion here