Unnecessary files included in PR
opened this issue · comments
Bug description
On the log, it says
👋 Exiting process with 0 found differences & 3 baselines to update
Sending anonymized telemetry data.
After this, we are expecting the PR to have 3 new additional baseline images, but the PR includes other files as well.
Any possible reason why it includes files other than the 3 new additional baselines?
How to reproduce
Expected behavior
No response
Lost Pixel information
lost-pixel logs from CI
No response
hey! Which folder are we looking at now? are you using the automatic PR flow to update the images? Lost Pixel will generate all images in the current folder, and in the update mode it will move all of the images from current folder into the baseline folder thus making the baseline update.
Would be useful if you share the workflow file and little bit more info on where are you seeing this!
Thanks a lot!
These are from the baseline images folder, we added current and difference images folder to .gitignore
- name: Visual Test
id: lp
uses: lost-pixel/lost-pixel@v3.4.1
env:
LOST_PIXEL_MODE: update
- name: Create Pull request
id: cpr
uses: peter-evans/create-pull-request@v5
if: ${{ failure() && steps.lp.conclusion == 'failure' }}
with:
token: ${{ secrets.PULL_REQUEST_TOKEN }}
commit-message: Update Lost-Pixel baselines
committer: GitHub <noreply@github.com>
author: ${{ github.actor }} <${{ github.actor }}@users.noreply.github.com>
signoff: false
base: ${{ github.head_ref || github.ref_name }}
branch: "lost-pixel-baselines/${{ github.ref_name }}"
delete-branch: true
title: "Lost Pixel updating baselines - ${{ github.ref_name }}"
body: Automated baseline update PR - lost pixel
draft: false
We have this workflow to run VRT, which creates a PR when the testing fails. In this scenario, we expect it to fail since we added 3 new stories.
👋 Exiting process with 0 found differences & 3 baselines to update
Sending anonymized telemetry data.
Based on the log, there's no difference on the existing stories. However, after the PR was generated it included other updated images as well.
Judging from what I am seeing it shouldn't happen. What exactly are the changes that you are seeing on unrelated stories? Do you use some thresholds for the stories(I can think of some random bug that sees the stories as no diff but still having a diff as a threshold related)
In this case, only these 3 are expected to be in the created PR.
Yes, we do have threshold in lostpixel.config.ts which is 0.5%
What's the actual change on those stories that you say should not be there? Can you please verify if they have changes at all(if they show up on gh it means they have different binary representation so they should)
Adding more context to this, what we observed is as follows, lost-pixel's visual test picked up a change for one of our components:
According to the image comparison engine (odiff) there is only 1 visual change
However, when the pull request is create, multiple false positives are presented:
Looking a little closer into why Create PR picked those files up, we noticed it was because there are file size differences, and they seem to be inconsistent every time the lost pixel PR is created. For example, these are from 2 different lost pixel PRs requesting to merge into a branch with the same image baseline
We have added a fairly generous waitBeforeScreenshot configuration of 2 seconds and double checked that no animation was running for those stories when run in visual regression testing mode.
I still don't see what exactly is wrong with the images, I see that there are differences, but I don't see what kind of differences in your screenshot. maybe you could try run lost pixel locally and debug this way. npx lost-pixel update
it's hard to say what might be wrong without closer look at the images, we are merely doing screenshots and comparing and there is not much that could go wrong there.
you could also try running on app.lost-pixel.com and see if it will be any better, it's a managed platform and we are free while in beta
We were able to get around the issue by doing the following
- turn off
LOST_PIXEL_MODE update - write a script to manually check what files were detected and generated in the
imagePathDifferencefolder - copy said files from
imagePathCurrentfolder intoimagePathBaselinefolder
I suspect that the update mode is not taking threshold ignored images into consideration, hence when imagePathBaseline is updated, it includes all changes including the threshold ignored ones.
Now that I think about that - you are absolutely right! We are taking the thresholds into account on the platform, but when writing files to the file system we might be totally ignoring it!
If you feel like it I would appreciate the PR so you could just use lp update and not write custom logic for that! Thanks a lot for investigation and being elaborate about this!
Another small note is that you have with this example the inclusion of a gha-creds file, at least in my branch.
I added to .gitignore:
# output from github action auto-generated workflow
gha-creds*.json
@d-ivashchuk @kchuah @krrevilla18 opened a PR with a colleague on this: #309
(We're reviewing contribution guidelines, but shouldn't change much)
