Comparing with oddiff
headdu opened this issue · comments
Bug description
The percentage threshold value (less than 1) is not being properly calculated when comparing with odiff
How to reproduce
- Configure a project that uses odiff as a comparison tool and uses a threshold lower than 1 (ex: 0.1)
- Use lost-pixel v2.3.4 image
- Build storybook
- Try comparing snapshots on CI pipeline
- Observe that we compare the threshold value with it's absolute value instead of percentage (example, if we use threshold of 0.1 we're expecting values with difference percentage less than 10% to pass, but we can observe that only values with percentage less than 0.1% are green)
Expected behavior
The threshold values under 1 should be used as percentages
Lost Pixel information
I'm using the recipe as is
This is my config file
import { CustomProjectConfig } from 'lost-pixel';
export const config: CustomProjectConfig = {
storybookShots: {
storybookUrl: './storybook-static',
mask: [{ selector: '#storybook-docs' }]
},
// OSS mode
generateOnly: true,
failOnDifference: true,
threshold: 0.11,
compareEngine: 'odiff'
};
lost-pixel logs from CI
Here you can see that even though I have the threshold as 0.11, the 0.03% passes but the 0.79% doesn't
[14/60] Difference of 317 pixels (0.03%) found but within threshold. (base-forms-input-with-inner-addon--default-input-addon-wrapper)
[23/60] Comparing 'base-forms-selects--default' (base-forms-selects--default)
[13/60] Difference of 7260 pixels (0.79%) found. Difference image saved to: .lostpixel/difference/base-forms-plain-text-input--plain-text-input-with-addon.png (base-forms-plain-text-input--plain-text-input-with-addon)
heya @headdu! do you mind running the latest lost pixel just to be on the safe side please? Also I will take a look at the run logs & your setup. in the meantime please look at this, maybe it will get clearer:
https://docs.lost-pixel.com/user-docs/recipes/general-recipes/thresholds
Thank you for replying so quickly.
I'm getting this error with the latest version. And I did read the docs and I think I understood them.
I'm using 0.11 and I'm expecting it to ignore differences below 11%
I took a look at the code and I saw that for pixelMatch you're calling checkThreshold which does the conversion for values below 1.
But for odiff you're using threshold and the diffPercentage. I'm guessing the diffPercentage there is the actual percentage [0, 100] and that it should be converted to [0, 1] just like in the return
Btw, but I couldn't get the local environment to run. Just to test. Happy to open a separate ticket for that but basically:
The command in contributing to build the storybook is no longer available. I did try using build-example-storybook-v6.4 or build-example-storybook-v6.5-storystore-v7 but the results were always the same
- Running
npm run devgives me
Loading project config ...
Current working directory: /Users/eduardomartins/Projects/lost-pixel
Looking for config file: /Users/eduardomartins/Projects/lost-pixel/lostpixel.config.(ts|js|cjs|mjs)
✅ Found config file: /Users/eduardomartins/Projects/lost-pixel/lostpixel.config.ts
🚀 Starting Lost Pixel in 'generateOnly' mode
📂 Creating shot folders
📸 Creating shots
Removing 0 files from .lostpixel/current/
Removing 0 files from .lostpixel/difference/
=== [Storybook Mode] ./storybook-static ===
Trying to collect stories via window object
Fallback to /stories.json
❌ page.waitForFunction: Timeout 30000ms exceeded.
at collectStoriesViaWindowApi (/Users/eduardomartins/Projects/lost-pixel/src/crawler/storybook.ts:90:14)
at async collectStories (/Users/eduardomartins/Projects/lost-pixel/src/crawler/storybook.ts:227:20)
at async createShots (/Users/eduardomartins/Projects/lost-pixel/src/createShots.ts:86:26)
at async runner (/Users/eduardomartins/Projects/lost-pixel/src/runner.ts:43:23)
at async /Users/eduardomartins/Projects/lost-pixel/src/bin.ts:69:7 {
name: 'TimeoutError'
}
❌ Unexpected token < in JSON at position 0
Sending anonymized telemetry data.```
Thanks a lot for the contribution! I am little bit busy today and tomorrow, but on the weekend I will review the PR and also guide you to improve the setup for the local setup if you'd be willing to contribute!
Nice catch @headdu 👍
You are correct, we had the right percentage conversions as a return value but forgot to use the same conversion on the threshold.
Thank you for extending the tests in your PR. 🙏