False positives for Storybook shots

[Benjamin Forster]

Bug description

We are noticing a lot of false positives in our visreg tests for storybook. The height of the story seems to be variable. Sometimes truncating the stories.

Please see attached image:

How to reproduce

using simple config like:

module.exports = {
  storybookShots: {
    storybookUrl: './storybook-static',
  },
  lostPixelProjectId: 'your-id',
  apiKey: process.env.LOST_PIXEL_API_KEY,
};

And storybook config:

const config: StorybookConfig = {
  stories: ['../src/**/*.stories.@(ts|tsx)'],
  addons: [
    '@storybook/addon-links',
    '@storybook/addon-essentials',
    '@storybook/addon-interactions',
  ],
  framework: {
    name: '@storybook/nextjs',
    options: {},
  },
  docs: {
    autodocs: 'tag',
  },
  features: {
    storyStoreV7: true,
  },
};

1. Build storybook
2. Use lost-pixel v3.4.0 image to upload diffs
3. See false negatives in lost-pixel app.

Expected behavior

We would expect consistent image size between builds

Lost Pixel information

No response

lost-pixel logs from CI

No response

[Dima Ivashchuk]

hey @e-e-e! could you please try the most up to date version of lost pixel first. If you storybook build is not public and you are on the platform we could look at it together in our shared slack or private discord channel, this could move things way faster!

[Dima Ivashchuk]

It would be 3.13.0

[Benjamin Forster]

Thanks @d-ivashchuk I will update our version and see if it addresses the issue

[Maxime BAUMANN]

I have the same issue with version 3.16.0, my pipelines are failing randomly

Here the margins appeared on the right and the bottom.

Edit: My Storybook is on Storybook 8, build with Vite

[Dima Ivashchuk]

I guess guys bumped up their runners to beefier machines and it started working pretty well for them. So in this regard I don't see a problem in the lost pixel or playwright. Still we can investigate and see what's wrong? Any public repo or reproduction of this I can look at?

[Bright Lee]

having the same issue with LostPixel 3.16 + Storybook 8 + nextjs 14.1.3 (all latest for the time being)

[Maxime BAUMANN]

I guess guys bumped up their runners to beefier machines and it started working pretty well for them. So in this regard I don't see a problem in the lost pixel or playwright. Still we can investigate and see what's wrong? Any public repo or reproduction of this I can look at?

I'll try to make a plublic repo with a repro tomorrow, I can't bump my runners for the moment, I'll just use the ones provided by GH

[Dima Ivashchuk]

yeah, if we can somehow remedy this on our side - would be happy to do so! reproduction would be very helpful!

@benevbright @mbaumanndev - you are running in OSS mode, right?

[Maxime BAUMANN]

Yes, I'm in OSS mode running on GitHub Actions

[Dima Ivashchuk]

then please do a repro - would allow us to tackle it properly 🙇🏼‍♂️ thanks a lot!

[Bright Lee]

Yes, we're also OSS mode.

one finding is that when I reduced shotConcurrency to 5 from 10, it gives the correct result. shotConcurrency: 10 was working ok with Storybook 6 + Nextjs 13.0. But after updating these dependencies, the worker seems a bit overwhelmed.

So from my end, it might not be LostPixel's problem. But what I do want to suggest is how LP starts concurrent jobs. As just my quick guess, it you put a small arbitrary delay between each jobs in one concurrent, it might help to increase this number in our config (just total guess)

[Bright Lee]

Sorry, I might have said it too early.... even with shotConcurrency: 5, sometimes it gives result with wrong height as this issue described.

[Bright Lee]

and it's happening only with heavy pages.

[Dima Ivashchuk]

Concurrency is definitely at play here. How heavy are the pages?

We recommend 4-5 for oss users if your gh action runners are not super beefy. But as you legitimately noticed the heaviness of the pages also matter.

Your runner just can't handle rendering all of the stuff concurrently and there is no easy solution to that apart from:

• Splitting visual tests into more sensible(smaller) ones
• reduce concurrency
• bump up machine resources

I am not sure what storybook did across the update but it might in a minor way influence something. Still I can't imagine anything major

[Dima Ivashchuk]

Closing this as it seems like the problem is clear. If anybody needs more assistance - feel free to reopen!