Problems finding correct last visual change

Question

Problems finding correct last visual change

soulgalore opened this issue 6 years ago · comments

Hi Pat,
we have a problem with finding the correct last visual change when we test our mobile version of the site, running Chrome in emulated mode (or setting the viewport to mobile phone size). The last visual change is always a lot higher but visually I can't see any change.

Here's two example videos:
https://www.dropbox.com/s/i3ch0ic9jnit28j/en.m.wikipedia.org-wiki-Facebook.mp4?dl=0
https://www.dropbox.com/s/vfya08i7t46d6gv/en.m.mediawiki.org-wiki-Download.mp4?dl=0

If I run latest master:
../visualmetrics.py -i en.m.mediawiki.org-wiki-Download.mp4 --orange --force --viewport --dir a

I get the Last Visual Change 4817 but it looks like it should be something like 1100 when I look at the video and the screenshots.

If I increase the 10% difference in individual pixels to a little higher (14%) then it looks better but is that the way to go? I got a feeling the problem for us is higher when we have more plain text on the screenshots.

Best
Peter

Patrick Meenan · Answer 1 · Thu Aug 30 2018 03:25:58 GMT+0800 (China Standard Time)

I'll have to look closer at the videos and run it through some testing to see what the diffs are. Sometimes it seems that with text content Chrome will turn on antialiasing or something like that when onload fires and there are slight differences around all of the text. A higher percentage worries me because it can miss some larger visible changes.

Peter Hedenskog · Answer 2 · Thu Aug 30 2018 14:56:02 GMT+0800 (China Standard Time)

Chrome will turn on antialiasing or something like that when onload fires

Ah I see, that match my experience that something happens really late. Yep I see your point with the percentage.

Peter Hedenskog · Answer 3 · Fri Aug 31 2018 20:14:12 GMT+0800 (China Standard Time)

Thanks @pmeenan ! I've installed another font set on the server where we run the test and it seems to work now. I'll keep the issue open during the weekend when I continue to test.

Patrick Meenan · Answer 4 · Fri Aug 31 2018 20:16:58 GMT+0800 (China Standard Time)

Which fonts did you install? The wptagent ubuntu installer installs "ttf-mscorefonts-installer fonts-noto*" but if there are others that are commonly installed on desktops or mobile devices that should be there I'm happy to expand the default set as well.

Peter Hedenskog · Answer 5 · Fri Aug 31 2018 20:36:22 GMT+0800 (China Standard Time)

I've installed fonts-noto* and that seems to help I but will look to see if I can narrow it down a bit since it adds almost 300mb to the container.

Patrick Meenan · Answer 6 · Fri Aug 31 2018 21:53:53 GMT+0800 (China Standard Time)

FWIW, the dockerfile for wptagent should also include fonts-noto*

Peter Hedenskog · Answer 7 · Sat Sep 01 2018 01:33:51 GMT+0800 (China Standard Time)

I was wrong. Locally on my machine the last visual change looked right, but when I pushed it to the server and let it test out different URLs, the last visual change again started to be really high. It seems to be where onLoad happens as you said with anti alias, do you know if there's a way to turn it off? I've only found the switch to turn it on that I guess was needed in older Chrome.