Pixel rounding
anders0l opened this issue · comments
Hello!
First of all thank you for this awesome tool.
We (AutoScout24.com) switched to it from annoying and slowly galen-framework.
We move all our test setup to quixote but there is one unsolvable thing - pixel rounding.
In our UI we've got at least button, that renders in Chrome (chromium) and Safari differently then in IE, Edge and FireFox.
It's about 0.80 pixels differ.
Some other elements renders little bit different in IE and it can be 0.10 or -0.70.
As it's such differ we can't provide Math.floor or Math.ceil or Math.round or Math.trunc.
And we can't define height in styles as it's flexible and the element can break from one to two lines.
In Galen, we had approximation for this thing.
http://galenframework.com/docs/getting-started-configuration/ (top of the Config example)
Is there anything like that in quixote?
If not, we'll fork quixote and add it.
Thanks!
Thanks! Glad you like it.
Pixel-rounding is built into Quixote. (Here.) Pixels that are within +/- 0.5 pixels are considered identical.
If your test browsers are zoomed to any level other than 100%, that can make pixel rounding issues worse. Please double-check your browser zoom level before continuing.
If your browsers are at zoom level 100% and you're still seeing the issue, please attach a minimal test case and I'll see what I can do. I can increase the pixel-rounding tolerance or possibly introduce a browser detect depending on the situation.
Thank you for fast reply!
We test on local machines and using BrowserStack and get same problem.
I'm sure on 99%, that they are at 100% zoom.
I'll check tomorrow and if we'll have same issue, I'll provide you an example.
But it will be great to define somewhere and somehow approximation:
var quixote = require("quixote")({approximation: 3});
or the perfect solution but hardly to implement:
logo.bottom.plus(10).diff(3)
What's your opinion about it?
A "tolerance" descriptor is an interesting idea, and I don't think it would be too hard to implement. I want to see what's going on here before deciding, though.
The 0.5 tolerance we have now has been enough for all the cases I've tried. I want to dig into the rounding issue you're seeing and make sure there isn't some sort of deeper issue.
Just dummy thing to show you different
https://autoscout24.github.io/showcar-ui/#button-link
in Chrome (any versions)
window.getComputedStyle(document.querySelector('.sc-btn-bob')).height 40px
in FireFox (any versions)
window.getComputedStyle(document.querySelector('.sc-btn-bob')).height 40.8px
And same we get in tests.
Can you attach/paste a specific HTML+CSS snippet that demonstrates the issue?
I made a simple example for you. Please try it out on Chrome and Firefox. I get 3.20px mismatch.
https://github.com/anders0l/test-quixote/tree/master/example
/example/src/_media_css_test.js
/example/src/screen.css
Thanks for the example. It looks like there's multiple issues here.
- There's a race condition with the web font. Sometimes the font loads and sometimes it doesn't. With the font loaded, the difference in height is actually 2.5px.
You can wait out the race condition by introducing a timeout, like this:
it('Buttons have the correct height', function (done) {
setTimeout(function() {
button.assert({
height: 38
});
done();
}, 1000);
});Timeouts aren't the right way to solve race conditions, so this needs a proper fix. I'll open that as a separate issue.
- Even with the race condition resolved, there are genuine differences between Firefox and Chrome. Quixote is correctly identifying this difference. It's not a pixel-rounding issue, it's a cross-browser rendering difference--exactly what Quixote tests are designed to catch.
2a) The first cross-browser difference is that the font sizes are different between Firefox and Chrome. On Chrome, the font is 20px tall. On Firefox, it's 20.5px tall.
This is within Quixote's pixel-rounding tolerance, but if you were to use a different font size, it wouldn't be. You can fix the issue by using Quixote's relative comparisons, like this:
(function () {
"use strict";
var quixote = require("../vendor/quixote.js");
describe("Button", function () {
var frame;
var button;
var font;
before(function (done) {
frame = quixote.createFrame({
stylesheet: "/base/src/screen.css"
}, done);
});
after(function () {
frame.remove();
});
beforeEach(function () {
frame.reset();
frame.add(
"<button class='sc-btn-bob'><span id='font'>I am Bob!</span></button>",
"button"
);
button = frame.get('.sc-btn-bob');
font = frame.get("#font");
});
it('Buttons have the correct height', function (done) {
setTimeout(function() {
button.assert({
height: font.height.plus(20)
});
done();
}, 1000);
});
});
}());2b) However, even this won't work, because Firefox adds an extra 2px to buttons for some reason. See this StackOverflow question.
As that question states, the following CSS will fix it:
button::-moz-focus-inner {
padding: 0;
border: 0
}And if you add that, the test passes.
I'll add a browser detect for this situation.
Thank you for such a detailed answer.
- We use some hacks to test 320, 768, 1024 dimension on 1 page (it speed ups the test). So we didn't fail in race condition issue. It takes little bit longer for warmup, but at the end it works as expected.
window.__karma__.loaded = function () {}; // prevent of execution mocha https://zerokspot.com/weblog/2013/07/12/delay-test-execution-in-karma/
var quixote = require('quixote');
var assert = require('chai').assert;
var frame;
var deviceWidth = [320, 768, 1024];
var runTests = function (browserWidth, index) {
frame = quixote.createFrame({
src: 'http://localhost:9876/test/', // karma url with port
width: browserWidth
}, function () {
if (index === 0) {
setTimeout(function () { // browserStack extra timeout
window.__karma__.start(); //execute mocha
}, 1000);
}
});
describe('Device width: ' + browserWidth, function () {
require('./src/**/specs/*layout.js', { mode: 'list' }).forEach(function (file) {
file.module(frame, assert, browserWidth);
});
});
beforeEach(function () {
frame.reset();
});
after(function () {
frame.remove();
});
};
deviceWidth.forEach(runTests);
2a. We can't add inner to each button for now, because we've got about 50 unique pages that uses buttons and 9 teams in 4 countries. So it will be too overhead. And we can't use :before or :after as it used by other teams.
On our website we've got only 0.8 pixel mismatch, so it didn't break anything.
2b. It didn't help us with 0.8 pixel problem.
I fork your lib, change to 1 pixel tolerance and everything works perfectly. Even we solve some IE 11 bugs and problems with bigger tolerance.
It's hard to reproduce all the things, that we get on our rendered page.
I'll try after Easter holidays.
Thank you so much for helping!
Are you aware of QFrame.resize()? It may be faster than creating three separate frames.
The key idea behind 2a) is to have relative comparisons rather than hard-coded pixel values. This is the core innovation behind Quixote compared to other tools. Rather than adding tolerance workarounds, we want to give you the ability to make precise comparisons, even when there are cross-browser differences.
In your case, it's likely that the issues you're seeing are caused by font-rendering differences across browsers. The trick is to get the font metrics and use that as the basis for your tests, not a hard-coded pixel value.
So your test becomes height: font.height.plus(20) rather than height: 38.
You don't have to modify your actual HTML to do this. You should be able to do it in your test. One way would be to programmatically add the <span> to your button, but it's probably easier to just add an element to the bottom of the frame:
beforeEach(function() {
...
font = frame.add("<span style='font: my-font;'>Arbitrary text</span>");
});This would be something worth making easier. I'll create an issue for it.
Today we remove frame.reset() and gain at least 30% speed. Now all tests run in few seconds.
We don't use frame.add() at all. We check directly on the page.
And it's really fast.
We'll keep 1 pixel rounding as we had this issue in IE, Edge and Firefox.
Today we try to make 70 tests on 3 window width, it takes only 10 seconds on 1 browder.
That's really awesome!
Glad it's working for you! I'll consider this issue closed, then.