BackstopJS
Catch CSS curve balls.
BackstopJS automates visual regression testing of your responsive web UI by comparing DOM screenshots over time.
Version 3 Features
- Render with Chrome Headless, Phantom and Slimer
- Simulate user interactions with ChromyJS and CasperJS scripts
- Browser reports with visual diffs
- CLI reports
- JUnit reports
- Plays nice with CI and source control
- Run globally or locally as a standalone package app or
require('backstopjs')
right into your node app - Incredibly easy to use: just 3 commands go a long long way!
Install BackstopJS now
$ npm install -g backstopjs
Contents
Tutorials, Extensions and more
The BackstopJS workflow
-
backstop init
: Set up a new BackstopJS instance -- specify URLs, cookies, screen sizes, DOM selectors, interactions etc. (see examples directory) -
backstop test
: BackstopJS creates a set of test screenshots and compares them with your reference screenshots. Any changes show up in a visual report. (Run this after making CSS changes as many times as needed.) -
backstop approve
: If the test you ran looks good, then go ahead and and approve it. Approving changes will update your reference files with the results from your last test. Future tests are compared against your most recent approved test screenshots.
Getting started
Installation
Global installation (recommended)
$ npm install -g backstopjs
Local installation
BackstopJS will run as a totally stand alone app -- but installing locally allows you to do this...
const backstop = require('backstopjs');
See Integration Options to learn about cool BackstopJS integration options!
Initializing your project
If you don't already have BackstopJS set up... BackstopJS can create a default configuration file and project scaffolding in your current working directory. Please note: this will overwrite any existing files!
cd
to your projects's directory and run...
$ backstop init
Working with your config file
By default, BackstopJS places backstop.json
in the root of your project. And also by default, BackstopJS looks for this file when invoked.
Required config properties
As a new user setting up tests for your project, you will be primarily concerned with these properties...
id
– Used for screenshot naming. Set this property when sharing reference files with teammates -- otherwise omit and BackstopJS will auto-generate one for you to avoid naming collisions with BackstopJS resources.
viewports
– An array of screen size objects your DOM will be tested against. Add as many as you like -- but add at least one.
scenarios
– This is where you set up your actual tests. The important sub properties are...
scenarios[n].label
– Required. Also used for screenshot naming.scenarios[n].url
– Required. Tells BackstopJS what endpoint/document you want to test. This can be an absolute URL or local to your current working directory.
Generating test bitmaps
$ backstop test
This will create a new set of bitmaps in bitmaps_test/<timestamp>/
Once the test bitmaps are generated, a report comparing the most recent test bitmaps against the current reference bitmaps will display.
Pass a --config=<configFilePathStr>
argument to test using a different config file.
Tip To use a js-module as a config file, just explicitly specify your config filepath and point to a .js
file. Just be sure to export your config object as a node module.
Pass a --filter=<scenarioLabelRegex>
argument to just run scenarios matching your scenario label.
Approving changes
$ backstop approve
Change can be good! When running this command, all images (with changes) from your most recent test batch will be promoted to your reference collection. Subsequent tests will be compared against your updated reference files.
Tip: Remember to pass a --config=<configFilePathStr>
argument if you passed that when you ran your last test.
Using BackstopJS
Advanced Scenarios
Scenario properties are described throughout this document and processed sequentially in the following order...
label // [required] Tag saved with your reference images
onBeforeScript // Used to set up browser state e.g. cookies.
url // [required] The url of your app state
referenceUrl // Specify a different state or enviornment when creating reference.
readyEvent // Wait until this string has been logged to the console.
readySelector // Wait until this selector exists before continuing.
delay // Wait for x millisections
hideSelectors // Array of selectors set to visibility: hidden
removeSelectors // Array of selectors set to display: none
onReadyScript // After the above conditions are met -- use this script to modify UI state prior to screen shots e.g. hovers, clicks etc.
hoverSelector // Move the pointer over the specified DOM element prior to screen shot (available with default onReadyScript)
clickSelector // Click the specified DOM element prior to screen shot (available with default onReadyScript)
postInteractionWait // Wait for a selector after interacting with hoverSelector or clickSelector (optionally accepts wait time in ms. Idea for use with a click or hover element transition. available with default onReadyScript)
selectors // Array of selectors to capture. Defaults to document if omitted. Use "viewport" to capture the viewport size. See Targeting elements in the next section for more info...
selectorExpansion // See Targeting elements in the next section for more info...
misMatchThreshold // Around of change before a test is marked failed
requireSameDimensions // If set to true -- any change in selector size will trigger a test failure.
Targeting elements
BackstopJS makes it super easy to capture screenshots of your entire layout or just parts of your layout. This is defined in the your scenario.selectors array. Elements are defined with standard CSS notation. By default BackstopJS takes a screenshot of the first occurrence of any selector found in your DOM. e.g. If you have three li
tags in your layout only the first will used.
selectorExpansion
If you want BackstopJS to find and take screenshots of all matching selector instances then there is a handy switch for that. Set selectorExpansion
to true
like so...
scenarios: [
{
"selectors": [
".aListOfStuff li"
],
"selectorExpansion": true
}
]
// captures all <li> tags inside .aListOfStuff
(Default behavior) If you want very explicit control of what you capture then you can disable selectorExpansion
and explicitly select what you want...
scenarios: [
{
"selectors": [
".aListOfStuff li"
],
"selectorExpansion": false
}
]
// Just captures the first <li> tag inside .aListOfStuff
Testing Progressive apps, SPAs and AJAX content
It is very common for client-side web apps is to initially download a small chunk of bootstrapping code/content/state and render it to the screen as soon as it arrives at the browser. Once this has completed, various JS components often take over to progressively load more content/state.
The problem testing these scenarios is knowing when to take the screenshot. BackstopJS solves this problem with two config properties: readySelector
, readyEvent
and delay
.
Trigger screen capture via selector
The readySelector
property tells BackstopJS to wait until a selector exists before takeing a screenshot. For example, the following line will delay screen capture until a selctor with the id '#catOfTheDayResult' is present somewhere in the DOM.
"readySelector": "#catOfTheDayResult"
Another approach might look like this...
"readySelector": "body.ember-has-rendered"
Trigger screen capture via console.log()
The readyEvent
property enables you to trigger the screen capture by logging a predefined string to the console. For example, the following line will delay screen capture until your web app calls console.log("backstopjs_ready")
...
"readyEvent": "backstopjs_ready"
In the above case it would be up to you to wait for all dependencies to complete before calling logging "backstopjs_ready"
string to the console.
Delay screen capture
The delay
property enables you to pause screen capturing for a specified duration of time. This delay is applied after readyEvent
(if also applied).
"delay": 1000 //delay in ms
In the above case, BackstopJS would wait for one second before taking a screenshot.
In the following case, BackstopJS would wait for one second after the string backstopjs_ready
is logged to the console.
{
// ...
"readyEvent": "backstopjs_ready",
"delay": 1000 //delay in ms
// ...
}
Dealing with dynamic content
For obvious reasons, this screenshot approach is not optimal for testing live dynamic content. The best way to test a dynamic app would be to use a known static content data stub – or ideally many content stubs of varying lengths which, regardless of input length, should produce certain specific bitmap output.
Hiding selectors
That said, for a use case where you are testing a DOM with say an ad banner or a block of dynamic content which retains static dimensions, we have the hideSelectors
property in capture/config.json
which will set the corresponding DOM to visibility:hidden
, thus hiding the content from our Resemble.js analysis but retaining the original layout flow.
"hideSelectors": [
"#someFixedSizeDomSelector"
]
Removing selectors
There may also be elements which need to be completely removed during testing. For that we have removeSelectors
which removes them from the DOM before screenshots.
"removeSelectors": [
"#someUnpredictableSizedDomSelector"
]
Changing test sensitivity
"misMatchThreshold"
(percentage 0.00%-100.00%) will change the amount of difference BackstopJS will tolerate before marking a test screenshot as "failed". The default setting is 0.1
, this may need to be adjusted based on the kinds of testing you're doing.
More info on how misMatchThreshold is derrived can be found here... https://github.com/Huddle/Resemble.js/blob/af57cb2f4edfbe718d24b350b2be1d956b764298/resemble.js#L495
"requireSameDimensions"
(true || false) will change whether BackstopJS will accept any change in dimensions. The default setting is true
. If set to true then the test must be the same dimensions as the reference. If set to false the test does not have to be the same dimensions as the reference.
This setting can be used in conjunction with "misMatchThreshold"
, for example, when setting a "misMatchThreshold"
of more than 0.00% and the mismatch causing a change in dimensions, setting "requireSameDimensions"
to false will allow the test to still pass, setting it to true would still make it fail.
Capturing the entire document or just the viewport, or just an element, or a combination.
BackstopJS recognizes two magic selectors: document
and viewport
-- these capture the entire document and just the current specified viewport respectively. e.g.
"scenarios": [
{
"selectors": [
"document",
"viewport",
"#myFeature",
// ...
],
// ...
}
]
Testing across different environments
Comparing against different environments is easy. (e.g. compare a production environment against a staging environment).
You can create reference files (without previewing) by using the command backstop reference
. By default this command calls the url
property specified in your config. Optionally, you can add a referenceUrl
property to your scenario configuration. If found, BackstopJS will use referenceUrl
for screen grabs when running $ backstop reference
.
"scenarios": [
{
"label": "cat meme feed sanity check",
"url": "http://www.moreCatMemes.com",
"referenceUrl": "http://staging.moreCatMemes.com:81",
// ...
}
]
Running custom scripts
Simulate user actions (click, scroll, hover, wait, etc.) or states (cookie values) by running your own Casper.js script on ready. For each scenario, the custom .js file you specify is imported and run when the BackstopJS ready events are fulfilled.
From your project root, place your scripts in...
./backstop_data/engine_scripts
at the root of your config or in your scenario...
"onReadyScript": "filename.js" // Runs after onReady event on all scenarios -- use for simulating interactions (.js suffix is optional)
"onBeforeScript": "filename.js" // Runs before each scenario -- use for setting cookies or other env state (.js suffix is optional)
"scenarios": [
{
"label": "cat meme feed sanity check",
"onReadyScript": "filename.js" // If found will run instead of onReadyScript set at the root (.js suffix is optional)
"onBeforeScript": "filename.js" // If found will run instead of onBeforeScript at the root (.js suffix is optional)
// ...
}
]
Inside filename.js
, structure it like this:
// onBefore example
module.exports = function(casper, scenario, vp) {
// scenario is the current scenario object being run from your backstop config
// vp is the current viewport object being run from your backstop config
// Example: setting cookies
casper.echo("Setting cookies");
casper.then(function(){
casper.page.addCookie({name: 'cookieName', value: 'cookieValue'});
});
}
// onReady example
module.exports = function(casper, scenario, vp) {
// Example: Adding script delays to allow for things like CSS transitions to complete.
casper.echo( 'Clicking button' );
casper.click( '.toggle' );
casper.wait( 250 );
// Example: changing behavior based on config values
if (vp.label === 'phone') {
casper.echo( 'doing stuff for just phone viewport here' );
}
// ...do other cool stuff here, see Casperjs.org for a full API and many ideas.
}
Setting the base path for custom onBefore and onReady scripts
By default the base path is a folder called engine_scripts
inside your BackstopJS installation directory. You can override this by setting the paths.scripts
property in your backstop.json
file to point to somewhere in your project directory (recommended).
NOTE: SlimerJS currently requires an absolute path -- so be sure to include the full path when using the "engine": "slimer"
configuration option.
"paths": {
"engine_scripts": "backstop_data/engine_scripts"
}
Reporting workflow tips
One testing approach to consider is incorporating BackstopJS into your build process and just let the CLI report run on each build or before each deploy.
It's natural for your layout to break while you're in feature development -- in that case you might just run a backstop test
when you feel things should be shaping up.
Using the report
property in your config to enable or disable browser including/excluding the respective properties. E.G. The following settings will open a browser and write a junit report.
"report": ["browser", "CI"]
If you choose the CI-only reporting or even no reporting (CLI is always on) you can always enter the following command to see the latest test run report in the browser.
$ backstop openReport
Test report integration with a build system like Jenkins/Travis
The following config would enable the CI - report (default: junit format)
"report" : [ "CI" ],
The regression test report will be generated in the JUnit format and the report will be placed in the given directory (default: [backstopjs dir]/test/ci_report/xunit.xml).
You may customize the testsuite name and/or a report file (xunit.xml) path to your build report directory by using the below configuration overrides,
"paths": {
"ci_report" : "backstop_data/ci_report"
},
"ci": {
"format" : "junit" ,
"testReportFileName": "myproject-xunit", // in case if you want to override the default filename (xunit.xml)
"testSuiteName" : "backstopJS"
},
CLI error handling
When a layout error is found in CLI mode, BackstopJS will let you know in a general report displayed in the console. In addition, BackstopJS will return a 1 (error) to the calling CLI process.
Setting the bitmap and script directory paths
By default, BackstopJS saves generated resources into the backstop_data
directory in parallel with your backstop.json
config file. The location of the various resource types are configurable so they can easily be moved inside or outside your source control or file sharing environment. See below for options...
Tip: these file paths are relative to your current working directory._
...
"paths": {
"bitmaps_reference": "backstop_data/bitmaps_reference",
"bitmaps_test": "backstop_data/bitmaps_test",
"engine_scripts": "backstop_data/engine_scripts",
"html_report": "backstop_data/html_report",
"ci_report": "backstop_data/ci_report"
}
...
Changing the rendering engine
BackstopJS supports using Chrome-Headless, PhantomJS or SlimerJS for web app rendering. PhantomJS is currently the default value and will be installed by default.
Chrome-Headless (The latest webkit library)
This will also enable the very cool chromy.js (https://github.com/OnetapInc/chromy) library. (When creating onBefore and onReady scripts please make sure you are referring to the Chromy script documentation. Casper features will not work with this setting.)
You must also have Chrome installed, version 59 or greater is required.
"engine": "chrome"
Slimer (Gecko/Mozilla rendering)
To run in Slimer, be sure to have SlimerJS installed. From your root directory run...
$ npm install -g slimerjs
Then, in your backstop.json
config file, update the engine property to...
"engine": "slimerjs"
Thats it.
Setting Casper command-line flags
This is for you if for some reason you find yourself needing advanced configuration access to CasperJS. You can set CasperJS flags via casperFlags
like so...
"casperFlags": [
"--engine=slimerjs",
"--proxy-type=http",
"--proxy=proxyIp:port",
"--proxy-auth=user:pass"
]
Integration options (local install)
If you are going to call backstop from another app you will probably want to do a local install in the project directory of your choice...
$ npm install backstopjs
Once installed you can simply require a local backstop installation in your project.
const backstop = require('backstopjs');
Invoke default behavior in the current working directory context
backstop('test')
.then(() => {
// test successful
}).catch(() => {
// test failed
});
Pass options to the command
backstop('test', {config:'custom/backstop/config.json'});
Pass a config to the command
// you can also pass
backstop('test', {
config: {
id: "foo",
scenarios: [
//some scenarios here
]
}
});
Parse a config file explicitly
backstop('test', {
config: require("./backstop.config.js")({
"foo": "bar"
})
});
// Inside of `backstop.config.js` we export a function that returns the configuration object
module.exports = options => {
return {
//you can access options.foo here
}
}
Since the backstop returns promises so it can run natively as a task in build systems like gulp
const gulp = require('gulp');
const backstopjs = require('backstopjs');
gulp.task('backstop_reference', () => backstopjs('reference'));
gulp.task('backstop_test', () => backstopjs('test'));
Using npm run scripts
When BackstopJS is installed locally, NPM will recognize the backstop <command>
pattern originating from your own NPM package.json
scripts. The following would enable you to run the respective npm <command>
commands locally in your project.
"scripts": {
"approve": "backstop approve",
"test": "backstop test",
"init": "backstop init"
}
The above is a crude example -- there are other fancy mappings you can create as well -- check out the NPM documentation for more info.
Tuning BackstopJS performance
During a test, BackstopJS processes image capture and image comparisons in parallel. You can adjust how much BackstopJS does at one time by changing
Capturing screens in parallel
By default, this value is limited to 10. This value can be adjusted as needed to increase/decrease the amount of RAM required during a test.
The example below would capture 5 screens at a time...
asyncCaptureLimit: 5
Comparing screens in parallel
By default, this value is limited to 50. This value can be adjusted as needed to increase/decrease the amount of RAM required during a test.
As a (very approximate) rule of thumb, BackstopJS will use 100MB RAM plus approximately 5 MB for each concurrent image comparison.
To adjust this value add the following to the root of your config...
"asyncCompareLimit": 100
// Would require 600MB to run tests. Your milage most likely will vary ;)
Creating reference files
This Utility command will by default delete all existing screen references and create new ones based on the referenceUrl
or url
config config. It will not run any file comparisons.
Use this when you...
- create references from another enviornment (e.g. staging vs prod)
- or clean out your reference files and start fresh with all new reference
- or just create references without previewing
From your project directory...
$ backstop reference
optional parameters
--config=<configFilePath>
point to a specific config file
--filter=<scenario.name>
filter on scenario.name via regex string
--i
incremental flag -- use if you don't want BackstopJS do first delete all files in your reference directory
Modifying output settings of image-diffs
By specifying resembleOutputOptions
in your backstop.json file you can modify the image-diffs transparency, errorcolor, etc. (See Resemble.js outputSettings for the full list.
"resembleOutputOptions": {
"errorColor": {
"red": 255,
"green": 0,
"blue": 255
},
"errorType": "movement",
"transparency": 0.3
}
Troubleshooting
Migrating to 2.0
Filename issue: Projects don't work when I share with other users or run in different environments.
If you just upgraded to 2.x from 1.x
Filename formats have changed. To use the 1.x (compatible) file format, use the fileNameTemplate
property like so...
{
// ...
fileNameTemplate: '{scenarioIndex}_{scenarioLabel}_{selectorIndex}_{selectorLabel}_{viewportIndex}_{viewportLabel}',
// ...
}
If you are not migrating scripts but have recently upgraded BackstopJS
Be sure to use a config id
in your config file. See garris#291
Windows users...
PhantomJS needs Python -- please make sure you have Python installed... see garris#185
The dreaded command-not-found error...
Did you install BackstopJS with the global option? If installing globally remember to add that -g
when installing with npm i.e. npm install backstop -g
. If you installed locally, remember that the backstop <command>
pattern will only be available to your npm scripts -- see the local installation section above for more info.
Debugging
If you are using Chrome-Headless engine then you have the option of displaying the Chrome window as tests are running. This can be helpful for visually monitoring your app state at the time of your test. To enable use...
"debugWindow": true
For all engines there is also the debug
setting. This enables verbose console output.This will also output your source payload to the terminal so you can make sure to check that the server is sending what you expect.
"debug": true
Tutorials, Extensions and more
-
(RECOMMEDED Updated for version 2) Regression testing with BackstopJS, in-depth tutorial by Angela Riggs http://www.metaltoad.com/blog/regression-testing-backstopjs
-
BackstopJS tutorial on css-tricks.com
-
A lovely article on Making Visual Regression Useful by Phillip Gourley
-
Automated regression testing for AngularJS (and other) web-apps -- article on DWB
-
Grunt fans -- check out grunt-backstop and this very nicely written article by Joe Watkins
-
Generate a BackstopJS configuration file from sitemap.xml with BackstopJS Scenarios Constructor by Enzo Sterro
-
BackstopJS brochure at http://BackstopJS.org/.
Credits
BackstopJS was created and is maintained by Garris Shipon
Features by:
- Shinji Yamada for Chrome Headless & Chromy.JS integration support in 3.0.0.
- Shane McGraw for testing and awesomeness during 3.0 development.
- Steve Fischer, uğur mirza zeyrek, Sven Wütherich, Alex Bondarev for concurrency support, JS config passing, JPEG support, CLI Auth support.
- Klaus Bayrhammer for making BackstopJS a "requireable" node module in 2.3.1
- Suresh Kumar. M for selector expansion in 1.3.2
- @JulienPradet, @onigoetz, @borys-rudenko, @ksushik, @dmitriyilchgmailcom, @Primajin for giving the world BackstopJS version 2.0!
- Suresh Kumar. M for help on the 1.3.2 release
- Klaus Bayrhammer for all the incredible effort leading up to 1.0 -- the cli reports and compatibility fixes are awesome!
- Evan Lovely and Klaus Bayrhammer for help on the 0.9.0 release
- Robert O'Rourke for help on the 0.8.0 release
- Klaus Bayrhammer for help on the 0.7.0 release
- Benedikt Rötsch for help on the 0.6.0 release
- Yulia Tsareva for help on the 0.5.0 release -- windows support
- Lewis Nyman and Stoutie for help with 0.4.0 release -- you guys are responsible for really getting the ball rolling!
BackstopJS uses icons from the Noun Project