Vizzy is a powerful Ruby on Rails web server that facilitates Visual Automation, a continuous integration testing strategy that aims to prevent visual regressions. It does this by performing pixel by pixel comparisons of screenshots captured during test runs. In doing so, it tests application data as well as application views. In order to harness the full power of Vizzy, there are two major prerequisites:
- A build pipeline that runs tests on pull requests.
- A client testing framework that enables developers to easily write tests that can access the application's views, capture images of them, and save those images to a directory.
The process Vizzy goes through is as follows:
-
First a test suite is run against the master branch. This generates a set of images which becomes the baseline to test against. These "Base Images" are uploaded and managed by Vizzy.
-
Then for every subsequent pull request changeset, the same test suite runs and generates a new set of images called "Test Images". MD5 hashes are created from the images so only the images that are different from their base image get uploaded. Vizzy goes through each matching test, which contains two images, one base image and one test image, and performs a pixel by pixel diff of the two images using a tool called ImageMagick. A third image is generated which highlights the differences in red.
-
After all diffs are calculated, Vizzy updates a Github status with the link to the visual overview. It displays: diffs, new tests, missing tests, test history, comments, and the ability to approve test images. The developer is then presented with the option to approve the differences if the changes were intentional, or to fix, commit, and rerun visual automation if the changes were unintentional or indicate a bug. Once the pull request is merged and subsequent master build finishes, the approved images replace the base images for those tests, and subsequent visual tests are compared against these new images. Large development teams can run multiple builds simultaneously because the base image set always matches the master branch.
Vizzy is built with a plugin architecture. Plugins implement build hooks (build_created, build_committed, build_failed) to achieve a task. Slack, Jira, Bamboo, and Jenkins plugins are provided and configurable.
Ruby 2.3.7 and Rails 5.1.6.2. See http://installrails.com/ for install instructions.
Install ImageMagick, a command line image processing tool (version 6 is required as the latest version causes bundle install to fail):
brew unlink imagemagick
brew install imagemagick@6 && brew link imagemagick@6 --force
Install the PostgreSQL database:
brew install postgres
PostgreSQL is a database server, so you'll need to start it up to run Vizzy.
brew services start postgresql
Vizzy uses icu4c version 62. The default installation from brew will not work and version 62 is not listed under recents. To replace the default installation with version 62 you can run:
`brew reinstall https://raw.githubusercontent.com/Homebrew/homebrew-core/575eb4bbef683551e19f329f60456b13a558132f/Formula/icu4c.rb
Fork the repository. Then clone your fork: git clone https://github.com/<your-name>/vizzy.git
Vizzy uses Rails 5.1.6.2 encrypted secrets for its configuration. From the Vizzy project directory, run these commands
Generate the encrypted secrets file and key.
bin/rails secrets:setup
Do not check in the encryption key into the repository, add it to the project with an environment variable called RAILS_MASTER_KEY. For more information you can go to this blog post.
Add secrets to the yaml file by running the following command: Note: you may specify for your favorite editor.
EDITOR=vi bin/rails secrets:edit
If you see an error Devise.secret_key was not set you will have to add the sample secret_key to devise.rb, run the setup/edit, then remove the Devise.secret_key. Devise now uses the project secret_key_base.
Here are the possible secrets:
shared:
# Required Secrets
POSTGRES_USER: "postgres_database"
POSTGRES_PWD: "postgres_database_password"
GITHUB_AUTH_TOKEN: "auth_token_value"
ADMIN_EMAILS: "email1@gmail.com, email2@gmail.com"
# Optional Plugin Secrets
BAMBOO_USERNAME: "bamboo_username"
BAMBOO_PASSWORD: "bamboo_password"
JIRA_USERNAME: "jira_username"
JIRA_PASSWORD: "jira_password"
SLACK_WEBHOOK: "slack_webhook_value"
BUGSNAG_API_KEY: "bugsnag_api_key_value"
# Required To Run System Tests
GITHUB_ROOT_URL: "live_github_url"
GITHUB_REPO: "live_github_repo"
JIRA_PROJECT: "live_jira_project"
JIRA_BASE_URL: "live_jira_base_url"
BAMBOO_BASE_URL: "live_bamoo_base_url"
development:
secret_key_base: "XXXXX1"
test:
secret_key_base: "XXXXX2"
ci_test:
secret_key_base: "XXXXX3"
production:
secret_key_base: "XXXXX4"To generate new instances of secret_key_base run rake secret 4 times and copy/paste the values above. Upon saving this file, a file called secrets.yml.enc will be created.
To configure open vizzy.yaml
For local authentication with user registration, no setup is required. The default auth strategy is local.
defaults: &default
devise:
auth_strategy: 'local'For LDAP authentication/account lookup, configure your LDAP auth server.
defaults: &default
devise:
auth_strategy: 'LDAP'
ldap_email_domain: '@domain.com'
ldap_host: 'host_ip_address'
ldap_port: 'host_port'
ldap_base: 'DC=domaininternal,DC=com'
ldap_email_internal_domain: '@domaininternal.com'
password_placeholder: "LDAP Password"From the Vizzy project directory run
rake db:create db:migrate
From the Vizzy project directory run
rails server
Vizzy projects allow you to run visual automation on multiple branches (ex. Master, Develop), each branch represented by a project. Each project has its own set of base images. There is typically a one to one mapping of a build plan to a Vizzy project. NOTE: Pull Requests opened against the master branch should be uploaded to the master Vizzy project so the correct set of base images are used to calculated the diffs.
To create a test project for development, I recommend seeding the database. Vizzy uses FactoryBot to add test data. Add your Github information (required) as well as plugin settings (optional) to the project factory project.rb.
Then run rake db:reset which will clear the database, run the migrations, and populate the database with the test data in seeds.rb(sample user and project). You could also achieve this by running rake db:seed if you don't want to clear the database.
You can also create a user account and navigate to http://0.0.0.0:3000/projects to create a new project once the server is running.
Create a Rails Run Configuration in your IDE (RubyMine)
Run the server in development mode. Settings are straightforward.
Navigate to http://0.0.0.0:3000 to see the running server.
There are 3 sets of sample images in /test-image-upload. There is an utility script run_test_push.rb to make running test builds easier. It only takes 2 parameters: Test Case and Is Master. Here are the 6 possible combinations which are easy to setup as Ruby Run Configurations.
| Name | Test Case | Is Master |
|---|---|---|
| Master 1 | 1 | 1 |
| Master 2 | 2 | 1 |
| Master 3 | 3 | 1 |
| Pull Request 1 | 1 | 2 |
| Pull Request 2 | 2 | 2 |
| Pull Request 3 | 3 | 2 |
Here is a sample run config for Master 1:
This server is docker ready and can be deployed with any tool you want. Be sure to complete the configuration steps in Setup before deploying the server.
- Dockerfile: Builds dependencies, adds the source code, precompiles the assets, and exposes port 3000.
Travis CI publishes the latest master image to Docker Hub
We recommend a tool called Kubernetes (k8s), an opened sourced container cluster manager originally designed by Google, now owned by Cloud Native Computing. This tool aims to provide a platform for automating deployment, scaling, and operations of application containers across clusters of hosts.
Helm, a tool used for Kubernetes deployments, is recommended for deployment and the chart is provided at in vizzy. More information on how to deploy Vizzy with helm can be found here
Vizzy scales well, we use 5 replica pods each with 8Gi memory.
Other options: Google Kubernetes Engine [AWS EKS] (https://aws.amazon.com/eks/)
Create an account, and select new project and choose a name. Choose type "Container Registry" Note: requires Heroku CLI. Select the tap "Resources" and choose the add-on Heroku Postgres to provision the database.
Since you have the configurations in your repo, build and push the docker container to heroku.
docker build --build-arg RAILS_MASTER_KEY=[encryption-key-here] --tag registry.heroku.com/[project-name-here]/web .
docker push registry.heroku.com/[project-name-here]/web
To get complete the setup of the database you need to run the rails migrations
heroku run rake db:create db:migrate
You should now be able to see Vizzy running at https://[your-app-name-here].herokuapp.com/
Deployment infrastructure depends on the size of the team, for a team of 30, we use
1 Pod for Postgres database 5 Pods for the Vizzy server
All pods are assigned an IP/Host in the data center and share:
- A persistent volume (PVC - on the main host machine) which is mounted on every pod. All pods read and write images with this volume so as a user, no matter which of the 5 pods you connect to, images will all be loaded correctly.
- A single Postgres database shared for all pods.
If deployed on AWS, you can use [Amazon RDS] (https://aws.amazon.com/rds/postgresql/) for the database pod.
Once deployed, you can start adding Vizzy to your continuous integration builds.
Vizzy contains the upload script in the public directory upload_images_to_server.rb. To obtain the upload script in your CI builds, you can download it from the running server with this shell script
vizzy_endpoint="$1"
if [ -z "$vizzy_endpoint" ]; then
echo "No Vizzy endpoint given. Usage: './download_upload_script.sh <vizzy-endpoint>'"
exit 1
fi
echo "Downloading from $vizzy_endpoint/upload_images_to_server.rb"
curl -O "$vizzy_endpoint/upload_images_to_server.rb"
chmod a+x ./upload_images_to_server.rbThis allows the upload script to be versioned with the server.
There are 4 script options. Open is only used for creating Dev Builds and is not needed on CI builds.
Usage: upload_images_to_server.rb command [options] Available commands are: create: Creates a new build with the server. Saves build information in json into a provided file path upload: Upload provided images. Reads in build information from a file (as generated by the 'create' command) open: Opens a visual build in a browser referenced in the build information file as generated by the 'create' command. fail: Fail the build referenced in the build information file as generated by the 'create' command )
See 'upload_images_to_server.rb COMMAND --help' for more information on a specific command. -h, --help Show this message
VISUAL_HOST=https://vizzy.com
PLAN_KEY=${bamboo.planKey}
BUILD_NUMBER=${bamboo.buildNumber}
PLAN_TITLE=$PLAN_KEY-$BUILD_NUMBER
BUILD_URL="https://bamboo.com/browse/$PLAN_TITLE"
GIT_HASH=${bamboo.planRepository.1.revision}
VIZZY_USER_EMAIL=${bamboo.VIZZY_USER_EMAIL}
VIZZY_USER_TOKEN=${bamboo.VIZZY_USER_TOKEN_PASSWORD}
ruby ./upload_images_to_server.rb create $VISUAL_HOST --title "$PLAN_TITLE" --project 1 --commit "$GIT_HASH" --file ./visual-build-info --url "$BUILD_URL" --user-email "$VIZZY_USER_EMAIL" --user-token "$VIZZY_USER_TOKEN"#!/bin/sh
VISUAL_HOST=https://vizzy.com
TEST_IMAGE_DIR="../../android/application/visual-automation-test-images/visual-automation-device-images"
VIZZY_USER_EMAIL=${bamboo.VIZZY_USER_EMAIL}
VIZZY_USER_TOKEN=${bamboo.VIZZY_USER_TOKEN_PASSWORD}
ruby ./upload_images_to_server.rb upload $VISUAL_HOST --directory "$TEST_IMAGE_DIR" --file ./visual-build-info --user-email "$VIZZY_USER_EMAIL" --user-token "$VIZZY_USER_TOKEN"If your CI build fails at any time, you can notify Vizzy with a failure message
VISUAL_HOST=https://vizzy.com
VIZZY_USER_EMAIL=${bamboo.VIZZY_USER_EMAIL}
VIZZY_USER_TOKEN=${bamboo.VIZZY_USER_TOKEN_PASSWORD}
ruby ./upload_images_to_server.rb fail "$VISUAL_HOST" --message "Build Failed!" --file ./visual-build-info --user-email "$VIZZY_USER_EMAIL" --user-token "$VIZZY_USER_TOKEN"Squashing your commits when merging a pull request will break pre-approvals. This is because Vizzy uses the commit sha of the pull request and stores it with each image approval. Squash and merge will create a NEW merge commit and remove the commit sha that was used for approvals.
To disable this setting, go to the Github repository settings, and uncheck Allow squash merging
Turn off Animations for test suites: taking screenshots of animations does not always capture the same image which will cause visual diffs.
Mock Dynamic Data: taking screenshots that contain dates will change day to day and cause visual diffs.
Unit tests are run with the command
rake test
In order to run System tests, fill out the system test encrypted secrets. System tests are run with the command
rails test:system
Join the developer community https://join.slack.com/t/vizzy-dev/shared_invite/enQtMzQxMzI4MjE5MTA3LTNmM2U5MzgzN2U4NzIxZTMzNDI2ZjE5ZDNmNTBhYzUxYzFiMGIzNjE0YWNiYjRlZjhhNWM5YjAzOGViNDA5YzQ
https://medium.com/workday-engineering/testing-mobile-application-views-at-workday-c5f96ee14dc6
- Fork the repo!
- Create your feature branch:
git checkout -b my-new-feature - Commit your changes:
git commit -am 'Add some feature' - Push to the branch:
git push origin my-new-feature - Submit a pull request :D