baconbao / aim

Aalto Interface Metrics (AIM): A Service and Codebase for Computational GUI Evaluation.

Aalto Interface Metrics (AIM)

Aalto Interface Metrics (AIM) is a new service that lets you automatically test the usability and aesthetics of your website using several verified user interface metrics. Simply enter a web page URL or upload a screenshot, and select desired metrics to get started. AIM segments the page, and displays a preview. As results are computed, they are presented along with detailed explanations, and histograms for comparison. AIM is fully open-sourced, and we invite you to extend the service with your own contributions. Head over to interfacemetrics.aalto.fi to give it a try, and for more information.

Architecture

AIM codebase is divided into three distinct parts:

• Backend - web application
• Frontend - web application
• Legacy - miscellaneous files

The backend is written in Python using the Tornado web application framework having MongoDB as a database, whereas the frontend is built with Vue.js. The backend and the frontend communicate with each other in real-time via WebSocket. The legacy folder contains miscellaneous files of AIM version 1 that have not been integrated into the new AIM version yet.

The most important files and folders in the AIM codebase are:

.
├── backend               : AIM backend files
│   ├── aim               : Source code (incl. metrics)
│   ├── data              : Data files (incl. datasets)
│   ├── tests             : Unit tests
│   ├── evaluator.py      : Evaluator utility app
│   ├── screenshoter.py   : Screenshoter utility app
│   └── server.py         : Server app
├── frontend              : AIM frontend files
│   ├── build             : Build scripts
│   ├── src               : Sources code (incl. assets)
│   ├── static            : Static files
│   ├── test              : Unit tests, etc.
│   └── config.js         : Configuration module
├── legacy                : AIM legacy files (version 1)
├── scripts               : AIM utility scripts
├── .env(.example)        : AIM environment variables
├── docker-compose.yml    : AIM environment and services
├── metrics.json          : AIM metrics configuration file
├── ...
.

Quick Start with Docker

The easiest and fastest way to get AIM up and running on your computer is by using Docker and following the instructions below.

Prerequisites

1. Docker Engine
2. Docker Compose

Configuration

Go to the project home directory and copy the .env.example to a .env file:

cp .env.example .env

Then, configure AIM environment variables, if needed:

nano .env

Usage

AIM uses Docker Compose (see docker-compose.yml) to define its environment and services. The services consist of AIM backend (backend) and AIM frontend (frontend) as well as MongoDB (mongo) and Mongo Express (mongo-express). The latter is a web application for managing the MongoDB database, and it can be accessed at http://localhost:8081.

To build and start the services (i.e., the entire AIM web application), run:

# Note: The --build option can be omitted on subsequent runs
docker-compose up --build

Upon completion, you can start using the AIM web application by opening your web browser and navigating to http://localhost:8080.

The setup also supports smooth frontend and backend development, as it uses bind mounts to most essential source files. However, some actions such as installing new dependencies might require rebuilding the images (with the same command as above).

To stop the services, just hit Ctrl+C. You can also stop services by their ID as follows:

# List all running containers
docker container ls
# Stop a specific container by its ID
docker container stop <ID>

Installation

AIM can also be installed and run without Docker. While this installation process requires some additional effort, it may work better for some practitioners. Moreover, it offers additional utility apps and tools that ain't currently available in docker mode.

Prerequisites

Make sure you have the following software already installed on your computer before proceeding!

The backend dependencies include Python 3.7, pip, MongoDB, and Chrome. In addition, it is highly recommended to install virtualenv or Pipenv to create a dedicated Python virtual environment (see instructions). Other dependencies include Node.js 16.14.2 and npm for the frontend and git to track changes made to the codebase.

Configuration

Go to the project home directory and copy the .env.example to a .env file:

cp .env.example .env

Then, configure AIM environment variables, if needed:

nano .env

The variables of interest (with default values) are as follows:

# Frontend-specific variables
NODE_ENV=development
FRONTEND_PORT=8080
WS_URL=ws://localhost:8888/
AUTO_OPEN_BROWSER=false

# Backend-specific variables
ENVIRONMENT=development
NAME=aim-dev
PORT=8888
DATA_INPUTS_DIR=data/webapp/inputs/
DATA_RESULTS_DIR=data/webapp/results/
DB_USER=root
DB_PASS=example
DB_HOST=mongo
DB_PORT=27017
DB_NAME=aim

Database

Create a new database called $DB_NAME (see the AIM environment variables above) in MongoDB with the following three collections in it: errors, inputs, and results. Further, create a new user called $DB_USER with a password $DB_PASS in the admin database, or use the user credentials of an existing one. You also have to change $DB_HOST and $DB_PORT to match the host (e.g., localhost) and port (e.g., 27017) used by your MonggoDB, respectively.

Note: AIM backend will attempt to authenticate the user to the admin database in MongoDB (for details, see authSource). Therefore, make sure that your $DB_USER exists in the admin database (not in your $DB_NAME) and it has appropriate roles (e.g., readWrite) on your $DB_NAME database.

Backend

AIM backend uses Selenium to take screenshots of web pages. Our Selenium code, in turn, depends on Chrome and a matching version of ChromeDriver, of which the latter needs to be placed into the ./backend/webdrivers folder. For Linux and macOS, we provide a script that automatically downloads a matching version of ChromeDriver and places it into the above-mentioned folder. To execute the script, run:

bash ./scripts/get_webdrivers.sh

Alternatively, on Windows, you need to manually download a matching version of ChromeDriver and place it into the above-mentioned folder. For more instructions or what the necessary steps are, please check the comments in the script file.

In the next step, we will create a new Python virtual environment (recommended) for the backend. Before proceeding, go to the backend directory.

Working with virtualenv

Create a new virtual environment:

virtualenv ../venv

Activate the virtual environment:

source ../venv/bin/activate

Install all dependencies, including development packages:

pip install -r requirements.txt

Re-activate the virtual environment to update paths (see Stack Overflow for details):

deactivate && source ../venv/bin/activate

To deactivate the virtual environment, run:

deactivate

Working with Pipenv

Install all dependencies, including development packages:

pipenv install --dev

Activate your Pipenv environment:

pipenv shell

To deactivate your Pipenv environment, run:

exit

Frontend

Go to the frontend directory and install the dependencies by running:

npm install

Usage

Backend

To start the backend server, go to the backend directory and run:

python server.py

Frontend

To start the frontend HTTP server in development mode, go to the frontend directory and run:

npm run dev

To build the frontend for production, run the following command in the same directory:

npm run build

After the build is complete, the files (for production) can be found under the newly created dist directory. These files are meant to be served over an HTTP server, such as Apache HTTP Server.

It is highly recommended to use a load balancer (e.g., Apache HTTP Server) in a production environment, as certain metrics are extremely CPU intensive. This means that the backend needs to be launched with multiple instances, each listening to a different port. A process manager (e.g., pm2) will come in handy at that point.

Tests

AIM backend uses pytest, a Python testing framework, to run tests on the backend source code, including metrics. To configure and run the tests, go to the backend directory and follow the instructions below.

Configuration

Configure pytest, if needed:

nano pytest.ini

Usage

Run all tests:

pytest .

Run a specific test file:

pytest [FILEPATH]

Screenshoter App

AIM backend provides a utility app for taking web page screenshots in specified dimensions (i.e., width, height, and/or full page). To configure and run the app, go to the backend directory and follow the instructions below.

Configuration

Configure Loguru, if needed:

nano loguru.ini

Usage

Show the help message:

python screenshoter.py -h

Example of taking web page screenshots:

python screenshoter.py -i data/screenshots/ALEXA_500/urls.csv -sw 1280 -sh 800 -f -o data/screenshots/results/

Evaluator App

AIM backend also provides a utility app for evaluating GUI designs (i.e., web page screenshots) against selected metrics. The app generates three files: results.csv, results.json, and quantiles.csv. Two former files contain evaluation results, while the latter file contains statistics. Optionally, it also generates histogram figures for each metric. To configure and run the app, go to the backend directory and follow the instructions below.

Configuration

Configure Loguru, if needed:

nano loguru.ini

Datasets

Alexa Top Global Sites (ALEXA_500) currently serves as our test dataset. Additional datasets can be downloaded, for instance, from https://doi.org/10.7910/DVN/XEYNYW.

Usage

Show the help message:

python evaluator.py -h

Example of evaluating GUI designs:

python evaluator.py -i data/screenshots/ALEXA_500/ -m m1,m2,m3 -p -o data/evaluations/results/

Note: A set of screenshots in the input directory can be excluded from the evaluation by listing their file names in the exclude.txt file.

Utility Tools

In addition, AIM backend supports the following utility tools to (i) ease development and (ii) unify and improve code quality. Their installation and use is optional, but highly recommended.

• isort. Python utility to automatically sort imports. https://pypi.org/project/isort/
• Black. Python code formatter. https://pypi.org/project/black/
• Mypy. Static type checker for Python. https://pypi.org/project/mypy/
• Flake8. Python tool for style guide enforcement. https://pypi.org/project/flake8/

Installation

Go to the backend directory. No additional installation is needed.

Configuration

Configure isort, if needed:

nano .isort.cfg

Configure Black, if needed:

nano pyproject.toml

Configure mypy, if needed:

nano mypy.ini

Configure flake8, if needed:

nano .flake8

Usage

Sort imports:

isort .

Format code:

black .

Type check code:

mypy .

Lint code:

flake8 .

Pre-Commit

Git hook scripts for AIM backend that are automatically run on every commit to point out issues in code such as missing semicolons, trailing whitespace, and debug statements. Its installation and use is optional, but highly recommended.

• pre-commit. Package manager for pre-commit hooks. https://pypi.org/project/pre-commit/

Installation

Go to the project home directory and install pre-commit into your git hooks:

pre-commit install --install-hooks --overwrite

To uninstall pre-commit from your git hooks, run:

pre-commit uninstall

Configuration

Configure pre-commit, if needed:

nano .pre-commit-config.yaml

Usage

Run all pre-commit hooks against currently staged files:

pre-commit run

Run a single pre-commit hook against currently staged files:

pre-commit run [HOOK ID]

Run all pre-commit hooks against all files:

pre-commit run --all-files

Run all pre-commit hooks against specific files:

pre-commit run --files [FILES [FILES ...]]

Troubleshooting

1. PROBLEM: Running mypy fails and the following error message is shown ./venv/lib/python3.7/site-packages is in the PYTHONPATH. Please change directory so it is not. SOLUTION: Create your virtual environment outside of the AIM backend directory. For example, virtualenv ../venv. Alternatively, configure mypy to exclude your virtual environment directory.
2. PROBLEM: Running mypy finds errors in files that are not part of the AIM backend. For example, venv/bin/activate_this.py:28: error: "str" has no attribute "decode"; maybe "encode"? SOLUTION: Create your virtual environment outside of the AIM backend directory. For example, virtualenv ../venv. Alternatively, configure mypy to exclude your virtual environment directory.
3. PROBLEM: Running python screenshoter.py or submitting an URL in the web application fails and an error message similar to the following is shown selenium.common.exceptions.SessionNotCreatedException: Message: session not created: This version of ChromeDriver only supports Chrome version 89. Current browser version is 91.0.4472.77 with binary path /Applications/Google Chrome.app/Contents/MacOS/Google Chrome SOLUTION: Download a matching version of ChromeDriver and replace the appropriate chromedriver_xxx file in the webdrivers directory.

Contributing

Please read CONTRIBUTING.md for details on contributing to AIM.

Changelog

Detailed changes for each release are documented in the release notes.

Funding Information and Contact

AIM and related research have been funded by the Technology Industries of Finland Future Makers project SOWP (2017-2019) and the European Research Council starting grant (COMPUTED 2015-2020).

For questions and further information, please contact us via email at [e-mail].

License

Copyright (c) 2018-present, User Interfaces group, Aalto University, Finland

This software is distributed under the terms of the MIT License. See LICENSE.txt for details.