superterran scanner

This project uses a Chrome webdriver to fetch all requests being generated by the page. Chrome renders the page and executes the javascript, giving malicious code a change to expose itself. The scanner reviews each request, rules out ones from trusted domains, then highlights what is suspicious. A human can then evalute each offending request one-by-one. All assets are scanned against the global whitelist, and also the whitelists defined in this repo.

The pipelines in this repo are configured to run this script every day. and after every build.

Usage

There are two ways to use this tool:

Run from This Repo

To run this scanner as-is, clone down this repo, and then run from within:

$ composer install
$ make selenium-start-docker
$ bin/scanner scan

Quick Scan

You can run the scanner against any url with the default settings with bin/scanner scan:url example.com

Creating and running from a Targets repo

Probably the best way to run this tool is to create a private repo with a set of targets, and run the tool against that.

$ mkdir targets-repo
$ cd targets-repo

let's create a composer project and add our repo (these will streamline as this project matures)

$ composer init
$ composer config repositories.scanner vcs https://github.com/superterran/scanner
$ composer require superterran/scanner:dev-master
$ composer update

Now let's add some example targets

$ mkdir targets/
$ cp vendor/superterran/scanner/targets/example.yml targets/example.yml

Optionalyl, if you want to create a git repo from this...

$ git init
$ cp vendor/superterran/scanner/.gitignore .
$ git add .
$ git commit -m 'initial commit!'

Now we can scan the targets in this directory with:

$ vendor/bin/scanner

Selenium

This project uses selenium to manage a chrome webdriver to collect network data. For now, you must have selenium listening on your local port 4444. We can make this configurable. The makefile in vendor/superterran/scanner/ demonstrates other ways to run Selenium but most don't fully work, or are for firefox where this is built for the chrome webdriver. make selenium-start-docker works completely, is my preference, and is implemented in this repo's pipeline.

Targets

Targets are yml files that describe how to scan a site. They also provide a site-level whitelist, use this for assets you don't fully trust, or anything site specific.

You can run an individual target file with the following:

$vendor/bin/scanner scan <target>

Example target.yml

site-level whitelists are better for when you can't match all subdomains, when it's a client-specific resource, or maybe something deserves further review but seems safe.

---
whitelist:
  - //www.example.com/
  - .cloudmaestro.com/
  - .cdninstagram.com/
  - .doubleclick.net/
targets:
  - url: https://www.example.com/
  - url: https://www.example.com/checkout/

Whitelists

./whitelist.yml contains a global whitelist, this will be included with the upstream whitelist, and sitemap whitelist when executing. These should validate full domains, so only put domain checks in there starting with //, for example...

---
- //blueacorn.io/

superterran / scanner