MOAR URLs - MURL

MURL takes a list of URLs and maps their domains to any indentified adtech vendors.

To use MURL, you need a .txt file with a list of urls or domains. If you are testing multiple sites or scenarios, split each site into a separate .txt file.

A common use case for MURL is to export all urls seen in a proxy capture or a pcap file, and then map those calls to the companies that own the domains.

MURL generates a report that lists out each tested site, with each company contacted during testing for each site. Any domains that were contacted but not mapped to an owner are listed at the end of the report.

MURL will also export all URLs that have parameters in a single section. This can simplify searching for and finding sensitive or meaningful information exposed in urls.

MURL makes use of DuckDuckGo's Tracker Radar, which is included as a submodule.

Running MURL

MURL was developed against Python 3.7, but it should work on Pyton 3.6 or greater.

Clone the repository onto your local machine:

git clone git@github.com:billfitzgerald/murl.git

Once you have the repository cloned, run git submodule init to initialize submodule config file. Then, run git submodule update to bring in the code from the Tracker Radar.

For more information and background on working with submodules, see the book.

Quick start

Make sure the source and results directories exist. See Usage details below for more information.
Create .txt files with the URLs you want to map. Generally, these files will be exported from an intercepting proxy capture or a pcap file. See Naming your .txt files below for more information.
Put the .txt files in the source directory.
Optional: adjust the output settings as needed. This isn't necessary, and the default settings are good for most cases. For more information, see Adjust output preferences below.

Usage details

After cloning the repo, create two directories in the base: source and results.

You will place all source .txt files in the source directory, and all reports will be stored in the results directory.

You can change these locations, but be sure to adjust the corresponding values in the python script.

Naming your .txt files

When creating the text files that store urls, your file name should contain the name of the domain you are testing. For example, if you are testing nytimes.com, your file name should be nytimes_com.txt.
Create a separate .txt file for each domain or scenario you are testing. Every individual file will have its own section in the report generated by MURL.
In the .txt files, put one url on each line.
Don't worry about long and ugly urls in your .txt files. They will be managed.

Adjust output preferences

In most cases, this isn't needed, as the defaults work cleanly. However, if you want to fidget, three settings can be adjusted. These values can be set in the beginning of the analyse_urls.py file.

include_params = "yes" # set as a yes/no value. Setting to "no" excludes all parameters from the report
params_separate_file = "yes" # set to "yes" to only write paramaters to a separate file. This is a sane default
include_subdomains = "yes" # set as a yes/no value. Setting to "no" excludes all subdomains from the report

I see a problem, or I want to help

If you see something that appears to be inaccurate, please open an issue.

billfitzgerald / murl