IXP Country Jedi

IXP Country Jedi is a project that shows if the Internet traffic paths stay within the country, and if the paths between two networks within the country go through an Internet eXchange Point (IXP).

You can get the pre-processed data here: jedi.ripe.net

You can read more about the use-cases in these articles:

Probe mesh measurements

This codebase contains a couple of scripts to make probe mesh measurements feasible/easy.

Monthly runs of this code for all countries with enough diversity in RIPE Atlas probes can be found here: jedi.ripe.net

The way the scripts are used/tested is to first create a directory for your measurement campaign, and run the scripts as follows:

mkdir 'SE-2015-03'
cd SE-2015-03 
<create config.json file here>
<run scripts relative to current directory, ie. ../prepare.py ; 
../measure.py ...>

The scripts depend on this convention.

Dependencies on external python modules are in 'requirements.txt', so you can run

pip install -r requirements.txt

to fulfill these.

Before you use the scripts, create a config file specifying what mesh you would like to do.

config.json

This file contains base data used for creating measurements and analysing the results. It can contain various parts:

country: ISO 2 letter code for the country under analysis, or list of countries under analysis (ie. NL, IT, CH);
ixps: List of IXPs one wants to detect/report on;
locations: List of cities on which probe selection will be based (if applicable)
probetag: Probetag on which probe selection criteria is based.
probe_ids: List of probes you want to measure from.

examples are provided in the examples directory. For probe selection it is mandatory to either have country , probetag , or probe_ids defined. It is possible to combine probetab and country (which will take all the probes with that probetag in the specified country/countries). It is also possible to combine probe_ids and country: after selecting the probes in the specified country/ies, the probe_ids specified will be added (any duplicates will be removed). Without a locations section, the capital of country (or the first country, if country is a list) is used for probe selection (see below).

One can set an extra 'location-constraint' config key. The value of this configuration directive needs to be an integer. When this is set, the probe selection part of ixp-country-jedi will only select probes that are within 'location-constraint' kilometers from any of the given locations. This can be useful if you only want to measure a specific city. In that case, set the 'location'-list to that specific city, and set a 'location-constraint' to a reasonable value (for instance 50, so you'll cover 50 kilometers from the city centre).

Note: The interaction between various of the configuration settings hasn't been thoroughly tested, so beware of combining. What is known to work is probetag as a stand-alone probe selection option, and country is known to work well together with all other config options.

prepare.py

This script reads config.json and generates two output files:

basedata.json
probeset.json

For each one of the IXPs listed in config.json it will try to find the member ASNs.

Based on the country and locations specified it will do a probe selection. It will:

only select public probes (because the measurements we are going to create will use their their public IP address);
for every ASN, it will select up to twice the number of locations specified.

If an ASN hosts more than 2 probes, the selection will be a maximum of 2x the number of locations in config.json: the closest as well as the furthest probe from each location will be selected. Note that this can result in less than 2 times the number-of-locations probes per ASN even when there are more probes available in that ASN, because a single probe could be the furthest away from multiple locations.

In the preparation/data gathering phase this script uses the GeoNames service for geocoding. Please put a valid geonames username in ~/.geonames/auth.

for more information on GeoNames see http://www.geonames.org/export/web-services.html

This script also relies on Google Maps Geocoding API to geocode a location from list of probes. The limit is 2500 free requests. You can pass your own API key by using environment variable GOOGLEMAPS_API_KEY.

measure.py

This script runs one-off measurements for the probes specified in probeset.json and stores their results in measurementset.json

This uses the RIPE Atlas measurement API for measurement creation, and it needs a valid measurement creation API key in ~/.atlas/auth .

For more information on RIPE Atlas API keys see https://atlas.ripe.net/docs/keys/

get-ips.py

This script gathers metadata for all the IPs in the collected data. This is done separately from the rest of the analysis-code because it is time-consuming and ideally is done pretty soon after running all the measurements. If done too soon, not all measurement results would be in yet, if done much later (think days) meta-data - like reverse hostname mapping - might have changed, so is don't get-ips a few minutes after measure.py. This step will create a file called ips.json-fragments file with reverse DNS lookups, ASNs and geoloc (via OpenIPMap, not MaxMind) of IPs encountered in the traceroutes.

get-measurements.py

This script fetches measurements (from measurementset.json) and does some initial analysis on them using information from config.json, basedata.json, probeset.json and ips.json-fragments. It creates a local results directory and outputs a single json file per measurement (analysed..json) which is a list of analysis results, one result per src/dst combination.

analyse-results.py

This script produces text and/or webpages with analysis and visualisation in a local analysis directory. For webpages, these need to be on an actual webserver for some of the javascript in them to work. One can easily create a local webserver that would work for this purpose like this:

cd analysis ; python -m SimpleHTTPServer 3333

and then pointing your browser at localhost:3333/

Note that some visualisations use libraries in a common directory located in the analysis directory, so the webserver needs to run with the 'analysis'-directory (or lower) as root.

Templates (HTML,javascript,CSS) for webpages are in the templates directory and copied over each time the analyse-results.py script is run.

If you want to tweak the visualisations, tweaking them in the templates directory and then running the analyse-results.py script again will probably do what you want.

doit.py

This script is a convenient replacement for running the scripts above in sequence for every country (running individual Jedis for each country). When finished it packs the configs into ixp-country-jedi-confs.tgz .

Install the user-to-user graph locally

The user-to-user graph (dixit: 'eyeballsgraph') is a js/react application that can be ran locally with webpack-dev-server.

To do so:

You'll need node.js installed to make this work. Go to https://nodejs.org/en/download/package-manager/ to see an overview of options for all OSes. I greatly prefer using nodesource.com for linux distros. For MacOS I prefer using the bash install script instead of homebrew, but YMMV.
Clone this repo somewhere on you local machine with git clone git@github.com:emileaben/ixp-country-jedi.git
Go into the eyeballsgraph directory: cd template/eyeballsgraph/
Install all dependencies with: npm install
Right now we don't have a (public) npm package for one dependency, the @ripe-rnd/ui-components library. You'll have to link this manually like this:
- Clone the ui-components library git clone git@github.com:RIPE-NCC/rnd-ui-components.git in a directory of your liking.
- Go into the root of the library with cd ui-components
- Make a linked local package out of this repo with npm link
- Now go (back) into the template/eyeballsgraph/ diretory of the ixp-country-jedi repo and install the linked package with npm link @ripe-rnd/ui-components.
Now you can start the dev server with npm start and go with you favourite web-browser to localhost:4042

In the future we will have a npm package of the @ripe-rnd/ui-components and the link steps will go away.

Note that you might run into trouble with the CORS settings of the RIPE NCC webservers when making XHR calls to the json files hosted on the webservers of the RIPE NCC. A workaround is to locally edit your /etc/hosts file to point localhost to something.ripe.net and change the webpack-dev-server port to port 80. Now if you go to https://something.ripe.net on your browser you'll be able to download the JSON files (the CORS headers are set to *.ripe.net).

Misc

The misc directory has various prototype/ideas that are partly implemented. It has a separate README

emileaben / ixp-country-jedi