netdiff
Netdiff is an experimental Python library that provides utilities for parsing network topologies of open source dynamic routing protocols and calculating changes in these topologies.
It was developed to abstract the differences between the different JSON structures of the open source dynamic routing protocols (like OLSR and batman-advanced).
It's currently also used in Nodeshot to update the network links that are shown on the map.
If you are a developer of another community network node-db project and you want to use netdiff to update the topology stored in your database, please get in touch!
Install stable version from pypi
Install from pypi:
pip install netdiff
Install development version
Install tarball:
pip install https://github.com/ninuxorg/netdiff/tarball/master
Alternatively you can install via pip using git:
pip install -e git+git://github.com/ninuxorg/netdiff#egg=netdiff
If you want to contribute, install your cloned fork:
git clone git@github.com:<your_fork>/netdiff.git
cd netdiff
python setup.py develop
Basic Usage Example
Calculate diff of an OLSR 0.6.x topology:
from netdiff import OlsrParser
from netdiff import diff
old = OlsrParser('./stored-olsr.json')
new = OlsrParser('telnet://127.0.0.1:9090')
diff(old, new)
In alternative, you may also use the subtraction operator:
from netdiff import OlsrParser
from netdiff import diff
old = OlsrParser('./stored-olsr.json')
new = OlsrParser('telnet://127.0.0.1:9090')
old - new
The output will be an ordered dictionary with three keys:
- added
- removed
- changed
Each key will contain a dict compatible with the NetJSON NetworkGraph format representing respectively:
- the nodes and links that have been added to the topology
- the nodes and links that have been removed from the topology
- links that are present in both topologies but their cost changed
If no changes are present, keys will contain None
.
So if between old
and new
there are no changes, the result will be:
{
"added": None
"removed": None,
"changed": None
}
While if there are changes, the result will look like:
{
"added": {
"type": "NetworkGraph",
"protocol": "OLSR",
"version": "0.6.6",
"revision": "5031a799fcbe17f61d57e387bc3806de",
"metric": "ETX",
"nodes": [
{
"id": "10.150.0.7"
},
{
"id": "10.150.0.6"
}
],
"links": [
{
"source": "10.150.0.3",
"target": "10.150.0.7",
"cost": 1.50390625
},
{
"source": "10.150.0.3",
"target": "10.150.0.6",
"cost": 1.0
}
]
},
"removed": {
"type": "NetworkGraph",
"protocol": "OLSR",
"version": "0.6.6",
"revision": "5031a799fcbe17f61d57e387bc3806de",
"metric": "ETX",
"nodes": [
{
"id": "10.150.0.8"
}
],
"links": [
{
"source": "10.150.0.7",
"target": "10.150.0.8",
"cost": 1.0
}
]
},
"changed": {
"type": "NetworkGraph",
"protocol": "OLSR",
"version": "0.6.6",
"revision": "5031a799fcbe17f61d57e387bc3806de",
"metric": "ETX",
"nodes": [],
"links": [
{
"source": "10.150.0.3",
"target": "10.150.0.2",
"cost": 1.0
}
]
}
}
Parsers
Parsers are classes that extend netdiff.base.BaseParser
and implement a parse
method
which is in charge of converting a python data structure into networkx.Graph
object and return the result.
Parsers also have a json
method which returns valid NetJSON output.
The available parsers are:
netdiff.OlsrParser
: parser for the olsrd jsoninfo pluginnetdiff.BatmanParser
: parser for the batman-advanced alfred toolnetdiff.Bmx6Parser
: parser for the BMX6 b6m toolnetdiff.CnmlParser
: parser for CNML 0.1netdiff.NetJsonParser
: parser for theNetworkGraph
NetJSON object.
Initialization arguments
data: the only required argument, different inputs are accepted:
- JSON formatted string representing the topology
- python dict (or subclass of dict) representing the topology
- string representing a HTTP URL where the data resides
- string representing a telnet URL where the data resides
- string representing a file path where the data resides
timeout: integer representing timeout in seconds for HTTP or telnet requests, defaults to None
verify: boolean indicating to the request library whether to do SSL certificate verification or not
Initialization examples
Local file example:
from netdiff import BatmanParser
BatmanParser('./my-stored-topology.json')
HTTP example:
from netdiff import NetJsonParser
url = 'https://raw.githubusercontent.com/interop-dev/netjson/master/examples/network-graph.json'
NetJsonParser(url)
Telnet example with timeout
:
from netdiff import OlsrParser
OlsrParser('telnet://127.0.1:8080', timeout=5)
HTTPS example with self-signed SSL certificate using verify=False
:
from netdiff import NetJsonParser
OlsrParser('https://myserver.mydomain.com/topology.json', verify=False)
NetJSON output
Netdiff parsers can return a valid NetJSON
NetworkGraph
object:
from netdiff import OlsrParser
olsr = OlsrParser('telnet://127.0.0.1:9090')
# will return a dict
olsr.json(dict=True)
# will return a JSON formatted string
print(olsr.json(indent=4))
Output:
{
"type": "NetworkGraph",
"protocol": "OLSR",
"version": "0.6.6",
"revision": "5031a799fcbe17f61d57e387bc3806de",
"metric": "ETX",
"nodes": [
{
"id": "10.150.0.3"
},
{
"id": "10.150.0.2"
},
{
"id": "10.150.0.4"
}
],
"links": [
{
"source": "10.150.0.3",
"target": "10.150.0.2",
"cost": 2.4
},
{
"source": "10.150.0.3",
"target": "10.150.0.4",
"cost": 1.0
}
]
}
Exceptions
All the exceptions are subclasses of netdiff.exceptions.NetdiffException
.
ConversionException
netdiff.exceptions.ConversionException
Raised when netdiff can't recognize the format passed to the parser.
Not necessarily an error, should be caught and managed in order to support additional formats.
The data which was retrieved from network/storage can be accessed via the "data" attribute, eg:
def to_python(self, data):
try:
return super(OlsrParser, self).to_python(data)
except ConversionException as e:
return self._txtinfo_to_jsoninfo(e.data)
ParserError
netdiff.exceptions.ParserError
Raised when the format is recognized but the data is invalid.
NetJsonError
netdiff.exceptions.NetJsonError
Raised when the json
method of netdiff.parsers.BaseParser
does not have enough data
to be compliant with the NetJSON NetworkGraph specification.
TopologyRetrievalError
netdiff.exceptions.TopologyRetrievalError
Raised when it is not possible to retrieve the topology data (eg: the URL might be temporary unreachable).
Known Issues
ConnectionError: BadStatusLine
If you get a similar error when performing a request to the jsoninfo plugin of olsrd (version 0.6 to 0.9) chances are high that http headers are disabled.
To fix it turn on http headers in your olsrd configuration file, eg:
LoadPlugin "olsrd_jsoninfo.so.0.0" { PlParam "httpheaders" "yes" # add this line PlParam "Port" "9090" PlParam "accept" "0.0.0.0" }
Running tests
Install your forked repo:
git clone git://github.com/<your_fork>/netdiff
cd netdiff/
python setup.py develop
Install test requirements:
pip install -r requirements-test.txt
Run tests with:
./runtests.py
Alternatively, you can use the nose
command (which has a ton of available options):
nosetests
nosetests tests.test_olsr # run only olsr related tests
nosetests tests/test_olsr.py # variant form of the previous command
nosetests tests.test_olsr:TestOlsrParser # variant form of the previous command
nosetests tests.test_olsr:TestOlsrParser.test_parse # run specific test
See test coverage with:
coverage run --source=netdiff runtests.py && coverage report
Contributing
- Join the ninux-dev mailing list
- Fork this repo and install it
- Follow PEP8, Style Guide for Python Code
- Write code
- Write tests for your code
- Ensure all tests pass
- Ensure test coverage is not under 90%
- Document your changes
- Send pull request