Merlin: Network Magic

A Django implementation of Merlin

Build using

REST API provided by

Scheduling coordinated by

And

Featuring presentation enhancements with

And Mind Maps by

Metrics by

Featuring Advanced Search and more with

Containerized with

Presented with

Merlin Installation Guide

Install Docker Engine and Docker Compose (already included on macOS and Windows)

First make sure you have Docker Compose https://docs.docker.com/compose/install/

From your terminal

git clone https://github.com/automateyournetwork/merlin_unchained.git &&
cd merlin_unchained/merlin/merlin

Note: If you will be accessing Merlin from a different PC, edit ALLOWED_HOSTS in the settings.py file before building the container.

docker-compose build merlin_unchained &&
docker-compose up -d merlin_unchained

To gracefull shutdown Merlin:

$ docker-compose down --volumes

Merlin User Guide

The Admin Panel - https://youtu.be/svuKgtc4mWI

DevNet Sandbox Integration - https://www.youtube.com/watch?v=VBAqD9SiHxg

Working with Devices - https://www.youtube.com/watch?v=d7g2ywKf1PM

Considerations for current Alpha build

Please note - these instructions are currently to install all of the required packages, clone the repository, and start a pre-packaged database and server.
In the near future this project will move to a Docker container in hopes of eliminating these manual steps
In the interest of agility and accessibility and due to demand I've decided to make a working README for the project in it's current Alpha state so others can start to use the tool
The project has ** only ** been tested against a single device and has not taken scale into consideration at this point yet - once I scale out and confirm 1+n devices work this bullet point will be removed

Homepage

Simply Visit http://localhost:80 to navigate the world of Merlin

Log In - Admin Panel

You should now be able to visit http://localhost:8000/admin

Where you will be presented with a login

Default Username and Password:

admin

You can change this default password as well as add your own users and groups via the admin panel

Tables

Each pyATS function - learn config, parse show ip interface brief, learn platform, etc - are all mapped to indivudal database tables

As an administrator you can perform Create Read Update Delete (CRUD) operations directly from the administrator panel

Default Dataset

This utility is currently being built using the Cisco DevNet Sandbox - Nexus 9000

I have included a single record set per table for demonstration from the Cisco DevNet Sandbox - Nexus 9000

In order to be YAML FREE we convert a traiditonal "testbed.yaml" file into a database Model and Table

Traditional pyATS solutions typically would include a file like

testbed_DevNet_Nexus9k_Sandbox.yaml

That would look like this:

devices:
    sbx-n9kv:
      alias: 'DevNet_Sandbox_Nexus9k'
      type: Nexus 9000
      os: 'nxos'
      platform: n9k
      credentials:
        default:
          username: admin
          password: Cisco123
      connections:        
        cli:
          protocol: ssh
          ip: 10.10.20.58
          port: 22
          arguments:
            connection_timeout: 360

Merlin has transformed this into a database table via a Django Model which allows us to be YAML FREE!

Again, this alpha has been tested on a Nexus 9000, should you choose to try this on your own (hopefully) LAB equipment

Delete this device from the database

Add a new device

Populate the following fields:

Hostname - (required) The actual hostname of the device
Alias - (required) A friendly name for the device
Type - (optional) Platform type friendly name (ie - Nexus 9000)
OS - (required) Either 'nxos', 'ios', or 'iosxe'
Platform (required) Plese check pyATS documentation but Platforms include:
- n9k, n7k, n5k, cat4500, cat3850, cat9300
Username - (required) A valid username
Password - (required) A valid password
Protocol - (required) Typically 'ssh'; could be 'telnet'
IP - (required) A valid IP address (ie - 10.10.20.58)
Port - (required) Typically '22'
Connection Timeout - (optional) Typically 360 seconds

Again, you can populate multiple devices and mix operating systems, but in this Alpha scale (1+n devices) and non-NXOS platforms (IOS, IOS-XE), have NOT been tested, will NOT be supported, but will eventually become part of Merlin. Your mileage may vary.

Bulk Import

Using this format


Hostname,Alias,Device_type,OS,Platform,Username,Password,Protocol,IP Address,Port,Connection Timeout

You can visit http://localhost:8000/DeviceImport/ and bulk-upload your devices into Merlin

Please refer to the bulk_import_template.csv file in this repository

Scheduling Merlin

In the Admin Panel, using the Django Celery Beat package, you can setup periodic scheduled hands-free Merlin.

Scheduling Notes

It is recommended to use the Scheduler in the Admin panel, however, to manually start a 5 minute full state capture run the 2 following lines at the CLI:

(merlin)$ celery -A merlin beat -l info --logfile=celery.beat.log --detach  

(merlin)$ celery -A merlin worker -l info --logfile=celery.log --detach

To stop manually scheduled period tasks:

(merlin)$ pkill -f "celery worker"  

(merlin)$ kill -9 $(ps aux | grep celery | grep -v grep | awk '{print $2}' | tr '\n'  ' ') > /dev/null 2>&1

pyATS Command Coverage

As of this build the following pyATS functions have been transformed into Merlin:

Learn ACL
Learn ARP
Learn ARP Statistics
Learn BGP Instances
Learn BGP Routes
Learn BGP Tables
Learn Config
Learn Interface
Learn Platform
Learn Platform Slots
Learn Platform Virtual Devices
Learn VLAN
Learn VRF
Show Inventory
Show IP Interface Brief
Show Version

On-Demand Centre

http://localhost:8000/OnDemand/

Merlin allows users to capture either the full network state or individual functions using the On-Demand Centre.

Users can press the button will capture the requested records for all devices in the database

Users can also filter the scope of the pyATS job at a group level using Device Type, Operating System or Username; or at an individual device level using Hostname, Alias, or IP Address

While the pyATS job collects and transforms the network state a spinner will be displayed in the Status field

Search Engine

http://localhost:8000/Search/

The Merlin Search Engine allows users to keyword search against the database.

Search All Records will search both Network State and Network Configuration tables. Using this search will return the 'full' JSON of a running-config if the search hits against the Learn Configuration table.

This search omits Learn Config and only searches the "State" of the network

This search only looks up Learn Config and, different from the All Records Search, will break down any hits in a Configuration to each individual key, value pair found.

Using the same "vty" example:

Network State Change Centre

http://localhost:8000/Changes/

The Merlin Network State Change Centre can detect changes in the network state or configuration by gathering the latest state from the network and comparing it against the previous latest record set in the database.

Much like the On-Demand Centre users can press a button to compare against all devices in the database or they can use keyword filtering at a group or host level to scope the network state capture against specific devices.

No Changes Detected

If Merlin does not detect any changes in network state or configuration it will tell you!

Additions Found

If Merlin detects additions - that is to say, state or configuration found in the network but not found in the database - you will be notified of these additions

Removals Found

If Merlin detects removals - that is to say, state or configuration not found in the network but found in the database - you will be notified of these removals

REST API

http://localhost:8000/API/

Merlin contains a fully functional, stand-alone REST API !

As you can see - each pyATS function has it's own dedicated REST API

http://localhost:8000/API/LearnConfig/

cURL, Postman, Python

Provided Merlin is running you can automate and program against the REST API !

Without ever even launching your browser you can use other tools to access the REST API

cURL

curl http://localhost:8000/API/LearnConfig/

Postman

Popular API tools like Postman can also be used!

Python

Pythonically you can use the requests library

$ source merlin/bin/activate 
(merlin)$ python

# import requests
>>> Import requests

# GET Learn Config Tables

>>> learn_config = requests.get("http://localhost:8000/API/LearnConfig")

# Validate response
>>> config
<Response [200]>

# Dump content of API to screen
>>> config.content

Triggering Automation with the API

It should be noted that all of the pyATS automation behind "the buttons" and filters can be triggered with REST API GET requests against the appropriate URL / API

For example if you want to repopulate the entire database with fresh network state and configuration you could do a GET against

http://localhost:8000/OnDemand/GetAll/get_all_all_result/

Once the pyATS job completes Postman will return the following success confirming the database has been refreshed with new network state.

Business Ready Document Centre

http://localhost:8000/CSV/

At the heart of Merlin is the ability to provide network state and configuration information in Business Ready Documents - Spreadsheets !

Either download All Records or only the Latest Records with the click of a button!

All Records

Latest Records

These CSV files will be in your default browser Download location and can be opened with Excel, VS Code with Excel Preview, or any CSV-viewer

Latest Records

http://localhost:8000/Latest/

Often users may want only the latest records - the so-called "current" state - of the database and ideally the network state and running-configuration

Users can visit any /Latest/{{ any function }} to get the Latest records.