JeevesTakesOver/Railtrack

Railtrack

A building block for a cloud based infrastructure.
A distributed L2 switch across multiple cloud providers.
A resilient, meshed VPN/Network layer.
Built on Tinc
Self-serviceable.
Access control provided by GitHub/GitLab/Bitbucket.
Manages Pets
Static or DHCP IP allocation of Pets on the VPN.
Python based, built on Fabric and Bookshelf

The code in this repository provisions a fully meshed, geographically distributed, resilient, Layer 2 encrypted virtual network.

This allows for deploying any networked service across the internet, including multi-cloud setups with a single network namespace. As it is a layer 2 VPN, it also supports multicast,unicast services or floating virtual-ips.

Services can be added to the virtual network simply by binding the listening ports of those services to the local virtual network ip addresses. This allows for increased security as access to services is only available to hosts connected in the VPN.

Management of the hosts allowed to access the virtual network is performed through git and pull requests. To add/remove a host of the virtual network, install the tinc client on that box, generate a private/public key, update the tinc config to connect to the central tinc nodes, and add the public key to the following repository:

https://github.com/JeevesTakesOver/menagerie

Workflow

The expected workflow is that any team member can add new team members or services to the virtual network by commiting the tinc public keys of the tinc clients to the github repository and submit a pull request. This PR is peer reviewed and once approved, brings the new service online on the virtual network. This workflow allows for teams to self-service themselves, adding new hosts or team members or removing their access when they leave the project.

Usage cases

A new starter which upon joining his team, submits a pull request asking for access to the company services by providing his public key.
Allow teams to be self-sufficient in adding/removing services to the virtual network.
Full audits and history through git repository history.
Single Network segment across multi-cloud vendors
Highly available VPN services
Cross-Region/Global network
Deployment of secure services only available through the VPN.
Mobile workers

How does it do it?

There are four hosts:

3 core hosts (core01..03), running tinc, isc-dhcp-server, bind9, consul, fsconsul.

1 git2consul host, running git2consul and a consul client

As soon a public key is commited to the menagerie github repository a process running in the git2consul host will forward the new key to a consul cluster running on the three core vpn nodes. On each one of the core vpn master, another process fsconsul will write down the ssh keys into the tinc host directories and reload the daemons. At this point the new tinc client is allowed access to the VPN and all its services.

On the client side, the tinc interface-up scripts will ask for a DHCP ip address through the VPN link. On the first core-vpn server isc-dhcp-server is running and will provide an ip address from a DHCP pool. That ip address is then added to a local bind9 DNS zone. The DNS servers are configured as caching DNS servers and will forward queries upstream, so they can be configured on the client as the main DNS servers, or simply for the VPN zone domains.

Requirements

python virtualenv

Configuration and Deployment

Set the following environment variables

export AWS_ACCESS_KEY_ID=MY_AWS_KEY
export AWS_SECRET_ACCESS_KEY=MY_SECRET_KEY

prepare a git repository for holding your secrets

mkdir deployment-secrets
cd deployment-secrets
echo "Railtrack" > .gitignore
git init
git add .gitignore
git commit -m "ignore all Railtrack dir"
mkdir key-pairs

Generate private and public keys for the different hosts:

openssl genrsa -out key-pairs/core01.priv 4096
openssl rsa -pubout -in key-pairs/core01.priv -out key-pairs/core01.pub

openssl genrsa -out key-pairs/core02.priv 4096
openssl rsa -pubout -in key-pairs/core02.priv -out key-pairs/core02.pub

openssl genrsa -out key-pairs/core03.priv 4096
openssl rsa -pubout -in key-pairs/core03.priv -out key-pairs/core03.pub

openssl genrsa -out key-pairs/git2consul.priv 4096
openssl rsa -pubout -in key-pairs/git2consul.priv -out key-pairs/git2consul.pub

ssh-keygen -f key-pairs/tinc-vpn.pem

commit your keys

git add key-pairs
git commit -m "added tinc keys"

clone the Railtrack repo

git clone git@github.com:JeevesTakesOver/Railtrack.git
cd Railtrack
git checkout v1.0.1
cd -

copy the config.yaml sample file

cp Railtrack/config/config.yaml config.yaml
export CONFIG_YAML=$PWD/config.yaml

edit the config.yaml deployment file:
- update any file paths so that they point to the newly created key-pair files
- Add new public DNS names, IP addresses of the EC2 instances.
- Add the public key contents to the different blocks.
- Choose a Consul Encryption Key.
```
vim $CONFIG_YAML
```

commit your config.yaml

git add config.yaml
git commit -m "added config.yaml"

Prepare a python virtualenv

cd Railtrack
virtualenv venv
. venv/bin/activate
pip install -r requirements.txt

Edit the main.tf if needed.

To deploy, run the following:

fabric -f tasks/fabfile.py step_01_create_hosts
fabric -f tasks/fabfile.py run_it
fabric -f tasks/fabfile.py acceptance_tests

Laptop Configuration

To consume a DHCP IP address from the VPN, see the provision block for the laptop, and the up_laptop task in the Makefile. The laptop VM is an example for configuring a client to obtain an IP address from the VPN which is automatically registered in DNS.

NIXOS

My local development laptop is NixOS, there's is a local default.nix file to help with consuming the python code in a more standard virtualenv way.

just run:

nix-shell

Jenkins Builds on NixOS using Mesos

This is my Jenkins build job for RailTrack CI

#!/usr/bin/env bash

source /etc/profile

export HOME=/var/lib/mesos
export PYTHONUNBUFFERED=no

set -e

export AWS_ACCESS_KEY_ID=XXXXXXXXXXXXXXXXXXXX
export AWS_SECRET_ACCESS_KEY=YYYYYYYYYYYYYYYYY

export CONFIG_YAML=config/config.yaml

nix-shell --run "fab -f tasks/fabfile.py jenkins_build"

License

This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with this program. If not, see <http://www.gnu.org/licenses/>.

JeevesTakesOver / Railtrack