curtisz / ssd

Docker Swarm Service Driller

Docker Swarm Service Driller(ssd)

ssd is a troubleshooting utility for Docker swarm networks.

control-plane and datapath consistency check on a node

ssd checks for the consistency between docker network control-plane (from the docker daemon in-memory state) and kernel data path programming. Currently the tool checks only for the consistency of the Load balancer (implemented using IPVS).

In a three node swarm cluser ssd status for a overlay network ov2 which has three services running, each replicated to 3 instances.

vagrant@net-1:~/code/go/src/github.com/docker/docker-e2e/tests$ docker run -v /var/run/docker.sock:/var/run/docker.sock -v /var/run/docker/netns:/var/run/docker/netns --privileged sanimej/ssd ov2
Verifying LB programming for containers on network ov2
Verifying container /s2.3.ltrdwef0iqf90rqauw3ehcs56...
service s2... OK
service s3... OK
service s1... OK
Verifying container /s3.3.nyhwvdvnocb4wftyhb8dr4fj8...
service s2... OK
service s3... OK
service s1... OK
Verifying container /s1.3.wwx5tuxhnvoz5vrb8ohphby0r...
service s2... OK
service s3... OK
service s1... OK
Verifying LB programming for containers on network ingress
Verifying container Ingress...
service web... OK

control-plane consistency check across nodes in a cluster

Docker networking uses a gossip protocol to synchronize networking state across nodes in a cluster. ssd's gossip-consistency command verifies if the state maintained by all the nodes are consistent.

In a three node cluster with services running on an overlay network ov2 ssd consistency-checker shows 

vagrant@net-1:~/code/go/src/github.com/docker/docker-e2e/tests$ docker run -v /var/run/docker.sock:/var/run/docker.sock -v /var/run/docker/netns:/var/run/docker/netns --privileged sanimej/ssd ov2 gossip-consistency
Node id: sjfp0ca8f43rvnab6v7f21gq0 gossip hash c57d89094dbb574a37930393278dc282

Node id: bg228r3q9095grj4wxkqs80oe gossip hash c57d89094dbb574a37930393278dc282

Node id: 6jylcraipcv2pxdricqe77j5q gossip hash c57d89094dbb574a37930393278dc282

This is hash digest of the control-plane state for the network ov2 from all the cluster nodes. If the values have a mismatch docker network inspect --verbose on the individual nodes can help in identifying what the specific difference is.