dineshba/docker-swarm-canary

Canary Deployment in Docker Swarm

Steps:

Deploy the application that needs the canary deployment

docker stack deploy -c docker-compose-app-only.yml myapplication

Run with proper replicas and update_config

deploy:
      replicas: 2
      update_config:
        delay: 60s
        parallelism: 1

Deploy a utils container which can be used for debugging

docker stack deploy -c docker-compose-utils.yml utils

# Exec into the container
$(docker ps | grep utils | awk '{print "docker exec -it " $1  " bash" }')

# continously ping the application
while true; do curl myapplication:8123; sleep 1; date; echo ""; done;

Deploy the infra components that helps the canary.

In our case, we are going to use envoy proxy for the dynamic traffic shifting. We are going to use Envoy-Pilot which listens to the consul and updates the envoy proxies on the fly. So without restart, we can achieve the canary deployment. So this will install a consul (for production we need to run consul in cluster mode) and envoy-pilot

docker stack deploy -c docker-compose-infra.yml infra

Inject envoy-proxy before our application so that we can start controlling the traffic

All the traffic that uses myapplication for dns will flow through the envoy
If you want to send the traffic from interlock to envoy, remove the labels related to interlock from myapplication and add it to envoy. Labels such as

com.docker.lb.hosts: dns-name-to-register-in-interlock
com.docker.lb.port: port

10/20 sec downtime during this step

docker stack deploy -c docker-compose-app-with-envoy.yml myapplication

Start the version 2 of the application in the new docker stack and change the config in consul, so that envoy-proxy can start splitting the traffic between two versions.

docker stack deploy -c docker-compose-app-version-two.yml myapplication_v2

If everything goes well, install version of the application

5/10 sec downtime during this step

docker stack deploy -c docker-compose-app-version-two-only.yml myapplication --prune

Note: --prune will clean the no longer referenced service in yaml file. In our case it will remove consul-import and myapplication_behind_envoy

Clean the extra components which are used for canary

docker stack rm infra
docker stack rm myapplication_v2
docker stack rm utils

Latency test after introducing envoy proxy:

Command used for testing:

while true; do curl -w "%{time_total}\n" myapplication:8123/return/200?delay=1000 -o /dev/null -s; sleep 1; done;

without_proxy:
  1.006084s
  1.006302s
  1.006514s
  1.006616s
  1.006759s
  1.006860s
  1.006884s

with_proxy:
  1.006429s
  1.006636s
  1.006747s
  1.006885s
  1.007227s
  1.008819s
  1.009144s
  1.010470s

so max delay(approx): 0.004 seconds (4ms)

Refer here for better Benchmarking Envoy Proxy, HAProxy, and NGINX Performance results.

dineshba / docker-swarm-canary