kind
, like docker:dind
but aims to make a full Kubernetes cluster available to test against in your CI pipeline. It is fast to startup and totally ephemeral, so you get a clean start for each CI run.
Use prebuilt images from Dockerhub: https://hub.docker.com/r/bsycorp/kind/
Run:
docker run -it --privileged -p 8443:8443 -p 10080:10080 bsycorp/kind:latest-1.12
Or more likely run CI in, see examples
We run workloads in Kubernetes, and run our CI on Kubernetes too, but in our CI pipeline we wanted a way to have a quick and reliable way of having the current code under test, built, deployed and tested but then have the environment be torn down after testing. We initially considered deploying to a proper cluster but the overhead of pushing images to a registry just for a CI test, and the setup / teardown of resources was too much, we wanted something totally ephemeral so building on top of docker's dind made sense.
Sort of, but not optimised for a fast-starting single-node cluster. There are examples of other solutions to this problem, it seems mostly from the Kubernetes devs themselves, as a way to automate development of integrations or Kubernetes itself. We wanted something that was both ephemeral, reliable and really fast, none of the available options gave us all of that.
Other projects:
- https://github.com/kubernetes/minikube/tree/master/deploy/docker (Deprecated)
- https://github.com/kubernetes/test-infra/tree/master/dind (Replaces no.1)
- https://github.com/kubernetes-sigs/kubeadm-dind-cluster/ (more info: http://callistaenterprise.se/blogg/teknik/2017/12/20/kubernetes-on-docker-in-docker/)
Excluding no.1 which uses localkube
(which is deprecated), both no.2 and no.3 create a proper cluster, with separate instances for a master and node(s). Because they start a multi node cluster they need to use dind
inside dind
, creating docker in docker in docker. As you can imagine adding another levels of abstraction can change how things behave, so disk access can be even slower than normal as we are two levels deep. Depending on the setup this can be caused because a fast storage driver like overlay2 can't be used on overlay2, so docker will fall back to slower / more compatiable choices like AUFS (depends on Docker version). It can also have an impact on reliability, if you are hitting docker+kernel edge cases that trigger strange behaviour, it is likely you will see more strange behaviour with docker in docker in docker.
As an applcation sitting on top of kubernetes, we don't really need a full multi-node cluster to run our CI tests, we just need a single node+master. Since we only need a single instance, we don't need the complexity of docker-in-docker-in-docker, docker-in-docker is just fine (as we only have 1 instance vs 2+). This should make disk access faster (only 1 dind
, so can use overlay2
) which should help make our kube deploy and tests fast and it should be easier and more reliable to prime/bootstrap everything during build time and just start it up at runtime.
tl;dr; Building on docker-in-docker it uses minikube
and kubeadm
to bootstrap and pre-configure a cluster at build time that works at runtime.
The simplest way to get a Kubernetes cluster running in CI is to use minikube
and start with --vm-driver none
, this uses kubeadm
to bootstrap a set of local processes to start Kubernetes. This doesn't work out of the box in dind
as kubeadm
assumes it is running in a systemd environment, which alpine is not. It also downloads binaries and bootstraps the cluster everytime it is run which depending on your network and resources takes around 4 minutes.
To make this process fast, kind
aims to move all the cluster bootstrapping to the container build phase, so when you run kind
it is already bootstrapped and is effectively just starting the kubelet
process with a preconfigured etcd
, apiserver
etc. To achieve this we need to initially configure kubeadm
with a static IP that will be routable both during the build phase, and the run phase, we have arbitrarily chosen 172.30.99.1
for that address.
During the build phase kind
adds 172.30.99.1
to the default network interface for the container eth0
and forces kubeadm
and friends to use this address when bootstrapping the cluster. During the container run phase (when you are running kind
in your environment) this static IP address is again attached to the default network interface eth0
for the container so when kubelet
is run the IP that it has been configured against is still routable.
Doing this network trickery means we can move all the hard work into the build phase, and kind
can startup fast. In our CI environment using kind
a single node cluster comes up and is ready to use in 30 seconds, down from 4 minutes in the simple minikube implementation (3+ minutes is a lot in a CI pipeline!).
A further optimisation is to have the build phase docker pull
any dependent images your Kubernetes resources will require, so when your CI process is deploying your Kubernetes resources it doesn't have to pull in any images over the network. To do this you will need to build your own version of kind
and just overwrite the /images.sh
file with the images your want to pull in.
kind
is a docker image that only runs as --privileged
, that is designed to be run as a CI service, where by it is accessible over a known interface, normally localhost
. Much like docker:dind
on which it is based.
Running it as a service means the container running your tests needs to know when kind
is ready, and how to get the kubectl
config to make cli calls. This is achieved via the config endpoint. By default this is exposed over port 10080
and is just a simple http server hosting files.
There are few important events that kind
exposes:
- When the docker host is ready,
http://localhost:10080/docker-ready
will return 200 - When the Kubernetes is ready,
http://localhost:10080/kubernetes-ready
will return 200 - When kubectl config is ready,
http://localhost:10080/config
will return 200
As you want your docker images you want to test to be built into kind
docker host, the general process is to wait for .../docker-ready
before doing docker build..
, then wait for .../kubernetes-ready
before deploy, then test.
Not sure as we are running this in an on-premise GitLab install, but interested to hear feedback from people where it does or doesn't work. As above it is designed to be like docker:dind
but with Kubernetes, so in theory anywhere docker:dind
runs this should run, and like docker:dind
it requires the container be launched as --privileged
which generally cloud providers don't like.
Work:
Should work:
- GitLab.com with BYO Docker or Kubenetes runners
- Codeship Pro
Unlikely to work:
- Bitbucket Pipelines, has a magic
docker: true
flag so will likely not work
Pre-built images are available on dockerhub (https://hub.docker.com/r/bsycorp/kind/), but if you want to bake in your own images to make it as fast as possible, you will want to built it yourself.
Run ./build.sh <image name>
to build the image. Add your custom images to /images.sh
to have them be available at runtime. These environment variables are available to configure the build:
- DOCKER_IMAGE: defaults to
stable-dind
- MINIKUBE_VERSION: defaults to
v0.28.0
- KUBERNETES_VERSION: defaults to
v1.10.5
- STATIC_IP: defaults to
172.30.99.1
We use git submodules to pull in this project and then add images and CI configuration around it, but there are other ways to do it.
There are two hooks available during the kind
build, before-cluster.sh
and after-cluster.sh
. As their names suggest they are run directly before and after the kube cluster is created.
Examples of this scripts exist in the repo already, but they can be overwritten / extended to add extra functionality.
To use generated kube config:
wget http://localhost:10080/config
cp config ~/.kube/config
kubectl get nodes
To create/add custom local kube config with creds:
wget http://localhost:10080/ca.crt
wget http://localhost:10080/client.crt
wget http://localhost:10080/client.key
kubectl config set-cluster kind-cluster --server=https://localhost:8443 \
--certificate-authority=ca.crt
kubectl config set-credentials kind-admin \
--certificate-authority=ca.pem \
--client-key=client.key \
--client-certificate=client.crt
kubectl config set-context kind --cluster=kind-cluster --user=kind-admin
kubectl config use-context kind
kubectl get nodes
Depending how you authenticate to your registry this can be a bit tricky. As the ./build.sh
script actually starts a docker container and configures it, because of this the container running your build hook scripts won't have access to all the environment variables / binaries it normally would. To work around this we just write out the build hook script in our outer CI process dynamically during the build.
Your kind
CI build script might look like:
echo "$(aws ecr get-login --no-include-email --region some-region)" >> ./kind/before-cluster.sh
echo "docker pull 12345.ecr.amazonaws.com/smth:latest" >> ./kind/before-cluster.sh
or
echo "docker login -u username -p $REGISTRY_TOKEN_FROM_CI registry.smth.com" >> ./kind/before-cluster.sh
echo "docker pull registry.smth.com/app:latest" >> ./kind/before-cluster.sh
Then the before-cluster.sh
hook will fire during the build and have all the details it needs to login and pull the private images.
or dynamically supply the credentials when running the image:
docker run --privileged \
-e REGISTRY="registry.smth.com" \
-e REGISTRY_USER="username" \
-e REGISTRY_PASSWORD='$REGISTRY_TOKEN' \
bsycorp/kind:latest-1.10