Please use kraken, the intended user interface to kraken-lib. The following instructions are intended for developers working on kraken-lib.
kraken-lib is an orchestration and cluster-level management system for Kubernetes that creates a production-scale Kubernetes cluster on a range of platforms using default settings. When you're ready to optimize your cluster for your own environment and use case, you can deploy with kraken-lib's rich set of configurable options.
We (Samsung CNCT) built this tool to aid in our own research into high performance and reliability for the Kubernetes control plane. Realizing this would be a useful tool for the public at large, we released it as kraken (now kraken-v1) in mid 2016. This first release was great, but we had developed it quickly and just for research. After using it ourselves for almost a year and identifying some pain points, we deemed it best to build anew, bringing the best parts forward. Thus sprouted kraken-lib our second release.
It continues to use Ansible and Terraform because of the flexible and powerful abstractions these tools provide at the right layers. kraken-lib provides the same functionality as kraken-v1 but with much cleaner internal abstractions. This more easily facilitates external and internal contributions. It also enables us to quickly improve and evolve with the Kubernetes ecosystem as a whole.
kraken-lib is targeted at operations teams who support Kubernetes, a practice becoming known as "ClusterOps." It provides a single interface where ClusterOps teams can manage Kubernetes clusters across all environments.
kraken-lib uses a single file to drive cluster configuration, enabling you to check the file into a VCS of your choice and solving two major problems:
- Use version control for your cluster configuration as you promote changes from dev through production, for either existing cluster configurations or brand-new ones;
- Enable continuous integration for developer applications against sandboxed and transient Kubernetes clusters. kraken-lib provides a destroy command that cleans up all traces of the temporary infrastructure.
We believe solving these two problems is a baseline for effectively and efficiently nurturing a Kubernetes-based infrastructure.
To support our efforts to make kraken-lib a fault-tolerant, reliable tool, we collect data if kraken-lib crashes on up, down or update. If you are running it with the kraken-tools Docker container and the program exits with a failure, the following data will be collected by kraken-lib crash-app
- Logs
- The failing task
This data remains internal for the Samsung-CNCT team to use for data-driven development. We do not collect personal information from users.
kraken-lib also supports a number of Samsung CNCT-supported add-ons in the form of Helm charts. These charts, tested and maintained by Samsung CNCT, can be found in the following repositories:
To deploy a lightweight, configurable logging stack you can install:
This chart deploys:
- Fluent-bit daemonset
- ElasticSearch cluster
- ElasticSearch curator
- Kibana
Additionally, you can deploy some or all of these charts individually with the following charts:
These charts should work on any Kubernetes cluster.
Another Kubernetes add-on that kraken-lib supports is cluster-autoscaling this can be 'enabled' in the following config.yaml
block:
autoScalingConfigs:
- &defaultAutoScaling
minNodes: 3
maxNodes: 10
enabled: true
You will need to have the following:
- A machine that can run Docker
- A text editor
- Amazon credentials with the following privileges:
- Launch EC2 instances
- Create VPCs
- Create ELBs
- Create EBSs
- Create Route 53 records
- Create IAM roles for EC2 instances
You will need the following installed on your machine:
- Python 2.x (virtualenv strongly suggested)
- pip
- boto
- netaddr
- Ansible (see kraken-tools for the version)
- Cloud SDKs
- AWS cli
- gcloud SDK
- Terraform and Providers (see kraken-tools for the versions)
- kubectl
- Helm
For the specific version of Python modules (including Ansible) that are expected, see kraken-tools. For the versions of all other dependecies, see the kraken-tools Dockerfile.
The easiest way to get started with kraken-lib directly is to use a kraken-lib container image:
docker pull quay.io/samsung_cnct/k2:latest
Add/configure the environment variables below; K2OPTS is used to pass Docker the specified Docker volumes (note -v in the K2OPTS variable). Ensure each of these files or directories exist:
KRAKEN=${HOME}/.kraken # This is the default output directory for Kraken
SSH_ROOT=${HOME}/.ssh
AWS_ROOT=${HOME}/.aws
AWS_CONFIG=${AWS_ROOT}/config # Use these files when using the aws provider
AWS_CREDENTIALS=${AWS_ROOT}/credentials
SSH_KEY=${SSH_ROOT}/id_rsa # This is the default rsa key configured
SSH_PUB=${SSH_ROOT}/id_rsa.pub
K2OPTS="-v ${KRAKEN}:${KRAKEN}
-v ${SSH_ROOT}:${SSH_ROOT}
-v ${AWS_ROOT}:${AWS_ROOT}
-e HOME=${HOME}
--rm=true
-it"
If this is your first time using kraken-lib, use the kraken-lib Docker image to generate a 'sensible defaults' configuration:
With the Docker container:
docker run $K2OPTS quay.io/samsung_cnct/k2:latest ./bin/up.sh --generate
With the cloned repo:
./bin/up.sh --generate
This will generate a config.yaml file located here:
${KRAKEN}/config.yaml
In this section, the variable YOURCLUSTER
refers to the name you must assign to your cluster at the bottom of the generated config.yaml
in the deployments section. Once you assign the name, it will look like:
deployment:
clusters:
- name: YOURCLUSTER
Then rename the config.yaml
file to YOURCLUSTER.yaml
. This is best practice.
For the rest of the discussion, we will assume the environmental variable ${CLUSTER}
has been set to the name of your cluster.
It is particularly useful when trying to create and manage multiple clusters, each of which must have unique names.
Follow the instructions above for creating an AWS configuration file but include the following flag:
--provider GKE
While all configuration options are available for a reason, some are more important than others. In addition to the region and subnet selections under provider clauses, some key options include:
clusters[x].providerConfig
clusters[x].nodePools[x].count
kubeConfig[x].version
kubeConfig[x].hyperkubeLocation
helmConfigs[x].charts
For a detailed explanation of all configuration variables, please consult our configuration documentation
To add a human-readable domain name to your Kubernetes API server, uncomment customApiDns
and add the desired domain name to your config.yaml:
deployment:
clusters:
...
customApiDns: YOURDOMAINNAME
Configure the custom domain name to point to your cluster's Kubernetes API server ELB. This can be found in the cluster's admin.kubeconfig file under clusters/cluster/server:
apiVersion: v1
clusters:
- cluster:
certificate-authority-data: ....
server: <Kubernetes API server ELB Address>
name: CLUSTER_NAME
If you already have configured your machine to use AWS, you can skip this step.
To configure the environment with your AWS credentials, run one of the following commands:
Using a Docker container:
docker run $K2OPTS quay.io/samsung_cnct/k2:latest bash -c 'aws configure'
Using the local AWS CLI tool:
aws configure
To bring your cluster up, run:
docker run $K2OPTS quay.io/samsung_cnct/k2:latest ./bin/up.sh --config $HOME/.kraken/${CLUSTER}.yaml
This will take a while and will generate a lot of output.
After creating a cluster, to use the kubectl shipped with kraken-lib, run commands in the following fashion:
docker run $K2OPTS quay.io/samsung_cnct/k2:latest kubectl --kubeconfig $HOME/.kraken/${CLUSTER}/admin.kubeconfig get nodes
With locally installed kubectl:
`kubectl --kubeconfig ~/.kraken/${CLUSTER}/admin.kubeconfig get nodes`
After creating a cluster, to use the Helm shipped with kraken-lib, run:
docker run $K2OPTS -e HELM_HOME=$HOME/.kraken/${CLUSTER}/.helm -e KUBECONFIG=$HOME/.kraken/${CLUSTER}/admin.kubeconfig quay.io/samsung_cnct/k2:latest helm list
With locally installed kubectl:
export KUBECONFIG=~/.kraken/${CLUSTER}/admin.kubeconfig
`helm list --home ~/.kraken/${CLUSTER}/.helm`
After creating a cluster, you will be able to SSH to various cluster nodes:
ssh master-3 -F ~/.kraken/${CLUSTER}/ssh_config
Cluster creating process generates an SSH config file at:
~/.kraken/${CLUSTER}/ssh_config
Host names are based on node pool names from your config file. For example, if you had a config file with a node pool section as below:
nodePools:
- name: etcd
count: 5
...
- name: etcdEvents
count: 5
...
- name: master
count: 3
...
- name: clusterNodes
count: 3
...
- name: specialNodes
count: 2
...
Then, the SSH host names available will be:
- etcd-1 through etcd-5
- etcdEvents-1 through etcdEvents-5
- master-1 through master-3
- clusterNodes-1 through clusterNodes-3
- specialNodes-1 through specialNodes-2
To boot up a cluster per your configuration, execute the following command:
docker run $K2OPTS quay.io/samsung_cnct/k2:latest ./bin/up.sh --config $HOME/.kraken/${CLUSTER}.yaml
Normally kraken-lib will look at your configuration, generate artifacts such as cloud-config files and deploy VMs that will become your cluster. During this time, errors can occur if the configuration file contains unexpected settings. If needed, fix any errors and restart the cluster deployment.
The amount of time for deploying a new cluster varies, but you can expect roughly 5 minutes from starting the command to the cluster becoming available for use.
After kraken-lib has run, you will have a working cluster waiting for workloads. To verify it is functional, run the commands described in this section.
docker run $K2OPTS quay.io/samsung_cnct/k2:latest kubectl --kubeconfig ~/.kraken/${CLUSTER}/admin.kubeconfig get nodes
The result should resemble the following:
NAME STATUS AGE
ip-10-0-113-56.us-west-2.compute.internal Ready,SchedulingDisabled 2m
ip-10-0-164-212.us-west-2.compute.internal Ready 2m
ip-10-0-169-86.us-west-2.compute.internal Ready,SchedulingDisabled 3m
ip-10-0-194-57.us-west-2.compute.internal Ready 2m
ip-10-0-23-199.us-west-2.compute.internal Ready 3m
ip-10-0-36-28.us-west-2.compute.internal Ready,SchedulingDisabled 2m
ip-10-0-58-24.us-west-2.compute.internal Ready 3m
ip-10-0-65-77.us-west-2.compute.internal Ready 2m
docker run $K2OPTS quay.io/samsung_cnct/k2:latest kubectl --kubeconfig ~/.kraken/${CLUSTER}/admin.kubeconfig get deployments --all-namespaces
NAMESPACE NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
default central-fluentd-deployment 3 3 3 3 3m
default kafka-monitor 1 1 1 1 3m
default kibana-logging 3 3 3 3 3m
kube-system heapster-standalone 1 1 1 1 3m
kube-system kube-dns 1 1 1 1 3m
kube-system tiller-deploy 1 1 1 1 3m
Optional step
Kraken-lib comes with a lot of built-in services but you can also deploy additional services.
In the past, we used github repositories for our Helm charts. These are deprecated in favor of Quay's app registry. The app registry allows for tagging the latest versions of charts and flexibly pulling up-to-date chart images for dependencies elsewhere. Additionally the registry can grant login-based access for private Helm charts. Information on Quay app registry
As an example for using Helm to install a new service, try installing the Kubernetes dashboard:
docker run $K2OPTS -e HELM_HOME=$HOME/.kraken/${CLUSTER}/.helm -e KUBECONFIG=$HOME/.kraken/${CLUSTER}/admin.kubeconfig quay.io/samsung_cnct/k2:latest helm registry list quay.io | grep kubernetes-dashboard
quay.io/samsung_cnct/kubernetes-dashboard 0.1.0-0
This indicates the chart to install is samsung_cnct/kubernetes-dashboard
from the quay.io
registry.
Or for the legacy repo (deprecated):
docker run $K2OPTS -e HELM_HOME=$HOME/.kraken/${CLUSTER}/.helm -e KUBECONFIG=$HOME/.kraken/${CLUSTER}/admin.kubeconfig quay.io/samsung_cnct/k2:latest helm search kubernetes-dashboard
NAME VERSION DESCRIPTION
atlas/kubernetes-dashboard 0.1.0 A kubernetes dashboard Helm chart
In this case the chart to install is kubernetes-dashboard
from the atlas
repo.
docker run $K2OPTS -e HELM_HOME=$HOME/.kraken/${CLUSTER}/.helm -e KUBECONFIG=$HOME/.kraken/${CLUSTER}/admin.kubeconfig quay.io/samsung_cnct/k2:latest helm registry install --namespace kube-system samsung_cnct/kubernetes-dashboard
NAME: innocent-olm
LAST DEPLOYED: Thu May 18 22:04:03 2017
NAMESPACE: kube-system
STATUS: DEPLOYED
RESOURCES:
==> v1/Service
NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes-dashboard 10.46.101.182 <pending> 80:31999/TCP 0s
==> extensions/v1beta1/Deployment
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
kubernetes-dashboard 1 1 1 0 0s
The chart has been installed. It will take a moment for AWS ELB DNS to propagate, but you can get the DNS now.
docker run $K2OPTS quay.io/samsung_cnct/k2:latest kubectl --kubeconfig ~/.kraken/${CLUSTER}/admin.kubeconfig describe service kubernetes-dashboard --namespace kube-system
Name: kubernetes-dashboard
Namespace: kube-system
Labels: app=kubernetes-dashboard
Selector: app=kubernetes-dashboard
Type: LoadBalancer
IP: 10.46.101.182
LoadBalancer Ingress: ae7a0bae03c1511e78f8f06148e55c0f-1296896684.us-west-2.elb.amazonaws.com
Port: <unset> 80/TCP
NodePort: <unset> 31999/TCP
Endpoints: 10.129.84.6:9090
Session Affinity: None
Events:
FirstSeen LastSeen Count From SubobjectPath Type Reason Message
--------- -------- ----- ---- ------------- -------- ------ -------
3m 3m 1 {service-controller } Normal CreatingLoadBalancer Creating load balancer
2m 2m 1 {service-controller } Normal CreatedLoadBalancer Created load balancer
After a few minutes, you can view the Kubernetes dashboard. In this example, it is located at here.
For AWS clusters, kraken-lib creates a storage class that is assigned the default
Namespace. GKE clusters come with a GKE-provided storageclass. We are working to add support for storage classes with other providers.
If kraken-lib hangs during deployment, hit CTRL-C to break out of the application and try again. Note that some steps are slow and may give a false indication that the deployment is hung up. In particular, the TASK [/kraken/ansible/roles/kraken.provider/kraken.provider.aws : Run cluster up] ***
step and the wait for a cluster to come up can take some time.
You can use the AWS console to log into the created VMs. There you will see various items, such as:
- EC2 instances that include the
cluster
value in their name - Auto-scaling groups that include the
cluster
value in their name - ELB (for API server) that includes the
cluster
value in its name - VPC that includes the
cluster
value in its name - Route 53 Zone that includes the
clusterDomain
value in its name
Using the EC2 instance list, you #can SSH# into VMs and do further debugging.
You can make some changes to the cluster configuration by first making appropriate changes in the config file, and then running the kraken-lib update command as described below. Please be aware of which changes can be safely made to your cluster.
- cluster name
clusters:
- name: YOURCLUSTER
- etcd settings (beyond machine type)
- Node pools
- Node pool counts and instance types
- Cluster services desired to be run
- Kubernetes version
- Location of the hyperkube container
Below we discuss some differences between clusters hosted on AWS versus clusters hosted on GKE.
On AWS, your nodes will still reflect the version they had upon creation. When you run the update
command, kraken-lib will delete nodes one by one, waiting for updated replacement nodes to come online before deleting the next node. This will ensure no information gets lost and the control plane remains up and running.
You can update all or some of your control plane and cluster nodes (but not etcd nodes, as mentioned above).
On GKE nodes, it is not possible to update the control plane. Cluster node updates are possible. The mechanics of deleting and updating nodes are handled by GKE in this case, not kraken-lib.
You can specify different versions of Kubernetes in each node pool. This may affect the compatibility of your cluster's kraken-lib services (see below). You can also update node pool counts and instance types. The update action has a required --nodepools
or -n
flag followed by a comma-separated list of the names of the node pools you want to update. Please be patient; this process may take a while.
- Step 1: Make appropriate changes to configuration file
- Step 2: Run
docker run $K2OPTS quay.io/samsung_cnct/k2:latest ./bin/update.sh --config $HOME/.Kraken/${CLUSTER}.yaml --nodepools clusterNodes,specialNodes
If you change your configuration file to add or remove a node pool, kraken-lib's update action can handle this as well. Adding a node pool will create a new one with the number and type of nodes specified in the config file. Removing a node pool will irretrievably delete any nodes in that node pool, and anything scheduled on those nodes will be lost. This process is much faster than updating individual nodes.
- Step 1: Make appropriate changes to configuration file
- Step 2: Run
docker run $K2OPTS quay.io/samsung_cnct/k2:latest ./bin/update.sh --config $HOME/.kraken/${CLUSTER}.yaml --addnodepools <nodepools,you,wish,to,add> --rmnodepools <nodepools,you,wish,to,remove>
kraken-lib will use the versions of Helm and kubectl appropriate for the Kubernetes version of each cluster. It does so by determining each cluster's currently set Kubernetes minor version. Because node pools can have different versions from each other, the minor version is set according to the version of the control-plane node pool in AWS clusters. For GKE clusters, kraken-lib uses the Kubernetes version of the last node pool in the node pools list.
New Helm releases may be shipped after new Kubernetes releases resulting in Helm possibly not supporting the latest features of Kubernetes. Kraken-lib will use the latest Helm release.
To destroy a cluster created with kraken-lib, do the following:
docker run $K2OPTS quay.io/samsung_cnct/k2:latest ./bin/down.sh --config $HOME/.kraken/${CLUSTER}.yaml
To create a small, low resource-consuming cluster, alter your configuration to the following:
Role | # | Type |
---|---|---|
etcd nodes | 1 | t2.small |
Master nodes | 1 | m4.large |
Cluster nodes | 1 | c4.large |
deployment:
clusters:
- name: YOURCLUSTER
...
nodePools:
- name: etcd
count: 1
...
- name: master
count: 1
...
- name: clusterNodes
count: 1
...
# - name: specialNodes
# count: 2
Delete or comment out 'Special nodes'.
You can find further information here:
This document is maintained by Patrick Christopher (@coffeepac) at Samsung SDS.