Getting familiar with Kubernetes

Introduction

Kubernetes is probably the premiere Orchestration framework for the Cloud.

[For a short overview, the Wikipedia article on Kubernetes is pretty good.]

As of August, 2017 all of the major cloud providers belong to the Cloud Native Computing Foundation (CNCF) which supports Kubernetes. In fact, Kubernetes was the first project to graduate from the CNCF, in March 2018.

The main cloud providers (AWS, Azure, GCP) all support Kubernetes (generally as a premium offering).

[As Google had already used GCE for their compute engine, their container TLA is GKE.]

[For those who like such terms, Kubernetes is a PaaS (Platform as a Service).]

Thoughts on the best cloud provider

There's an old photography joke about what the best camera is: "the one you have with you" (the one at home is no use).

Likewise, the best cloud provider is whichever one you know best.

They all have huge eco-systems so learning each providers set of offerings is definitely a non-trivial exercise.

My read on things is that AWS is the leader and likely to stay that way; Azure is the corporate choice for the MS world; and GCP is a late-comer (in terms of commercial offerings) but cannot be discounted (or ignored). For one thing, the presence of GCP has led everyone to discount their prices (which is probably not a bad thing).

So in the end overall cost is probably not a good criterion as charges are likely to continue to go down.

[UPDATE: For probably much the same reasons as Amazon (their customers were already running it on their hardware), Digital Ocean are getting into the Kubernetes market. DO announced their upcoming Early Access Program im May, 2018 with clusters free through September 2018. DO also upgraded their CNCF membership from Silver to Gold.]

AWS

It does not seem to be that easy to create a Kubernetes cluster with AWS. While there are kops and kube-aws, which seem to be actively supported and full-featured, it is a concern that there is no AWS-native tooling. This situation may improve now that Amazon has joined the Cloud Native Computing Foundation (CNCF) - or it may not. Amazon's intentions are not clear but their Adrian Cockcroft has expressed interest in Containerd and Linkerd (Linkerd is a service mesh, much like Istio). Plus they seem to be experimenting with kops.

Amazon of course have their own ECS (EC2 Container Service) which uses different terminology than Kubernetes (for instance I believe a task approximates to a pod) but largely offers the same sorts of services.

However according to the CNCF the bulk of AWS customers appear to have opted for Kubernetes over ECS:

http://www.cncf.io/blog/2017/06/28/survey-shows-kubernetes-leading-orchestration-platform/

Adrian Cockcroft on Cloud Native Computing and AWS:

http://www.linuxfoundation.org/blog/2018/02/adrian-cockcroft-convergence-cloud-native-computing-aws/

[Apparently 63 percent of Kubernetes workloads run on AWS.]

Adrian Cockcroft on Cloud Native Computing and the CNCF:

For customers who are trying to track a fast-moving and confusing world, it’s helpful to regard CNCF as a brand endorsement for a loose collection of interesting, sometimes competing projects, rather than a single, integrated cloud native architecture. There’s no particular endorsement of any one project over another for members of CNCF, or for users of projects.

From:

http://aws.amazon.com/blogs/opensource/cloud-native-computing/

While the Amazon ECS offering can be expected to continue to evolve, they also seem to be hedging their bets with Kubernetes.

UPDATE: As of November 29, 2017 Amazon announced their new Containers-as-a-Service service Fargate as well as their new managed Kubernetes offering Amazon Elastic Container Service for Kubernetes (for which the TLA is EKS). It's probably too early to say what these services offer over and above what was previously offered, but it does seem to be proof that Amazon Marketing likes the Kubernetes brand. Fargate, at least, looks like a product that will really appeal to the ML (Machine learning) crowd.

Azure and Firefox

If you use firefox as your browser, you will need to add a popup exception for portal.azure.com to allow it to open pop-up windows, otherwise Azure's Cloud Shell window will not open (I have left feedback so perhaps this will be fixed). It does not seem to be possible to paste into the Cloud Shell window either, which is annoying.

Running in the Cloud versus running locally

The cloud providers listed all provide either free credits or free services (presumably for evaluation purposes). The charges for cloud services are likely to continue to go down.

They all require a valid credit card too (presumably for identification/authentication/idemnification purposes).

So tread carefully: some of the allocation settings are not set by default to the free tier offerings.

[Generally the free offerings are limited to the 'micro' or 'small' machine images.]

Premium products (such as Google's Cloud Spanner - which is pretty cool) are generally not free.

Using these will eat into your free credits, so remember to tear everything down when finished.

As with Docker it is possible to run Kubernetes locally, which definitely has some advantages.

For one thing, all of the cloud providers have extensive (and very cluttered) dashboards whereas the command minikube dashboard will pop open a browser populated with a much less cluttered dashboard (making it much easier to see what is going on).

Tools

There are 3 main tools: kubeadm, kubectl, and minikube.

For setting up local clusters or for provisioning VMs, kubeadm is probably useful.

[If you go this route, make sure to use your best machine for the master node, as it is a single point of failure (multi-master clusters may be in the works but are not yet a reality as of Kubernetes 1.8). If the etcd on the master node breaks, not much else will work properly either. You should probably back up this etcd on a regular basis too. Maybe with a Kubernetes cronjob.]

However, for most purposes - including dealing with cloud providers (such as AWS, Azure, GCP, etc) - kubeadm is probably not necessary. It seems to be more of an installation and administration tool.

Having dabbled with all 3 of the listed providers, I can confirm that it is not necessary to install kubectl locally either.

[Each of the cloud providers recommends that you install their CLI (or command-line toolset), which definitely makes a lot of things simpler, however it should be possible to perform all needed functions from a web interface (navigating said dashboard is generally non-trivial however). When operating in the cloud you will use a provided kubectl, so generally no need to have it installed locally.]

For local familiarization, minikube is the way to go - and it requires kubectl.

[minikube is really the local equivalent of a cloud provider's command-line toolset.]

minikube options

There are three options for deploying minikube:

as a VM (virtual machine)
as a container
bare-metal

The third option (bare-metal) is considered risky.

The second option is the recommended one; it can use Docker.

Choosing the first option requires installing some form of virtualization; for linux either VirtualBox or KVM2.

Using either of these probably requires enabling either VT-x or AMD-v hardware virtualization in your BIOS.

[Don't worry, the minikube startup process will tell you if this setting needs to be enabled or not. Hopefully not.]

[In my BIOS this was Advanced -> CPU Configuration -> Intel Virtualization Technology]

[While I previously used the VirtualBox option, the Docker option is now the recommended one - so that is what I use.]

minikube configuration

Which option is chosen above may affect how easy it is to configure minikube's limits.

VirtualBox provides a nice GUI:

If using Docker, the command line must be used:

In the end both get the job done so it boils down to personal preference.

My Projects

These are not in the same order that I went through them, as I progressed I had to backtrack from time to time (when I needed a better grounding on basic concepts) but the order below corresponds to what I think is the difficulty level; in other words the order presented is the order to follow.

These all require kubectl and minikube.