title

author

patat

$ kubectl hacking - my journey with kate

Tobias Schneck, Loodse GmbH

wrap

pandocExtensions

margins

theme

true

patat_extensions

autolink_bare_uris

left	right
5	5

syntaxHighlighting

code

codeBlock

decVal

bold

onDullRed

dullBlack

onDullWhite

dullBlack

onDullWhite

My journey to Kubernetes

Java programmer -> Testautomation -> Docker -> OpenShift -> Kubernetes

                              ..              
                              ..              
                           ........           
                   ...  ..++..++..++..  ...   
                    ..+++..  .++.  ..+++..    
                     .+.++.. .++. ..++.+.     
                    .+.  .++++++++++.  .+.    
                    ++.  ..++....++..  .++    
                  ..++.+++++++..+++++++.++..  
                .....++..  .++++++.  ..++.....
                     .++.  ++.  .++  .++.     
                      ..+..+.    .+..+..      
                         .+.++++++.+.         
                        .+.        .+.        
                        +.          .+

📧 mailto:tobi@loodse.com

🐦 https://twitter.com/toschneck

🐙 https://github.com/toschneck

`kubectl` - what's this about?

Main command line tool for interact with Kubernetes writen in Golang
- Maintained by SIG API Machinery (part of official community)
- Source code: https://github.com/kubernetes/kubernetes/tree/master/pkg/kubectl
- Uses REST HTTPS calls
- Caches results under ~/.kube/cache/
Other clients
- client-go: https://github.com/kubernetes/client-go/
- other main languages https://kubernetes.io/docs/reference/using-api/client-libraries
- Generate client from Swagger specs
- curl or other REST webclient
```
curl --cert userbob.pem --key userBob-key.pem \
   --cacert /path/to/ca.pem \
   https://k8sServer:6443/api/v1/pods
 
# as helper use:
kubectl get pod -v 10
```

basic bash config

Enable bash auto completion (update compatible)

# add to ~/.bashrc
source <(kubectl completion bash)
alias k=kubectl
complete -F __start_kubectl k

beautify your bash prompt -> powerline go
1. Install binary go get -v -u github.com/justjanne/powerline-go
2. ensure powerline-go is available in the path: export PATH=$PATH:$GOPATH/bin
3. config ~/.bashrc -> source ./powerline-go/.bashrc

`kubectl` help?

kubectl [command] --help is very helpful!
kubectl explain [object][.field][...] information about spec fields!
kubectl api-resources shows available objects - with CRDs!

Examples

kubectl get --help
#options for all commands
kubectl options

kubectl explain pod.spec.containers.ports
kubectl explain svc.spec.type

kubectl api-resources --api-group=apps
kubectl api-resources -o wide

kubeconfig

Default configuration file: ~/.kube/config
- endpoints
- SSL keys
- contexts
$KUBECONFIG environment variable or --kubeconfig flag
- useful to manage multiple cluster
- merge multiple config files
```
KUBECONFIG=conf1:conf2 kubectl config view --flatten > merged.conf
```

Configures your current working environment.

Attention: set-context modifies context values! use-context change the current context!

# list and change context
kubectl config get-context
kubectl config use-context CONTEXT_NAME

# set the used default namespace
kubectl config set-context --current --namespace=default

kubeconfig - tooling

Special thx to Ahmet Alp Balkan

Installation

Fastest way to install kubectx, kubens and fzf

cd $HOME/bin
wget https://raw.githubusercontent.com/ahmetb/kubectx/master/kubectx
wget https://raw.githubusercontent.com/ahmetb/kubectx/master/kubens
chmod 755 kubectx kubens
wget https://github.com/junegunn/fzf-bin/releases/download/0.18.0/fzf-0.18.0-linux_amd64.tgz
tar xf fzf-0.18.0-linux_amd64.tgz
# → kubectx, kubens, fzf

kubeconfig - tooling

Usage `kubectx`, `kubens`

fast context switching

# fuzzy search list
kubectx
# direct select e.g. `default` context 
kubectx default
# select last context
kubectx -

fast namespace switching

# fuzzy search list
kubens
# direct select e.g. `kube-system` namespace 
kubens kube-system
# select last namespace
kubens -

kubeconfig - tooling

Use fuzzy search `fzf`

Search a file the fuzzy way

fzf
# with preview
fzf --preview 'cat {}'

Key binding → add ./fzf/.fzf.bash to your ~/.bashrc
```
[ -f ~/.fzf.bash ] && source ~/.fzf.bash
```

Pipe kubectl output, e.g. logs, config

k logs POD_NAME | fzf
k get pod POD_NAME -o yaml | fzf

`kubectl` output parameter

--v=9 Debug verbosity 0-10
-o wide, -o yaml shows more important information about an object
--show-labels and --label-columns=k8s-app structure your output
-l k8s-app=my-app, --field-selector=status.phase=Running select objects
-o json | jq 'expresion' combine JSON and jq to get more details (useful for scripting)
jsonpath=JSONPATH_EXP powerful one line helper to get multiple valuesx of a json output
kubectl describe OBJECT shows information and events

`kubectl` output parameter

Examples

# all runnings pods
k get pod --field-selector=status.phase=Running
 
# node kernel version
k get nodes -o json | jq '.items[].status.nodeInfo.kernelVersion' -r

# all used images
kubectl get pods --all-namespaces \
  -o jsonpath='{range .items[*]}{@.metadata.name}{" "}{@.spec.containers[*].image}{"\n"}{end}'

# Check which nodes are ready
JSONPATH='{range .items[*]}{"\n---\n"}{@.metadata.name}: 
{"\n"}{range @.status.conditions[*]}{@.type}={@.status}; {"\n"}{end}{end}' \
 && kubectl get nodes -o jsonpath="$JSONPATH"

# troubleshoot node state
kubectl describe node NODE_NAME

Quick wins - let `kubectl` help you!

use `run` for resource creation (deprecated)

--image=image Docker image
--env="key=value" environment variable(s)
--port=port exposing port of container
--replicas=replicas count of replicas
--label="myapp=app1" add some label(s)

--restart trigger different kind of object creation:

kubectl run # without flag creates a deployment
kubectl run --restart=Never  # creates a Pod
kubectl run --restart=OnFailure # creates a job
kubectl run --restart=OnFailure -schedule="* * * * *" # creates a cronjob

run ... -- argument pass the arguments directly to the container

# start a simple web image and test it with bussy box
kubectl run --image=loodse/demo-www --port 80 web-deployment
kubectl run --image=busybox --restart=Never --rm -it -- bash
# ... inside the conainer: wget $WEB_DEPLOYMENT_SERVICE_HOST -O -

Quick wins - let `kubectl` help you!

new `create` for resource creation

kubectl create 
clusterrole          deployment           priorityclass        secret
clusterrolebinding   job                  quota                service
configmap            namespace            role                 serviceaccount
cronjob              poddisruptionbudget  rolebinding

--image=image Docker image
... less options see kubectl create OBJECT --help

Example

kubectl create deployment web-deployment --image=loodse/demo-www

Quick wins - let `kubectl` help you!

use `expose` for service creation

Can reference pod (po), service (svc), replicationcontroller (rc), deployment (deploy), replicaset (rs).

--port listing port to match at referenced resource
--type type of Service: ClusterIP (default), NodePort, LoadBalancer, ExternalName
--traget-port port at the service
--selector specify label selector

k expose deployment web-deployment --type=NodePort --port=80
k expose deployment web-deployment --type=LoadBalancer --port=80

k get nodes -o wide
k get nodes --selector=kubernetes.io/role!=master \
  -o jsonpath={.items[0].status.addresses[?\(@.type==\"ExternalIP\"\)].address}

Combine with port-forward for quick testing or debugging
- can target pod, deployment, service
- use localport:remoteport for port mapping

k port-forward svc/web-deployment 8080:80 &
curl localhost:8080

Quick wins - let `kubectl` help you!

create templates

--dry-run combined with -o yaml and run --restart or create creates a template for common resource

# create a deployment yaml file
kubectl run --image=loodse/demo-www --port 80 --dry-run -o yaml web-template > dep.yaml
kubectl create deployment web-template --image=loodse/demo-www --dry-run -o yaml > dep.yaml

# job with 10 sleep 
kubectl run --image=busybox --restart=OnFailure --dry-run -o yaml job -- /bin/sleep 10 > job.yaml

--export get a pod's YAML without cluster specific information

#deployment
k get deployment web-deployment -o yaml --export > dep.export.yaml
vim dep.export.yaml 
k apply -f dep.export.yaml   

#service
k get service web-deployment --export -o yaml > svc.export.yaml

Quick wins - let `kubectl` help you!

Modify resources

Use inplace editor functionality
- KUBE_EDITOR sets the local editor
- kubectl edit TYP OBJECT open in cluster resource
Use apply for mutable objects, replace for immutable objects. Note: You can use -f FOLDER for using multi manifests!
```
kubectl apply -f dep.yaml
# delete resource and recreates it
kubectl replace --force -f pod.yaml
```
Use scaling functions
- k autoscale deployment foo --min=2 --max=10 add HPA
- k scale deployment --replicas=10 web-deployment scales up

Quick wins - let `kubectl` help you!

Modify resources

Manipulate current objects, e.g. the image value

# use set for common modification
k set image deployment/web-deployment web-deployment=loodse/demo-www
k set env deployment/web-deployment TEST=val

# use patch for all other, e.g. service type
kubectl patch svc/web-deployment -p '{"spec":{"type":"LoadBalancer"}}' 

# Update a container's image; spec.containers[*].name is required because it's a merge key
kubectl patch pod/podname -p \
 '{"spec":{"containers":[{"name":"web-deployment","image":"loodse/demo-www"}]}}'

Basic cluster information

What cluster do I use?
```
kubectl cluster-info
```

Whats about the components?

kubectl get componentstatuses
kubectl get cs

Troubleshoot the whole cluster

# download the state
kubectl cluster-info dump --output-directory=./output/cluster-state

# diagnose it
tree ./output/cluster-state

grep -r Error output/cluster-state
grep -C 5 -r Error output/cluster-state

Extend `kubectl` with plugins

Enable kubectl plugin manager krew

# add to ~/.bashrc
#
# export KREW_ROOT=/path/to/krew-folder
export PATH="${KREW_ROOT:-$HOME/.krew}/bin:$PATH"

Plugin management

kubectl-krew search
kubectl-krew insatll view-secret

Example: decode base64 secrets

kubectl get secret
kubectl view-secret default-token-976rc namespace

Use `kubectl` the fuzzy way with fubectl

bash/zsh wrapper based on kubectl, jq and fzf
Interactive search and interaction with Kubernetes objects
- support's also CRD's
- No in-cluster installation needed

Installation

curl -LO https://rawgit.com/kubermatic/fubectl/master/fubectl.source
# add to `~/.bashrc`
[ -f <path-to>/fubectl.source ] && source <path-to>/fubectl.source

Usage of fubectl:

khelp
# [ka] get all pods in namespace
# [kall] get all pods in cluster
# [kwa] watch all pods in the current namespace
# [kwall] watch all pods in cluster
# [kp] open kubernetes dashboard with proxy
# [kwatch] watch resource
# [kdebug] start debugging in cluster
# [kube_ctx_name] get the current context
# [kube_ctx_namespace] get current namespace
# [kget] get a resource by its YAML
# [ked] edit a resource by its YAML
# [kdes] describe resource
# [kdel] delete resource
# [klog] fetch log from container
# [kex] execute command in container
# [kfor] port-forward a container port to your local machine
# [ksearch] search for string in resources
# [kcl] context list
# [kcs] context set
# [kcns] context set default namespace
# [kwns] watch pods in a namespace

Troubleshooting

Take a look for objects in state Pending, Error, CrashLoopBackOff
Use port-forward to test different connections, e.g. service or pod

Use prepared debug container for e.g. network debugging

kubectl run --image=amouat/network-utils --restart=Never --rm -it -- bash

top for resource usage, requires metrics-server
```
kubectl top node
kubectl top pod   
```
Reproduce the event and stream all matching logs, e.g. with label name=myLabel
- kubectl logs -f -l name=myLabel --all-containers
exec into running container
- kubectl exec my-pod -- ls -la /
- kubectl exec my-pod -it -- sh

Cluster Inspection Tools

++ need no running components in the cluster

`k9s`

provides a curses based terminal UI
interactive view similar to htop

`popeye`

Kubernetes Cluster Sanitizer
Find errors and warnings

Cluster Management by Cluster API

Manage Cluster's by CRDs in depedent of the provider (cloud/on-prem)
Currently mostly used machine creation, see as e.g. machine-controller implementations
- Used by e.g. HA cluster management tool kubeOne

Immutable machine objects handle cluster nodes similar to pods

Deployment -> ReplicaSet -> Pod -> Container
MachineDeployment -> MachineSet -> Machine -> Node

# see the machine definition
k describe machine -n kube-system MACHINE_NAME
k get machinedeployment,machineset,machine,node -n kube-system

# update e.g. kubernetes version, machine size, ...
k edit machinedeployment

# machine to node reference:
k get machine -n kube-system \
  -o jsonpath='{range .items[*]}{@.metadata.name}{" >> "}{@.status.nodeRef.name}{"\n"}{end}}'

Manage VMs with kubevirt

New open source project to manage virtual Machines
Approach to manage VMs inside of kubernetes
Example: https://github.com/kubevirt/demo/blob/master/manifests/vm.yaml

Questions?

I'm happy to answer!

Slides?

Take a look at https://github.com/loodse/kubectl-hacking

Something to add?

Open a pull request 😉

Thx for your attention!

References:

My journey to Kubernetes

kubectl - what's this about?

basic bash config

kubectl help?

Examples

kubeconfig

kubeconfig - tooling

Installation

kubeconfig - tooling

Usage kubectx, kubens

kubeconfig - tooling

Use fuzzy search fzf

kubectl output parameter

kubectl output parameter

Examples

Quick wins - let kubectl help you!

use run for resource creation (deprecated)

Quick wins - let kubectl help you!

new create for resource creation

Quick wins - let kubectl help you!

use expose for service creation

Quick wins - let kubectl help you!

create templates

Quick wins - let kubectl help you!

Modify resources

Quick wins - let kubectl help you!

Modify resources

Basic cluster information

Extend kubectl with plugins

Use kubectl the fuzzy way with fubectl

Troubleshooting

Cluster Inspection Tools

k9s

popeye

Cluster Management by Cluster API

Manage VMs with kubevirt

Questions?

Slides?

Something to add?

About

Languages

`kubectl` - what's this about?

`kubectl` help?

Usage `kubectx`, `kubens`

Use fuzzy search `fzf`

`kubectl` output parameter

`kubectl` output parameter

Quick wins - let `kubectl` help you!

use `run` for resource creation (deprecated)

Quick wins - let `kubectl` help you!

new `create` for resource creation

Quick wins - let `kubectl` help you!

use `expose` for service creation

Quick wins - let `kubectl` help you!

Quick wins - let `kubectl` help you!

Quick wins - let `kubectl` help you!

Extend `kubectl` with plugins

Use `kubectl` the fuzzy way with fubectl

`k9s`

`popeye`