eksler

stuff with eksctl, helm etc.

bin/eksler create eu-west-1 test-1 ipv6 1.22 1

issues

[EKS] [request]: Managed Nodes scale to 0 - aws/containers-roadmap#724
[EKS] [request]: create managed nodegroups without scaling one instance when minimum is 0 - aws/containers-roadmap#1684
autoscaler uses nodegroups which are not healthy - kubernetes/autoscaler#4751
memory usage increases heavily with pods - kubernetes/autoscaler#4746

scale-down-delay-after-add: 5m # helmer: NOTE: nodegroup is being created it is being registered etc in nodegroup, so do not scaledown immediately

related issues

predicate checking error: node(s) didn't match Pod's node affinity; predicateName=NodeAffinity; reasons: node(s) didn't match Pod's node affinity; debugInfo= - kubernetes/autoscaler#4052 kubernetes/autoscaler#3802 Cluster Autoscaler does not start new nodes when Taints and NodeSelector are used in EKS -

other issues

[EKS/Fargate] Support for Wildcard * Namespaces aws/containers-roadmap#621

kubernetes/autoscaler#3802

kubectl delete pod -n prometheus-1 -l app=prometheus,component=server

features

eksler create eu-north-1 ipv6|ipv4 - create ipv6 or ipv4 clusters without yaml
eksler list- lists all clusters in all regions
eksler delete - ensures cluster deletion even when cloudformation fails
eksler with eu-north-1 mycluster list - lists all nodegroups
eksler with eu-north-1 mycluster create
- --ami pin down AMI version without yaml
- --volumeIOPS and --volumeThroughput without yaml
eksler with eu-north-1 mycluster kubectl get pod
export KUBECONFIG=$(eksler with eu-north-1 mycluster kubeconfig:path)
https://github.com/aws/amazon-ec2-instance-selector without yaml
sets root password to badpassword for ec2 serial console access when SSM or SSH etc is not accessible
installs htop iotop nano screen bind-utils to the host for faster debugging
great & tested configs for cluster-autoscaler, ingress-nginx, aws-load-balancer-controller and prometheus
output shows elapsed seconds
runs commands until they succeed
parallel creation / deletion of nodegroups

upcoming featurs

pin down vpc-cni, coredns and kube-proxy versions instead of latest

bin/eksler delete $REGION $CLUSTER bin/eksler with $REGION $CLUSTER pool 1 all-1-pre create

POOLS="bot-1-pre bot-2-pre bot-3-pre" POOL_REFRESH=no bin/main eu-north-1 eka-1 ipv6 1

TODO

does topologySpreadConstraints work with skew 1
cross-zone load balancing
https://github.com/grpc-ecosystem/grpc-health-probe/releases/download/v0.4.7/grpc_health_probe-linux-amd64
call ec2-instance-selector to fetch a list and use it --> deprecate eksctl instance selector
if only 2 zones are available during cluster creation time, those will be the zones (?)
metrics server has emptydir which prevents ca remove --> "fixed" by ignoring emptydirs in cluster-autoscaler

notes

6.8gb used disk after bot

aws issue envs

POOLS_REFRESH=yes POOLS="x-1-pre t-64-128-pre-p4-1" bin/test-main eu-north-1 sixtyfour ipv6 1 1.21 POOLS_REFRESH=yes POOLS="x-1-pre t-96-192-pre-p3-1" bin/test-main eu-north-1 ninetysix ipv6 1 1.21 POOLS_REFRESH=yes bin/test-main eu-north-1 sixtyfour-ninetysix ipv6 1 1.21

test

bin/test-cluster eu-north-1 test-1 ipv6 1.22
bin/test-nodes   eu-north-1 test-1
bin/test-test    eu-north-1 test-1 ipv6

matti / eksler