Hibernate shrinks and expands your cluster on schedule
This utility currently is on best effort support by @Leon Kuperman and @Augustinas Stirbis through community slack
Install hibernate
Run this command to install Hibernate CronJobs
kubectl apply -f https://raw.githubusercontent.com/castai/hibernate/main/deploy.yaml
Change API key
Create API token with Full Access permissions and encode base64
echo -n "98349587234524jh523452435kj2h4k5h2k34j5h2kj34h5k23h5k2345jhk2" | base64
use this value to update Secret
apiVersion: v1
kind: Secret
metadata:
name: castai-Hibernate
namespace: castai-agent
type: Opaque
data:
API_KEY: >-
CASTAI-API-KEY-REPLACE-ME-WITH-ABOVE==
OR for convenience use one liner
kubectl get secret castai-hibernate -n castai-agent -o json | jq --arg API_KEY "$(echo -n 9834958-CASTAI-API-KEY-REPLACE-ME-5k2345jhk2 | base64)" '.data["API_KEY"]=$API_KEY' | kubectl apply -f -
Set Cloud env variable
AKS is set by default, but requires changing in both CronJobs "Cloud" env variable to [EKS|GKE|AKS]
How it works
Hibernate-pause Job will
- Disable Unscheduled Pod Policy (to prevent growing cluster)
- Prepare Hibernation node (node that will stay hosting essential components)
- Mark essential Deployments with Hibernation toleration (system critical and with NAMESPACES_TO_KEEP env var)
- Delete all other nodes (only hibernation node should stay running)
Hibernate-resume Job will
- Renable Unscheduled Pod Policy to allow cluster to expand to needed size
Override default hibernate-node size
- Set the HIBERNATE_NODE environment variable to override the default node sizing selections. Make sure the size selected is appropriate for your cloud.
Override default NAMESPACES_TO_KEEP
- Set the NAMESPACES_TO_KEEP environment variable to override, "opa,istio""
Override default "PROTECT_EVICTION_DISABLED" and set to "true" to prevent the removal of removal-disabled nodes from being removed during hibernate. This looks for the autoscaling.cast.ai/removal-disabled="true"
label on a node and if it exists excludes it from being cordoned and deleted.
TODO
- Auto detect Cloud
Development
Create [aks|eks|gke] K8s cluster
- create file hack/aks/tf.vars from example
- run "make aks"
- connect to cluster (az/gcloud) / switch kubectl context
Run code locally
- copy cluster_id from console.cast.ai to .env file (example .env.example)
- uncomment in main.py # local_development = True
- run end2end tests