Unable to configure disruption controls for karpenter

[Clay Risser]

I am unable to figure out how to add a disruption consolidationPolicy and expireAfter in my karpenter node pools for kops. Where do I configure this?

The karpenter docs discuss this here.

https://karpenter.sh/v0.32/concepts/nodepools/#specdisruption

I'm not even able to see a CRD for karpenter NodePools, so I'm guessing kops has another way of managing the disruption controls?

  disruption:
    consolidationPolicy: WhenUnderutilized
    expireAfter: 720h # 30 * 24h = 720h

[Moshe Vayner]

From what I can tell right now, kOps installs karpenter version 0.31.3 by default which didn't support the nodePools concept yet, according to what I'm seeing in the docs (I hope I'm not wrong there), ref:

kops/pkg/model/components/karpenter.go

Lines 38 to 40 in d489024

 if c.Image == "" { 
 	c.Image = "public.ecr.aws/karpenter/controller:v0.31.3" 
 } 

.
This brings me to believe that it's not supported in kOps right now, and thus, we might need to put in some effort to add this.

I don't mind taking a stab at this one, wdyt @hakman @rifelpet @olemarkus ?

[Ciprian Hacman]

I don't mind taking a stab at this one, wdyt @hakman @rifelpet @olemarkus ?

My impression is that, if we want to move Karpenter support to a newer version, we would need to move from providing the LaunchTemplates to doing everything via Karpenter objects.

kops/upup/models/cloudup/resources/addons/karpenter.sh/k8s-1.19.yaml.template

Lines 1796 to 1874 in d489024

 {{ range $name, $spec := GetNodeInstanceGroups }} 
 {{ if eq $spec.Manager "Karpenter" }} 
 --- 
 apiVersion: karpenter.k8s.aws/v1alpha1 
 kind: AWSNodeTemplate 
 metadata: 
   name: {{ $name }} 
 spec: 
   subnetSelector: 
     kops.k8s.io/instance-group/{{ $name }}: "*" 
     kubernetes.io/cluster/{{ ClusterName }}: "*" 
   launchTemplate: {{ $name }}.{{ ClusterName }} 
 --- 
 apiVersion: karpenter.sh/v1alpha5 
 kind: Provisioner 
 metadata: 
   name: {{ $name }} 
 spec: 
   consolidation: 
     enabled: true 
 {{ with $spec.Kubelet }} 
 {{ if or .MaxPods .SystemReserved .KubeReserved }} 
   kubeletConfiguration: 
     {{ if .MaxPods }} 
     maxPods: {{ .MaxPods }} 
     {{ end }} 
     {{ if .SystemReserved }} 
     systemReserved: 
       {{ range $key, $val := .SystemReserved}} 
       {{ $key }}: "{{ $val }}" 
       {{ end }} 
     {{ end }} 
     {{ if .KubeReserved }} 
     kubeReserved: 
       {{ range $key, $val := .KubeReserved}} 
       {{ $key }}: "{{ $val }}" 
       {{ end }} 
     {{ end }} 
 {{ end }} 
 {{ end }} 
   requirements: 
     - key: karpenter.sh/capacity-type 
       operator: In 
       values: ["spot", "on-demand"] 
     - key: kubernetes.io/arch 
       operator: In 
       values: ["{{ ArchitectureOfAMI $spec.Image }}"] 
     - key: "node.kubernetes.io/instance-type" 
       operator: In 
       values: 
       {{ range $type := KarpenterInstanceTypes $spec }} 
       - {{ $type }} 
       {{ end }} 
 {{ with $spec.Taints }} 
   taints: 
     {{ range $taintString := $spec.Taints }} 
     {{ $taint := ParseTaint $taintString }} 
   - key: {{ $taint.key }} 
     effect: {{ $taint.effect }} 
     {{ if $taint.value }} 
     value: "{{ $taint.value }}" 
     {{ end }} 
     {{ end }} 
 {{ end }} 
 {{ if $.ExternalCloudControllerManager }} 
   startupTaints: 
     - key: node.cloudprovider.kubernetes.io/uninitialized 
       effect: NoSchedule 
 {{ end }} 
 {{ with $spec.NodeLabels }} 
   labels: 
   {{ range $key, $value := . }} 
     {{ $key }}: "{{ $value }}" 
   {{ end }} 
 {{ end }} 
   providerRef: 
     name: {{ $name }} 
 {{ end }} 
 {{ end }} 

[Moshe Vayner]

My impression is that, if we want to move Karpenter support to a newer version, we would need to move from providing the LaunchTemplates to doing everything via Karpenter objects.

Yeah, that makes sense to me.
So, would that be (theoretically) a somewhat similar process to any other cloudup add-on such as aws-cni, in which we'll update the template (and potentially supporting resources such as template functions etc.) according to the vendor chart?

[Ciprian Hacman]

Yes. The good part is that we have a Karpenter e2e test, so should be easy to test via WIP PR.

[Moshe Vayner]

Sounds good! I'll give that a try. Thanks!

/assign

[teocns]

From my understanding it's unlikely possible but doesn't hurt to ask if there is any workaround for getting upstream Karpenter to manage current kOps's release InstanceGroups?