Kubernetes on Windows - Notes

NOTE: My Goal is to get Windows working with AWS EKS or at least a cluster running in AWS. Some items defined in configuration will be targeted at AWS specifically.

Versions

Current supported versions:

Windows Server 2016 - 1803

Targeted for General Availability:

Window Server 2019 - 1809?

People that are working on the #sig-windows

Shared Works for Windows

Troubleshooting

Microsoft Docs

Kubernetes Notes and helpful links

Docker Notes

Cleanup of Dead Docker Container

Implementations of k8s for windows

Microsoft SDN

Microsoft/SDN

This is the core repo that MS uses for publishing stuff for CNI.

When working to setup my first k8s cluster I read over this repo and it felt a bit old and none functional. I tried using their flannel.exe and it did not work in my tests. Overall not sure where to go without some more commits from them. The kublet.exe stuff works correctly.

Ptylenda

k8s-for-windows

I have implemented this from the ground up. It seems to work alright but I struggle getting flannel.exe working correctly. Nodes register and function correctly but they restart due to pod-sandbox needing to be reconfigured continually. This seems to even happen with SDN/Rancher if the flannel conf is defined wrong.

Rancher

This seems to work alright but flannel again did not work correctly tracking down the files seemed to not work very well but at least it was fruitful and gave some new files.

WARNING: These IP Addresses where just to get valid configs out they are not accurate.

.\hyperkube.ps1 `
  -KubeClusterCIDR 10.244.0.0/16 `
  -KubeClusterDomain cluster.local `
  -KubeServiceCIDR 127.0.0.0 `
  -KubeDnsServiceIP 172.20.0.10 `
  -KubeCNIComponent flannel `
  -KubeCNIMode win-bridge `
  -KubeletCloudProviderName aws `
  -KubeletOptions $(@"
--v=4;
--pod-infra-container-image=kubeletwin/pause;
--allow-privileged=true;
--cloud-provider=aws;
--cluster-dns=172.20.0.10;
--cluster-domain=cluster.local;
--register-node=true;
--anonymous-auth=false;
--kubeconfig='C:\etc\kubernetes\kubelet.conf';
--pod-manifest-path='C:\etc\kubernetes\manifests';
--authentication-token-webhook;
--authorization-mode=Webhook;
--client-ca-file='C:\etc\kubernetes\pki\ca.crt';
--image-pull-progress-deadline=20m;
--resolv-conf='';
--enable-debugging-handlers;
--feature-gates=RotateKubeletServerCertificate=true;
"@ -replace "`t|`n|`r","") `
  -NodeIP 10.14.34.71 `
  -NodeName "ip-10-14-34-71.us-west-2.compute.internal" `
  -KubeproxyOptions $(@"
--v=4;
--proxy-mode=userspace;
--kubeconfig='C:\etc\kubernetes\kubelet.conf'
"@ -replace "`t|`n|`r","")

net-cni.conf

-KubeClusterCIDR 10.244.0.0/16 `
-KubeClusterDomain cluster.local `
-KubeServiceCIDR 127.0.0.0 `
-KubeDnsServiceIP 172.20.0.10 `
-NodeIP 10.14.34.71 `
# -NetworkRange 10.14.34.0/25 # Note this value is not defined and auto computed by hyperkube.ps1

flannel conf

{  
   "capabilities":{  
      "dns":true
   },
   "delegate":{  
      "dns":{  
         "search":[  
            "svc.cluster.local"
         ],
         "nameservers":[  
            "172.20.0.10"
         ]
      },
      "policies":[  
         {  
            "value":{  
               "ExceptionList":[  
                  "10.244.0.0/16",
                  "127.0.0.0",
                  "10.14.34.0/25"
               ],
               "Type":"OutBoundNAT"
            },
            "name":"EndpointPolicy"
         },
         {  
            "value":{  
               "DestinationPrefix":"127.0.0.0",
               "NeedEncap":true,
               "Type":"ROUTE"
            },
            "name":"EndpointPolicy"
         },
         {  
            "value":{  
               "DestinationPrefix":"10.14.34.71/32",
               "NeedEncap":true,
               "Type":"ROUTE"
            },
            "name":"EndpointPolicy"
         }
      ],
      "type":"win-l2bridge"
   },
   "name":"cbr0",
   "type":"flannel",
   "cniVersion":"0.2.0"
}

Compiled files - thxCode

flannel.exe, win-cni.exe etc.

Add dns capabilities for Windows CNI plugins - Merged [67435]

flanneld.exe

Windows "host-gw" & "vxlan" support - Merged [1042]

Currently releases have not been cut with the merged code. Until then it sounds like custom builds of master might work.

About

Notes for running Windows with Kubernetes