service account token missing in pod

[Bich C. Le]

Pods using in-cluster authentication are failing because /run/secrets/kubernetes.io/serviceaccount/token is missing from their file system. I am using this KIP provider version:

    Image:         elotl/kip:latest
    Image ID:      docker-pullable://elotl/kip@sha256:8182c30c9751290ce319eaa27f82109bd907c5f265095c37d0520b82bd7fa510

on a AWS cluster recently created using KIP's terraform script.

One way to reproduce this problem is to start a simple ubuntu pod that runs an infinite loop:

kubectl run xenial --image=ubuntu:xenial --command -- bash -c 'while true; do echo `date`; sleep 5; done'

Then, this command shows the contents of /run/secrets/kubernetes.io/serviceaccount/:

kubectl exec xenial -- bash -c "ls -l /run/secrets/kubernetes.io/serviceaccount/"

The output shows that the token file is missing:

total 4
-rw-r--r-- 1 root root 1066 Sep 14 04:00 ca.crt

[Bich C. Le]

This command shows that the serviceaccount directory appears correctly mounted. It's just that its contents are missing the token file:

$ kubectl describe pod xenial

Namespace:    default
Priority:     0
Node:         kip-provider-0/10.0.29.198
Start Time:   Mon, 13 Sep 2021 20:59:18 -0700
Labels:       run=xenial
Annotations:  <none>
Status:       Running
IP:           10.0.30.142
IPs:
  IP:  10.0.30.142
Containers:
  xenial:
    Container ID:  
    Image:         ubuntu:xenial
    Image ID:      ubuntu:xenial
    Port:          <none>
    Host Port:     <none>
    Command:
      bash
      -c
      while true; do echo `date`; sleep 5; done
    State:          Running
      Started:      Mon, 01 Jan 0001 00:00:00 +0000
    Ready:          True
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-lb7j8 (ro)