Improve the jitter value for kubelet's serviceAccountToken projected volume source
ahmedtd opened this issue · comments
The current jittering logic in kubelet token_manager refreshes the token once it has 20% of its lifetime remaining, with 10 seconds of jitter.
Experimentally, this doesn't seem to be enough to really blunt spikes of refresh requests that come from many workloads having a synced refresh. This can occur when:
- A large deployment is scaled up. All the new replicas will have synced-up refresh traffic.
- A kubelet restarts. Because kubelet only holds the tokens in an in-memory cache, it will need to acquire new tokens for all pods on the node after it restarts. After that, all of the pods will have synced-up refresh traffic.
As an example, here's serviceaccount/token API traffic from a real cluster. Note how there is a repeating spike every 48 minutes, which is what we would expect from the "refresh after 80%" logic, and the default setting of token lifetime for 1 hour. Effective jittering logic would be noticeably smoothing these peaks.
We should increase the maxJitter value. Perhaps 5 minutes would be a good value.
/sig auth
/kind bug
/triage accepted
Need to change the jitter value to take in the token expiration into consideration. It can't be a fixed 5 mins because user can request a token with 10mins expiry.