Paul Dally
2 min readJun 8, 2023

--

Thank you so much for your feedback! I really appreciate it, and am very glad that you find the articles useful.

With large volumes of evicted Pods, the impacts can be:

  1. Usability — a large number of evicted Pods can make it more difficult to find the Pods that aren’t evicted (if you have 99 evicted Pods cluttering up the results of kubectl get po, finding the 1 non-evicted Pod is a bit more of a challenge. This can also impact related systems — for example systems displaying metrics for current Pods on a worker node may have significantly more Pod names in the legend of graphs presenting those usage metrics.
  2. Performance — automation using Kubernetes APIs against Pods may perform less efficiently simply because of the volume of Pods. I’ve seen scenarios, albeit not commonly, where developers generated 1000s of evicted Pods in each of multiple namespaces. Your etcd will have metadata related to all evicted Pods, which is not adding much value and depending on the value of your terminated-pod-gc-threshold setting, may impact its performance.
  3. Stability — the more keys stored in your etcd, the more resources that etcd will require to run properly. These resource requirements aren’t very high compared to many other types of applications or software, but you may nonetheless find that with very large numbers of evicted Pods that the resources that you have given to etcd are no longer sufficient. If etcd isn’t healthy, your cluster won’t be healthy. In certain cluster types, the IP addresses from evicted Pods are not reused until the evicted Pod is cleaned up, and this may lead to IP address starvation and the inability to start new Pods.

--

--

Paul Dally
Paul Dally

Written by Paul Dally

AVP, IT Foundation Platforms Architecture at Sun Life Financial. Views & opinions expressed are my own, not necessarily those of Sun Life

No responses yet