Are your Kubernetes Pods running where you want them to?

Globe: https://www.publicdomainpictures.net/en/view-image.php?image=127827&picture=america-globe public domain, Pod icon: https://github.com/kubernetes/community/tree/master/icons Apache 2.0

Sometimes you should care about where your Pods are running.

Your cluster may have worker nodes in different availabity zones (or perhaps even different regions) and be using a topologySpreadConstraint to provide high-availability for your application. You might be using a podAntiAffinity to prevent multiple Pods from being scheduled on the same worker node. You may be using nodeAffinity or a nodeSelector so that your Pods execute only on particular worker nodes that are appropriately licensed or that provide certain hardware capabilities.

You may not have thought about this before — but often, you really should. There are lots of potential scenarios that could make sense.

To see where Kubernetes has scheduled your Pods, add “-o=wide” to your get pod command. For example:

You can then describe the node(s) to make sure that they have the labels that you expected they should:

$ kubectl describe node worker1.example.com
Name:               worker1.example.com
Roles:              worker
Labels:             beta.kubernetes.io/arch=amd64
                    beta.kubernetes.io/os=linux
                    kubernetes.io/arch=amd64
                    kubernetes.io/hostname=worker1.example.com
                    kubernetes.io/os=linux
                    topology.kubernetes.io/region=ca-central-1
                    topology.kubernetes.io/zone=ca-central-1a

If your Pods aren’t being scheduled where you want, you might have one of the following common problems:

• do you actually have any affinity, nodeSelectors or topologySpreadConstraints? Kubernetes can’t read your mind.
• do your worker nodes have the labels that you think they do?
• are your node labels accurate? If a node is labelled with topology.kubernetes.io/zone: ca-central-1a but the node is actually in ca-central-1b, that would be a problem…
• do you have mutually-exclusive nodeSelectors?
• do your labelSelectors for Pods in affinity clauses or topologySpreadConstraints actually match labels on your Pods?
• Are you using preferredDuringSchedulingIgnoredDuringExecution when you intended to be using requiredDuringSchedulingIgnoredDuringExecution?