Kubernetes — Deployments Don’t Create Pods

When applying a Deployment in Kubernetes, you may find that no Pods are created but the Deployment doesn’t appear to give any indication of the reason. In this situation, you might even think “this is dumb… if a Deployment can’t create Pods, it should be obvious from the events of the Deployment”.

Indeed, that might be helpful — but this way of thinking suggests that you are missing some context on what a Deployment does. Specifically, a Deployment does not create Pods.

That may sound ludicrous, especially if you have successfully applied a Deployment and seen Pods get created in the past. Nonetheless, a Deployment does not (directly) create Pods. Instead, a Deployment creates ReplicaSets, and ReplicaSets create Pods.

If you invoke kubectl edit pod <podname> on a successfully scheduled Pod associated with a Deployment , you’ll see an ownerReferences stanza in the metadata that confirms that the “parent” of the Pod is a ReplicaSet:

apiVersion: v1
kind: Pod
metadata:
 labels:
    pod-template-hash: 548f955bf
  name: test-548f955bf-j8wng
  ownerReferences:
  - apiVersion: apps/v1
    blockOwnerDeletion: true
    controller: true
    kind: ReplicaSet
    name: test-548f955bf
    ...

So… the reason that you can’t find any information on why Pods aren’t being created is because in looking at the Deployment, you are looking at the wrong thing. You should probably be looking at the ReplicaSet.

For example, consider the case of a Namespace with the following ResourceQuota:

apiVersion: v1
kind: ResourceQuota
metadata:
  name: default-resource-quotas
spec:
  hard:
    requests.cpu: "50m"

If you try to apply a Deployment where the Pod’s requests.cpu resources usage exceeds 50m, Pods will not be created but the Deployment’s events will not contain any errors.

In the case of the following Deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: helloworld-deployment
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: helloworld-deployment
  replicas: 1
  template:
    metadata:
      labels:
        app.kubernetes.io/name: helloworld-deployment
    spec:
      containers:
        - name: hello-world
          image: helloworld-webserver:1.0.0
          resources:
            requests:
              cpu: 100m
              memory: 32Mi
            limits:
              cpu: 500m
              memory: 64Mi

After applying the deployment, there will be no Pods created but a describe of the Deployment will look more or less fine (other than that it does indicate that 0 Replicas are available and 1 is unavailable):

>kubectl -n default get po
No resources found in default namespace.
>kubectl -n default describe deployment helloworld-deployment
Name:                   helloworld-deployment
Namespace:              default
CreationTimestamp:      Wed, 19 Jan 2022 14:49:19 -0500
Labels:                 <none>
Annotations:            deployment.kubernetes.io/revision: 1
Selector:               app.kubernetes.io/name=helloworld-deployment
Replicas:               1 desired | 0 updated | 0 total | 0 available | 1 unavailable
StrategyType:           RollingUpdate
MinReadySeconds:        0
RollingUpdateStrategy:  25% max unavailable, 25% max surge
Pod Template:
  Labels:  app.kubernetes.io/name=helloworld-deployment
  Containers:
   hello-world:
    Image:      helloworld-webserver:1.0.0
    Port:       <none>
    Host Port:  <none>
    Limits:
      cpu:     500m
      memory:  64Mi
    Requests:
      cpu:        100m
      memory:     32Mi
    Environment:  <none>
    Mounts:       <none>
  Volumes:        <none>
Conditions:
  Type             Status  Reason
  ----             ------  ------
  Progressing      True    NewReplicaSetCreated
  Available        False   MinimumReplicasUnavailable
  ReplicaFailure   True    FailedCreate
OldReplicaSets:    <none>
NewReplicaSet:     helloworld-deployment-7ccbf84fdb (0/1 replicas created)
Events:
  Type    Reason             Age   From                   Message
  ----    ------             ----  ----                   -------
  Normal  ScalingReplicaSet  63s   deployment-controller  Scaled up replica set helloworld-deployment-7ccbf84fdb to 1

However, looking carefully at the describe of the Deployment, you can see that a ReplicaSet was created (helloworld-deployment-7ccbf84fdb) and it has 0/1 replicas created.

Describing this ReplicaSet, on the other hand, provides some very useful information:

>kubectl -n default describe rs helloworld-deployment-7ccbf84fdb
Name:           helloworld-deployment-7ccbf84fdb
Namespace:      default
Selector:       app.kubernetes.io/name=helloworld-deployment,pod-template-hash=7ccbf84fdb
Labels:         app.kubernetes.io/name=helloworld-deployment
                pod-template-hash=7ccbf84fdb
Annotations:    deployment.kubernetes.io/desired-replicas: 1
                deployment.kubernetes.io/max-replicas: 2
                deployment.kubernetes.io/revision: 1
Controlled By:  Deployment/helloworld-deployment
Replicas:       0 current / 1 desired
Pods Status:    0 Running / 0 Waiting / 0 Succeeded / 0 Failed
Pod Template:
  Labels:  app.kubernetes.io/name=helloworld-deployment
           pod-template-hash=7ccbf84fdb
  Containers:
   hello-world:
    Image:      helloworld-webserver:1.0.0
    Port:       <none>
    Host Port:  <none>
    Limits:
      cpu:     500m
      memory:  64Mi
    Requests:
      cpu:        100m
      memory:     32Mi
    Environment:  <none>
    Mounts:       <none>
  Volumes:        <none>
Conditions:
  Type             Status  Reason
  ----             ------  ------
  ReplicaFailure   True    FailedCreate
Events:
  Type     Reason        Age                  From                   Message
  ----     ------        ----                 ----                   -------
  Warning  FailedCreate  3m55s                replicaset-controller  Error creating: pods "helloworld-deployment-7ccbf84fdb-vvchx" is forbidden: exceeded quota: default-resource-quotas, requested: requests.cpu=100m, used: requests.cpu=0, limited: requests.cpu=50m
  Warning  FailedCreate  3m55s                replicaset-controller  Error creating: pods "helloworld-deployment-7ccbf84fdb-fc8lx" is forbidden: exceeded quota: default-resource-quotas, requested: requests.cpu=100m, used: requests.cpu=0, limited: requests.cpu=50m

The ReplicaSet events tell us exactly what the problem is. We just need to know where to look…