Kubernetes – You Should ALWAYS Specify A RollingUpdate Strategy On Your Deployments!

Are you finding that when you apply updates to your Deployment, nothing happens? It seems that Kubernetes is refusing to do what you are telling it, and that can be quite frustrating.

If you were to dig deeper, you might realize that while new Pods are not created and your old Pods continue running — a ReplicaSet is created, but that new ReplicaSet is not creating any Pods. The issue may be your Deployment’s rolloutStrategy interacting with your Namespace’s ResourceQuota.

Consider the following Deployment (if you do not explicitly specify a strategy for a Deployment, the default is the strategy shown below):

apiVersion: apps/v1
kind: Deployment
metadata:
  name: helloworld-deployment
spec:
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
  selector:
    matchLabels:
      app.kubernetes.io/name: helloworld-deployment
  replicas: 2
  template:
    metadata:
      labels:
        app.kubernetes.io/name: helloworld-deployment
    spec:
      containers:
        - name: hello-world
          image: helloworld-webserver:1.0.0
          resources:
            requests:
              cpu: 25m
              memory: 32Mi
            limits:
              cpu: 50m
              memory: 64Mi

And the following ResourceQuota:

apiVersion: v1
kind: ResourceQuota
metadata:
  name: default-resource-quotas
spec:
  hard:
    requests.cpu: "50m"

When you first apply the Deployment, Kubernetes creates the requested 2 Pods successfully.

>kubectl -n local-demo-rolloutstrategy-ns get pod
NAME                                     READY   STATUS    RESTARTS   AGE
helloworld-deployment-6469589dcf-l9sfl   1/1     Running   0          72s
helloworld-deployment-6469589dcf-rmbnv   1/1     Running   0          75s

However, what happens if we change the image tag or add an environment variable in the Deployment yaml?

apiVersion: apps/v1
kind: Deployment
metadata:
  name: helloworld-deployment
spec:
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
  selector:
    matchLabels:
      app.kubernetes.io/name: helloworld-deployment
  replicas: 2
  template:
    metadata:
      labels:
        app.kubernetes.io/name: helloworld-deployment
    spec:
      containers:
        - name: hello-world
          image: helloworld-webserver:1.0.1
          resources:
            requests:
              cpu: 25m
              memory: 32Mi
            limits:
              cpu: 50m
              memory: 64Mi

Kubernetes will try to rollout the Deployment. The maxSurge configuration takes precedence over maxUnavailable, however there is insufficient ResourceQuota available for Kubernetes to create a new Pod, which you will see in the events of the newly created ReplicaSet:

>kubectl -n local-demo-rolloutstrategy-ns get rs
NAME                               DESIRED   CURRENT   READY   AGE
helloworld-deployment-6469589dcf   2         2         2       3m11s
helloworld-deployment-6c8c486b5c   1         0         0       119s
> kubectl -n local-demo-rolloutstrategy-ns describe rs helloworld-deployment-6c8c486b5c
....
Events:
  Type     Reason        Age                     From                   Message
  ----     ------        ----                    ----                   -------
  Warning  FailedCreate  7s                      replicaset-controller  Error creating: pods "helloworld-deployment-6c8c486b5c-lk87v" is forbidden: exceeded quota: default-resource-quotas, requested: requests.cpu=25m, used: requests.cpu=50m, limited: requests.cpu=50m
  Warning  FailedCreate  7s                      replicaset-controller  Error creating: pods "helloworld-deployment-6c8c486b5c-x5czs" is forbidden: exceeded quota: default-resource-quotas, requested: requests.cpu=25m, used: requests.cpu=50m, limited: requests.cpu=50m

Kubernetes will patiently wait forever for the ReplicaSet to do its thing — but the ReplicaSet can never create its Pod because of the ResourceQuota. The old Pods continue taking traffic, and your change is stuck.

>kubectl -n local-demo-rolloutstrategy-ns get pod
NAME                                     READY   STATUS    RESTARTS   AGE
helloworld-deployment-6469589dcf-l9sfl   1/1     Running   0          4m56s
helloworld-deployment-6469589dcf-rmbnv   1/1     Running   0          4m59s

Setting maxSurge to 0%, however, changes the behavior:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: helloworld-deployment
spec:
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 0%
      maxUnavailable: 25%
  selector:
    matchLabels:
      app.kubernetes.io/name: helloworld-deployment
  replicas: 2
  template:
    metadata:
      labels:
        app.kubernetes.io/name: helloworld-deployment
    spec:
      containers:
        - name: hello-world
          image: helloworld-webserver:1.0.1
          resources:
            requests:
              cpu: 25m
              memory: 32Mi
            limits:
              cpu: 50m
              memory: 64Mi

Kubernetes will round the 25% maxUnavailable setting to the nearest integer, and terminate one of the existing Pods, create a new Pod, terminate the remaining old Pod and create another new Pod (although, it might be nice to set maxUnavailable to 50%, since that makes more sense when replicas=2):

>kubectl -n local-demo-rolloutstrategy-ns get pod
NAME                                     READY   STATUS        RESTARTS   AGE
helloworld-deployment-6469589dcf-l9sfl   1/1     Terminating   0          5m31s
helloworld-deployment-6469589dcf-rmbnv   1/1     Running       0          5m34s
...
>kubectl -n local-demo-rolloutstrategy-ns get pod
NAME                                     READY   STATUS        RESTARTS   AGE
helloworld-deployment-5d4b48b96d-4sw7x   1/1     Running       0          3s
helloworld-deployment-6469589dcf-rmbnv   1/1     Terminating   0          5m55s
...
>kubectl -n local-demo-rolloutstrategy-ns get pod
NAME                                     READY   STATUS    RESTARTS   AGE
helloworld-deployment-5d4b48b96d-4sw7x   1/1     Running   0          83s
helloworld-deployment-5d4b48b96d-b24c7   1/1     Running   0          39s

As it turns out, Kubernetes is doing exactly what you are telling it to do… and now you know how to tell it to do what you really want!

Save yourself some grief — I recommend always specifying a rollout strategy explicitly for all Deployments, and if your Namespace has a ResourceQuota, you should consider setting maxSurge to 0% (and you may want to ensure that you have replicas ≥ 2, to ensure that rollouts don’t cause outages).

See https://github.com/psdally/k8s-rollout-strategy for source code used in this article.