The Most Common Problem I See When Using Json6902 Patching With Kustomize

When is an “add” actually a “replace”?

The problem

The most common problem that I see when people use Json6902 patching with kubectl kustomize is not realizing that “add” sometimes behaves as a “replace”.

For example, they’ll define a Deployment in their base that looks like this:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: helloworld-deployment
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: helloworld-deployment
  replicas: 2
  template:
    metadata:
      labels:
        app.kubernetes.io/name: helloworld-deployment
    spec:
      containers:
        - name: hello-world
          image: helloworld-webserver:1.0
      nodeSelector:
        example.com/nodelabel1: "somevalue"

They then define an overlay, and attempt to add an additional nodeSelector term with a patch that looks like this:

- op: add
  path: /spec/template/spec/nodeSelector
  value: 
    example.com/nodelabel2: "anothervalue"

What they are expecting is that the resulting Deployment will have 2 nodeSelector terms:

    # This is what they expect, but it is NOT what they get
    ...
    spec:
      containers:
        - name: hello-world
          image: helloworld-webserver:1.0
      nodeSelector:
        example.com/nodelabel1: "somevalue"
        example.com/nodelabel2: "anothervalue"

Instead, the resulting nodeSelector does not retain the values from the base and is entirely overwritten by the patch (notice that the nodeSelector for example.com/nodelabel1 has disappeared):

> kubectl kustomize overlay1
    # This is what they actually get
    ...
    spec:
      containers:
      - image: helloworld-webserver:1.0
        name: hello-world
      nodeSelector:
        example.com/nodelabel2: "anothervalue"

This issue is not restricted to patching just nodeSelectors — I’ve seen this mistake when patching almost everything you can imagine. The reason I highlight nodeSelectors, however, is that incorrect nodeSelectors may not be immediately obvious and I’ve seen that developers often tend to not test/verify them at all… but you should watch out for this mistake in any patching that you do.

The solution

What the patch should look like instead is:

- op: add
  path: /spec/template/spec/nodeSelector/example.com~1nodelabel2
  value: "anothervalue"

If we kustomize an overlay with this patch, we see the desired behaviour:

> kubectl kustomize overlay2
    # This is what we want, and we get it when we use
    # the right patch
    ...
    spec:
      containers:
      - image: helloworld-webserver:1.0
        name: hello-world
      nodeSelector:
        example.com/nodelabel1: somevalue
        example.com/nodelabel2: anothervalue

Notice the use of the ~1 to escape the slash in the nodeSelector term name. If you don’t escape the slash, you’ll get an error like this:

>kubectl kustomize overlay3
Error: replace operation does not apply: doc is missing path: /spec/template/spec/nodeSelector/example.com/nodelabel2: missing value

As you can see, kustomize is assuming that underneath nodeSelector there is a object called example.com, and underneath that is where a nodeLabel2 object should go — which makes sense, as the slash is used to delineate levels of the json/yaml heirarchy. Escaping the slash with ~1 tells kustomize that you are actually looking for a slash in the object name.

The source code for this article is on github.