Kubernetes — Using Gatekeeper For Operational Guardrails — Part 2 (Mutation)

If you haven’t read part 1, please give it a quick go-over — it will provide some context for part 2 (this article).

In the previous part of this article, we looked at how to validate objects being deployed to Kubernetes to ensure that they conform with our policies and governance.

Rather than rejecting objects that are not acceptable, however, in some cases we may be able to make modifications automatically to make the object acceptable. If Kubernetes knows what an object has to look like, wouldn’t you rather it just made the changes for you rather than telling you to get lost and go make the changes yourself? Kubernetes refers to this as “mutating” the object.

First the slightly bad news

Gatekeeper support for mutation graduated to “beta” (I would characterize this as “production-worthy beta” but your risk tolerances and practices may vary) with version v3.7.0, which was released on November 15, 2021. If you are running an earlier version, you’ll want to upgrade.

Also, Gatekeeper does not support rego policies for mutation. A feature called External Data is in alpha which looks interesting, but at this point any mutations will need to be done with data that is either static, built-in or user-defined within the object being mutated.

Now the good news

Alpha features are probably best confined to the lab, but external data is not so critical for our use case described in part 1 — or for many (most?) other use cases of mutation as well.

In our case, all that we need to do is create 1 mutation for each business unit that we have. For example:

apiVersion: mutations.gatekeeper.sh/v1beta1
kind: Assign
metadata:
  name: inject-widget-nodeselector
spec:
  applyTo:
  - groups: [""]
    kinds: ["Pod"]
    versions: ["v1"]
  match:
    scope: Namespaced
    kinds:
    - apiGroups: ["*"]
      kinds: ["Pod"]
    namespaceSelector:
      matchLabels:
        businessunit: "widget"
  location: "spec.nodeSelector"
  parameters:
    assign:
      value:
        businessunit: "widget"

and:

apiVersion: mutations.gatekeeper.sh/v1beta1
kind: Assign
metadata:
  name: inject-doodad-nodeselector
spec:
  applyTo:
  - groups: [""]
    kinds: ["Pod"]
    versions: ["v1"]
  match:
    scope: Namespaced
    kinds:
    - apiGroups: ["*"]
      kinds: ["Pod"]
    namespaceSelector:
      matchLabels:
        businessunit: "doodad"
  location: "spec.nodeSelector"
  parameters:
    assign:
      value:
        businessunit: "doodad"

A similar Mutation would be created for the “shared” business unit, as well as any future business units that we deploy dedicated worker Nodes for.

The results

After deploying the mutations above, if we look at the “bad” widget Deployment from part 1 we can see that the Deployment does not have a nodeSelector:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: widget-deployment
  namespace: local-demo-widget-ns
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: widget-deployment
  replicas: 2
  template:
    metadata:
      labels:
        app.kubernetes.io/name: widget-deployment
    spec:
      containers:
        - name: widget-app
          image: widget-app:1.0.0
          resources:
            requests:
              cpu: 50m
              memory: 32Mi
            limits:
              cpu: 100m
              memory: 64Mi

However, the resulting Pod does have the nodeSelector:

>kubectl -n local-demo-widget-ns describe pod widget-deployment-764d55c6bc-s8hqm
Name:         widget-deployment-764d55c6bc-s8hqm
Namespace:    local-demo-widget-ns
Priority:     0
...
Node-Selectors:              businessunit=widget
...

This is fantastic! Now our developers don’t get an error when they don’t have the nodeSelector (or if they assign an invalid value to the nodeSelector) — it gets added (or corrected) as if by magic! Sometimes what is better than a good fence… is a good fence that keeps the developers on the right track without them even realizing it!