Kubernetes — Why Am I Getting ImagePullBackOff When I’m Testing Locally? My Image Exists!

You’ve built your image on your local desktop (Docker Desktop, Minikube, etc.), and you’ve created a Pod (perhaps by creating a Deployment, StatefulSet, etc.). Unfortunately, you see that your Pod is in an ImagePullBackoff loop.

>kubectl -n local-demo-sidecars-ns describe pod helloworld-deployment-1-8456546869-vvt6d
<snip>
Events:
  Type     Reason     Age                    From               Message
  ----     ------     ----                   ----               -------
  Normal   Scheduled  5m46s                  default-scheduler  Successfully assigned local-demo-sidecars-ns/helloworld-deployment-1-8456546869-vvt6d to minikube
  Normal   Pulling    4m1s (x4 over 5m36s)   kubelet            Pulling image "helloworld-webserver:1.0.0"
  Warning  Failed     3m59s (x4 over 5m33s)  kubelet            Failed to pull image "helloworld-webserver:1.0.0": rpc error: code = Unknown desc = Error response from daemon: pull access denied for helloworld-webserver, repository does not exist or may require 'docker login': denied: requested access to the resource is denied
  Warning  Failed     3m59s (x4 over 5m33s)  kubelet            Error: ErrImagePull
  Warning  Failed     3m47s (x6 over 5m32s)  kubelet            Error: ImagePullBackOff
  Normal   BackOff    30s (x20 over 5m32s)   kubelet            Back-off pulling image "helloworld-webserver:1.0.0"

You know the image exists:

>docker image ls
REPOSITORY             TAG     IMAGE ID       CREATED         SIZE
helloworld-webserver   1.0.0   5fdcec72e8df   20 hours ago    24.8MB

You’ve built and used local images before so you are confident that there isn’t anything wrong with the image and you know it doesn’t require a login. What is going on?

Is The Problem Your imagePullPolicy?

The reason may be because of your imagePullPolicy on the Pod spec. For example, you might be deploying a Deployment that looks like this:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: helloworld-deployment-1
  namespace: local-demo-sidecars-ns
spec:
  replicas: 2
  selector:
    matchLabels:
      app.kubernetes.io/name: helloworld-deployment-1
  template:
    spec:
      containers:
      - image: helloworld-webserver:1.0.0
        imagePullPolicy: Always

When imagePullPolicy is Always, Kubernetes will never use your locally built image without first accessing the registry to do an image digest lookup. If the kubelet has a container image with that exact digest cached locally, the kubelet uses its cached image; otherwise, the kubelet pulls the image with the resolved digest, and uses that image to launch the container. If you haven’t already pushed your image to Docker Hub, unless there happens to be an image with the same respository name and tag present, you’ll get the error message shown above.

If an image exists on Docker hub that does have the same repository and tag, you’ll pull that image and run it instead. Or, if the image name was prefixed with a registry, you would connect to that registry and pull the image. Unless, your local image was the same as the image in the registry, you would pull and run the remote image and not run the image that you just built locally. Which is a case where success == failure…

If the image in the registry isn’t the same as the image that you’ve just built locally (and it probably wouldn’t be, because you’ve just built it locally and are trying to test locally, and so you likely won’t have pushed it to the registry yet) then you might end up spending a lot of time trying to figure out why your changes aren’t working as you expect, but your changes may work just fine — you just aren’t testing what you think you are testing).

Of course, the image tags used in your podSpecs should almost always be immutable, and so if you find yourself in this second scenario you should probably reconsider your tagging practices…

What should you do about this?

• consider using an imagePullPolicy of IfNotPresent instead of Always when deploying to your local environment and
• avoid reusing image tags, if you are using that tag in your Pod specs