Kubernetes Image Pull Policy

Talk to Our Team
Course Catalog
Open Enrollment Schedule

Introduction

In this article we will discuss Kubernetes image pull policy. One of the critical decisions you will make is when container images are pulled from image registries; Kubernetes image pull policies govern this decision. These policies are essential for ensuring that your applications run the desired versions of container images, which can significantly affect your applications’ reliability and security.

Kubernetes Image Pull Policy Diagram

How does Kubernetes pull container images?

When you create a pod, whether that pod is a standalone pod or deployed as part of a Deployment, StatefulSet, etc., you specify the container image (or images if it is a multi-container pod) for the container(s) in the pod. The image(s) are pulled from the specified registry, like the Docker Hub, by their fully-qualified image name (FQIN) and cached by the container runtime on a Kubernetes worker node. The image pull policy is a configuration setting that tells the kubelet when to direct the container runtime to pull an image from a registry.

Image pull policies

Note: if no image pull policy is explicitly defined in a pod specification, Kubernetes automatically chooses a policy based on the image tag. If the image tag is :latest or if the tag is omitted, Kubernetes defaults to the Always policy. For any other tag, the default policy is IfNotPresent.

Always

This policy instructs Kubernetes always to pull the image from the registry, ensuring that the latest version is always used. It is worth noting that this policy doesn’t mean the container runtime will always pull the image! Pulling an image is a series of API calls to the registry; the first to the /manifests endpoint and the second (if necessary) to the /blob endpoint:

  • First the container runtime will call and download the image manifest from the /manifest endpoint; the manifest is a lightweight json document requiring very little network bandwidth
  • It will then try to resolve the image digest with an existing image in its cache; if it finds a matching digest, it uses that locally-cached image to create a container, avoiding downloading the heavyweight image archive (tarball).
  • If no locally-cached image is a match, it calls the /blob endpoint, downloads the image archive, extracts it, and creates a container from it.

By “always” downloading and reconciling the image manifest against the local cache, Kubernetes guarantees it always has the latest version of the image, whether or not it has to download a new tarball.

IfNotPresent

With this policy, the kubelet checks if the image already exists on the node. If the image is present, kubelet will not direct the container runtime to pull the image from the registry.

Never

When this policy is set, Kubernetes will never pull an image from the registry. It will solely rely on the locally-cached images.

Use cases

The most asked question is when to use which policy. Here are some use cases for each policy:

Always policy

In production environments, where you want to ensure that the latest version of an image is always used, the Always policy is the best policy to use. This policy ensures that the latest version of the image is always used, including all the latest security patches and features. The container runtime will reconcile the image digest to ensure the most up-to-date image is used. If the digest is the same as the one on the node, the image will not be pulled from the registry. This method gives you a caching mechanism but is not the same as the IfNotPresent policy.

Here is what the container runtime uses to check if the image is the same as the one on the node: @sha256:

$ crictl pull busybox:1.36.1
pulling image (busybox:1.36.1)
$ crictl images --digests
IMAGE                            TAG                  DIGEST              IMAGE ID            SIZE
docker.io/library/busybox        1.36.1               6d9ac9237a84a       3e4fd538a9a0b       1.93MB

Based on that our Busybox image will look like this: busybox@sha256:6d9ac9237a84a

Real-world scenario: An application is set to automatically deploy to a staging environment after each commit to the main branch. By setting the image pull policy to Always and using the :latest tag (or a specific version tag that gets updated with each build), you ensure that every deployment in the staging environment reflects the latest codebase.

IfNotPresent policy

In development and testing environments, where rapid iteration is common, and network bandwidth might be limited, the IfNotPresent policy can significantly speed up deployment times. This policy avoids unnecessary downloads if the image already exists on the node, allowing developers to quickly test their changes without waiting for images to be pulled from the registry unless absolutely necessary.

Real-world scenario: You are working on a new feature and frequently updating the application code. By using an image tagged with a version other than :latest and the IfNotPresent policy, you can reuse the locally cached version of the container image, reducing the time spent waiting for the image to be pulled from the registry.

Never policy

This policy is less commonly used but can be useful in isolated, secure, or air-gapped environments where you want to ensure complete control over the images being used. It can also be useful in environments where you want to ensure that the image is not pulled from the registry but only from the local cache. In highly secure, air-gapped, or restricted environments where internet access is limited or non-existent, the Never policy ensures that only pre-approved, locally available images are used. This policy is crucial for maintaining security and compliance, as it prevents any external image from being pulled into the environment.

Real-world scenario: A financial institution operates a Kubernetes cluster within a secure environment that cannot access external image registries for security reasons. By using the Never policy, the institution ensures that only images that have been manually vetted and loaded onto the worker nodes are used, maintaining strict compliance with security policies.

Setting the image pull policy

Here is an example of a pod yaml file with the image pull policy set to IfNotPresent:

apiVersion: v1
kind: Pod
metadata:
  name: pod-ifnotpresent
  labels:
    app: pod-ifnotpresent
spec:
  containers:
  - name: image-pull
    image: busybox:1.36.1
    imagePullPolicy: IfNotPresent     # << this is our image policy
    command: ["/bin/sh", "-c", "echo image is successfully pulled"]

Serial vs parallel image pulling

By default, Kubernetes pulls images serially.

Serial pull

Serial image pulls mean that each image required by the pods on a node is downloaded one after the other. This conservative approach ensures that the network and disk IO are not overwhelmed by multiple simultaneous downloads, which can be particularly important in environments with limited resources or bandwidth.

Parallel image pulls

Parallel image pulls allow multiple images to be downloaded simultaneously across the nodes in a Kubernetes cluster. This approach can significantly speed up the deployment process, especially for applications with many service components or when scaling up a deployment quickly.

Configuring image pulling

You can configure the way how kubernetes pulls images by setting the serializeImagePulls option in the kubelet.

Hands-on

To simplify our demo, we have a single node K8s cluster. Let’s see it in action. We will do couple of things:

  • Monitor kubelet logs using journalctl
  • Monitor containerd logs using journalctl
  • Create a pod in Kubernetes

Open a terminal and follow the logs for the kubelet.

$ journalctl -u kubelet -f
Feb 13 14:29:14 ubuntu kubelet[97671]: I0213 14:29:14.643476   97671 pod_startup_latency_tracker.go:102] "Observed pod startup duration" pod="kube-system/coredns-76f75df574-dftzk" podStartSLOduration=36.643428544 podStartE2EDuration="36.643428544s" podCreationTimestamp="2024-02-13 14:28:38 +0100 CET" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2024-02-13 14:29:13.631123059 +0100 CET m=+41.322879535" watchObservedRunningTime="2024-02-13 14:29:14.643428544 +0100 CET m=+42.335185021"

Open another terminal and follow the logs for containerd:

$ journalctl -u containerd -f
Feb 13 14:29:32 ubuntu containerd[96261]: time="2024-02-13T14:29:32.427218888+01:00" level=info msg="TearDown network for sandbox \"18bf27d68d31aa2cebc8bb5f6aa71dc9752b0011f76a88ad1c9ea2d67e5bc103\" successfully"
Feb 13 14:29:32 ubuntu containerd[96261]: time="2024-02-13T14:29:32.429362870+01:00" level=info msg="RemovePodSandbox \"18bf27d68d31aa2cebc8bb5f6aa71dc9752b0011f76a88ad1c9ea2d67e5bc103\" returns successfully"

Image pull policy: Always

Let’s create a pod with the image pull policy set to Always.

apiVersion: v1
kind: Pod
metadata:
  name: pod-always
  labels:
    app: pod-always
spec:
  containers:
  - name: image-pull
    image: busybox:1.36.1
    imagePullPolicy: Always     # << this is our image policy
    command: ["/bin/sh", "-c", "sleep 600"]

Create the pod and examine the logs for kubelet and containerd.

$ kubectl apply -f pod-always.yaml
pod/pod-always created

Let’s check the kubelet logs:

Feb 13 13:52:42 ip-172-31-22-199 kubelet[4830]: I0213 13:52:42.076888    4830 topology_manager.go:215] "Topology Admit Handler" podUID="6f769ce8-7820-4195-a3e4-044ed3f31962" podNamespace="default" podName="pod-always"
Feb 13 13:52:42 ip-172-31-22-199 kubelet[4830]: I0213 13:52:42.158665    4830 reconciler_common.go:258] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6jmf\" (UniqueName: \"kubernetes.io/projected/6f769ce8-7820-4195-a3e4-044ed3f31962-kube-api-access-p6jmf\") pod \"pod-always\" (UID: \"6f769ce8-7820-4195-a3e4-044ed3f31962\") " pod="default/pod-always"

As we can see, the kubelet itself does not pull the image. Let’s check the containerd logs:

...
Feb 13 13:52:46 ip-172-31-22-199 containerd[3070]: time="2024-02-13T13:52:46.750719023Z" level=info msg="ImageUpdate event &ImageUpdate{Name:docker.io/library/busybox:1.36.1,Labels:map[string]string{io.cri-containerd.image: managed,},XXX_unrecognized:[],}"
Feb 13 13:52:46 ip-172-31-22-199 containerd[3070]: time="2024-02-13T13:52:46.754684457Z" level=info msg="ImageCreate event &ImageCreate{Name:docker.io/library/busybox@sha256:6d9ac9237a84afe1516540f40a0fafdc86859b2141954b4d643af7066d598b74,Labels:map[string]string{io.cri-containerd.image: managed,},XXX_unrecognized:[],}"
Feb 13 13:52:46 ip-172-31-22-199 containerd[3070]: time="2024-02-13T13:52:46.758021970Z" level=info msg="PullImage \"busybox:1.36.1\" returns image reference \"sha256:3f57d9401f8d42f986df300f0c69192fc41da28ccc8d797829467780db3dd741\""
Feb 13 13:52:46 ip-172-31-22-199 containerd[3070]: time="2024-02-13T13:52:46.770091312Z" level=info msg="CreateContainer within sandbox \"de1ec2e591ea5c288f9cbcb69296e5254193da20ea48f2830d9cb487a01c0b1d\" for container &ContainerMetadata{Name:image-pull,Attempt:0,}"
Feb 13 13:52:46 ip-172-31-22-199 containerd[3070]: time="2024-02-13T13:52:46.798923701Z" level=info msg="CreateContainer within sandbox \"de1ec2e591ea5c288f9cbcb69296e5254193da20ea48f2830d9cb487a01c0b1d\" for &ContainerMetadata{Name:image-pull,Attempt:0,} returns container id \"5d13a0947805a88b1c87565cea392f8c98468a3d82e5b87271843824fe9d34b4\""
...

You can see the line PullImage "busybox:1.36.1" which means that the image was pulled from the registry.

You can see how containerd is pulling the image @sha256:
{Name:docker.io/library/busybox@sha256:6d9ac9237a84afe ………

Remember, we are using image digests. We can verify that again with the crictl command.

$ sudo crictl image --digests | grep busy
docker.io/library/busybox                 1.36.1              6d9ac9237a84a       3f57d9401f8d4       2.23MB

As you can see the 6d9ac9237a84a digest matches the one from the containerd logs.

Image pull policy: IfNotPresent

Let’s create another pod with the IfNotPresent policy and see what happens. We will not change the image tag because we already have the image cached on the node.

apiVersion: v1
kind: Pod
metadata:
  name: pod-ifnotpresent
  labels:
    app: pod-ifnotpresent
spec:
  containers:
  - name: image-pull
    image: busybox:1.36.1
    imagePullPolicy: IfNotPresent     # << this is our image policy
    command: ["/bin/sh", "-c", "sleep 600"]

Create the pod and examine the logs for containerd once again.

$ kubectl apply -f pod-ifnotpresent.yaml
pod/pod-ifnotpresent created

Let’s check the containerd logs:

Feb 13 13:59:08 ip-172-31-22-199 containerd[3070]: time="2024-02-13T13:59:08.326678546Z" level=info msg="RunPodSandbox for &PodSandboxMetadata{Name:pod-ifnotpresent,Uid:51b066c2-8105-4b93-a05d-cc8ed71bee94,Namespace:default,Attempt:0,}"
Feb 13 13:59:08 ip-172-31-22-199 containerd[3070]: time="2024-02-13T13:59:08.537776859Z" level=info msg="loading plugin \"io.containerd.event.v1.publisher\"..." runtime=io.containerd.runc.v2 type=io.containerd.event.v1
Feb 13 13:59:08 ip-172-31-22-199 containerd[3070]: time="2024-02-13T13:59:08.537999833Z" level=info msg="loading plugin \"io.containerd.internal.v1.shutdown\"..." runtime=io.containerd.runc.v2 type=io.containerd.internal.v1
Feb 13 13:59:08 ip-172-31-22-199 containerd[3070]: time="2024-02-13T13:59:08.538447490Z" level=info msg="loading plugin \"io.containerd.ttrpc.v1.task\"..." runtime=io.containerd.runc.v2 type=io.containerd.ttrpc.v1
Feb 13 13:59:08 ip-172-31-22-199 containerd[3070]: time="2024-02-13T13:59:08.539178282Z" level=info msg="starting signal loop" namespace=k8s.io path=/run/containerd/io.containerd.runtime.v2.task/k8s.io/6c5c065007c6bbc7586350ee46a0cb9218928c26cb291baaea21a523c40a055a pid=6600 runtime=io.containerd.runc.v2
Feb 13 13:59:08 ip-172-31-22-199 containerd[3070]: time="2024-02-13T13:59:08.657345449Z" level=info msg="RunPodSandbox for &PodSandboxMetadata{Name:pod-ifnotpresent,Uid:51b066c2-8105-4b93-a05d-cc8ed71bee94,Namespace:default,Attempt:0,} returns sandbox id \"6c5c065007c6bbc7586350ee46a0cb9218928c26cb291baaea21a523c40a055a\""
Feb 13 13:59:08 ip-172-31-22-199 containerd[3070]: time="2024-02-13T13:59:08.661396293Z" level=info msg="CreateContainer within sandbox \"6c5c065007c6bbc7586350ee46a0cb9218928c26cb291baaea21a523c40a055a\" for container &ContainerMetadata{Name:image-pull,Attempt:0,}"
Feb 13 13:59:08 ip-172-31-22-199 containerd[3070]: time="2024-02-13T13:59:08.683918941Z" level=info msg="CreateContainer within sandbox \"6c5c065007c6bbc7586350ee46a0cb9218928c26cb291baaea21a523c40a055a\" for &ContainerMetadata{Name:image-pull,Attempt:0,} returns container id \"b085b39197a03eaefc711a8d86ce425fc39b0e871aed72bdff97a5a085d8b9a3\""
Feb 13 13:59:08 ip-172-31-22-199 containerd[3070]: time="2024-02-13T13:59:08.685203651Z" level=info msg="StartContainer for \"b085b39197a03eaefc711a8d86ce425fc39b0e871aed72bdff97a5a085d8b9a3\""
Feb 13 13:59:08 ip-172-31-22-199 containerd[3070]: time="2024-02-13T13:59:08.769318927Z" level=info msg="StartContainer for \"b085b39197a03eaefc711a8d86ce425fc39b0e871aed72bdff97a5a085d8b9a3\" returns successfully"

As we can see, there is no image pull event in the logs. This is because the image is already on the worker node.

Image pull policy: Never

What about the Never policy? Let’s create a pod with the Never policy and see what happens.
The only difference is we will change the busybox version to 1.36.0 because we already have the 1.36.1 on the node, and we want to show you what will happen when we don’t have it.

apiVersion: v1
kind: Pod
metadata:
  name: pod-never
  labels:
    app: pod-never
spec:
  containers:
  - name: image-pull
    image: busybox:1.36.0
    imagePullPolicy: Never     # << this is our image policy
    command: ["/bin/sh", "-c", "sleep 600"]

Before we create the pod, let’s see the current pods.

$ kubectl get pods
NAME               READY   STATUS    RESTARTS         AGE
pod-always         1/1     Running   50 (2m37s ago)   8h
pod-ifnotpresent   1/1     Running   49 (8m7s ago)    8h

Let’s create our pod with the image pull policy Never and check what will happen.

$ kubectl apply -f pod-never.yaml
pod/pod-never created

Now, let’s check the pods again.

$ kubectl get pods
NAME               READY   STATUS              RESTARTS        AGE
pod-always         1/1     Running             50 (6m1s ago)   8h
pod-ifnotpresent   1/1     Running             50 (91s ago)    8h
pod-never          0/1     ErrImageNeverPull   0               11s

As you can see, the pod is in the ErrImageNeverPull state. This is because the image is not on the node. We instructed Kubernetes never to pull the image from the registry and use the local one. If the image is not on the node, the pod will be in the ErrImageNeverPull state.

Let’s pull the image using crictl and see what happens.

$ sudo crictl pull busybox:1.36.0
Image is up to date for sha256:af2c3e96bcf1a80da1d9b57ec0adc29f73f773a4a115344b7e06aec982157a33
$ kubectl get pods
NAME               READY   STATUS    RESTARTS         AGE
pod-always         1/1     Running   50 (7m56s ago)   8h
pod-ifnotpresent   1/1     Running   50 (3m26s ago)   8h
pod-never          1/1     Running   0                2m6s

As you can see, when we manually pull the image after a short period, Kubernetes will try to start the pod again and will be successful. Now, our pod is in the Running state.

Bust-a-kube challenge

Want to try this on your own? Try out our related Bust-a-kube challenge. It’s fun, a bit tricky, and full of surprises. Think you can crack it? Give it a try and see what you find! Check out Bust-a-kube here.

Secret Link