Debugging running applications in Kubernetes starts by retrieving simple status information about the pods.
Here are a few places to start looking:
- Pod status – is the pod running, pending, or crash looping?
- Pod restart count – does the pod have many recent restarts?
- Pod resources – is the pod requesting more resources available in its namespace or on its node?
Pod details are retrieved by introspecting the pod with
kubectl describe pod pod_name.
Take a look at how common pod issues are debugged using the pod’s description.
Here are key segments in a description of a pending pod:
$ kubectl describe pod busybox Name: busybox Namespace: default ... Status: Pending IP: 10.32.0.5 IPs: IP: 10.32.0.5 Containers: myapp-container: Container ID: Image: busyBOX ... Conditions: Type Status Initialized True Ready False ContainersReady False PodScheduled True ... Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 10s default-scheduler Successfully assigned default/busybox to ubuntu Warning InspectFailed 9s (x2 over 9s) kubelet, ubuntu Failed to apply default image tag "busyBOX": couldn't parse image reference "busyBOX": invalid reference format: repository name must be lowercase Warning Failed 9s (x2 over 9s) kubelet, ubuntu Error: InvalidImageName $
Events section of the description points out the image name is invalid. Correcting the image name will resolve this issue.
Here’s another pending pod:
$ kubectl get pods NAME READY STATUS RESTARTS AGE nginx 0/1 Pending 0 2m15s $
This nginx pod has been pending for over 2 minutes.
Let’s look at the pod’s
$ kubectl describe pod nginx | grep -A4 Events Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning FailedScheduling 76s (x4 over 3m54s) default-scheduler 0/1 nodes are available: 1 Insufficient cpu. $
Not enough resources are available in the cluster. We can see if the pod is making a hard resource request by looking at the
Containers section of the pod description.
$ kubectl describe pod nginx | grep -A10 Containers Containers: nginx: Image: nginx Port: Host Port: Limits: cpu: 2 memory: 2Gi Requests: cpu: 1500m memory: 1536Mi $
The pod is requesting 1500m of cpu. There are no nodes in the cluster with 1500m free of cpu so the pod stays pending. There are a few ways to resolve this issue: reduce the container’s cpu request, free up cpu on a cluster node, or add a new worker node to the cluster.
Learn more about application debugging.