305-998-7702 | 415-800-2922 info@rx-m.com

Debugging

Learn how to put the latest open source technology into practice with hands-on training, delivered by industry experts, aligned to your desired business outcomes

Debugging running applications in Kubernetes starts by retrieving simple status information about the pods.
Here are a few places to start looking:

  • Pod status – is the pod running, pending, or crash looping?
  • Pod restart count – does the pod have many recent restarts?
  • Pod resources – is the pod requesting more resources available in its namespace or on its node?

Pod details are retrieved by introspecting the pod with kubectl describe pod pod_name.

Take a look at how common pod issues are debugged using the pod’s description.

Here are key segments in a description of a pending pod:

$ kubectl describe pod busybox

Name:         busybox
Namespace:    default
...
Status:       Pending
IP:           10.32.0.5
IPs:
  IP:  10.32.0.5
Containers:
  myapp-container:
    Container ID:  
    Image:         busyBOX
  ...
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
...
Events:
  Type     Reason         Age              From               Message
  ----     ------         ----             ----               -------
  Normal   Scheduled      10s              default-scheduler  Successfully assigned default/busybox to ubuntu
  Warning  InspectFailed  9s (x2 over 9s)  kubelet, ubuntu     Failed to apply default image tag "busyBOX": couldn't parse image reference "busyBOX": invalid reference format: repository name must be lowercase
  Warning  Failed         9s (x2 over 9s)  kubelet, ubuntu     Error: InvalidImageName

$

The Events section of the description points out the image name is invalid. Correcting the image name will resolve this issue.

Here’s another pending pod:

$ kubectl get pods

NAME    READY   STATUS    RESTARTS   AGE
nginx   0/1     Pending   0          2m15s

$

This nginx pod has been pending for over 2 minutes.
Let’s look at the pod’s Events.

$ kubectl describe pod nginx | grep -A4 Events

Events:
  Type     Reason            Age                  From               Message
  ----     ------            ----                 ----               -------
  Warning  FailedScheduling  76s (x4 over 3m54s)  default-scheduler  0/1 nodes are available: 1 Insufficient cpu.

$

Not enough resources are available in the cluster. We can see if the pod is making a hard resource request by looking at the Containers section of the pod description.

$ kubectl describe pod nginx | grep -A10 Containers

Containers:
  nginx:
    Image:      nginx
    Port:       
    Host Port:  
    Limits:
      cpu:     2
      memory:  2Gi
    Requests:
      cpu:        1500m
      memory:     1536Mi

$

The pod is requesting 1500m of cpu. There are no nodes in the cluster with 1500m free of cpu so the pod stays pending. There are a few ways to resolve this issue: reduce the container’s cpu request, free up cpu on a cluster node, or add a new worker node to the cluster.

Learn more about application debugging.