305-998-7702 | 415-800-2922 info@rx-m.com

Troubleshoot Networking

Learn how to put the latest open source technology into practice with hands-on training, delivered by industry experts, aligned to your desired business outcomes

Every pod running in a Kubernetes cluster must have its own IP address. Pods receive their unique IPs from the cluster Container Network Interface (CNI) plugin. In order to properly use a CNI plugin, each Kubelet must be configured to expect CNI plugins in their systemd service.

Without a functional CNI, the kubelet is unable to configure the container runtime’s network and does not report a “ready” status to the cluster:

$ sudo systemctl status kubelet

● kubelet.service - kubelet: The Kubernetes Node Agent
   Loaded: loaded (/lib/systemd/system/kubelet.service; enabled; vendor preset: enabled)
  Drop-In: /etc/systemd/system/kubelet.service.d
           └─10-kubeadm.conf
   Active: active (running) since Fri 2020-02-28 08:50:27 PST; 5min ago
     Docs: https://kubernetes.io/docs/home/
 Main PID: 19302 (kubelet)
    Tasks: 16 (limit: 512)
   Memory: 45.2M
      CPU: 3.761s
   CGroup: /system.slice/kubelet.service
           └─19302 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kubelet/config.yaml --cgroup-driver=cgroupfs 

Feb 28 08:55:58 labsys kubelet[19302]: E0228 08:55:58.668145   19302 kubelet.go:2183] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin i
Feb 28 08:56:02 labsys kubelet[19302]: W0228 08:56:02.452412   19302 cni.go:237] Unable to update cni config: no networks found in /etc/cni/net.d

$

Other symptoms of a non-functional or misconfigured CNI plugin include:

  • Pods receiving a Docker local IP
  • New pods staying in the ContainerCreating phase even if the image is pulled

Other networking failures may involve factors within the cluster’s underlying infrastructure, such as firewall rules or full network partitions preventing communication.

Learn more about debugging networking at various levels of the cluster and checking if the kube-proxy is working