CKA Mod 5: Troubleshoot services and networking

Learn how to put the latest open source technology into practice with hands-on training, delivered by industry experts, aligned to your desired business outcomes

Talk to Our Team
Course Catalog
Open Enrollment Schedule

Services

Services should be available by their DNS name within the cluster. Tools like nslookup and dig should be able to resolves services by their name in the same namespace or by the pattern: service-name.namespace in any namespace in the cluster. If DNS resolution fails, examine the /etc/resolv.conf file in the client pod to make sure the settings are correct. The nameserver should be a clusterIP (10.96.0.10 in kubeadm clusters with the default configuration) and the search domains should include the pod’s namespace and the cluster’s domain.

To eliminate DNS from your debugging efforts, use the clusterIP of the service instead of its DNS name. Sending requests to the clusterIP of the service should work from any pod and any node in the cluster since clusterIPs are written to each node’s IPTables or IPVS tables.

Listing endpoints with kubectl get ep <service-name> can reveal if the endpoints list is empty. If it is, you can replicate the service’s selector by using kubectl get po -l <selector-key>=<selector-value>. Using this technique you may discover if any/all the target pods are not ready due to probe failures, nodes where pods run being down or not reporting ready, etc.

Kube-proxy needs to be actively updating the IPTables or IPVS tables of each node. There are several steps to check if the kube-proxy is working.

Lastly, ensure the network policies are not blocking ingress/egress traffic to and from services and clients within the cluster.

Learn more about troubleshooting services.

Networking

Every pod running in a Kubernetes cluster must have its own IP address. Pods receive their unique IPs from the cluster Container Network Interface (CNI) plugin. In order to properly use a CNI plugin, each Kubelet must be configured to expect CNI plugins in their systemd service.

Without a functional CNI, the kubelet is unable to configure the container runtime’s network and does not report a “ready” status to the cluster:

$ sudo systemctl status kubelet

● kubelet.service - kubelet: The Kubernetes Node Agent
   Loaded: loaded (/lib/systemd/system/kubelet.service; enabled; vendor preset: enabled)
  Drop-In: /etc/systemd/system/kubelet.service.d
           └─10-kubeadm.conf
   Active: active (running) since Fri 2020-02-28 08:50:27 PST; 5min ago
     Docs: https://kubernetes.io/docs/home/
 Main PID: 19302 (kubelet)
    Tasks: 16 (limit: 512)
   Memory: 45.2M
      CPU: 3.761s
   CGroup: /system.slice/kubelet.service
           └─19302 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kubelet/config.yaml --cgroup-driver=cgroupfs

Feb 28 08:55:58 labsys kubelet[19302]: E0228 08:55:58.668145   19302 kubelet.go:2183] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin i
Feb 28 08:56:02 labsys kubelet[19302]: W0228 08:56:02.452412   19302 cni.go:237] Unable to update cni config: no networks found in /etc/cni/net.d

Note the message in the output above: Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady.

Other symptoms of a non-functional or misconfigured CNI plugin include: new pods staying in the ContainerCreating phase even if the image is pulled.

Other networking failures may involve factors within the cluster’s underlying infrastructure, such as firewall rules or full network partitions preventing communication.

Learn more about debugging networking at various levels of the cluster.

Secret Link