305-998-7702 | 415-800-2922 info@rx-m.com

Troubleshoot Worker Node Failure

Learn how to put the latest open source technology into practice with hands-on training, delivered by industry experts, aligned to your desired business outcomes

A node that runs pods in a Kubernetes cluster is represented by a kubelet. Each kubelet requires a container runtime (like Docker) to run containers. Since the kubelet and container runtime run as system agents, you must use host-level tools like systemctl or journalctl to view their logs.

$ sudo systemctl status kubelet

● kubelet.service - kubelet: The Kubernetes Node Agent
   Loaded: loaded (/lib/systemd/system/kubelet.service; enabled; vendor preset: enabled)
  Drop-In: /etc/systemd/system/kubelet.service.d
           └─10-kubeadm.conf
   Active: activating (auto-restart) (Result: exit-code) since Fri 2020-02-28 08:08:03 PST; 8s ago
     Docs: https://kubernetes.io/docs/home/
  Process: 2175 ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_KUBE
 Main PID: 2175 (code=exited, status=255)

Feb 28 08:08:03 labsys systemd[1]: kubelet.service: Main process exited, code=exited, status=255/n/a
Feb 28 08:08:03 labsys systemd[1]: kubelet.service: Unit entered failed state.
Feb 28 08:08:03 labsys systemd[1]: kubelet.service: Failed with result 'exit-code'.

$ journalctl -u kubelet | tail -5

Feb 28 08:09:15 labsys kubelet[2771]: I0228 08:09:15.417176    2771 server.go:641] --cgroups-per-qos enabled, but --cgroup-root was not specified.  defaulting to /
Feb 28 08:09:15 labsys kubelet[2771]: F0228 08:09:15.417372    2771 server.go:273] failed to run Kubelet: running with swap on is not supported, please disable swap! or set --fail-swap-on flag to false. /proc/swaps contained: [Filename                                Type                Size        Used        Priority /dev/dm-1                               partition        4194300        0        -1]
Feb 28 08:09:15 labsys systemd[1]: kubelet.service: Main process exited, code=exited, status=255/n/a
Feb 28 08:09:15 labsys systemd[1]: kubelet.service: Unit entered failed state.
Feb 28 08:09:15 labsys systemd[1]: kubelet.service: Failed with result 'exit-code'.

$

Kubelets (and thus worker nodes) require functional container runtimes to operate pods. In default installations, they also need to have system swap disabled.
 
Learn more about debugging your worker nodes and kubelets