RX-M, a cloud-native training, consulting, and advisory firm, provides vital strategies, insights, and tactical approaches in its Cloud Native Short Takes series. In this take, instructor Christian Lacsina reviews the Kubernetes Observability Overview module. Christian discusses the Kubernetes Resource- and Custom Metrics APIs and looks at autoscaling applications with the Horizontal Pod Autoscaler (HPA).
Video Summary
Welcome to another RX-M cloud native short take! My name is Christian Lacsina and today we’ll short take the RX-M kubernetes Observability module. In this module we:
- Discuss the concept of observability by comparing and contrasting metrics tracing and logging
- Take a look at how Kubernetes implements observability with the Resource Metrics API and the Custom Metrics API
- Explore auto-scaling workloads using the Kubernetes Horizontal Pod Autoscaler (HPA) and Vertical Pod Autoscaler (VPA)
For this demo, the Kubernetes cluster is running a Deployment with a PHP demo application. There is an additional pod that will act as a client named “driver”. In addition to those pods, there is a Horizontal Pod Autoscaler that’s currently watching the Deployment. This HPA will scale up the workload when it’s resource threshold–in this case cpu–reaches or breaches a target of 50% usage. This HPA is set up to maintain at least one pod and will scale up to five pods depending on the current, or observed, usage versus the target threshold.
The application runs PHP code that calculates a million square roots each time a request is received on its listening port. This code is run as pods on the cluster maintained by a Deployment. To enable this workload to scale, the pod specification has resource request in the Deployment’s pod specification, which helps the HPA calculate a resource target to scale against.
In the demo, the driver pod generates load by running a for loop that sends a request to the Deployment’s Service. Once a request is received, the application calculates a million square roots, taking CPU calculation time. As this calculation is completed, the server responds with “OK!”
In the background, the Kubernetes Metrics Server contacts each of the kubelets, collecting cpu and memory metrics from those (gathered by the kubelet’s cAdvisor component). Once the Metric Server collects those metrics from the kubelets, it publishes those metrics to the Resource Metrics API. HPAs uses metrics stored in the Resource Metrics API to calculate the current cpu usage and compare it against a declared target.
When the HPA finds it has breached the target, it increases the desired replicas by mutating the “replicas” key of the target Deployment config. The Deployment’s ReplicaSet controller then requests additional pods to handle the increased load. After the scaling event, the Deployment now has five ready pods, five up to date pods, and five available pods. The HPA will only scale up to five max pods, so it will no longer scale up despite the continuous load.
Now that the workload is scaled up, we cancel the for loop from the “driver” pod. One would expect that the workload should scale down immediately, however in order to prevent inappropriate scaling up and scaling down in response to spiking workloads, the HPA waits (by default) five minutes before it allows the workload to scale down. This ensures that workloads maintain capacity in the face of intermittent bursts of activity.
Eventually the HPA mutates the Deployment again in response to the current resource use being well under the scaling target, decreasing the desired replicas and the ReplicaSet scales the pods down by requesting pods be deleted in last-in-first-out (LIFO) order, completing the demo. At this point, the demo shows how Kubernetes HPAs dynamically scale workloads based on metrics provided by Kubernetes itself!
This is a just sample of what you can learn in the Kubernetes Observability Overview module. You can create your own customized Kubernetes courses with this and many other modules using the courseware builder here on the RX-M website.
That’s our Cloud Native Short Take on the kubernetes Observability Overview module!