One-Way Pod Scaling with the Kubernetes HPA
(Jul. 30 2019) – Kubernetes has the ability to dynamically scale sets of pods based on resource usage. This is great for ensuring that applications always have the resources they need as loads vary. Autoscaling the number of pods in Kubernetes is most often accomplished using a Horizontal Pod Autoscaler (HPA). HPAs dynamically adjust the replica count of a pod controller, like a Deployment or a StatefulSet. Scaling is driven by metrics made available through the Kubernetes API by the Kubernetes metrics server and other third-party tools like Prometheus.
Metrics collected by Kubernetes can be used by autoscaler objects to evaluate resource usage against a configured threshold. If the resource usage exceeds or is significantly below the configured threshold, then the application deployments are scaled up or down accordingly.
While autoscalers do a great job of “automating the toil” in most situations, out-of-the-box autoscaling behavior may not be a perfect fit for all application types. For example, we often find the need to automate scale-out in stateful services but with the need to avoid automated scale-in. Scaling down a stateful set of pods implementing a clustered storage engine like Cassandra or Kafka could risk data loss, reduced availability, and other problems.
While one-way scaling could be implemented with a custom operator, a modified HPA or some other engineering shenanigans, we try to solve problems with plain vanilla Kubernetes features whenever possible. The resulting solutions tend to be simpler, easier to maintain and more cost-effective to implement; making our client’s lives easier down the road. For example, when it comes to one-way scaling, a postStart hook and a little bash shell scripting are all it takes.
This blog will walk you through the steps necessary to create a One-Way HPA.
1. Sample implementation Preparation
For simplicity, this example will utilize Alpine containers that have wget, grep and bash installed. Those three tools are central in our implementation of one-way pod scaling.
$ vim Dockerfile && cat Dockerfile
FROM alpine:latest
RUN apk add --no-cache wget grep bash
$Build the Dockerfile and tag the image rxmllc/hpamod, though feel free to use any other tag. Be sure to utilize that tag when preparing statefulSet specs. The regular Alpine docker image may be used as well.
$ docker build -t rxmllc/hpamod .
Sending build context to Docker daemon 10.24kB
Step 1/2 : FROM alpine:latest
---> 5cb3aa00f899
Step 2/2 : RUN apk add --no-cache wget grep bash
---> Running in 6fbbab8cf84a
fetch http://dl-cdn.alpinelinux.org/alpine/v3.9/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.9/community/x86_64/APKINDEX.tar.gz
(1/8) Installing ncurses-terminfo-base (6.1_p20190105-r0)
(2/8) Installing ncurses-terminfo (6.1_p20190105-r0)
(3/8) Installing ncurses-libs (6.1_p20190105-r0)
(4/8) Installing readline (7.0.003-r1)
(5/8) Installing bash (4.4.19-r1)
Executing bash-4.4.19-r1.post-install
(6/8) Installing pcre (8.42-r1)
(7/8) Installing grep (3.1-r2)
(8/8) Installing wget (1.20.1-r0)
Executing busybox-1.29.3-r10.trigger
OK: 15 MiB in 22 packages
Removing intermediate container 6fbbab8cf84a
---> c78866d22b68
Successfully built c78866d22b68
Successfully tagged rxmllc/hpamod:latest
$To begin, use the following spec to launch a 1-replica statefulSet using the alpine image built above:
$ vim statefulSet.yaml && cat statefulSet.yaml
apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
name: alpine-sts
spec:
serviceName: alpine-sts
replicas: 1
selector:
matchLabels:
run: alpine-sts
updateStrategy:
type: RollingUpdate
template:
metadata:
creationTimestamp: null
labels:
run: alpine-sts
spec:
containers:
- image: rxmllc/hpamod:latest
imagePullPolicy: Never
name: alpine-sts
command:
- /bin/sh
- -c
- tail -f /dev/null
$2. Getting the API calls
The ability to have pods communicate with the API Server and make effective requests is central to the one-way scaling approach. Pods themselves cannot benefit from the use of kubectl, but they can issue the same requests to the API that kubectl makes.
In a Kubernetes Cluster, apply the following HPA spec:
$ vim hpa.yaml && cat hpa.yaml
apiVersion: autoscaling/v2beta2
kind: HorizontalPodAutoscaler
metadata:
name: alpine-sts
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: StatefulSet
name: alpine-sts
maxReplicas: 3
minReplicas: 1
metrics:
- type: Resource
resource:
name: cpu
target:
type: Utilization
averageUtilization: 150
$ kubectl apply -f hpa.yaml
horizontalpodautoscaler.autoscaling/alpine-sts created
$If the Kubernetes Metrics Server or a similar solution has not been installed, this HPA will not perform any automatic scaling based on metrics. It can still be deployed, modified, and can even manipulate the number of replicas, just not against any real-time resource usage by the pod.
Changing the minReplicas parameter in the HPA is key to the whole solution. To do this, use kubectl patch to modify the current value.
Try utilizing kubectl patch to change the minimum replicas:
$ kubectl patch hpa alpine-sts -p '{"spec":{"minReplicas": 1}}'
horizontalpodautoscaler.autoscaling/alpine-sts patched (no change)
$That kubectl patch command targets the minReplicas parameter under the spec of the selected HPA. It uses a JSON map to pinpoint the exact parameter within the spec we need to modify.
Now, run this command to change the minReplicas count to 3, using the -v8 option to increase the verbosity:
$ kubectl patch hpa alpine-sts -p '{"spec":{"minReplicas": 1}}' -v8
I0312 10:03:59.395775 49368 loader.go:359] Config loaded from file /home/user/.kube/config
I0312 10:03:59.396310 49368 loader.go:359] Config loaded from file /home/user/.kube/config
I0312 10:03:59.398065 49368 loader.go:359] Config loaded from file /home/user/.kube/config
I0312 10:03:59.402645 49368 loader.go:359] Config loaded from file /home/user/.kube/config
I0312 10:03:59.404345 49368 round_trippers.go:383] GET https://192.168.229.133:6443/apis/autoscaling/v1/namespaces/default/horizontalpodautoscalers/alpine-sts
I0312 10:03:59.404381 49368 round_trippers.go:390] Request Headers:
I0312 10:03:59.404395 49368 round_trippers.go:393] Accept: application/json
I0312 10:03:59.404411 49368 round_trippers.go:393] User-Agent: kubectl/v1.12.2 (linux/amd64) kubernetes/17c77c7
I0312 10:03:59.412554 49368 round_trippers.go:408] Response Status: 200 OK in 8 milliseconds
I0312 10:03:59.412592 49368 round_trippers.go:411] Response Headers:
I0312 10:03:59.412597 49368 round_trippers.go:414] Content-Length: 1567
I0312 10:03:59.412600 49368 round_trippers.go:414] Date: Tue, 12 Mar 2019 17:03:59 GMT
I0312 10:03:59.412603 49368 round_trippers.go:414] Content-Type: application/json
I0312 10:03:59.412651 49368 request.go:942] Response Body: {"kind":"HorizontalPodAutoscaler","apiVersion":"autoscaling/v1","metadata":{"name":"alpine-sts","namespace":"default","selfLink":"/apis/autoscaling/v1/namespaces/default/horizontalpodautoscalers/alpine-sts","uid":"5e3a470f-44dd-11e9-8b9b-000c294ee651","resourceVersion":"1072382","creationTimestamp":"2019-03-12T15:41:57Z","annotations":{"autoscaling.alpha.kubernetes.io/conditions":"[{\"type\":\"AbleToScale\",\"status\":\"True\",\"lastTransitionTime\":\"2019-03-12T15:42:12Z\",\"reason\":\"SucceededGetScale\",\"message\":\"the HPA controller was able to get the target's current scale\"},{\"type\":\"ScalingActive\",\"status\":\"False\",\"lastTransitionTime\":\"2019-03-12T15:42:12Z\",\"reason\":\"FailedGetResourceMetric\",\"message\":\"the HPA was unable to compute the replica count: missing request for cpu\"}]","kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"autoscaling/v2beta2\",\"kind\":\"HorizontalPodAutoscaler\",\"metadata\":{\"annotations\":{},\"name\":\"alpine-sts\",\"namespace\":\"def [truncated 543 chars]
I0312 10:03:59.413359 49368 loader.go:359] Config loaded from file /home/user/.kube/config
I0312 10:03:59.413583 49368 request.go:942] Request Body: {"spec":{"minReplicas": 1}}
I0312 10:03:59.413637 49368 round_trippers.go:383] PATCH https://192.168.229.133:6443/apis/autoscaling/v1/namespaces/default/horizontalpodautoscalers/alpine-sts
I0312 10:03:59.413644 49368 round_trippers.go:3] Request Headers:
I0312 10:03:59.413647 49368 round_trippers.go:393] Accept: application/json
I0312 10:03:59.413650 49368 round_trippers.go:393] Content-Type: application/strategic-merge-patch+json
I0312 10:03:59.413653 49368 round_trippers.go:393] User-Agent: kubectl/v1.12.2 (linux/amd64) kubernetes/17c77c7
I0312 10:03:59.415818 49368 round_trippers.go:408] Response Status: 200 OK in 2 milliseconds
I0312 10:03:59.415831 49368 round_trippers.go:411] Response Headers:
I0312 10:03:59.415835 49368 round_trippers.go:414] Content-Type: application/json
I0312 10:03:59.415837 49368 round_trippers.go:414] Content-Length: 1567
I0312 10:03:59.415840 49368 round_trippers.go:414] Date: Tue, 12 Mar 2019 17:03:59 GMT
I0312 10:03:59.415863 49368 request.go:942] Response Body: {"kind":"HorizontalPodAutoscaler","apiVersion":"autoscaling/v1","metadata":{"name":"alpine-sts","namespace":"default","selfLink":"/apis/autoscaling/v1/namespaces/default/horizontalpodautoscalers/alpine-sts","uid":"5e3a470f-44dd-11e9-8b9b-000c294ee651","resourceVersion":"1072382","creationTimestamp":"2019-03-12T15:41:57Z","annotations":{"autoscaling.alpha.kubernetes.io/conditions":"[{\"type\":\"AbleToScale\",\"status\":\"True\",\"lastTransitionTime\":\"2019-03-12T15:42:12Z\",\"reason\":\"SucceededGetScale\",\"message\":\"the HPA controller was able to get the target's current scale\"},{\"type\":\"ScalingActive\",\"status\":\"False\",\"lastTransitionTime\":\"2019-03-12T15:42:12Z\",\"reason\":\"FailedGetResourceMetric\",\"message\":\"the HPA was unable to compute the replica count: missing request for cpu\"}]","kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"autoscaling/v2beta2\",\"kind\":\"HorizontalPodAutoscaler\",\"metadata\":{\"annotations\":{},\"name\":\"alpine-sts\",\"namespace\":\"def [truncated 543 chars]
horizontalpodautoscaler.autoscaling/alpine-sts patched (no change)
$There were two calls made when running the kubectl patch command above. Running with level 8 verbosity has given the following information about each call:
- For the GET request:
- URL: https://192.168.229.133:6443/apis/autoscaling/v1/namespaces/default/horizontalpodautoscalers/alpine-sts
- The following headers were provided:
- Accept: application/json
- User-Agent: kubectl/v1.12.2 (linux/amd64) kubernetes/17c77c7
- For the PATCH request:
- URL: https://192.168.229.133:6443/apis/autoscaling/v1/namespaces/default/horizontalpodautoscalers/alpine-sts
- The following headers were provided:
- Accept: application/json
- Content-Type: application/strategic-merge-patch+json
- User-Agent: kubectl/v1.12.2 (linux/amd64) kubernetes/17c77c7
- And some body data:
- {“spec”:{“minReplicas”: 3}}
By running kubectl patch with increased verbosity, we were able to determine the structure of the API request that the pod needs to run in order to modify the HPA.
3. Getting permission: Handling Kubernetes RBAC elements
There are some role-based access control elements that need to be in place to allow the pod to make the direct API call.
Take a look at the following spec:
$ vim rbac.yaml && cat rbac.yaml
kind: Role
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
namespace: default
name: hpa-patcher
rules:
- apiGroups: ["*"]
resources: ["horizontalpodautoscalers"]
verbs:
- patch
- get
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
name: patch-hpa
namespace: default
subjects:
- kind: ServiceAccount
name: default
namespace: default
roleRef:
kind: Role
name: hpa-patcher
apiGroup: rbac.authorization.k8s.io
$ In this spec, two elements are defined: A role and a rolebinding. Both elements combined make up the RBAC portion of the autoscaling equation: giving a pod permission to make a request equivalent to kubectl patch.
The role, hpa-patcher, will be created in the default namespace. The hpa-patcher role will grant any users that receive it the ability to run PATCH and GET requests against a specific resource: HorizontalPodAutoscalers, in this case.
For one-way autoscaling, the ability to GET information about the existing HPA and the ability to PATCH it are necessary, so those are the only verbs (capabilities) the role will have. It’s good practice to be specific with role permissions: only assign roles the exact permissions needed to fulfill an operation.
The role binding will tie the hpa-patcher role to a user. In this case, it will assign it to the default namespace’s ServiceAccount – a universal account (within a namespace) that all pods within that namespace inherit. By inheriting the ServiceAccount, each pod will gain access to that ServiceAccount’s certificate, access token and namespace information: the components required to make a valid call to the API server.
Apply the rbac.yaml to your cluster:
$ kubectl apply -f rbac.yaml
role.rbac.authorization.k8s.io/hpa-patcher created
rolebinding.rbac.authorization.k8s.io/patch-hpa created
$The default namespace’s ServiceAccount now has the GET and PATCH permissions against any HorizontalPodAutoscaler objects.
4. Creating the API call
Now that the permissions are in place to modify HPAs, it’s time to formulate the API call. This will be done using the URL that we collected from kubectl patch. The call needs to be made from within the cluster from one of the application’s pods.
Exec into one of the statefulset pods:
$ kubectl exec -it alpine-sts-0 -- sh
/ #If you are using the public Alpine container, use
apk add wgetto retrieve a more current version of wget. The one that it is deployed with may not be able to pass all the headers and other flags needed for this solution.
First, use wget to try to access the same URL that was provided by Kubectl:
/ # wget -O - https://192.168.229.133:6443/apis/autoscaling/v1/namespaces/default/horizontalpodautoscalers/alpine-sts \
--2019-03-12 15:24:57-- https://192.168.229.133:6443/apis/autoscaling/v1/namespaces/default/horizontalpodautoscalers/alpine-sts
Connecting to 192.168.229.133:6443... connected.
ERROR: cannot verify 192.168.229.133's certificate, issued by 'CN=kubernetes':
Unable to locally verify the issuer's authority.
To connect to 192.168.229.133 insecurely, use `--no-check-certificate'.
/ #Since wget will download a file by default to the container’s filesystem, use the -O - to output the results to STDOUT.
In order to authenticate itself, the pod must present a valid certificate to the API server. Pods inherit authentication and authorization elements from their namespace’s service account. These elements are mounted to /var/run/secrets/kubernetes.io/serviceaccount/ within the container’s filesystem.
Inspect the /var/run/secrets/kubernetes.io/serviceaccount/ directory within the pod:
/ # ls -l /var/run/secrets/kubernetes.io/serviceaccount/
total 0
lrwxrwxrwx 1 root root 13 Mar 12 15:20 ca.crt -> ..data/ca.crt
lrwxrwxrwx 1 root root 16 Mar 12 15:20 namespace -> ..data/namespace
lrwxrwxrwx 1 root root 12 Mar 12 15:20 token -> ..data/token
/ #Each container inside a pod has the cluster CA certificate, a file containing the namespace name, and an authorization token. The certificate will allow the pod to authenticate against the API server and have the request be accepted, and the token will ensure that the Pod will perform the action as the intended user – the service account.
The token is written in a long, encrypted string. For the sake of this exercise (and reusability), it will be stored inside the variable TOKEN.
/ # TOKEN=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)
/ # echo $TOKEN
eyJhbGciOiJSUzI1NiIsImtpZCI6IiJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3V...
/ #To utilize these elements, they are attached to the request URL as headers, which the API server will read before taking any action on the request body.
Append the CA certificate and the authentication token to the wget command as headers:
/ # wget -O - https://192.168.229.133:6443/apis/autoscaling/v1/namespaces/default/horizontalpodautoscalers/alpine-sts \
> --ca-certificate=/var/run/secrets/kubernetes.io/serviceaccount/ca.crt --header="Authorization: Bearer $TOKEN"
--2019-03-12 15:46:17-- https://192.168.229.133:6443/apis/autoscaling/v1/namespaces/default/horizontalpodautoscalers/alpine-sts
Connecting to 192.168.229.133:6443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1685 (1.6K) [application/json]
Saving to: 'STDOUT'
- 0%[ ] 0 --.-KB/s {
"kind": "HorizontalPodAutoscaler",
"apiVersion": "autoscaling/v1",
"metadata": {
"name": "alpine-sts",
"namespace": "default",
"selfLink": "/apis/autoscaling/v1/namespaces/default/horizontalpodautoscalers/alpine-sts",
"uid": "5e3a470f-44dd-11e9-8b9b-000c294ee651",
"resourceVersion": "1066323",
"creationTimestamp": "2019-03-12T15:41:57Z",
"annotations": {
"autoscaling.alpha.kubernetes.io/conditions": "[{\"type\":\"AbleToScale\",\"status\":\"True\",\"lastTransitionTime\":\"2019-03-12T15:42:12Z\",\"reason\":\"SucceededGetScale\",\"message\":\"the HPA controller was able to get the target's current scale\"},{\"type\":\"ScalingActive\",\"status\":\"False\",\"lastTransitionTime\":\"2019-03-12T15:42:12Z\",\"reason\":\"FailedGetResourceMetric\",\"message\":\"the HPA was unable to compute the replica count: missing request for cpu\"}]",
"kubectl.kubernetes.io/last-applied-configuration": "{\"apiVersion\":\"autoscaling/v2beta2\",\"kind\":\"HorizontalPodAutoscaler\",\"metadata\":{\"annotations\":{},\"name\":\"alpine-sts\",\"namespace\":\"default\"},\"spec\":{\"maxReplicas\":3,\"metrics\":[{\"resource\":{\"name\":\"cpu\",\"target\":{\"averageUtilization\":150,\"type\":\"Utilization\"}},\"type\":\"Resource\"}],\"minReplicas\":1,\"scaleTargetRef\":{\"apiVersion\":\"apps/v1\",\"kind\":\"StatefulSet\",\"name\":\"alpine-sts\"}}}\n"
}
},
"spec": {
"scaleTargetRef": {
"kind": "StatefulSet",
"name": "alpine-sts",
"apiVersion": "apps/v1"
},
"minReplicas": 1,
"maxReplicas": 3,
"targetCPUUtilizationPercentage": 150
},
"status": {
"currentReplicas": 1,
"desiredReplicas": 0
}
- 100%[========================================================================================================================================================================>] 1.65K --.-KB/s in 0s
2019-03-12 15:46:17 (119 MB/s) - written to stdout [1685/1685]
/ #--ca-certificatedeclares the path to the certificate file that will be used for authentication.--header="Authorization: Bearer $TOKEN"will present a valid login from the pod (as the Service Account)
Running that command performed a GET request on the horizontal pod autoscalers. The type of request is also determined by a header, which wasn’t provided in this first request. Since it was successful, it also indicates that one of the pods within the StatefulSet can successfully GET information about HPAs within its namespace.
If you receive a 403 error, make sure you applied the RBAC elements shown in the previous step to the Kubernetes cluster. Also, make sure that the HPA from the spec above exists, or else the command may throw a 404. Remember that a service account has no permissions to do anything until it is assigned some through a role.
Now it’s time to attempt a PATCH action by adding '{"spec":{"minReplicas": 1}}' to the request body. For wget, the equivalent to the curl --data flag is --body-data. Because PATCH is a more complex request, a media type must be provided. In the original CURL command, that was provided by the content-type header: "Content-Type: application/strategic-merge-patch+json"
Append that to the command as another header:
/ # wget -O - https://192.168.229.133:6443/apis/autoscaling/v1/namespaces/default/horizontalpodautoscalers/alpine-sts -v \
--ca-certificate=/var/run/secrets/kubernetes.io/serviceaccount/ca.crt --header="Authorization: Bearer $TOKEN" \
--method=PATCH --body-data '{"spec":{"minReplicas": 1}}' \
--header="Accept: application/json, */*" --header="Content-Type: application/strategic-merge-patch+json"
--2019-03-12 17:13:28-- https://192.168.229.133:6443/apis/autoscaling/v1/namespaces/default/horizontalpodautoscalers/alpine-sts
Connecting to 192.168.229.133:6443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1730 (1.7K) [application/json]
Saving to: 'STDOUT'
- 0%[ ] 0 --.-KB/s {
"kind": "HorizontalPodAutoscaler",
"apiVersion": "autoscaling/v1",
"metadata": {
"name": "alpine-sts",
"namespace": "default",
"selfLink": "/apis/autoscaling/v1/namespaces/default/horizontalpodautoscalers/alpine-sts",
"uid": "5e3a470f-44dd-11e9-8b9b-000c294ee651",
"resourceVersion": "1073070",
"creationTimestamp": "2019-03-12T15:41:57Z",
"annotations": {
"autoscaling.alpha.kubernetes.io/conditions": "[{\"type\":\"AbleToScale\",\"status\":\"True\",\"lastTransitionTime\":\"2019-03-12T15:42:12Z\",\"reason\":\"SucceededGetScale\",\"message\":\"the HPA controller was able to get the target's current scale\"},{\"type\":\"ScalingActive\",\"status\":\"False\",\"lastTransitionTime\":\"2019-03-12T15:42:12Z\",\"reason\":\"FailedGetResourceMetric\",\"message\":\"the HPA was unable to compute the replica count: missing request for cpu\"}]",
"kubectl.kubernetes.io/last-applied-configuration": "{\"apiVersion\":\"autoscaling/v2beta2\",\"kind\":\"HorizontalPodAutoscaler\",\"metadata\":{\"annotations\":{},\"name\":\"alpine-sts\",\"namespace\":\"default\"},\"spec\":{\"maxReplicas\":3,\"metrics\":[{\"resource\":{\"name\":\"cpu\",\"target\":{\"averageUtilization\":150,\"type\":\"Utilization\"}},\"type\":\"Resource\"}],\"minReplicas\":1,\"scaleTargetRef\":{\"apiVersion\":\"apps/v1\",\"kind\":\"StatefulSet\",\"name\":\"alpine-sts\"}}}\n"
}
},
"spec": {
"scaleTargetRef": {
"kind": "StatefulSet",
"name": "alpine-sts",
"apiVersion": "apps/v1"
},
"minReplicas": 1,
"maxReplicas": 3,
"targetCPUUtilizationPercentage": 150
},
"status": {
"lastScaleTime": "2019-03-12T16:59:50Z",
"currentReplicas": 1,
"desiredReplicas": 3
}
- 100%[============================================================================================================>] 1.69K --.-KB/s in 0s
2019-03-12 17:13:28 (101 MB/s) - written to stdout [1730/1730]
/ #A 200 OK response means the command was successful, though since no change was provided
Now, try to change the body to edit the replicas to 3:
/ # wget -O - https://192.168.229.133:6443/apis/autoscaling/v1/namespaces/default/horizontalpodautoscalers/alpine-sts -v \
--ca-certificate=/var/run/secrets/kubernetes.io/serviceaccount/ca.crt --header="Authorization: Bearer $TOKEN" \
--method=PATCH --body-data '{"spec":{"minReplicas": 3}}' \
--header="Accept: application/json, */*" --header="Content-Type: application/strategic-merge-patch+json"
--2019-03-12 17:15:23-- https://192.168.229.133:6443/apis/autoscaling/v1/namespaces/default/horizontalpodautoscalers/alpine-sts
Connecting to 192.168.229.133:6443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1730 (1.7K) [application/json]
Saving to: 'STDOUT'
- 0%[ ] 0 --.-KB/s {
"kind": "HorizontalPodAutoscaler",
"apiVersion": "autoscaling/v1",
"metadata": {
"name": "alpine-sts",
...
"spec": {
"scaleTargetRef": {
"kind": "StatefulSet",
"name": "alpine-sts",
"apiVersion": "apps/v1"
},
"minReplicas": 3,
...
2019-03-12 17:15:23 (10.2 MB/s) - written to stdout [1730/1730]
/ #The pod can now gather information on and update HPAs using a request equivalent to kubectl patch.
For reusability, we parameterized each request as much as possible. The first thing that can be done is to replace the IP and port of the machine that was previously being used to kubernetes.default.svc. That is a cluster DNS entry that any internal resource (pod, HPA, etc.) can use to communicate back to the API server. This ensures that regardless of where this script is deployed with the pod, it will reach the API server.
/ # wget -O - https://kubernetes.default.svc/apis/autoscaling/v1/namespaces/default/horizontalpodautoscalers/alpine-sts \
--ca-certificate=/var/run/secrets/kubernetes.io/serviceaccount/ca.crt --header="Authorization: Bearer $TOKEN"
--2019-03-12 17:28:35-- https://kubernetes.default.svc/apis/autoscaling/v1/namespaces/default/horizontalpodautoscalers/alpine-sts
Resolving kubernetes.default.svc... 10.96.0.1
Connecting to kubernetes.default.svc|10.96.0.1|:443... connected.
HTTP request sent, awaiting response... 200 OK
...
/ #Next, environment variables can be used to make the request URL more flexible for the following elements:
- Namespace – As shown above, this is provided to each container in a pod by the serviceAccount
- HPA name – In this example, both the statefulSet and HPA share a name, so it can be retrieved from the pod by removing the ordinal with the following expression:
cat /etc/hostname | sed -E 's/-[0-9]*$//'. It willcatthe hostname file of the pod, and use SED to remove the ordinal number at the end. - CACERT path – another path to be used to point to the or to obscure it by using an environment variable
/ # NAMESPACE=$(cat /var/run/secrets/kubernetes.io/serviceaccount/namespace)
/ # HPA=$(cat /etc/hostname | sed -E 's/-[0-9]*$//')
/ # CACERT=/var/run/secrets/kubernetes.io/serviceaccount/ca.crt
/ # wget -O - https://kubernetes.default.svc/apis/autoscaling/v1/namespaces/"$NAMESPACE"/horizontalpodautoscalers/"$HPA" \
--ca-certificate=$CACERT --header="Authorization: Bearer $TOKEN" -v
--2019-03-12 17:33:29-- https://kubernetes.default.svc/apis/autoscaling/v1/namespaces/default/horizontalpodautoscalers/alpine-sts
Resolving kubernetes.default.svc... 10.96.0.1
Connecting to kubernetes.default.svc|10.96.0.1|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1730 (1.7K) [application/json]
Saving to: 'STDOUT'
...
/ #By parameterizing the request headers and using the internal API Server dns address, any pods that share the HPA name (or part of it) can get information about the HPA and modify it too.
5. Scripting the API calls
Now that the API calls have been formulated, it’s time to tie them together with some logic. This One-way scaling technique uses the following logic:
- The newest pod in a statefulSet updates the HPA minimum replica value based on its ordinal value + 1 (starting from 0)
- If the new replica value is below the current HPA minimum, then the number will not change; the new value must be greater. This prevents a pod with a lower ordinal value from setting the HPA minimum too low and causing unwanted scale down.
Following that logic, the scaling action must:
- Retrieve (GET) the current HPA minimum replicas value
- Calculate the new minimum replica value from the pod name
- Evaluate whether the new minimum replica value is greater than the current
- If it is higher, use the newly calculated value
- If it is lower, use the current minimum replica value
Now write a script around that logic:
$ vim oneway-scaler.sh && cat oneway-scaler.sh
#!/bin/bash
set -xe
# Retrieve the pod's stateful set name from the pod hostname and set the HPA variable
HPA=$(cat /etc/hostname | sed -E 's/-[0-9]*$//')
# Set variables for the namespace's Service Account token
NAMESPACE=$(cat /var/run/secrets/kubernetes.io/serviceaccount/namespace)
CACERT=/var/run/secrets/kubernetes.io/serviceaccount/ca.crt
TOKEN=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)
# Calculate the new minimum replication value from the pod's ordinal number from the host name.
NEWMINREP=$(expr $(cat /etc/hostname | sed 's/.*-//') + 1)
# Retrieve the current minimum replica value from the HPA
CURHPAMIN=$(wget -O - https://kubernetes.default.svc/apis/autoscaling/v1/namespaces/$NAMESPACE/horizontalpodautoscalers/$HPA \
--ca-certificate=$CACERT --method=GET \
--header="Accept: application/json, */*" \
--header="Authorization: Bearer $TOKEN" \
| awk '/"minReplicas"/{match($0, /"minReplicas"/); print substr($0, RSTART - 1, RLENGTH + 15)}' | grep -oP '\d*')
# Decide which minimum replica value to use.
# If the new count calculated from the pod ordinal + 1 is higher, it will use that value.
# Otherwise, it will use the current HPA minimum to prevent inappropriate scaling
if [ "$NEWMINREP" -gt "$CURHPAMIN" ]
then
MINREP="$NEWMINREP"
else
MINREP="$CURHPAMIN"
fi
# Execute an api request to update the HPA, which is named the same as the stateful set, with the calculated minimum replication value.
wget -O - https://kubernetes.default.svc/apis/autoscaling/v1/namespaces/$NAMESPACE/horizontalpodautoscalers/$HPA \
--ca-certificate=$CACERT --method=PATCH \
--header="Accept: application/json, */*" \
--header="Content-Type: application/strategic-merge-patch+json" \
--header="Authorization: Bearer $TOKEN" \
--body-data '{"spec":{"minReplicas":'${MINREP}'}}'Note that script is using
bashandgrep. If you are using the publicly available Alpine image, useapk add --no-cache bash grepto update them.
To achieve the logical goals, the script does the following:
- Gathers environment variables HPA, NAMESPACE, CACERT and TOKEN
- Use
catandsedto query the hostname and grab the ordinal number, then usesexprto increment it plus one before storing it inNEWMINREP - Issues a GET request on the HPA and uses
awkandgrepto pull the value from “minReplicas” and stores it inCURHPAMIN - Evaluates whether
NEWMINREPis greater thanCURHPAMINand inserts the selected value intoMINREP - Issues the PATCH request to the API using the
MINREPvalue to update the HPA minimum replica count
The easiest way to distribute this script to pods is as a Configmap. A configmap can be mounted to Kubernetes pods as a volume, which makes its contents (such as a file) available from within the pod’s filesystem.
Create a configMap from the script by using kubectl create --from-file oneway-scaler.sh:
$ kubectl create configmap oneway-scale --from-file oneway-scaler.sh
$ kubectl get configmap oneway-scale -o yaml
apiVersion: v1
data:
oneway-scaler.sh: "#!/bin/bash\n\nset -xe\n\n# Print the pod's stateful set name
from the pod hostname\nHPA=$(expr $(cat /etc/hostname | sed -E 's/-[0-9]*$//'))\n\n#
Set variables for the namespace's Service Account token\nNAMESPACE=$(cat /var/run/secrets/kubernetes.io/serviceaccount/namespace)\nCACERT=/var/run/secrets/kubernetes.io/serviceaccount/ca.crt\nTOKEN=$(cat
/var/run/secrets/kubernetes.io/serviceaccount/token)\n\n# Calculate the new minimum
replication value from the pod's ordinal number from the host name.\nNEWMINREP=$(expr
$(cat /etc/hostname | sed 's/.*-//') + 1)\n\n# Retrieve the current minimum replica
value from the HPA\nCURHPAMIN=$(wget -O - https://kubernetes.default.svc/apis/autoscaling/v1/namespaces/$NAMESPACE/horizontalpodautoscalers/$HPA
\\\n--ca-certificate=$CACERT --method=GET \\\n--header=\"Accept: application/json,
*/*\" \\\n--header=\"Authorization: Bearer $TOKEN\" \\\n| awk '/\"minReplicas\"/{match($0,
/\"minReplicas\"/); print substr($0, RSTART - 1, RLENGTH + 15)}' | grep -oP '\\d*')\n\n#
Decide which minimum replica value to use.\n# If the new count calculated from
the pod ordinal + 1 is higher, it will use that value.\n# Otherwise, it will use
the current HPA minimum to prevent inappropriate scaling\nif [ \"$NEWMINREP\"
-gt \"$CURHPAMIN\" ]\nthen\n\tMINREP=\"$NEWMINREP\"\nelse\n\tMINREP=\"$CURHPAMIN\"\nfi\n\n#
Execute an api request to update the HPA, which is named the same as the stateful
set, with the calculated minimum replication value.\nwget -O - https://kubernetes.default.svc/apis/autoscaling/v1/namespaces/$NAMESPACE/horizontalpodautoscalers/$HPA
\\\n--ca-certificate=$CACERT --method=PATCH \\\n--header=\"Accept: application/json,
*/*\" \\\n--header=\"Content-Type: application/strategic-merge-patch+json\" \\\n--header=\"Authorization:
Bearer $TOKEN\" \\\n--body-data '{\"spec\":{\"minReplicas\":'${MINREP}'}}'\n"
kind: ConfigMap
metadata:
creationTimestamp: 2019-03-12T00:35:39Z
name: oneway-scale
namespace: default
resourceVersion: "1058186"
selfLink: /api/v1/namespaces/default/configmaps/oneway-scale
uid: c2dc15c3-445e-11e9-8b9b-000c294ee651
$The configMap, and the script within, are now available for our StatefulSet pods to use.
6. Mounting and Executing the script
Now it’s time to tie things together. Add the following elements to the StatefulSet container spec:
lifecycle:
postStart:
exec:
command:
- /bin/sh
- -c
- ./utils/oneway-scaler.sh
volumeMounts:
- name: hpa-script
mountPath: /utils
volumes:
- name: hpa-script
configMap:
name: oneway-scale
defaultMode: 0777 - The
oneway-scaleconfigmap is defined as a mountable volume calledhpa-scriptin this spec. It will mount the script as a universally executable file inside the alpine container’s filesystem. - Then, the
hpa-scriptvolume defined in the volumes section is mounted to the/utilsdirectory inside the container. - To call the script, a postStart hook will execute a command immediately after the pod’s containers are created.
- This command will first ensure that wget, bash and grep – which are used by the script, are present and up to date.
- It will then run the script by calling it as a user would from inside the container.
The completed StatefulSet spec should now look as below:
$ vim alpine-sts.yaml && cat alpine-sts.yaml
apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
name: alpine-sts
spec:
serviceName: alpine-sts
replicas: 1
selector:
matchLabels:
run: alpine-sts
updateStrategy:
type: RollingUpdate
template:
metadata:
creationTimestamp: null
labels:
run: alpine-sts
spec:
containers:
- image: rxmllc/hpamod:latest
imagePullPolicy: Never
name: alpine-sts
command:
- /bin/sh
- -c
- tail -f /dev/null
lifecycle:
postStart:
exec:
command:
- /bin/sh
- -c
- ./utils/oneway-scaler.sh
volumeMounts:
- name: hpa-script
mountPath: /utils
volumes:
- name: hpa-script
configMap:
name: oneway-scale
defaultMode: 0777
$ All the elements are in place – RBAC, API calls, and the script are ready to be executed by the pod.
7. Testing the solution
Reconfigure the alpine-sts statefulset by applying the new, completed spec file:
$ kubectl apply -f alpine-sts.yaml
statefulset.apps/alpine-sts configured
$The statefulSet has been updated. Now, as pods come up, they will have the oneway-scaler.sh script present within their filesystems.
Use kubectl exec into one of the Pods and check if the oneway-scaler script is present:
$ kubectl exec -it alpine-sts-0 -- sh
/ #
/ # ls -l /utils
total 0
lrwxrwxrwx 1 root root 23 Mar 12 18:55 oneway-scaler.sh -> ..data/oneway-scaler.sh
/ #The script is in place, per the volume mapping.
Now, run the script manually:
/ # ./utils/oneway-scaler.sh
+++ cat /etc/hostname
+++ sed -E 's/-[0-9]*$//'
+ HPA=alpine-sts
++ cat /var/run/secrets/kubernetes.io/serviceaccount/namespace
+ NAMESPACE=default
+ CACERT=/var/run/secrets/kubernetes.io/serviceaccount/ca.crt
++ cat /var/run/secrets/kubernetes.io/serviceaccount/token
+ TOKEN=eyJhbGciOiJSUzI1NiIsImtpZCI6IiJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJkZWZhdWx0Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6ImRlZmF1bHQtdG9rZW4tOTRkOWIiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoiZGVmYXVsdCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6IjRkNjUxMjg4LWUxMzktMTFlOC1iMWE2LTAwMGMyOTRlZTY1MSIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDpkZWZhdWx0OmRlZmF1bHQifQ.vKxVaNWELJU4UO7DmLOd7Qbf_K9Jdtgt_3uRP3AEtVHzBysy1o9PEYxtT1djJA8UMnEYz2dgorDUjIdXNAwy6TVpo1SjXf4jtEMBQU4q64PQLgDRqw8P0E33T2hfzZTVxiZscrM8SpyQtgqVD1y0syveEGAUcWtGxMu5vihaZdxT_jNUgrvOpX3gNps36Ku7EYDf6lzG7lfH3krdenvuffDSASt3kvSirYPrTd-VpNftj8xTEbhkX-nZgGUdXTTBar9hOtx7zlS9kj9oYb7gcs99kVxcGkyjwrV33jzi3U5Hf5w8nHdnpQA9CuOxVYYD3vJD9B0o9uTAdjsCLuRPtQ
+++ cat /etc/hostname
+++ sed 's/.*-//'
++ expr 0 + 1
+ NEWMINREP=1
++ wget -O - https://kubernetes.default.svc/apis/autoscaling/v1/namespaces/default/horizontalpodautoscalers/alpine-sts --ca-certificate=/var/run/secrets/kubernetes.io/serviceaccount/ca.crt --method=GET '--header=Accept: application/json, */*' ++ '--header=Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IiJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJkZWZhdWx0Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6ImRlZmF1bHQtdG9rZW4tOTRkOWIiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoiZGVmYXVsdCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6IjRkNjUxMjg4LWUxMzktMTFlOC1iMWE2LTAwMGMyOTRlZTY1MSIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDpkZWZhdWx0OmRlZmF1bHQifQ.vKxVaNWELJU4UO7DmLOd7Qbf_K9Jdtgt_3uRP3AEtVHzBysy1o9PEYxtT1djJA8UMnEYz2dgorDUjIdXNAwy6TVpo1SjXf4jtEMBQU4q64PQLgDRqw8P0E33T2hfzZTVxiZscrM8SpyQtgqVD1y0syveEGAUcWtGxMu5vihaZdxT_jNUgrvOpX3gNps36Ku7EYDf6lzG7lfH3krdenvuffDSASt3kvSirYPrTd-VpNftj8xTEbhkX-nZgGUdXTTBar9hOtx7zlS9kj9oYb7gcs99kVxcGkyjwrV33jzi3U5Hf5w8nHdnpQA9CuOxVYYD3vJD9B0o9uTAdjsCLuRPtQ'awk
'/"minReplicas"/{match($0, /"minReplicas"/); print substr($0, RSTART - 1, RLENGTH + 15)}'
++ grep -oP '\d*'
--2019-03-12 20:31:00-- https://kubernetes.default.svc/apis/autoscaling/v1/namespaces/default/horizontalpodautoscalers/alpine-sts
Resolving kubernetes.default.svc... 10.96.0.1
Connecting to kubernetes.default.svc|10.96.0.1|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1685 (1.6K) [application/json]
Saving to: 'STDOUT'
- 100%[============================================================================================================>] 1.65K --.-KB/s in 0s
2019-03-12 20:31:00 (25.2 MB/s) - written to stdout [1685/1685]
+ CURHPAMIN=3
+ '[' 1 -gt 3 ']'
+ MINREP=3
+ wget -O - https://kubernetes.default.svc/apis/autoscaling/v1/namespaces/default/horizontalpodautoscalers/alpine-sts --ca-certificate=/var/run/secrets/kubernetes.io/serviceaccount/ca.crt --method=PATCH '--header=Accept: application/json, */*' '--header=Content-Type: application/strategic-merge-patch+json' '--header=Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IiJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJkZWZhdWx0Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6ImRlZmF1bHQtdG9rZW4tOTRkOWIiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoiZGVmYXVsdCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6IjRkNjUxMjg4LWUxMzktMTFlOC1iMWE2LTAwMGMyOTRlZTY1MSIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDpkZWZhdWx0OmRlZmF1bHQifQ.vKxVaNWELJU4UO7DmLOd7Qbf_K9Jdtgt_3uRP3AEtVHzBysy1o9PEYxtT1djJA8UMnEYz2dgorDUjIdXNAwy6TVpo1SjXf4jtEMBQU4q64PQLgDRqw8P0E33T2hfzZTVxiZscrM8SpyQtgqVD1y0syveEGAUcWtGxMu5vihaZdxT_jNUgrvOpX3gNps36Ku7EYDf6lzG7lfH3krdenvuffDSASt3kvSirYPrTd-VpNftj8xTEbhkX-nZgGUdXTTBar9hOtx7zlS9kj9oYb7gcs99kVxcGkyjwrV33jzi3U5Hf5w8nHdnpQA9CuOxVYYD3vJD9B0o9uTAdjsCLuRPtQ' --body-data '{"spec":{"minReplicas":3}}'
--2019-03-12 20:31:00-- https://kubernetes.default.svc/apis/autoscaling/v1/namespaces/default/horizontalpodautoscalers/alpine-sts
Resolving kubernetes.default.svc... 10.96.0.1
Connecting to kubernetes.default.svc|10.96.0.1|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1685 (1.6K) [application/json]
Saving to: 'STDOUT'
- 0%[ ] 0 --.-KB/s {
"kind": "HorizontalPodAutoscaler",
"apiVersion": "autoscaling/v1",
"metadata": {
"name": "alpine-sts",
"namespace": "default",
"selfLink": "/apis/autoscaling/v1/namespaces/default/horizontalpodautoscalers/alpine-sts",
"uid": "2c8743c9-44f8-11e9-8b9b-000c294ee651",
"resourceVersion": "1081301",
"creationTimestamp": "2019-03-12T18:53:50Z",
"annotations": {
"autoscaling.alpha.kubernetes.io/conditions": "[{\"type\":\"AbleToScale\",\"status\":\"True\",\"lastTransitionTime\":\"2019-03-12T18:54:05Z\",\"reason\":\"SucceededGetScale\",\"message\":\"the HPA controller was able to get the target's current scale\"},{\"type\":\"ScalingActive\",\"status\":\"False\",\"lastTransitionTime\":\"2019-03-12T18:54:05Z\",\"reason\":\"FailedGetResourceMetric\",\"message\":\"the HPA was unable to compute the replica count: missing request for cpu\"}]",
"kubectl.kubernetes.io/last-applied-configuration": "{\"apiVersion\":\"autoscaling/v2beta2\",\"kind\":\"HorizontalPodAutoscaler\",\"metadata\":{\"annotations\":{},\"name\":\"alpine-sts\",\"namespace\":\"default\"},\"spec\":{\"maxReplicas\":3,\"metrics\":[{\"resource\":{\"name\":\"cpu\",\"target\":{\"averageUtilization\":90,\"type\":\"Utilization\"}},\"type\":\"Resource\"}],\"minReplicas\":1,\"scaleTargetRef\":{\"apiVersion\":\"apps/v1\",\"kind\":\"StatefulSet\",\"name\":\"alpine-sts\"}}}\n"
}
},
"spec": {
"scaleTargetRef": {
"kind": "StatefulSet",
"name": "alpine-sts",
"apiVersion": "apps/v1"
},
"minReplicas": 3,
"maxReplicas": 3,
"targetCPUUtilizationPercentage": 90
},
"status": {
"currentReplicas": 3,
"desiredReplicas": 0
}
- 100%[============================================================================================================>] 1.65K --.-KB/s in 0s
2019-03-12 20:31:00 (135 MB/s) - written to stdout [1685/1685]
/ #The script works! Since it was run on alpine-sts-0, the MINREP value used was from the current HPA.
It’s time to test whether this script runs when a new pod is created.
First, reset the HPA back to one minimum replica with kubectl patch:
/ # exit
command terminated with exit code 130
$ kubectl patch hpa alpine-sts -p '{"spec":{"minReplicas": 1}}'
horizontalpodautoscaler.autoscaling/alpine-sts patched
$Use kubectl get to view the state of the statefulset and HPA:
$ kubectl get pods,hpa
NAME READY STATUS RESTARTS AGE
pod/alpine-sts-0 1/1 Running 1 54s
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
horizontalpodautoscaler.autoscaling/alpine-sts StatefulSet/alpine-sts <unknown>/150% 1 3 1 2m11s
$Scale the statefulset using kubectl scale on the alpine-sts StatefulSet to three replicas:
$ kubectl scale sts alpine-sts --replicas=3
statefulset.apps/alpine-sts scaledWatch the process using kubectl get:
$ kubectl get pods,hpa
NAME READY STATUS RESTARTS AGE
pod/alpine-sts-0 1/1 Running 1 2m
pod/alpine-sts-1 0/1 ContainerCreating 0 40s
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
horizontalpodautoscaler.autoscaling/alpine-sts StatefulSet/alpine-sts <unknown>/150% 1 3 2 3m17s
$ kubectl get pods,hpa
NAME READY STATUS RESTARTS AGE
pod/alpine-sts-0 1/1 Running 1 2m10s
pod/alpine-sts-1 1/1 Running 1 50s
pod/alpine-sts-2 0/1 ContainerCreating 0 1s
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
horizontalpodautoscaler.autoscaling/alpine-sts StatefulSet/alpine-sts <unknown>/150% 2 3 2 3m27s
$Once the pod has started and completed all of its post start processes, you should see the MINPODS of the HPA increment up! This process should continue until all of the statefulset replicas are up and running.
What if a pod in the middle of the statefulset (with a lower ordinal number) dies and is recreated?
Try deleting one of the pods:
$ kubectl delete pod/alpine-sts-1
pod "alpine-sts-1" deletedCheck to see that the old pod has been recreated:
$ kubectl get pods,hpa
NAME READY STATUS RESTARTS AGE
pod/alpine-sts-0 1/1 Running 1 5m58s
pod/alpine-sts-1 0/1 ContainerCreating 0 8s
pod/alpine-sts-2 1/1 Running 1 3m49s
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
horizontalpodautoscaler.autoscaling/alpine-sts StatefulSet/alpine-sts <unknown>/150% 3 3 3 7m15s
$And wait for it to reach “Running” status – it is at that point that the script should fire.
$ kubectl get pods,hpa
NAME READY STATUS RESTARTS AGE
pod/alpine-sts-0 1/1 Running 1 6m53s
pod/alpine-sts-1 1/1 Running 1 63s
pod/alpine-sts-2 1/1 Running 1 4m44s
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
horizontalpodautoscaler.autoscaling/alpine-sts StatefulSet/alpine-sts <unknown>/150% 3 3 3 8m10s
$Nothing happened. That’s because the logic in the script makes a decision: If the new minimum replica count is lower than the current HPA minimum replica count, no changes will be made. This will ensure that if a pod that could potentially lower the HPA minimum replica count goes down, it will not disrupt that number when it is recreated by the kubelet.
This is just one way to implement one-way scaling. One big takeaway from this process is that Kubernetes is very flexible and that many in-tree tools are just as capable of performing needed tasks with a proper implementation.