Skip to content

One-Way Pod Scaling with the Kubernetes HPA

(Jul. 30 2019) – Kubernetes has the ability to dynamically scale sets of pods based on resource usage. This is great for ensuring that applications always have the resources they need as loads vary. Autoscaling the number of pods in Kubernetes is most often accomplished using a Horizontal Pod Autoscaler (HPA). HPAs dynamically adjust the replica count of a pod controller, like a Deployment or a StatefulSet. Scaling is driven by metrics made available through the Kubernetes API by the Kubernetes metrics server and other third-party tools like Prometheus.

Metrics collected by Kubernetes can be used by autoscaler objects to evaluate resource usage against a configured threshold. If the resource usage exceeds or is significantly below the configured threshold, then the application deployments are scaled up or down accordingly.

While autoscalers do a great job of “automating the toil” in most situations, out-of-the-box autoscaling behavior may not be a perfect fit for all application types. For example, we often find the need to automate scale-out in stateful services but with the need to avoid automated scale-in. Scaling down a stateful set of pods implementing a clustered storage engine like Cassandra or Kafka could risk data loss, reduced availability, and other problems.

While one-way scaling could be implemented with a custom operator, a modified HPA or some other engineering shenanigans, we try to solve problems with plain vanilla Kubernetes features whenever possible. The resulting solutions tend to be simpler, easier to maintain and more cost-effective to implement; making our client’s lives easier down the road. For example, when it comes to one-way scaling, a postStart hook and a little bash shell scripting are all it takes.

This blog will walk you through the steps necessary to create a One-Way HPA.

1. Sample implementation Preparation

For simplicity, this example will utilize Alpine containers that have wget, grep and bash installed. Those three tools are central in our implementation of one-way pod scaling.

$ vim Dockerfile && cat Dockerfile

FROM alpine:latest

RUN apk add --no-cache wget grep bash

$

Build the Dockerfile and tag the image rxmllc/hpamod, though feel free to use any other tag. Be sure to utilize that tag when preparing statefulSet specs. The regular Alpine docker image may be used as well.

$ docker build -t rxmllc/hpamod .

Sending build context to Docker daemon  10.24kB
Step 1/2 : FROM alpine:latest
 ---> 5cb3aa00f899
Step 2/2 : RUN apk add --no-cache wget grep bash
 ---> Running in 6fbbab8cf84a
fetch http://dl-cdn.alpinelinux.org/alpine/v3.9/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.9/community/x86_64/APKINDEX.tar.gz
(1/8) Installing ncurses-terminfo-base (6.1_p20190105-r0)
(2/8) Installing ncurses-terminfo (6.1_p20190105-r0)
(3/8) Installing ncurses-libs (6.1_p20190105-r0)
(4/8) Installing readline (7.0.003-r1)
(5/8) Installing bash (4.4.19-r1)
Executing bash-4.4.19-r1.post-install
(6/8) Installing pcre (8.42-r1)
(7/8) Installing grep (3.1-r2)
(8/8) Installing wget (1.20.1-r0)
Executing busybox-1.29.3-r10.trigger
OK: 15 MiB in 22 packages
Removing intermediate container 6fbbab8cf84a
 ---> c78866d22b68
Successfully built c78866d22b68
Successfully tagged rxmllc/hpamod:latest

$

To begin, use the following spec to launch a 1-replica statefulSet using the alpine image built above:

$ vim statefulSet.yaml && cat statefulSet.yaml

apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
  name: alpine-sts
spec:
  serviceName: alpine-sts
  replicas: 1
  selector:
    matchLabels:
      run: alpine-sts
  updateStrategy:
    type: RollingUpdate
  template:
    metadata:
      creationTimestamp: null
      labels:
        run: alpine-sts
    spec:
      containers:
      - image: rxmllc/hpamod:latest
        imagePullPolicy: Never
        name: alpine-sts
        command:
          - /bin/sh
          - -c
          - tail -f /dev/null       

$

2. Getting the API calls

The ability to have pods communicate with the API Server and make effective requests is central to the one-way scaling approach. Pods themselves cannot benefit from the use of kubectl, but they can issue the same requests to the API that kubectl makes.

In a Kubernetes Cluster, apply the following HPA spec:

$ vim hpa.yaml && cat hpa.yaml

apiVersion: autoscaling/v2beta2
kind: HorizontalPodAutoscaler
metadata:
  name: alpine-sts
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: StatefulSet
    name: alpine-sts
  maxReplicas: 3
  minReplicas: 1
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 150

$ kubectl apply -f hpa.yaml

horizontalpodautoscaler.autoscaling/alpine-sts created

$

If the Kubernetes Metrics Server or a similar solution has not been installed, this HPA will not perform any automatic scaling based on metrics. It can still be deployed, modified, and can even manipulate the number of replicas, just not against any real-time resource usage by the pod.

Changing the minReplicas parameter in the HPA is key to the whole solution. To do this, use kubectl patch to modify the current value.

Try utilizing kubectl patch to change the minimum replicas:

$ kubectl patch hpa alpine-sts -p '{"spec":{"minReplicas": 1}}'

horizontalpodautoscaler.autoscaling/alpine-sts patched (no change)

$

That kubectl patch command targets the minReplicas parameter under the spec of the selected HPA. It uses a JSON map to pinpoint the exact parameter within the spec we need to modify.

Now, run this command to change the minReplicas count to 3, using the -v8 option to increase the verbosity:

$ kubectl patch hpa alpine-sts -p '{"spec":{"minReplicas": 1}}' -v8

I0312 10:03:59.395775   49368 loader.go:359] Config loaded from file /home/user/.kube/config
I0312 10:03:59.396310   49368 loader.go:359] Config loaded from file /home/user/.kube/config
I0312 10:03:59.398065   49368 loader.go:359] Config loaded from file /home/user/.kube/config
I0312 10:03:59.402645   49368 loader.go:359] Config loaded from file /home/user/.kube/config
I0312 10:03:59.404345   49368 round_trippers.go:383] GET https://192.168.229.133:6443/apis/autoscaling/v1/namespaces/default/horizontalpodautoscalers/alpine-sts
I0312 10:03:59.404381   49368 round_trippers.go:390] Request Headers:
I0312 10:03:59.404395   49368 round_trippers.go:393]     Accept: application/json
I0312 10:03:59.404411   49368 round_trippers.go:393]     User-Agent: kubectl/v1.12.2 (linux/amd64) kubernetes/17c77c7
I0312 10:03:59.412554   49368 round_trippers.go:408] Response Status: 200 OK in 8 milliseconds
I0312 10:03:59.412592   49368 round_trippers.go:411] Response Headers:
I0312 10:03:59.412597   49368 round_trippers.go:414]     Content-Length: 1567
I0312 10:03:59.412600   49368 round_trippers.go:414]     Date: Tue, 12 Mar 2019 17:03:59 GMT
I0312 10:03:59.412603   49368 round_trippers.go:414]     Content-Type: application/json
I0312 10:03:59.412651   49368 request.go:942] Response Body: {"kind":"HorizontalPodAutoscaler","apiVersion":"autoscaling/v1","metadata":{"name":"alpine-sts","namespace":"default","selfLink":"/apis/autoscaling/v1/namespaces/default/horizontalpodautoscalers/alpine-sts","uid":"5e3a470f-44dd-11e9-8b9b-000c294ee651","resourceVersion":"1072382","creationTimestamp":"2019-03-12T15:41:57Z","annotations":{"autoscaling.alpha.kubernetes.io/conditions":"[{\"type\":\"AbleToScale\",\"status\":\"True\",\"lastTransitionTime\":\"2019-03-12T15:42:12Z\",\"reason\":\"SucceededGetScale\",\"message\":\"the HPA controller was able to get the target's current scale\"},{\"type\":\"ScalingActive\",\"status\":\"False\",\"lastTransitionTime\":\"2019-03-12T15:42:12Z\",\"reason\":\"FailedGetResourceMetric\",\"message\":\"the HPA was unable to compute the replica count: missing request for cpu\"}]","kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"autoscaling/v2beta2\",\"kind\":\"HorizontalPodAutoscaler\",\"metadata\":{\"annotations\":{},\"name\":\"alpine-sts\",\"namespace\":\"def [truncated 543 chars]
I0312 10:03:59.413359   49368 loader.go:359] Config loaded from file /home/user/.kube/config
I0312 10:03:59.413583   49368 request.go:942] Request Body: {"spec":{"minReplicas": 1}}
I0312 10:03:59.413637   49368 round_trippers.go:383] PATCH https://192.168.229.133:6443/apis/autoscaling/v1/namespaces/default/horizontalpodautoscalers/alpine-sts
I0312 10:03:59.413644   49368 round_trippers.go:3] Request Headers:
I0312 10:03:59.413647   49368 round_trippers.go:393]     Accept: application/json
I0312 10:03:59.413650   49368 round_trippers.go:393]     Content-Type: application/strategic-merge-patch+json
I0312 10:03:59.413653   49368 round_trippers.go:393]     User-Agent: kubectl/v1.12.2 (linux/amd64) kubernetes/17c77c7
I0312 10:03:59.415818   49368 round_trippers.go:408] Response Status: 200 OK in 2 milliseconds
I0312 10:03:59.415831   49368 round_trippers.go:411] Response Headers:
I0312 10:03:59.415835   49368 round_trippers.go:414]     Content-Type: application/json
I0312 10:03:59.415837   49368 round_trippers.go:414]     Content-Length: 1567
I0312 10:03:59.415840   49368 round_trippers.go:414]     Date: Tue, 12 Mar 2019 17:03:59 GMT
I0312 10:03:59.415863   49368 request.go:942] Response Body: {"kind":"HorizontalPodAutoscaler","apiVersion":"autoscaling/v1","metadata":{"name":"alpine-sts","namespace":"default","selfLink":"/apis/autoscaling/v1/namespaces/default/horizontalpodautoscalers/alpine-sts","uid":"5e3a470f-44dd-11e9-8b9b-000c294ee651","resourceVersion":"1072382","creationTimestamp":"2019-03-12T15:41:57Z","annotations":{"autoscaling.alpha.kubernetes.io/conditions":"[{\"type\":\"AbleToScale\",\"status\":\"True\",\"lastTransitionTime\":\"2019-03-12T15:42:12Z\",\"reason\":\"SucceededGetScale\",\"message\":\"the HPA controller was able to get the target's current scale\"},{\"type\":\"ScalingActive\",\"status\":\"False\",\"lastTransitionTime\":\"2019-03-12T15:42:12Z\",\"reason\":\"FailedGetResourceMetric\",\"message\":\"the HPA was unable to compute the replica count: missing request for cpu\"}]","kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"autoscaling/v2beta2\",\"kind\":\"HorizontalPodAutoscaler\",\"metadata\":{\"annotations\":{},\"name\":\"alpine-sts\",\"namespace\":\"def [truncated 543 chars]
horizontalpodautoscaler.autoscaling/alpine-sts patched (no change)

$

There were two calls made when running the kubectl patch command above. Running with level 8 verbosity has given the following information about each call:

  • For the GET request:
  • URL: https://192.168.229.133:6443/apis/autoscaling/v1/namespaces/default/horizontalpodautoscalers/alpine-sts
  • The following headers were provided:
    • Accept: application/json
    • User-Agent: kubectl/v1.12.2 (linux/amd64) kubernetes/17c77c7
  • For the PATCH request:
  • URL: https://192.168.229.133:6443/apis/autoscaling/v1/namespaces/default/horizontalpodautoscalers/alpine-sts
  • The following headers were provided:
    • Accept: application/json
    • Content-Type: application/strategic-merge-patch+json
    • User-Agent: kubectl/v1.12.2 (linux/amd64) kubernetes/17c77c7
  • And some body data:
    • {“spec”:{“minReplicas”: 3}}

By running kubectl patch with increased verbosity, we were able to determine the structure of the API request that the pod needs to run in order to modify the HPA.

3. Getting permission: Handling Kubernetes RBAC elements

There are some role-based access control elements that need to be in place to allow the pod to make the direct API call.

Take a look at the following spec:

$ vim rbac.yaml && cat rbac.yaml

kind: Role
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  namespace: default
  name: hpa-patcher
rules:
  - apiGroups: ["*"]
    resources: ["horizontalpodautoscalers"]
    verbs:
      - patch
      - get
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: patch-hpa
  namespace: default
subjects:
  - kind: ServiceAccount
    name: default
    namespace: default
roleRef:
  kind: Role
  name: hpa-patcher
  apiGroup: rbac.authorization.k8s.io

$

In this spec, two elements are defined: A role and a rolebinding. Both elements combined make up the RBAC portion of the autoscaling equation: giving a pod permission to make a request equivalent to kubectl patch.

The role, hpa-patcher, will be created in the default namespace. The hpa-patcher role will grant any users that receive it the ability to run PATCH and GET requests against a specific resource: HorizontalPodAutoscalers, in this case.

For one-way autoscaling, the ability to GET information about the existing HPA and the ability to PATCH it are necessary, so those are the only verbs (capabilities) the role will have. It’s good practice to be specific with role permissions: only assign roles the exact permissions needed to fulfill an operation.

The role binding will tie the hpa-patcher role to a user. In this case, it will assign it to the default namespace’s ServiceAccount – a universal account (within a namespace) that all pods within that namespace inherit. By inheriting the ServiceAccount, each pod will gain access to that ServiceAccount’s certificate, access token and namespace information: the components required to make a valid call to the API server.

Apply the rbac.yaml to your cluster:

$ kubectl apply -f rbac.yaml

role.rbac.authorization.k8s.io/hpa-patcher created
rolebinding.rbac.authorization.k8s.io/patch-hpa created

$

The default namespace’s ServiceAccount now has the GET and PATCH permissions against any HorizontalPodAutoscaler objects.

4. Creating the API call

Now that the permissions are in place to modify HPAs, it’s time to formulate the API call. This will be done using the URL that we collected from kubectl patch. The call needs to be made from within the cluster from one of the application’s pods.

Exec into one of the statefulset pods:

$ kubectl exec -it alpine-sts-0 -- sh

/ #

If you are using the public Alpine container, use apk add wget to retrieve a more current version of wget. The one that it is deployed with may not be able to pass all the headers and other flags needed for this solution.

First, use wget to try to access the same URL that was provided by Kubectl:

/ # wget -O - https://192.168.229.133:6443/apis/autoscaling/v1/namespaces/default/horizontalpodautoscalers/alpine-sts \


--2019-03-12 15:24:57--  https://192.168.229.133:6443/apis/autoscaling/v1/namespaces/default/horizontalpodautoscalers/alpine-sts
Connecting to 192.168.229.133:6443... connected.
ERROR: cannot verify 192.168.229.133's certificate, issued by 'CN=kubernetes':
  Unable to locally verify the issuer's authority.
To connect to 192.168.229.133 insecurely, use `--no-check-certificate'.

/ #

Since wget will download a file by default to the container’s filesystem, use the -O - to output the results to STDOUT.

In order to authenticate itself, the pod must present a valid certificate to the API server. Pods inherit authentication and authorization elements from their namespace’s service account. These elements are mounted to /var/run/secrets/kubernetes.io/serviceaccount/ within the container’s filesystem.

Inspect the /var/run/secrets/kubernetes.io/serviceaccount/ directory within the pod:

/ # ls -l /var/run/secrets/kubernetes.io/serviceaccount/

total 0
lrwxrwxrwx    1 root     root            13 Mar 12 15:20 ca.crt -> ..data/ca.crt
lrwxrwxrwx    1 root     root            16 Mar 12 15:20 namespace -> ..data/namespace
lrwxrwxrwx    1 root     root            12 Mar 12 15:20 token -> ..data/token

/ #

Each container inside a pod has the cluster CA certificate, a file containing the namespace name, and an authorization token. The certificate will allow the pod to authenticate against the API server and have the request be accepted, and the token will ensure that the Pod will perform the action as the intended user – the service account.

The token is written in a long, encrypted string. For the sake of this exercise (and reusability), it will be stored inside the variable TOKEN.

/ # TOKEN=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)

/ # echo $TOKEN
eyJhbGciOiJSUzI1NiIsImtpZCI6IiJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3V...

/ #

To utilize these elements, they are attached to the request URL as headers, which the API server will read before taking any action on the request body.

Append the CA certificate and the authentication token to the wget command as headers:

/ # wget -O - https://192.168.229.133:6443/apis/autoscaling/v1/namespaces/default/horizontalpodautoscalers/alpine-sts \
> --ca-certificate=/var/run/secrets/kubernetes.io/serviceaccount/ca.crt --header="Authorization: Bearer $TOKEN"

--2019-03-12 15:46:17--  https://192.168.229.133:6443/apis/autoscaling/v1/namespaces/default/horizontalpodautoscalers/alpine-sts
Connecting to 192.168.229.133:6443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1685 (1.6K) [application/json]
Saving to: 'STDOUT'

-                                                                       0%[                                                                                                                                                                         ]       0  --.-KB/s               {
  "kind": "HorizontalPodAutoscaler",
  "apiVersion": "autoscaling/v1",
  "metadata": {
    "name": "alpine-sts",
    "namespace": "default",
    "selfLink": "/apis/autoscaling/v1/namespaces/default/horizontalpodautoscalers/alpine-sts",
    "uid": "5e3a470f-44dd-11e9-8b9b-000c294ee651",
    "resourceVersion": "1066323",
    "creationTimestamp": "2019-03-12T15:41:57Z",
    "annotations": {
      "autoscaling.alpha.kubernetes.io/conditions": "[{\"type\":\"AbleToScale\",\"status\":\"True\",\"lastTransitionTime\":\"2019-03-12T15:42:12Z\",\"reason\":\"SucceededGetScale\",\"message\":\"the HPA controller was able to get the target's current scale\"},{\"type\":\"ScalingActive\",\"status\":\"False\",\"lastTransitionTime\":\"2019-03-12T15:42:12Z\",\"reason\":\"FailedGetResourceMetric\",\"message\":\"the HPA was unable to compute the replica count: missing request for cpu\"}]",
      "kubectl.kubernetes.io/last-applied-configuration": "{\"apiVersion\":\"autoscaling/v2beta2\",\"kind\":\"HorizontalPodAutoscaler\",\"metadata\":{\"annotations\":{},\"name\":\"alpine-sts\",\"namespace\":\"default\"},\"spec\":{\"maxReplicas\":3,\"metrics\":[{\"resource\":{\"name\":\"cpu\",\"target\":{\"averageUtilization\":150,\"type\":\"Utilization\"}},\"type\":\"Resource\"}],\"minReplicas\":1,\"scaleTargetRef\":{\"apiVersion\":\"apps/v1\",\"kind\":\"StatefulSet\",\"name\":\"alpine-sts\"}}}\n"
    }
  },
  "spec": {
    "scaleTargetRef": {
      "kind": "StatefulSet",
      "name": "alpine-sts",
      "apiVersion": "apps/v1"
    },
    "minReplicas": 1,
    "maxReplicas": 3,
    "targetCPUUtilizationPercentage": 150
  },
  "status": {
    "currentReplicas": 1,
    "desiredReplicas": 0
  }
-                                                                     100%[========================================================================================================================================================================>]   1.65K  --.-KB/s    in 0s      

2019-03-12 15:46:17 (119 MB/s) - written to stdout [1685/1685]

/ #
  • --ca-certificate declares the path to the certificate file that will be used for authentication.
  • --header="Authorization: Bearer $TOKEN" will present a valid login from the pod (as the Service Account)

Running that command performed a GET request on the horizontal pod autoscalers. The type of request is also determined by a header, which wasn’t provided in this first request. Since it was successful, it also indicates that one of the pods within the StatefulSet can successfully GET information about HPAs within its namespace.

If you receive a 403 error, make sure you applied the RBAC elements shown in the previous step to the Kubernetes cluster. Also, make sure that the HPA from the spec above exists, or else the command may throw a 404. Remember that a service account has no permissions to do anything until it is assigned some through a role.

Now it’s time to attempt a PATCH action by adding '{"spec":{"minReplicas": 1}}' to the request body. For wget, the equivalent to the curl --data flag is --body-data. Because PATCH is a more complex request, a media type must be provided. In the original CURL command, that was provided by the content-type header: "Content-Type: application/strategic-merge-patch+json"

Append that to the command as another header:

/ # wget -O - https://192.168.229.133:6443/apis/autoscaling/v1/namespaces/default/horizontalpodautoscalers/alpine-sts -v \
--ca-certificate=/var/run/secrets/kubernetes.io/serviceaccount/ca.crt --header="Authorization: Bearer $TOKEN" \
--method=PATCH --body-data '{"spec":{"minReplicas": 1}}' \
--header="Accept: application/json, */*" --header="Content-Type: application/strategic-merge-patch+json"

--2019-03-12 17:13:28--  https://192.168.229.133:6443/apis/autoscaling/v1/namespaces/default/horizontalpodautoscalers/alpine-sts
Connecting to 192.168.229.133:6443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1730 (1.7K) [application/json]
Saving to: 'STDOUT'

-                                                   0%[                                                                                                             ]       0  --.-KB/s               {
  "kind": "HorizontalPodAutoscaler",
  "apiVersion": "autoscaling/v1",
  "metadata": {
    "name": "alpine-sts",
    "namespace": "default",
    "selfLink": "/apis/autoscaling/v1/namespaces/default/horizontalpodautoscalers/alpine-sts",
    "uid": "5e3a470f-44dd-11e9-8b9b-000c294ee651",
    "resourceVersion": "1073070",
    "creationTimestamp": "2019-03-12T15:41:57Z",
    "annotations": {
      "autoscaling.alpha.kubernetes.io/conditions": "[{\"type\":\"AbleToScale\",\"status\":\"True\",\"lastTransitionTime\":\"2019-03-12T15:42:12Z\",\"reason\":\"SucceededGetScale\",\"message\":\"the HPA controller was able to get the target's current scale\"},{\"type\":\"ScalingActive\",\"status\":\"False\",\"lastTransitionTime\":\"2019-03-12T15:42:12Z\",\"reason\":\"FailedGetResourceMetric\",\"message\":\"the HPA was unable to compute the replica count: missing request for cpu\"}]",
      "kubectl.kubernetes.io/last-applied-configuration": "{\"apiVersion\":\"autoscaling/v2beta2\",\"kind\":\"HorizontalPodAutoscaler\",\"metadata\":{\"annotations\":{},\"name\":\"alpine-sts\",\"namespace\":\"default\"},\"spec\":{\"maxReplicas\":3,\"metrics\":[{\"resource\":{\"name\":\"cpu\",\"target\":{\"averageUtilization\":150,\"type\":\"Utilization\"}},\"type\":\"Resource\"}],\"minReplicas\":1,\"scaleTargetRef\":{\"apiVersion\":\"apps/v1\",\"kind\":\"StatefulSet\",\"name\":\"alpine-sts\"}}}\n"
    }
  },
  "spec": {
    "scaleTargetRef": {
      "kind": "StatefulSet",
      "name": "alpine-sts",
      "apiVersion": "apps/v1"
    },
    "minReplicas": 1,
    "maxReplicas": 3,
    "targetCPUUtilizationPercentage": 150
  },
  "status": {
    "lastScaleTime": "2019-03-12T16:59:50Z",
    "currentReplicas": 1,
    "desiredReplicas": 3
  }
-                                                 100%[============================================================================================================>]   1.69K  --.-KB/s    in 0s      

2019-03-12 17:13:28 (101 MB/s) - written to stdout [1730/1730]

/ #

A 200 OK response means the command was successful, though since no change was provided

Now, try to change the body to edit the replicas to 3:

/ # wget -O - https://192.168.229.133:6443/apis/autoscaling/v1/namespaces/default/horizontalpodautoscalers/alpine-sts -v \
--ca-certificate=/var/run/secrets/kubernetes.io/serviceaccount/ca.crt --header="Authorization: Bearer $TOKEN" \
--method=PATCH --body-data '{"spec":{"minReplicas": 3}}' \
--header="Accept: application/json, */*" --header="Content-Type: application/strategic-merge-patch+json"

--2019-03-12 17:15:23--  https://192.168.229.133:6443/apis/autoscaling/v1/namespaces/default/horizontalpodautoscalers/alpine-sts
Connecting to 192.168.229.133:6443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1730 (1.7K) [application/json]
Saving to: 'STDOUT'

-                                                   0%[                                                                                                             ]       0  --.-KB/s               {
  "kind": "HorizontalPodAutoscaler",
  "apiVersion": "autoscaling/v1",
  "metadata": {
    "name": "alpine-sts",

...

  "spec": {
    "scaleTargetRef": {
      "kind": "StatefulSet",
      "name": "alpine-sts",
      "apiVersion": "apps/v1"
    },
    "minReplicas": 3,

...

2019-03-12 17:15:23 (10.2 MB/s) - written to stdout [1730/1730]

/ #

The pod can now gather information on and update HPAs using a request equivalent to kubectl patch.

For reusability, we parameterized each request as much as possible. The first thing that can be done is to replace the IP and port of the machine that was previously being used to kubernetes.default.svc. That is a cluster DNS entry that any internal resource (pod, HPA, etc.) can use to communicate back to the API server. This ensures that regardless of where this script is deployed with the pod, it will reach the API server.

/ # wget -O - https://kubernetes.default.svc/apis/autoscaling/v1/namespaces/default/horizontalpodautoscalers/alpine-sts \
--ca-certificate=/var/run/secrets/kubernetes.io/serviceaccount/ca.crt --header="Authorization: Bearer $TOKEN"

--2019-03-12 17:28:35--  https://kubernetes.default.svc/apis/autoscaling/v1/namespaces/default/horizontalpodautoscalers/alpine-sts
Resolving kubernetes.default.svc... 10.96.0.1
Connecting to kubernetes.default.svc|10.96.0.1|:443... connected.
HTTP request sent, awaiting response... 200 OK

...

/ #

Next, environment variables can be used to make the request URL more flexible for the following elements:

  • Namespace – As shown above, this is provided to each container in a pod by the serviceAccount
  • HPA name – In this example, both the statefulSet and HPA share a name, so it can be retrieved from the pod by removing the ordinal with the following expression: cat /etc/hostname | sed -E 's/-[0-9]*$//'. It will cat the hostname file of the pod, and use SED to remove the ordinal number at the end.
  • CACERT path – another path to be used to point to the or to obscure it by using an environment variable
/ # NAMESPACE=$(cat /var/run/secrets/kubernetes.io/serviceaccount/namespace)

/ # HPA=$(cat /etc/hostname | sed -E 's/-[0-9]*$//')

/ # CACERT=/var/run/secrets/kubernetes.io/serviceaccount/ca.crt

/ # wget -O - https://kubernetes.default.svc/apis/autoscaling/v1/namespaces/"$NAMESPACE"/horizontalpodautoscalers/"$HPA" \
--ca-certificate=$CACERT --header="Authorization: Bearer $TOKEN" -v

--2019-03-12 17:33:29--  https://kubernetes.default.svc/apis/autoscaling/v1/namespaces/default/horizontalpodautoscalers/alpine-sts
Resolving kubernetes.default.svc... 10.96.0.1
Connecting to kubernetes.default.svc|10.96.0.1|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1730 (1.7K) [application/json]
Saving to: 'STDOUT'

...

/ #

By parameterizing the request headers and using the internal API Server dns address, any pods that share the HPA name (or part of it) can get information about the HPA and modify it too.

5. Scripting the API calls

Now that the API calls have been formulated, it’s time to tie them together with some logic. This One-way scaling technique uses the following logic:

  • The newest pod in a statefulSet updates the HPA minimum replica value based on its ordinal value + 1 (starting from 0)
  • If the new replica value is below the current HPA minimum, then the number will not change; the new value must be greater. This prevents a pod with a lower ordinal value from setting the HPA minimum too low and causing unwanted scale down.

Following that logic, the scaling action must:

  • Retrieve (GET) the current HPA minimum replicas value
  • Calculate the new minimum replica value from the pod name
  • Evaluate whether the new minimum replica value is greater than the current
  • If it is higher, use the newly calculated value
  • If it is lower, use the current minimum replica value

Now write a script around that logic:

$ vim oneway-scaler.sh && cat oneway-scaler.sh

#!/bin/bash

set -xe

# Retrieve the pod's stateful set name from the pod hostname and set the HPA variable
HPA=$(cat /etc/hostname | sed -E 's/-[0-9]*$//')

# Set variables for the namespace's Service Account token
NAMESPACE=$(cat /var/run/secrets/kubernetes.io/serviceaccount/namespace)
CACERT=/var/run/secrets/kubernetes.io/serviceaccount/ca.crt
TOKEN=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)

# Calculate the new minimum replication value from the pod's ordinal number from the host name.
NEWMINREP=$(expr $(cat /etc/hostname | sed 's/.*-//') + 1)

# Retrieve the current minimum replica value from the HPA
CURHPAMIN=$(wget -O - https://kubernetes.default.svc/apis/autoscaling/v1/namespaces/$NAMESPACE/horizontalpodautoscalers/$HPA \
--ca-certificate=$CACERT --method=GET \
--header="Accept: application/json, */*" \
--header="Authorization: Bearer $TOKEN" \
| awk '/"minReplicas"/{match($0, /"minReplicas"/); print substr($0, RSTART - 1, RLENGTH + 15)}' | grep -oP '\d*')

# Decide which minimum replica value to use.
# If the new count calculated from the pod ordinal + 1 is higher, it will use that value.
# Otherwise, it will use the current HPA minimum to prevent inappropriate scaling
if [ "$NEWMINREP" -gt "$CURHPAMIN" ]
then
    MINREP="$NEWMINREP"
else
    MINREP="$CURHPAMIN"
fi

# Execute an api request to update the HPA, which is named the same as the stateful set, with the calculated minimum replication value.
wget -O - https://kubernetes.default.svc/apis/autoscaling/v1/namespaces/$NAMESPACE/horizontalpodautoscalers/$HPA \
--ca-certificate=$CACERT --method=PATCH \
--header="Accept: application/json, */*" \
--header="Content-Type: application/strategic-merge-patch+json" \
--header="Authorization: Bearer $TOKEN" \
--body-data '{"spec":{"minReplicas":'${MINREP}'}}'

Note that script is using bash and grep. If you are using the publicly available Alpine image, use apk add --no-cache bash grep to update them.

To achieve the logical goals, the script does the following:

  • Gathers environment variables HPA, NAMESPACE, CACERT and TOKEN
  • Use cat and sed to query the hostname and grab the ordinal number, then uses expr to increment it plus one before storing it in NEWMINREP
  • Issues a GET request on the HPA and uses awk and grep to pull the value from “minReplicas” and stores it in CURHPAMIN
  • Evaluates whether NEWMINREP is greater than CURHPAMIN and inserts the selected value into MINREP
  • Issues the PATCH request to the API using the MINREP value to update the HPA minimum replica count

The easiest way to distribute this script to pods is as a Configmap. A configmap can be mounted to Kubernetes pods as a volume, which makes its contents (such as a file) available from within the pod’s filesystem.

Create a configMap from the script by using kubectl create --from-file oneway-scaler.sh:

$ kubectl create configmap oneway-scale --from-file oneway-scaler.sh
$ kubectl get configmap oneway-scale -o yaml

apiVersion: v1
data:
  oneway-scaler.sh: "#!/bin/bash\n\nset -xe\n\n# Print the pod's stateful set name
    from the pod hostname\nHPA=$(expr $(cat /etc/hostname | sed -E 's/-[0-9]*$//'))\n\n#
    Set variables for the namespace's Service Account token\nNAMESPACE=$(cat /var/run/secrets/kubernetes.io/serviceaccount/namespace)\nCACERT=/var/run/secrets/kubernetes.io/serviceaccount/ca.crt\nTOKEN=$(cat
    /var/run/secrets/kubernetes.io/serviceaccount/token)\n\n# Calculate the new minimum
    replication value from the pod's ordinal number from the host name.\nNEWMINREP=$(expr
    $(cat /etc/hostname | sed 's/.*-//') + 1)\n\n# Retrieve the current minimum replica
    value from the HPA\nCURHPAMIN=$(wget -O - https://kubernetes.default.svc/apis/autoscaling/v1/namespaces/$NAMESPACE/horizontalpodautoscalers/$HPA
    \\\n--ca-certificate=$CACERT --method=GET \\\n--header=\"Accept: application/json,
    */*\" \\\n--header=\"Authorization: Bearer $TOKEN\" \\\n| awk '/\"minReplicas\"/{match($0,
    /\"minReplicas\"/); print substr($0, RSTART - 1, RLENGTH + 15)}' | grep -oP '\\d*')\n\n#
    Decide which minimum replica value to use.\n# If the new count calculated from
    the pod ordinal + 1 is higher, it will use that value.\n# Otherwise, it will use
    the current HPA minimum to prevent inappropriate scaling\nif [ \"$NEWMINREP\"
    -gt \"$CURHPAMIN\" ]\nthen\n\tMINREP=\"$NEWMINREP\"\nelse\n\tMINREP=\"$CURHPAMIN\"\nfi\n\n#
    Execute an api request to update the HPA, which is named the same as the stateful
    set, with the calculated minimum replication value.\nwget -O - https://kubernetes.default.svc/apis/autoscaling/v1/namespaces/$NAMESPACE/horizontalpodautoscalers/$HPA
    \\\n--ca-certificate=$CACERT --method=PATCH \\\n--header=\"Accept: application/json,
    */*\" \\\n--header=\"Content-Type: application/strategic-merge-patch+json\" \\\n--header=\"Authorization:
    Bearer $TOKEN\" \\\n--body-data '{\"spec\":{\"minReplicas\":'${MINREP}'}}'\n"
kind: ConfigMap
metadata:
  creationTimestamp: 2019-03-12T00:35:39Z
  name: oneway-scale
  namespace: default
  resourceVersion: "1058186"
  selfLink: /api/v1/namespaces/default/configmaps/oneway-scale
  uid: c2dc15c3-445e-11e9-8b9b-000c294ee651

$

The configMap, and the script within, are now available for our StatefulSet pods to use.

6. Mounting and Executing the script

Now it’s time to tie things together. Add the following elements to the StatefulSet container spec:

lifecycle:
  postStart:
    exec:
      command:
      - /bin/sh
      - -c
      - ./utils/oneway-scaler.sh
volumeMounts:
- name: hpa-script
  mountPath: /utils
volumes:
- name: hpa-script
configMap:
  name: oneway-scale
  defaultMode: 0777
  • The oneway-scale configmap is defined as a mountable volume called hpa-script in this spec. It will mount the script as a universally executable file inside the alpine container’s filesystem.
  • Then, the hpa-script volume defined in the volumes section is mounted to the /utils directory inside the container.
  • To call the script, a postStart hook will execute a command immediately after the pod’s containers are created.
  • This command will first ensure that wget, bash and grep – which are used by the script, are present and up to date.
  • It will then run the script by calling it as a user would from inside the container.

The completed StatefulSet spec should now look as below:

$ vim alpine-sts.yaml && cat alpine-sts.yaml

apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
  name: alpine-sts
spec:
  serviceName: alpine-sts
  replicas: 1
  selector:
    matchLabels:
      run: alpine-sts
  updateStrategy:
    type: RollingUpdate
  template:
    metadata:
      creationTimestamp: null
      labels:
        run: alpine-sts
    spec:
      containers:
      - image: rxmllc/hpamod:latest
        imagePullPolicy: Never
        name: alpine-sts
        command:
          - /bin/sh
          - -c
          - tail -f /dev/null
        lifecycle:
          postStart:
            exec:
              command:
              - /bin/sh
              - -c
              - ./utils/oneway-scaler.sh
        volumeMounts:
        - name: hpa-script
          mountPath: /utils
      volumes:
      - name: hpa-script
        configMap:
          name: oneway-scale
          defaultMode: 0777    

$

All the elements are in place – RBAC, API calls, and the script are ready to be executed by the pod.

7. Testing the solution

Reconfigure the alpine-sts statefulset by applying the new, completed spec file:

$ kubectl apply -f alpine-sts.yaml

statefulset.apps/alpine-sts configured

$

The statefulSet has been updated. Now, as pods come up, they will have the oneway-scaler.sh script present within their filesystems.

Use kubectl exec into one of the Pods and check if the oneway-scaler script is present:

$ kubectl exec -it alpine-sts-0 -- sh

/ #

/ # ls -l /utils

total 0
lrwxrwxrwx    1 root     root            23 Mar 12 18:55 oneway-scaler.sh -> ..data/oneway-scaler.sh

/ #

The script is in place, per the volume mapping.

Now, run the script manually:

/ # ./utils/oneway-scaler.sh

+++ cat /etc/hostname
+++ sed -E 's/-[0-9]*$//'
+ HPA=alpine-sts
++ cat /var/run/secrets/kubernetes.io/serviceaccount/namespace
+ NAMESPACE=default
+ CACERT=/var/run/secrets/kubernetes.io/serviceaccount/ca.crt
++ cat /var/run/secrets/kubernetes.io/serviceaccount/token
+ TOKEN=eyJhbGciOiJSUzI1NiIsImtpZCI6IiJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJkZWZhdWx0Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6ImRlZmF1bHQtdG9rZW4tOTRkOWIiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoiZGVmYXVsdCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6IjRkNjUxMjg4LWUxMzktMTFlOC1iMWE2LTAwMGMyOTRlZTY1MSIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDpkZWZhdWx0OmRlZmF1bHQifQ.vKxVaNWELJU4UO7DmLOd7Qbf_K9Jdtgt_3uRP3AEtVHzBysy1o9PEYxtT1djJA8UMnEYz2dgorDUjIdXNAwy6TVpo1SjXf4jtEMBQU4q64PQLgDRqw8P0E33T2hfzZTVxiZscrM8SpyQtgqVD1y0syveEGAUcWtGxMu5vihaZdxT_jNUgrvOpX3gNps36Ku7EYDf6lzG7lfH3krdenvuffDSASt3kvSirYPrTd-VpNftj8xTEbhkX-nZgGUdXTTBar9hOtx7zlS9kj9oYb7gcs99kVxcGkyjwrV33jzi3U5Hf5w8nHdnpQA9CuOxVYYD3vJD9B0o9uTAdjsCLuRPtQ
+++ cat /etc/hostname
+++ sed 's/.*-//'
++ expr 0 + 1
+ NEWMINREP=1
++ wget -O - https://kubernetes.default.svc/apis/autoscaling/v1/namespaces/default/horizontalpodautoscalers/alpine-sts --ca-certificate=/var/run/secrets/kubernetes.io/serviceaccount/ca.crt --method=GET '--header=Accept: application/json, */*' ++ '--header=Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IiJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJkZWZhdWx0Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6ImRlZmF1bHQtdG9rZW4tOTRkOWIiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoiZGVmYXVsdCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6IjRkNjUxMjg4LWUxMzktMTFlOC1iMWE2LTAwMGMyOTRlZTY1MSIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDpkZWZhdWx0OmRlZmF1bHQifQ.vKxVaNWELJU4UO7DmLOd7Qbf_K9Jdtgt_3uRP3AEtVHzBysy1o9PEYxtT1djJA8UMnEYz2dgorDUjIdXNAwy6TVpo1SjXf4jtEMBQU4q64PQLgDRqw8P0E33T2hfzZTVxiZscrM8SpyQtgqVD1y0syveEGAUcWtGxMu5vihaZdxT_jNUgrvOpX3gNps36Ku7EYDf6lzG7lfH3krdenvuffDSASt3kvSirYPrTd-VpNftj8xTEbhkX-nZgGUdXTTBar9hOtx7zlS9kj9oYb7gcs99kVxcGkyjwrV33jzi3U5Hf5w8nHdnpQA9CuOxVYYD3vJD9B0o9uTAdjsCLuRPtQ'awk
'/"minReplicas"/{match($0, /"minReplicas"/); print substr($0, RSTART - 1, RLENGTH + 15)}'
++ grep -oP '\d*'
--2019-03-12 20:31:00--  https://kubernetes.default.svc/apis/autoscaling/v1/namespaces/default/horizontalpodautoscalers/alpine-sts
Resolving kubernetes.default.svc... 10.96.0.1
Connecting to kubernetes.default.svc|10.96.0.1|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1685 (1.6K) [application/json]
Saving to: 'STDOUT'

-                                                 100%[============================================================================================================>]   1.65K  --.-KB/s    in 0s      

2019-03-12 20:31:00 (25.2 MB/s) - written to stdout [1685/1685]

+ CURHPAMIN=3
+ '[' 1 -gt 3 ']'
+ MINREP=3
+ wget -O - https://kubernetes.default.svc/apis/autoscaling/v1/namespaces/default/horizontalpodautoscalers/alpine-sts --ca-certificate=/var/run/secrets/kubernetes.io/serviceaccount/ca.crt --method=PATCH '--header=Accept: application/json, */*' '--header=Content-Type: application/strategic-merge-patch+json' '--header=Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IiJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJkZWZhdWx0Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6ImRlZmF1bHQtdG9rZW4tOTRkOWIiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoiZGVmYXVsdCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6IjRkNjUxMjg4LWUxMzktMTFlOC1iMWE2LTAwMGMyOTRlZTY1MSIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDpkZWZhdWx0OmRlZmF1bHQifQ.vKxVaNWELJU4UO7DmLOd7Qbf_K9Jdtgt_3uRP3AEtVHzBysy1o9PEYxtT1djJA8UMnEYz2dgorDUjIdXNAwy6TVpo1SjXf4jtEMBQU4q64PQLgDRqw8P0E33T2hfzZTVxiZscrM8SpyQtgqVD1y0syveEGAUcWtGxMu5vihaZdxT_jNUgrvOpX3gNps36Ku7EYDf6lzG7lfH3krdenvuffDSASt3kvSirYPrTd-VpNftj8xTEbhkX-nZgGUdXTTBar9hOtx7zlS9kj9oYb7gcs99kVxcGkyjwrV33jzi3U5Hf5w8nHdnpQA9CuOxVYYD3vJD9B0o9uTAdjsCLuRPtQ' --body-data '{"spec":{"minReplicas":3}}'
--2019-03-12 20:31:00--  https://kubernetes.default.svc/apis/autoscaling/v1/namespaces/default/horizontalpodautoscalers/alpine-sts
Resolving kubernetes.default.svc... 10.96.0.1
Connecting to kubernetes.default.svc|10.96.0.1|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1685 (1.6K) [application/json]
Saving to: 'STDOUT'

-                                                   0%[                                                                                                             ]       0  --.-KB/s               {
  "kind": "HorizontalPodAutoscaler",
  "apiVersion": "autoscaling/v1",
  "metadata": {
    "name": "alpine-sts",
    "namespace": "default",
    "selfLink": "/apis/autoscaling/v1/namespaces/default/horizontalpodautoscalers/alpine-sts",
    "uid": "2c8743c9-44f8-11e9-8b9b-000c294ee651",
    "resourceVersion": "1081301",
    "creationTimestamp": "2019-03-12T18:53:50Z",
    "annotations": {
      "autoscaling.alpha.kubernetes.io/conditions": "[{\"type\":\"AbleToScale\",\"status\":\"True\",\"lastTransitionTime\":\"2019-03-12T18:54:05Z\",\"reason\":\"SucceededGetScale\",\"message\":\"the HPA controller was able to get the target's current scale\"},{\"type\":\"ScalingActive\",\"status\":\"False\",\"lastTransitionTime\":\"2019-03-12T18:54:05Z\",\"reason\":\"FailedGetResourceMetric\",\"message\":\"the HPA was unable to compute the replica count: missing request for cpu\"}]",
      "kubectl.kubernetes.io/last-applied-configuration": "{\"apiVersion\":\"autoscaling/v2beta2\",\"kind\":\"HorizontalPodAutoscaler\",\"metadata\":{\"annotations\":{},\"name\":\"alpine-sts\",\"namespace\":\"default\"},\"spec\":{\"maxReplicas\":3,\"metrics\":[{\"resource\":{\"name\":\"cpu\",\"target\":{\"averageUtilization\":90,\"type\":\"Utilization\"}},\"type\":\"Resource\"}],\"minReplicas\":1,\"scaleTargetRef\":{\"apiVersion\":\"apps/v1\",\"kind\":\"StatefulSet\",\"name\":\"alpine-sts\"}}}\n"
    }
  },
  "spec": {
    "scaleTargetRef": {
      "kind": "StatefulSet",
      "name": "alpine-sts",
      "apiVersion": "apps/v1"
    },
    "minReplicas": 3,
    "maxReplicas": 3,
    "targetCPUUtilizationPercentage": 90
  },
  "status": {
    "currentReplicas": 3,
    "desiredReplicas": 0
  }
-                                                 100%[============================================================================================================>]   1.65K  --.-KB/s    in 0s      

2019-03-12 20:31:00 (135 MB/s) - written to stdout [1685/1685]

/ #

The script works! Since it was run on alpine-sts-0, the MINREP value used was from the current HPA.

It’s time to test whether this script runs when a new pod is created.

First, reset the HPA back to one minimum replica with kubectl patch:

/ # exit

command terminated with exit code 130

$ kubectl patch hpa alpine-sts -p '{"spec":{"minReplicas": 1}}'

horizontalpodautoscaler.autoscaling/alpine-sts patched

$

Use kubectl get to view the state of the statefulset and HPA:

$ kubectl get pods,hpa

NAME               READY   STATUS    RESTARTS   AGE
pod/alpine-sts-0   1/1     Running   1          54s

NAME                                             REFERENCE                TARGETS         MINPODS   MAXPODS   REPLICAS   AGE
horizontalpodautoscaler.autoscaling/alpine-sts   StatefulSet/alpine-sts   <unknown>/150%   1         3         1          2m11s

$

Scale the statefulset using kubectl scale on the alpine-sts StatefulSet to three replicas:

$ kubectl scale sts alpine-sts --replicas=3

statefulset.apps/alpine-sts scaled

Watch the process using kubectl get:

$ kubectl get pods,hpa
NAME               READY   STATUS              RESTARTS   AGE
pod/alpine-sts-0   1/1     Running             1          2m
pod/alpine-sts-1   0/1     ContainerCreating   0          40s

NAME                                             REFERENCE                TARGETS         MINPODS   MAXPODS   REPLICAS   AGE
horizontalpodautoscaler.autoscaling/alpine-sts   StatefulSet/alpine-sts   <unknown>/150%   1         3         2          3m17s

$ kubectl get pods,hpa
NAME               READY   STATUS              RESTARTS   AGE
pod/alpine-sts-0   1/1     Running             1          2m10s
pod/alpine-sts-1   1/1     Running             1          50s
pod/alpine-sts-2   0/1     ContainerCreating   0          1s

NAME                                             REFERENCE                TARGETS         MINPODS   MAXPODS   REPLICAS   AGE
horizontalpodautoscaler.autoscaling/alpine-sts   StatefulSet/alpine-sts   <unknown>/150%   2         3         2          3m27s
$

Once the pod has started and completed all of its post start processes, you should see the MINPODS of the HPA increment up! This process should continue until all of the statefulset replicas are up and running.

What if a pod in the middle of the statefulset (with a lower ordinal number) dies and is recreated?

Try deleting one of the pods:

$ kubectl delete pod/alpine-sts-1

pod "alpine-sts-1" deleted

Check to see that the old pod has been recreated:

$ kubectl get pods,hpa

NAME               READY   STATUS              RESTARTS   AGE
pod/alpine-sts-0   1/1     Running             1          5m58s
pod/alpine-sts-1   0/1     ContainerCreating   0          8s
pod/alpine-sts-2   1/1     Running             1          3m49s

NAME                                             REFERENCE                TARGETS         MINPODS   MAXPODS   REPLICAS   AGE
horizontalpodautoscaler.autoscaling/alpine-sts   StatefulSet/alpine-sts   <unknown>/150%   3         3         3          7m15s

$

And wait for it to reach “Running” status – it is at that point that the script should fire.

$ kubectl get pods,hpa
NAME               READY   STATUS    RESTARTS   AGE
pod/alpine-sts-0   1/1     Running   1          6m53s
pod/alpine-sts-1   1/1     Running   1          63s
pod/alpine-sts-2   1/1     Running   1          4m44s

NAME                                             REFERENCE                TARGETS         MINPODS   MAXPODS   REPLICAS   AGE
horizontalpodautoscaler.autoscaling/alpine-sts   StatefulSet/alpine-sts   <unknown>/150%   3         3         3          8m10s
$

Nothing happened. That’s because the logic in the script makes a decision: If the new minimum replica count is lower than the current HPA minimum replica count, no changes will be made. This will ensure that if a pod that could potentially lower the HPA minimum replica count goes down, it will not disrupt that number when it is recreated by the kubelet.

This is just one way to implement one-way scaling. One big takeaway from this process is that Kubernetes is very flexible and that many in-tree tools are just as capable of performing needed tasks with a proper implementation.

Posted in Cloud Native, Kubernetes and tagged , ,
Home
Call RX-M
Message Us