This blog was originally posted as a guest blog for Tigera by our consultant Christopher Hanson.

Deploying an application on Kubernetes can require a number of related deployment artifacts or spec files: Deployment, Service, PVCs, ConfigMaps, Service Account – to name just a few. Managing all of these resources and relating them to deployed apps can be challenging, especially when it comes to tracking changes and updates to the deployed application (actual state) and its original source (authorized or desired state). Versions of the application are locked up in the Kubernetes platform, making them totally decoupled to the versions of the specs themselves (which are typically tracked in external source code management repos).

Additionally, static specs aren’t typically reusable outside a given domain, environment or cloud provider but involve a significant time investment to author and debug. Tooling can be used to provide string replacement based on matching expressions but this kind of automation also needs to be authored or customized to perform the tasks we require and may be prone to errors.

Helm solves these problems by packaging related Kubernetes specs into one simple deployment artifact (called a chart) that can be parameterized for maximum flexibility. In addition, Helm enables users to customize app packages at runtime in much the same way that the helm of a ship enables a pilot to steer (hence the name). If you are familiar with OS package managers such as apt or yum and packages such as deb or rpm, then the concepts of Helm and Helm Charts should feel familiar.

This blog is a tutorial that will take you from basic Helm concepts in an example deployment of a chart to modifying charts to fit your needs; in the example we will add a network policy to the chart.

Prerequisites

Helm uses Kubernetes; you will need a Kubernetes cluster running somewhere, a local Docker client, and a pre-configured kubectl client and config to your K8s cluster. Helm will be using your kubectl context to deploy Kubernetes resources on the configured cluster. The cluster should be using an SDN that understands Kubernetes network policies, like Calico, which you can install from the Installing Calico on Kubernetes guide.

Helm’s default installation is insecure, so if you’re trying Helm for the first time, doing so on a cluster where you won’t adversely affect your friends and colleagues is best. A blog about how to secure Helm this is not.

Installing Helm

There are two parts to Helm: The Helm client (helm) and the Helm server (Tiller). To install Tiller on a Kubernetes cluster, we use the helm client; we don’t need to use the helm client to deploy the Tiller server but, as you’ll see, it is convenient to do so.

The helm client can be installed from source or pre-built binary releases, via Snap on Linux, Homebrew on macOS or Chocolatey on Windows. But the Helm GitHub repo also holds an installer shell script that will automatically grab the latest version of the helm client and install it locally. The examples here use an Ubuntu 16.04 instance where Kubernetes was installed locally using kubeadm.

$ curl https://raw.githubusercontent.com/helm/helm/master/scripts/get > get_helm.sh

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  7236  100  7236    0     0  29435      0 --:--:-- --:--:-- --:--:-- 29534
$

Make the script executable and run it to download and install the latest version of helm; this step will require sudo permissions.

$ chmod 700 get_helm.sh

$ ./get_helm.sh

Downloading https://kubernetes-helm.storage.googleapis.com/helm-v2.12.3-linux-amd64.tar.gz
Preparing to install helm and tiller into /usr/local/bin
helm installed into /usr/local/bin/helm
tiller installed into /usr/local/bin/tiller
Run 'helm init' to configure helm.
$

We can use the version command with the client only flag (-c) to make sure the client is available:

$ helm version -c

Client: &version.Version{SemVer:"v2.12.3", GitCommit:"eecf22f77df5f65c823aacd2dbd30ae6c65f186e", GitTreeState:"clean"}
$

This command would hang without specifying the client only flag, as it will look for tiller using our kubeconfig and we don’t have tiller just yet.

By default, the helm client wants to set up a port-forward to the tiller pod using socat (more information in this github issue). In our case, socat is already installed as part of the preliminary setup of the Kubernetes cluster using kubeadm.

You do not need to perform this step but here is an example of when it was installed.

# apt-get install -y kubelet kubeadm kubectl kubernetes-cni

Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following additional packages will be installed:
  cri-tools ebtables socat
The following NEW packages will be installed:
  cri-tools ebtables kubeadm kubectl kubelet kubernetes-cni socat

...

Setting up kubernetes-cni (0.6.0-00) ...
Setting up socat (1.7.3.1-1) ...
Setting up kubelet (1.13.2-00) ...
Setting up kubectl (1.13.2-00) ...
Setting up kubeadm (1.13.2-00) ...
Processing triggers for systemd (229-4ubuntu7) ...
Processing triggers for ureadahead (0.100.0-19) ...
#

(If you need to install socat you can do it from apt: sudo apt-get install socat).

At this point we should be all set to deploy Tiller on our cluster.

Tiller

Tiller typically runs on your Kubernetes cluster as a Deployment. For development, it can also be run locally and configured to talk to a remote Kubernetes cluster–that’s handy!

The easiest way to install Tiller into the cluster is simply to run helm init. Helm will validate that helm’s local environment is set up correctly (or set it up if necessary), use the current-context of the kubeconfig to connect to the same cluster as kubectl and install the tiller pod.

init has a bunch of options to influence its behavior:

  • --canary-image – install the canary build of Tiller (test out the latest features)
  • --client-only – locally configure, but not install Tiller
  • --kube-context – use the named context in place of the current-context from your ~/.kube/config file
  • --node-selectors – specify the node labels required for scheduling the Tiller pod
  • --override – manipulates the specified properties of the final Tiller manifest
    • Accepts any valid value for any valid property in the Tiller deployment manifest
  • --output – skip the installation of Tiller’s deployment manifest and simply output the deployment manifest to stdout in either JSON or YAML format
  • --tiller-image – use a particular Tiller version other than latest
  • --upgrade – upgrade Tiller to the newest version

You can find even more of them here: https://docs.helm.sh/helm/#helm-init

Let’s take a look at what init is going to deploy by using the --output flag and tell it to oputpt yaml (or json if you prefer):

$ helm init --output yaml

---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  creationTimestamp: null
  labels:
    app: helm
    name: tiller
  name: tiller-deploy
  namespace: kube-system
spec:
  replicas: 1
  strategy: {}
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: helm
        name: tiller
    spec:
      automountServiceAccountToken: true
      containers:
      - env:
        - name: TILLER_NAMESPACE
          value: kube-system
        - name: TILLER_HISTORY_MAX
          value: "0"
        image: gcr.io/kubernetes-helm/tiller:v2.12.3
        imagePullPolicy: IfNotPresent
        livenessProbe:
          httpGet:
            path: /liveness
            port: 44135
          initialDelaySeconds: 1
          timeoutSeconds: 1
        name: tiller
        ports:
        - containerPort: 44134
          name: tiller
        - containerPort: 44135
          name: http
        readinessProbe:
          httpGet:
            path: /readiness
            port: 44135
          initialDelaySeconds: 1
          timeoutSeconds: 1
        resources: {}
status: {}

---
apiVersion: v1
kind: Service
metadata:
  creationTimestamp: null
  labels:
    app: helm
    name: tiller
  name: tiller-deploy
  namespace: kube-system
spec:
  ports:
  - name: tiller
    port: 44134
    targetPort: tiller
  selector:
    app: helm
    name: tiller
  type: ClusterIP
status:
  loadBalancer: {}

...
$

Here we see the tiller deployment and its service; we could simply save these files and use kubectl to deploy but what fun would that be?

Note the two environment variables: TILLER_NAMESPACE, which can be influenced by the --tiller-namespace flag to use a namespace other than kube-system and TILLER_HISTORY_MAX which is used to limit the maximum number of revisions saved per release (0 means no limit). Having an unlimited number of revisions has performance impacts, so in practice it should be set to something reasonable using the --history-max flag.

Tiller and RBAC

It is a good idea to limit Tiller’s ability to install resources to certain namespaces. When using RBAC, we can scope any application’s access to the Kubernetes API by giving it an identity (a Kubernetes service account) and assigning it scoped permissions using Kubernetes roles and bindings.

This time out we can keep the configuration to a minimum, assigning Tiller the cluster-admin cluster role so it can deploy to any namespace.

Don’t do this at home if your cluster isn’t a local or test cluster!

First, create the service account:

$ kubectl create serviceaccount --namespace kube-system tiller

serviceaccount/tiller created
$

Now create the cluster role binding, assigning the cluster-admin role to the tiller service account:

$ kubectl create clusterrolebinding tiller-cluster-role \
--clusterrole=cluster-admin --serviceaccount=kube-system:tiller

clusterrolebinding.rbac.authorization.k8s.io/tiller-cluster-role created
$

Deploy Tiller

Now we can deploy Tiller, using the --service-account flag to use our tiller service account:

$ helm init --service-account tiller

Creating /home/user/.helm
Creating /home/user/.helm/repository
Creating /home/user/.helm/repository/cache
Creating /home/user/.helm/repository/local
Creating /home/user/.helm/plugins
Creating /home/user/.helm/starters
Creating /home/user/.helm/cache/archive
Creating /home/user/.helm/repository/repositories.yaml
Adding stable repo with URL: https://kubernetes-charts.storage.googleapis.com
Adding local repo with URL: http://127.0.0.1:8879/charts
$HELM_HOME has been configured at /home/user/.helm.

Tiller (the Helm server-side component) has been installed into your Kubernetes Cluster.

Please note: by default, Tiller is deployed with an insecure 'allow unauthenticated users' policy.
To prevent this, run `helm init` with the --tiller-tls-verify flag.
For more information on securing your installation see: https://docs.helm.sh/using_helm/#securing-your-helm-installation
Happy Helming!
$

Helm automatically places its config files in ~/.helm; to put the helm client files somewhere other than ~/.helm, set the $HELM_HOME environment variable before running helm init. Tiller is then deployed with an important note:

Please note: by default, Tiller is deployed with an insecure ‘allow unauthenticated users’ policy. To prevent this, run helm init with the –tiller-tls-verify flag. For more information on securing your installation see: https://docs.helm.sh/using_helm/#securing-your-helm-installation

Tiller is insecure by default – have we mentioned that?

We can find Tiller on our cluster like any other Kubernetes resource:

$ kubectl -n kube-system get po,deploy,svc -l name=tiller

NAME                                READY   STATUS    RESTARTS   AGE
pod/tiller-deploy-dbb85cb99-fl929   1/1     Running   1          2m

NAME                                  READY   UP-TO-DATE   AVAILABLE   AGE
deployment.extensions/tiller-deploy   1/1     1            1           2m

NAME                    TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)     AGE
service/tiller-deploy   ClusterIP   10.97.133.203   <none>        44134/TCP   2m
$

Running the version command without -c should reveal both helm and tiller versions and ensure that helm can find and talk to Tiller:

$ helm version

Client: &version.Version{SemVer:"v2.12.3", GitCommit:"eecf22f77df5f65c823aacd2dbd30ae6c65f186e", GitTreeState:"clean"}
Server: &version.Version{SemVer:"v2.12.3", GitCommit:"eecf22f77df5f65c823aacd2dbd30ae6c65f186e", GitTreeState:"clean"}
$

Time to start Helming!

Explore Charts

If you recall, a chart is a collection of spec files that define a set of Kubernetes resources (like Services, Deployments, etc.). Charts typically include all of the resources that you would need to deploy an application as templates. The chart resource templates enable a user to customize the way the rendered resources are deployed at install time by providing values for some (or all) of the variables defined in the templates. Charts also include default values for all of the defined variables, making it easy to deploy the chart with little (or no) customization required.

As with other package managers, we want to get the latest list of, and updates to, charts from our configured repos using the update command:

$ helm repo update

Hang tight while we grab the latest from your chart repositories...
...Skip local chart repository
...Successfully got an update from the "stable" chart repository
Update Complete. ⎈ Happy Helming!⎈
$

Note that helm skipped the “local chart repository” but received an update from our only other repository, the “stable” repo. When you first install Helm, it is preconfigured to talk to a local repo and the official Kubernetes charts repository. The official repository (named “stable”), contains a number of carefully curated and maintained charts for common software like elasticsearch, influxdb, mariadb, nginx, prometheus, redis, and many others.

List your helm repos to show what has been configured:

$ helm repo list

NAME    URL                                             
stable  https://kubernetes-charts.storage.googleapis.com
local   http://127.0.0.1:8879/charts
$

Other repos can be added at any time with the helm repo add command. To get us started we’ll use the stable repo.

The helm search command will show us all of the available charts in the official repository (since it is the only repo configured and updated):

$ helm search

NAME                                    CHART VERSION   APP VERSION        DESCRIPTION                                                 
stable/acs-engine-autoscaler            2.2.2           2.1.1              DEPRECATED Scales worker nodes within agent pools           
stable/aerospike                        0.2.1           v3.14.1.2          A Helm chart for Aerospike in Kubernetes                    
stable/airflow                          0.15.0          1.10.0             Airflow is a platform to programmatically author, schedul...
stable/anchore-engine                   0.11.0          0.3.2              Anchore container analysis and policy evaluation engine s...
stable/apm-server                       0.1.0           6.2.4              The server receives data from the Elastic APM agents and ...
stable/ark                              3.0.0           0.10.1             A Helm chart for ark                                        
stable/artifactory                      7.3.1           6.1.0              DEPRECATED Universal Repository Manager supporting all ma...
stable/artifactory-ha                   0.4.1           6.2.0              DEPRECATED Universal Repository Manager supporting all ma...
stable/atlantis                         1.1.2           v0.4.11            A Helm chart for Atlantis https://www.runatlantis.io        
stable/auditbeat                        0.4.2           6.5.4              A lightweight shipper to audit the activities of users an...
stable/aws-cluster-autoscaler           0.3.3                              Scales worker nodes within autoscaling groups.              
stable/bitcoind                         0.1.5           0.15.1             Bitcoin is an innovative payment network and a new kind o...
stable/bookstack                        1.0.1           0.24.3             BookStack is a simple, self-hosted, easy-to-use platform ...
stable/buildkite                        0.2.4           3                  DEPRECATED Agent for Buildkite                              
stable/burrow                           1.0.1           0.23.3             Burrow is a permissionable smart contract machine           

...

Note the use of stable/ prepending all of the available charts. In the helm/charts project, the stable folder contains all of the charts that have gone through a rigorous promotion process and meet certain technical requirements. Incubator charts are also available but are still being improved until they meet these criteria. You can add the incubator repository (like any other repo) using the helm repo add command and pointing it at the correct URL.

Also note the CHART VERSION and APP VERSION columns; the former is the version of the Helm chart and must follow SemVer 2 format per the rules of the Helm project. The latter is the version of the actual software and is freeform in Helm but tied to the software’s release rules.

With no filter, helm search shows you all of the available charts. You can narrow down your results by searching with a filter:

$ helm search ingress

NAME                            CHART VERSION   APP VERSION DESCRIPTION                                                 
stable/gce-ingress              1.1.2           1.4.0       A GCE Ingress Controller                                    
stable/ingressmonitorcontroller 1.0.48          1.0.47      IngressMonitorController chart that runs on kubernetes      
stable/nginx-ingress            1.3.0           0.22.0      An nginx Ingress controller that uses ConfigMap to store ...
stable/external-dns             1.6.0           0.5.9       Configure external DNS servers (AWS Route53, Google Cloud...
stable/kong                     0.9.2           1.0.2       The Cloud-Native Ingress and Service Mesh for APIs and Mi...
stable/lamp                     1.0.0           7           Modular and transparent LAMP stack chart supporting PHP-F...
stable/nginx-lego               0.3.1                       Chart for nginx-ingress-controller and kube-lego            
stable/traefik                  1.60.0          1.7.7       A Traefik based Kubernetes ingress controller with Let's ...
stable/voyager                  3.2.4           6.0.0       DEPRECATED Voyager by AppsCode - Secure Ingress Controlle...
$

Why is traefik in the list? Because its package description relates it to ingress. We can use helm inspect chart to see how:

$ helm inspect chart stable/traefik

apiVersion: v1
appVersion: 1.7.7
description: A Traefik based Kubernetes ingress controller with Let's Encrypt support
engine: gotpl
home: https://traefik.io/
icon: http://traefik.io/traefik.logo.png
keywords:
- traefik
- ingress
- acme
- letsencrypt
maintainers:
- email: kent.rancourt@microsoft.com
  name: krancour
- email: emile@vauge.com
  name: emilevauge
- email: daniel.tomcej@gmail.com
  name: dtomcej
- email: ludovic@containo.us
  name: ldez
name: traefik
sources:
- https://github.com/containous/traefik
- https://github.com/helm/charts/tree/master/stable/traefik
version: 1.60.0
$

The keywords section of the traefik chart includes the keyword “ingress” so it shows up in our search.

Spend a few moments performing some additional keyword searches – see what you come up with!

Deploy a Chart (a.k.a. Installing a Package)

We’ll explore the anatomy of a chart later but to illustrate how easy it is to deploy a chart we can use one from the stable repo. To install a the chart, use the helm install command which only requires one argument: the name of the chart. Let’s start by doing just that, using the containerized Docker registry available from the official helm repo; you can check it out here.

Deploy the registry:

$ helm install stable/docker-registry

NAME:   kissable-clownfish
LAST DEPLOYED: Tue Feb  5 16:39:29 2019
NAMESPACE: default
STATUS: DEPLOYED

RESOURCES:
==> v1/Secret
NAME                                       TYPE    DATA  AGE
kissable-clownfish-docker-registry-secret  Opaque  1     0s

==> v1/ConfigMap
NAME                                       DATA  AGE
kissable-clownfish-docker-registry-config  1     0s

==> v1/Service
NAME                                TYPE       CLUSTER-IP     EXTERNAL-IP  PORT(S)   AGE
kissable-clownfish-docker-registry  ClusterIP  10.107.153.56  <none>       5000/TCP  0s

==> v1beta1/Deployment
NAME                                DESIRED  CURRENT  UP-TO-DATE  AVAILABLE  AGE
kissable-clownfish-docker-registry  1        0        0           0          0s

==> v1/Pod(related)
NAME                                                READY  STATUS   RESTARTS  AGE
kissable-clownfish-docker-registry-5ccc49955-rh7vj  0/1    Pending  0         0s


NOTES:
1. Get the application URL by running these commands:
  export POD_NAME=$(kubectl get pods --namespace default -l "app=docker-registry,release=kissable-clownfish" -o jsonpath="{.items[0].metadata.name}")
  echo "Visit http://127.0.0.1:8080 to use your application"
  kubectl port-forward $POD_NAME 8080:5000

$

What just happened?

Helm renders the Kubernetes resource templates by injecting the default values for all of the variables then deploys the resources on our Kubernetes cluster by submitting them to the Kubernetes API as static spec files. The act of installing a chart creates a new Helm release object; this release above is named “kissable-clownfish” (if you want to use your own release name, simply use the --name flag with the install command).

A Helm Release is a set of deployed resources based on a chart; each time a chart is installed, it deploys a whole set of Kubernetes resources with its own release name. The unique naming helps us keep track of how the Kubernetes resources are related and lets us deploy the chart any number of times with different customizations.

During installation, the Helm will print useful information about which resources were created, in our case a ConfigMap, Deployment, Secret, and a Service. To see it again you can use helm status with the release name. Let’s use our new registry server; the NOTES section of the install output has some clues to using it, let’s try it out.

$ export POD_NAME=$(kubectl get pods --namespace default -l "app=docker-registry,release=kissable-clownfish" -o jsonpath="{.items[0].metadata.name}")

$ kubectl port-forward $POD_NAME 8080:5000

Forwarding from [::1]:8080 -> 5000
Forwarding from 127.0.0.1:8080 -> 5000

At this point your terminal should be hijacked for the port-forward. Start a new terminal and use Docker to interact with the registry. From the Docker client on the Kubernetes host, pull a lightweight image like alpine:

$ docker image pull alpine

Using default tag: latest
latest: Pulling from library/alpine
6c40cc604d8e: Pull complete
Digest: sha256:b3dbf31b77fd99d9c08f780ce6f5282aba076d70a513a8be859d8d3a4d0c92b8
Status: Downloaded newer image for alpine:latest
$

Now re-tag it, prepending the image repo name with the IP:Port of our port-forwarded registry and try pushing it:

$ docker image tag alpine 127.0.0.1:8080/myalpine

$ docker image push 127.0.0.1:8080/myalpine

The push refers to repository [127.0.0.1:8080/myalpine]
503e53e365f3: Pushed
latest: digest: sha256:25b4d910f4b76a63a3b45d0f69a57c34157500faf6087236581eca221c62d214 size: 528
$

Verify that the registry has our image by querying the registry API:

$ curl -X GET http://127.0.0.1:8080/v2/_catalog

{"repositories":["myalpine"]}
$

Success!

That was easy but we’re only using the default configuration options for this chart. Likely you will want to customize a chart prior to deployment. To see what options are configurable for a given chart, use helm inspect values.

Kill your port-forward with Ctrl+C (^C) and then inspect the chart values:

$ helm inspect values stable/docker-registry

# Default values for docker-registry.
# This is a YAML-formatted file.
# Declare variables to be passed into your templates.
replicaCount: 1

updateStrategy:
  # type: RollingUpdate
  # rollingUpdate:
  #   maxSurge: 1
  #   maxUnavailable: 0

podAnnotations: {}

image:
  repository: registry
  tag: 2.6.2
  pullPolicy: IfNotPresent
# imagePullSecrets:
    # - name: docker
service:
  name: registry
  type: ClusterIP
  # clusterIP:
  port: 5000
  # nodePort:
  annotations: {}
  # foo.io/bar: "true"
ingress:
  enabled: false
  path: /
  # Used to create an Ingress record.
  hosts:
    - chart-example.local
  annotations:
    # kubernetes.io/ingress.class: nginx
    # kubernetes.io/tls-acme: "true"
  labels: {}

...

$

There are a number of configuration changes we can make; most notably, we can deploy an ingress record for the registry (useful if we have an ingress controller deployed). We can also configure a number of different storage backends like an S3 bucket and related Secret for storing AWS access keys.

But where are these values coming from?

(Partial) Anatomy of a Chart

A chart is a collection of files inside a directory named for the chart. Thus far we have only deployed a chart from a remote repo but if you looked at the link to the docker-registry chart on GitHub, you saw these files. When the chart is installed, Helm downloads the contents of the directory as an archive and caches it locally in the helm client’s workspace directory. The default location is ~/.helm:

$ ls -l ~/.helm

total 16
drwxr-xr-x 3 user user 4096 Jan 28 17:28 cache
drwxr-xr-x 2 user user 4096 Jan 28 17:28 plugins
drwxr-xr-x 4 user user 4096 Jan 28 17:28 repository
drwxr-xr-x 2 user user 4096 Jan 28 17:28 starters
$

The cache directory contains local clones of remote chart repositories in archive format:

$ ls -l ~/.helm/cache/archive/

total 8
-rw-r--r-- 1 user user 6316 Feb  5 16:39 docker-registry-1.7.0.tgz
$

If we want to explore a chart we can expand the archive ourselves, or better yet, use a helm command to do it for us!

Using the fetch command with the --untar argument results in an unpacked chart on our local system:

$ helm fetch stable/docker-registry --untar

$ ls -l docker-registry/

total 24
-rw-r--r-- 1 user user  391 Feb  5 17:42 Chart.yaml
-rw-r--r-- 1 user user   62 Feb  5 17:42 OWNERS
-rw-r--r-- 1 user user 7396 Feb  5 17:42 README.md
drwxr-xr-x 2 user user 4096 Feb  5 17:42 templates
-rw-r--r-- 1 user user 2595 Feb  5 17:42 values.yaml

The Helm docs are pretty good at explaining what most of these are; for now we are going to concentrate on values.yaml. Previously we inspected the values with the helm inspect values command; looking at the values.yaml we can see exactly what we were shown:

$ cat docker-registry/values.yaml

# Default values for docker-registry.
# This is a YAML-formatted file.
# Declare variables to be passed into your templates.
replicaCount: 1

updateStrategy:
  # type: RollingUpdate
  # rollingUpdate:
  #   maxSurge: 1
  #   maxUnavailable: 0

podAnnotations: {}

image:
  repository: registry
  tag: 2.6.2
  pullPolicy: IfNotPresent
# imagePullSecrets:
    # - name: docker
service:
  name: registry
  type: ClusterIP
  # clusterIP:
  port: 5000
  # nodePort:
  annotations: {}
  # foo.io/bar: "true"
ingress:
  enabled: false
  path: /
  # Used to create an Ingress record.
  hosts:
    - chart-example.local
  annotations:
    # kubernetes.io/ingress.class: nginx
    # kubernetes.io/tls-acme: "true"
  labels: {}

...

$

The values file is where the author(s) of a chart set the default values for all chart variables so all you have to do is type helm install and the chart should work. Some charts have prerequisites but those are typically documented so you know ahead of time. For example, the WordPress chart declares these prerequisites:

Prerequisites – Kubernetes 1.4+ with Beta APIs enabled – PV provisioner support in the underlying infrastructure

Now that we know what can be changed, let’s change something!

Update a Release

When you want to change the configuration of a release, you can use the helm upgrade command. Helm will only update things that have changed since the last release. Upgrade works with the same override flags as install so that you can customize a chart on initial install or sometime later.

Our original Docker Registry Service is ClusterIP type which is why we needed the port-forward:

$ helm inspect values stable/docker-registry |grep -B2 -A4 ClusterIP

service:
  name: registry
  type: ClusterIP
  # clusterIP:
  port: 5000
  # nodePort:
  annotations: {}
  # foo.io/bar: "true"
$

To confirm it was deployed that way list the Kubernetes Service:

$ kubectl get service kissable-clownfish-docker-registry

NAME                                 TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE
kissable-clownfish-docker-registry   ClusterIP   10.107.153.56   <none>        5000/TCP   10m
$

Let’s update the Service to use a NodePort so that we can expose the registry to the outside world.

There are two ways to pass configuration data during an update or upon initial install:

  • --values (or -f) – specify a YAML file with overrides
  • --set – specify overrides on the command line,
    • Basic: --set name=value == name: value
      • Key value pairs are comma separated
    • Multiple complex values are supported by set:
      • --set servers.port=80 becomes:
servers:
  port: 80
- `--set servers[0].port=80,servers[0].host=example` becomes:
servers:
  - port: 80
    host: example

We know that type is a child of service, so we can set its value with --set service.type=

$ helm upgrade --set service.type=NodePort kissable-clownfish stable/docker-registry

Release "kissable-clownfish" has been upgraded. Happy Helming!
LAST DEPLOYED: Wed Feb  6 11:48:32 2019
NAMESPACE: default
STATUS: DEPLOYED

RESOURCES:
==> v1/Secret
NAME                                       TYPE    DATA  AGE
kissable-clownfish-docker-registry-secret  Opaque  1     11m

==> v1/ConfigMap
NAME                                       DATA  AGE
kissable-clownfish-docker-registry-config  1     19h

==> v1/Service
NAME                                TYPE      CLUSTER-IP     EXTERNAL-IP  PORT(S)         AGE
kissable-clownfish-docker-registry  NodePort  10.107.153.56  <none>       5000:31836/TCP  11m

==> v1beta1/Deployment
NAME                                DESIRED  CURRENT  UP-TO-DATE  AVAILABLE  AGE
kissable-clownfish-docker-registry  1        1        1           1          11m

==> v1/Pod(related)
NAME                                                READY  STATUS   RESTARTS  AGE
kissable-clownfish-docker-registry-5ccc49955-rh7vj  1/1    Running  0         11m


NOTES:
1. Get the application URL by running these commands:
  export NODE_PORT=$(kubectl get --namespace default -o jsonpath="{.spec.ports[0].nodePort}" services kissable-clownfish-docker-registry)
  export NODE_IP=$(kubectl get nodes --namespace default -o jsonpath="{.items[0].status.addresses[0].address}")
  echo http://$NODE_IP:$NODE_PORT

$

Based on the output above, Kubernetes has updated the configuration of our Service. The NOTES section has even changed, indicating that we can now access our docker-registry service via http://NODE_IP:NODE_PORT

We can use helm get values to see whether that new setting took effect (according to what helm knows).

$ helm get values kissable-clownfish

service:
  type: NodePort
$

Not as much information presented here. Helm only concerns itself with the changes to the yaml key/value pairs.

Let’s see if it worked:

$ export NODE_PORT=$(kubectl get --namespace default -o jsonpath="{.spec.ports[0].nodePort}" services kissable-clownfish-docker-registry)

$ export NODE_IP=$(kubectl get nodes --namespace default -o jsonpath="{.items[0].status.addresses[0].address}")

$ curl -X GET http://$NODE_IP:$NODE_PORT/v2/_catalog

{"repositories":["myalpine"]}
$

Success! Let’s test it by pushing an image.

By default, Docker will only trust a secure remote registry or an insecure registry found on the localhost. Since Kubernetes runs our registry in a container, even when the Docker daemon and the docker-registry are running on the same host the registry is considered “remote”. Our port-forward used localhost so Docker allowed us to push, but won’t let us this time around. Try it:

$ docker image tag alpine $NODE_IP:$NODE_PORT/extalpine

$ docker image push $NODE_IP:$NODE_PORT/extalpine

The push refers to repository [192.168.225.251:31836/extalpine]
Get https://192.168.225.251:31836/v2/: http: server gave HTTP response to HTTPS client
$

There are two ways to address this situation; one way is to configure the registry server to support TLS. Instead of securing the docker-registry we can tell Docker to trust our non-secure registry (only do this in non-production environments). This allows us to use the registry without using SSL certificates.

Doing this next step on the Kubernetes host has a high chance of breaking a kubeadm-deployed cluster because it requires restarting Docker and all the Kubernetes services are running in containers. So use a Docker installation that is external to your Kubernetes host; after all, that is why we exposed the registry as a nodePort service!

Configure our Docker daemon by creating a config file under /etc/docker that looks like this (replace the example IP with the IP of your node which you stored in NODE_IP earlier):

$ sudo cat /etc/docker/daemon.json

{
   "insecure-registries": [
     "192.168.225.251/32"
   ]
}
$

To put those changes into effect, you’ll need to restart Docker:

$ sudo systemctl restart docker

Now, whether your Docker daemon should trust our registry:

$ docker image push 192.168.225.251:31836/extalpine

The push refers to repository [192.168.225.251:31836/extalpine]
503e53e365f3: Pushed
latest: digest: sha256:25b4d910f4b76a63a3b45d0f69a57c34157500faf6087236581eca221c62d214 size: 528
$

See if it worked:

$ curl -X GET http://$NODE_IP:$NODE_PORT/v2/_catalog

{"repositories":["extalpine","myalpine"]}
$

Now you’ve got a registry that your team can share! But everyone else can use it too; let’s put a limit on that.

Network Policies

A network policy is a specification of how groups of pods are allowed to communicate with each other and other network endpoints. NetworkPolicy resources use labels to select pods and define rules which specify what traffic is allowed to the selected pods. Network policies are implemented by a CNI network plugin, so you must use a CNI networking solution which supports NetworkPolicy (like Calico).

By default, pods are non-isolated; they accept traffic from any source. Pods become isolated by having a NetworkPolicy that selects them. Adding a NetworkPolicy to a namespace selecting a particular pod causes that pod to become “isolated”, rejecting any connections that are not explicitly allowed by a NetworkPolicy. Other pods in the namespace that are not selected by any NetworkPolicy will continue to accept all traffic.

Create a Blocking Network Policy

For our first network policy we’ll create a blanket policy that denies all inbound connections to pods in the default namespace. Create a one that resembles the following policy:

$ cat networkpolicy.yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny
spec:
  podSelector: {}
  policyTypes:
  - Ingress

Submit it to the Kubernetes API:

$ kubectl create -f networkpolicy.yaml

networkpolicy.networking.k8s.io/default-deny created

user@ubuntu:~/svc$ kubectl get networkpolicy

NAME           POD-SELECTOR   AGE
default-deny   <none>         1m
$

This policy selects all pods ({}) and has no ingress policies (Ingress) for them. By creating any network policy however, we automatically isolate all pods.

From an external client, query the docker-registry for its stored images:

external-client:~$ curl -X GET http://192.168.225.251:31836/v2/_catalog

curl: (7) Failed to connect to 192.168.225.251 port 31836: Connection timed out

Perfect. The presence of a network policy shuts down our ability to reach the registry pod.

Create a Permissive Network Policy

To enable clients to access our registry pod we will need to create a network policy that selects the pod and allows ingress from a cidr. Network policies use labels to identify pods to target; the registry pod has the labels “app=docker-registry” and “release=kissable-clownfish” so we can use those to select it. However, these labels are from the current chart release and what we really want is a way to modify the chart with a NetworkPolicy template that uses a parameterized selector and an ingress rule that allows a user to customize an ingress cidr value.

A lot of what we need to author the NetworkPolicy spec file is in the existing templates directory of our chart. Let’s take a look at the local copy of the service.yaml template file as an example:

$ cat docker-registry/templates/service.yaml
apiVersion: v1
kind: Service
metadata:
  name: {{ template "docker-registry.fullname" . }}
  labels:
    app: {{ template "docker-registry.name" . }}
    chart: {{ .Chart.Name }}-{{ .Chart.Version | replace "+" "_" }}
    release: {{ .Release.Name }}
    heritage: {{ .Release.Service }}
{{- if .Values.service.annotations }}
  annotations:
{{ toYaml .Values.service.annotations | indent 4 }}
{{- end }}
spec:
  type: {{ .Values.service.type }}
{{- if (and (eq .Values.service.type "ClusterIP") (not (empty .Values.service.clusterIP))) }}
  clusterIP: {{ .Values.service.clusterIP }}
{{- end }}
  ports:
    - port: {{ .Values.service.port }}
      protocol: TCP
      name: {{ .Values.service.name }}
      targetPort: 5000
{{- if (and (eq .Values.service.type "NodePort") (not (empty .Values.service.nodePort))) }}
      nodePort: {{ .Values.service.nodePort }}
{{- end }}
  selector:
    app: {{ template "docker-registry.name" . }}
    release: {{ .Release.Name }}

The metadata name and labels sections can be copied verbatim. We want our new policy file to match what is set here (if you examine the other template files they are identical). Many of these variable values are generated automatically by Helm; template uses a named template defined in a file (_helpers.tpl in the templates directory) that can be used in other templates. See the helm docs for more info on named templates.

We can also use the Service’s selector to create the matchLabels for our NetworkPolicy spec.

Putting that all together in our new NetworkPolicy template looks like this:

$ cat docker-registry/templates/networkpolicy.yaml
kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
  name: {{ template "docker-registry.fullname" . }}
  labels:
    app: {{ template "docker-registry.name" . }}
    chart: {{ .Chart.Name }}-{{ .Chart.Version | replace "+" "_" }}
    release: {{ .Release.Name }}
    heritage: {{ .Release.Service }}
spec:
  podSelector:
    matchLabels:
      app: {{ template "docker-registry.name" . }}
      release: {{ .Release.Name }}
  ingress:
  - from:
    - ipBlock:
        cidr: {{ .Values.networkPolicy.cidr }}

Finally, we will add a section to the values.yaml file that lets a user specify a cidr, like this:

$ tail docker-registry/values.yaml
fsGroup: 1000

priorityClassName: ""

nodeSelector: {}

tolerations: []

networkPolicy:
  cidr: 192.168.225.0/24

If we want to see what Helm does to render our new spec we can use helm template and pass it the --execute argument and the path to our new template file so that it only renders the single template:

$ helm template docker-registry/ --execute templates/networkpolicy.yaml
---
# Source: docker-registry/templates/networkpolicy.yaml
kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
  name: release-name-docker-registry
  labels:
    app: docker-registry
    chart: docker-registry-1.7.0
    release: release-name
    heritage: Tiller
spec:
  podSelector:
    matchLabels:
      app: docker-registry
      release: release-name
  ingress:
  - from:
    - ipBlock:
        cidr: 192.168.225.0/24

$

LGTM!

Run the upgrade command again, this time using the local version of the chart:

$ helm upgrade --set service.type=NodePort kissable-clownfish docker-registry/

Release "kissable-clownfish" has been upgraded. Happy Helming!
LAST DEPLOYED: Wed Feb  6 14:32:41 2019
NAMESPACE: default
STATUS: DEPLOYED

RESOURCES:
==> v1/ConfigMap
NAME                                       DATA  AGE
kissable-clownfish-docker-registry-config  1     2h

==> v1/Service
NAME                                TYPE      CLUSTER-IP     EXTERNAL-IP  PORT(S)         AGE
kissable-clownfish-docker-registry  NodePort  10.107.153.56  <none>       5000:31836/TCP  2h

==> v1beta1/Deployment
NAME                                DESIRED  CURRENT  UP-TO-DATE  AVAILABLE  AGE
kissable-clownfish-docker-registry  1        1        1           1          21h

==> v1/NetworkPolicy
NAME                                POD-SELECTOR                                    AGE
kissable-clownfish-docker-registry  app=docker-registry,release=kissable-clownfish  0s

==> v1/Pod(related)
NAME                                                READY  STATUS   RESTARTS  AGE
kissable-clownfish-docker-registry-5ccc49955-rh7vj  1/1    Running  0         2h

==> v1/Secret
NAME                                       TYPE    DATA  AGE
kissable-clownfish-docker-registry-secret  Opaque  1     2h


NOTES:
1. Get the application URL by running these commands:
  export NODE_PORT=$(kubectl get --namespace default -o jsonpath="{.spec.ports[0].nodePort}" services kissable-clownfish-docker-registry)
  export NODE_IP=$(kubectl get nodes --namespace default -o jsonpath="{.items[0].status.addresses[0].address}")
  echo http://$NODE_IP:$NODE_PORT

Notice our new NetworkPolicy in the release message!

Ask Kubernetes what network policies are in place:

$ kubectl get networkpolicy

NAME                                 POD-SELECTOR                                     AGE
default-deny                         <none>                                           83m
kissable-clownfish-docker-registry   app=docker-registry,release=kissable-clownfish   16s
$

From an external client, query the docker-registry once more:

external-client:~$ curl -X GET http://192.168.225.251:31836/v2/_catalog

{"repositories":["extalpine","myalpine"]}
$

Great, we have access once more! Now, try the same query from a test pod running on the Kubernetes cluster:

$ kubectl run client --generator=run-pod/v1 --image busybox --command -- tail -f /dev/null

$ kubectl exec -it client -- wget -qO - http://192.168.225.251:31836/v2/_catalog

wget: can't connect to remote host (192.168.225.251): Connection timed out
command terminated with exit code 1
$

What happened?

Our client pod doesn’t belong to the approved cidr so it is not allowed to reach the registry pod. Now external Docker daemons from the given cidr can push and pull images to the registry but pods are unable to talk to it. Our policy and chart release are working as expected. There is a bit more work to do to make this example chart more friendly/reusable to others but that, and all the other steps taken should get you started using charts and augmenting them to fit your needs.

Happy Helming!