FRSCA: Vault

Talk to Our Team
Course Catalog
Open Enrollment Schedule

Overview

In our blog series about FRSCA we’ve already deployed a hardened Kubernetes cluster with the help of kOps, Trivy, the NSA/CISA Kubernetes Hardening Guide, and the CIS Benchmark, then we addded SPIFFE/SPIRE to our cluster for workload identities. In this entry in the FRSCA blog series, we set up Hashicorp Vault to serve as the secrets manager for the cluster. Vault is a popular system providing secure handling of sensitive data meant for application use. It provides an additional layer of security to help keep data safe but available.

Within FRSCA, Vault hosts the long-term image signing key, providing access through OIDC credentials from SPIRE. Any tasks that require image signing will use short-lived SPIRE keys based on the long term key stored in Vault.

In this blog, we’ll complete the following activities:

  • Install Vault in the Kubernetes cluster using Helm Charts
  • Integrate SPIRE with Vault

Setting up Vault

The recommended method to install Vault is via Helm, which you will certainly be using if you are trying to adopt FRSCA!

For a very basic installation of Vault we need to configure Vault to use persistent storage and enable service access to the Vault UI. We’ll create the following vault.yaml values file to define the storage and service:

global:
  dataStorage:
    storageClass: kops-csi-1-21

ui:
  enabled: true
  serviceType: "LoadBalancer"

This values file identifies one of the StorageClasses available in the cluster (kops-csi-1-21, defined in our Kubernetes setup blog). The available StorageClasses can be seen using kubectl get sc if you are unsure which StorageClasses you have in your cluster. 

Additionally, Vault comes with a UI that will enable you to manage secrets and interact with Vault graphically. Since our cluster has LoadBalancer support, setting the serviceType to LoadBalancer tells the cluster to request an externally accessible LoadBalancer service with a public IP address for Vault.

There are many more values that can be set to customize the Vault installation documented in the “vault-helm” repository on GitHub.

Once you have prepared the values file, add the “hashicorp” Helm repository to your Helm client:

$ helm repo add hashicorp https://helm.releases.hashicorp.com

"hashicorp" has been added to your repositories

$ helm repo update

Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "aqua" chart repository
...Successfully got an update from the "hashicorp" chart repository
Update Complete. ⎈Happy Helming!⎈

$

Next, install Vault on your cluster using Helm and the prepared values file.

One thing to note is that this Vault instance runs inside the cluster. Certain architectures may benefit from having Vault run in its own Kubernetes cluster or outside of Kubernetes. You can have multiple Kubernetes clusters use a single Vault instance to manage application secrets.

Running Vault within a cluster is a common pattern used when storing application secrets for the cluster.

Use helm install to install Vault, preferably in another namespace (like vault) in the cluster:

$ helm install vault hashicorp/vault -n vault --create-namespace -f vault.yaml

NAME: vault
LAST DEPLOYED: Fri Mar 8 18:44:49 2024
NAMESPACE: vault
STATUS: deployed
REVISION: 1
NOTES:
Thank you for installing HashiCorp Vault!

Now that you have deployed Vault, you should look over the docs on using
Vault with Kubernetes available here:

https://developer.hashicorp.com/vault/docs


Your release is named vault. To learn more about the release, try:

  $ helm status vault
  $ helm get manifest vault

$

This deploys the Vault service using a Kubernetes StatefulSet and the Vault Agent Injector using a Kubernetes Deployment.

$ kubectl get all -n vault

NAME                                        READY   STATUS              RESTARTS   AGE
pod/vault-0                                 0/1     ContainerCreating   0          14s
pod/vault-agent-injector-55748c487f-mlzzm   1/1     Running             0          14s

NAME                               TYPE           CLUSTER-IP       EXTERNAL-IP                                                               PORT(S)             AGE
service/vault                      ClusterIP      100.69.227.220   <none>                                                                    8200/TCP,8201/TCP   14s
service/vault-agent-injector-svc   ClusterIP      100.66.146.17    <none>                                                                    443/TCP             14s
service/vault-internal             ClusterIP      None             <none>                                                                    8200/TCP,8201/TCP   14s
service/vault-ui                   LoadBalancer   100.67.139.78    abe27898c041e46588625dee4256ea38-1029919661.us-east-1.elb.amazonaws.com   8200:31093/TCP      14s

NAME                                   READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/vault-agent-injector   1/1     1            1           14s

NAME                                              DESIRED   CURRENT   READY   AGE
replicaset.apps/vault-agent-injector-55748c487f   1         1         1       14s

NAME                     READY   AGE
statefulset.apps/vault   0/1     14s

$

The Vault service will be in the “Not Ready” state until we complete further configuration.

Unsealing Vault

Instances of Vault must be unsealed after booting. Until unsealed, the Vault instance will not be reachable through CLI or GUI.

You can confirm that Vault is pending additional action by viewing its logs:

$ kubectl logs -n vault vault-0

==> Vault server configuration:

Administrative Namespace:
             Api Address: http://100.96.2.171:8200
                     Cgo: disabled
         Cluster Address: https://vault-0.vault-internal:8201
   Environment Variables: GODEBUG, HOME, HOSTNAME, HOST_IP, KUBERNETES_PORT, KUBERNETES_PORT_443_TCP, KUBERNETES_PORT_443_TCP_ADDR, KUBERNETES_PORT_443_TCP_PORT, KUBERNETES_PORT_443_TCP_PROTO, KUBERNETES_SERVICE_HOST, KUBERNETES_SERVICE_PORT, KUBERNETES_SERVICE_PORT_HTTPS, NAME, PATH, POD_IP, PWD, SHLVL, SKIP_CHOWN, SKIP_SETCAP, VAULT_ADDR, VAULT_AGENT_INJECTOR_SVC_PORT, VAULT_AGENT_INJECTOR_SVC_PORT_443_TCP, VAULT_AGENT_INJECTOR_SVC_PORT_443_TCP_ADDR, VAULT_AGENT_INJECTOR_SVC_PORT_443_TCP_PORT, VAULT_AGENT_INJECTOR_SVC_PORT_443_TCP_PROTO, VAULT_AGENT_INJECTOR_SVC_SERVICE_HOST, VAULT_AGENT_INJECTOR_SVC_SERVICE_PORT, VAULT_AGENT_INJECTOR_SVC_SERVICE_PORT_HTTPS, VAULT_API_ADDR, VAULT_CLUSTER_ADDR, VAULT_K8S_NAMESPACE, VAULT_K8S_POD_NAME, VAULT_PORT, VAULT_PORT_8200_TCP, VAULT_PORT_8200_TCP_ADDR, VAULT_PORT_8200_TCP_PORT, VAULT_PORT_8200_TCP_PROTO, VAULT_PORT_8201_TCP, VAULT_PORT_8201_TCP_ADDR, VAULT_PORT_8201_TCP_PORT, VAULT_PORT_8201_TCP_PROTO, VAULT_SERVICE_HOST, VAULT_SERVICE_PORT, VAULT_SERVICE_PORT_HTTP, VAULT_SERVICE_PORT_HTTPS_INTERNAL, VAULT_UI_PORT, VAULT_UI_PORT_8200_TCP, VAULT_UI_PORT_8200_TCP_ADDR, VAULT_UI_PORT_8200_TCP_PORT, VAULT_UI_PORT_8200_TCP_PROTO, VAULT_UI_SERVICE_HOST, VAULT_UI_SERVICE_PORT, VAULT_UI_SERVICE_PORT_HTTP, VERSION
              Go Version: go1.21.3
              Listener 1: tcp (addr: "[::]:8200", cluster address: "[::]:8201", max_request_duration: "1m30s", max_request_size: "33554432", tls: "disabled")
               Log Level:
                   Mlock: supported: true, enabled: false
           Recovery Mode: false
                 Storage: file
                 Version: Vault v1.15.2, built 2023-11-06T11:33:28Z
             Version Sha: cf1b5cafa047bc8e4a3f93444fcb4011593b92cb

==> Vault server started! Log data will stream in below:

2024-03-08T18:45:17.070Z [INFO]  proxy environment: http_proxy="" https_proxy="" no_proxy=""
2024-03-08T18:45:17.071Z [INFO]  incrementing seal generation: generation=1
2024-03-08T18:45:17.072Z [INFO]  core: Initializing version history cache for core
2024-03-08T18:45:17.072Z [INFO]  events: Starting event system
2024-03-08T18:45:24.819Z [INFO]  core: security barrier not initialized
2024-03-08T18:45:24.819Z [INFO]  core: seal configuration missing, not initialized

$

The core: security barrier not initialized and seal configuration missing, not initialized lines will repeat until the Vault is unsealed.

To unseal Vault and begin using it, it must be initialized. Initializing Vault is done only at the first install. Initialization creates a series of keys that must be provided to the Vault instance in order to unseal it. It will also generate a starter root credential.

The unsealing is done inside the Vault container by running vault operator init using kubectl exec on the Vault pod:

$ kubectl exec -n vault -it vault-0 -- vault operator init

Unseal Key 1: gMan…
Unseal Key 2: Oqbz…
Unseal Key 3: 9Sxvt…
Unseal Key 4: +vVF…
Unseal Key 5: Jwsz…

Initial Root Token: hvs.UoY…

Vault initialized with 5 key shares and a key threshold of 3. Please securely distribute the key shares printed above. When the Vault is re-sealed, restarted, or stopped, you must supply at least 3 of these keys to unseal it before it can start servicing requests.

Vault does not store the generated root key. Without at least 3 keys to reconstruct the root key, Vault will remain permanently sealed!

It is possible to generate new unseal keys, provided you have a quorum of existing unseal keys shares. See "vault operator rekey" for more information.

$

Per the command output, five keys were provided and three are required to unseal the Vault instance. There is also an initial root token, which you can use to interact with Vault later.

Copy one of the keys, then run vault operator unseal through kubectl exec on the Vault server:

$ kubectl exec -n vault -it vault-0 -- vault operator unseal gManYIV…

Key                Value
---                -----
Seal Type          shamir
Initialized        true
Sealed             true
Total Shares       5
Threshold          3
Unseal Progress    1/3
Unseal Nonce       3abba2f8-c1ab-96aa-15e8-56fdeb176709
Version            1.15.2
Build Date         2023-11-06T11:33:28Z
Storage Type       file
HA Enabled         false

$

By default, Vault requires at least three of the keys to be used. The two outputs of note in the above session are Initialized true and Sealed true, indicating that Vault needs additional keys to fully unseal.

Enter the next couple of keys in the same manner.

$ kubectl exec -n vault -it vault-0 -- vault operator unseal 9Sxvt…

Key                Value
---                -----
Seal Type          shamir
Initialized        true
Sealed             true
Total Shares       5
Threshold          3
Unseal Progress    2/3
Unseal Nonce       3abba2f8-c1ab-96aa-15e8-56fdeb176709
Version            1.15.2
Build Date         2023-11-06T11:33:28Z
Storage Type       file
HA Enabled         false


$ kubectl exec -n vault -it vault-0 -- vault operator unseal JwszJ…

Key             Value
---             -----
Seal Type       shamir
Initialized     true
Sealed          false
Total Shares    5
Threshold       3
Version         1.15.2
Build Date      2023-11-06T11:33:28Z
Storage Type    file
Cluster Name    vault-cluster-59f53ca7
Cluster ID      c0a7408a-93b7-ed61-90bc-29279708035f
HA Enabled      false

$

After entering the third key and fulfilling the unseal threshold, the Sealed false output indicates that the security barrier and configuration are finally complete.

Check the logs again and you will see more activity has occurred:

$ kubectl logs -n vault vault-0

...

2024-03-08T18:46:15.067Z [INFO]  core: seal configuration missing, not initialized
2024-03-08T18:46:15.068Z [INFO]  core: security barrier not initialized
2024-03-08T18:46:15.070Z [INFO]  core: security barrier initialized: stored=1 shares=5 threshold=3
2024-03-08T18:46:15.071Z [INFO]  core: post-unseal setup starting
2024-03-08T18:46:15.086Z [INFO]  core: loaded wrapping token key
2024-03-08T18:46:15.086Z [INFO]  core: successfully setup plugin runtime catalog
2024-03-08T18:46:15.086Z [INFO]  core: successfully setup plugin catalog: plugin-directory=""
2024-03-08T18:46:15.087Z [INFO]  core: no mounts; adding default mount table
2024-03-08T18:46:15.091Z [INFO]  core: successfully mounted: type=cubbyhole version="v1.15.2+builtin.vault" path=cubbyhole/ namespace="ID: root. Path: "
2024-03-08T18:46:15.092Z [INFO]  core: successfully mounted: type=system version="v1.15.2+builtin.vault" path=sys/ namespace="ID: root. Path: "
2024-03-08T18:46:15.093Z [INFO]  core: successfully mounted: type=identity version="v1.15.2+builtin.vault" path=identity/ namespace="ID: root. Path: "
2024-03-08T18:46:15.096Z [INFO]  core: successfully mounted: type=token version="v1.15.2+builtin.vault" path=token/ namespace="ID: root. Path: "
2024-03-08T18:46:15.097Z [INFO]  rollback: Starting the rollback manager with 256 workers
2024-03-08T18:46:15.097Z [INFO]  rollback: starting rollback manager
2024-03-08T18:46:15.098Z [INFO]  core: restoring leases
2024-03-08T18:46:15.099Z [INFO]  expiration: lease restore complete
2024-03-08T18:46:15.100Z [INFO]  identity: entities restored
2024-03-08T18:46:15.100Z [INFO]  identity: groups restored
2024-03-08T18:46:15.101Z [INFO]  core: usage gauge collection is disabled
2024-03-08T18:46:15.101Z [INFO]  core: Recorded vault version: vault version=1.15.2 upgrade time="2024-03-08 18:46:15.10078618 +0000 UTC" build date=2023-11-06T11:33:28Z
2024-03-08T18:46:15.673Z [INFO]  core: post-unseal setup complete
2024-03-08T18:46:15.674Z [INFO]  core: root token generated
2024-03-08T18:46:15.674Z [INFO]  core: pre-seal teardown starting
2024-03-08T18:46:15.674Z [INFO]  rollback: stopping rollback manager
2024-03-08T18:46:15.674Z [INFO]  core: pre-seal teardown complete
2024-03-08T18:47:19.066Z [INFO]  core.cluster-listener.tcp: starting listener: listener_address=[::]:8201
2024-03-08T18:47:19.066Z [INFO]  core.cluster-listener: serving cluster requests: cluster_listen_address=[::]:8201
2024-03-08T18:47:19.067Z [INFO]  core: post-unseal setup starting
2024-03-08T18:47:19.068Z [INFO]  core: loaded wrapping token key
2024-03-08T18:47:19.068Z [INFO]  core: successfully setup plugin runtime catalog
2024-03-08T18:47:19.068Z [INFO]  core: successfully setup plugin catalog: plugin-directory=""
2024-03-08T18:47:19.070Z [INFO]  core: successfully mounted: type=system version="v1.15.2+builtin.vault" path=sys/ namespace="ID: root. Path: "
2024-03-08T18:47:19.070Z [INFO]  core: successfully mounted: type=identity version="v1.15.2+builtin.vault" path=identity/ namespace="ID: root. Path: "
2024-03-08T18:47:19.071Z [INFO]  core: successfully mounted: type=cubbyhole version="v1.15.2+builtin.vault" path=cubbyhole/ namespace="ID: root. Path: "
2024-03-08T18:47:19.074Z [INFO]  core: successfully mounted: type=token version="v1.15.2+builtin.vault" path=token/ namespace="ID: root. Path: "
2024-03-08T18:47:19.075Z [INFO]  rollback: Starting the rollback manager with 256 workers
2024-03-08T18:47:19.075Z [INFO]  rollback: starting rollback manager
2024-03-08T18:47:19.075Z [INFO]  core: restoring leases
2024-03-08T18:47:19.075Z [INFO]  expiration: lease restore complete
2024-03-08T18:47:19.076Z [INFO]  identity: entities restored
2024-03-08T18:47:19.076Z [INFO]  identity: groups restored
2024-03-08T18:47:19.077Z [INFO]  core: usage gauge collection is disabled
2024-03-08T18:47:19.078Z [INFO]  core: post-unseal setup complete
2024-03-08T18:47:19.078Z [INFO]  core: vault is unsealed

$

With Vault unsealed, the pod’s readiness test will successfully complete:

$ kubectl get pods -n vault

NAME                                    READY   STATUS    RESTARTS   AGE
vault-0                                 1/1     Running   0          5m
vault-agent-injector-55748c487f-mlzzm   1/1     Running   0          5m

$

The unsealing process must be done each time Vault is restarted and on each Vault instance in your cluster. Per the output of the initialize command, make sure these seal keys are securely distributed and protected!

Now that Vault is up and running, it’s time to use Vault to secure our secrets!

Configuring the Secrets Engine

Before we put our Vault installation to use we need to configure a secrets engine. We can do this along with some experimenting by shelling into the Vault container itself.

Create a new /bin/sh session within the Vault container:

$ kubectl exec -n vault -it vault-0 --  /bin/sh

/ $

Take the initial root token from the Initialization and export it as an environment variable, then use vault login to authenticate with the server:

/ $ export VAULT_ROOT_KEY=hvs.UoYoI2i…

/ $ vault login $VAULT_ROOT_KEY

Success! You are now authenticated. The token information displayed below is already stored in the token helper. You do NOT need to run "vault login" again. Future Vault requests will automatically use this token.

Key                  Value
---                  -----
token                hvs.UoYoI2iUQZtwNZi1it6p7tbc
token_accessor       NJp0INIWNk74LRNszlXReAQy
token_duration       ∞
token_renewable      false
token_policies       ["root"]
identity_policies    []
policies             ["root"]

/ $

You are the root user in the Vault instance with permission to create and access any secret stored in Vault.

At this point Vault does not have anywhere to place secrets. Secrets in Vault are stored within paths, which are effectively endpoints in the Vault server that users issue requests against. Each path is backed by some kind of secrets engine.

Vault includes a “kv” secrets engine, which is a generic Key-Value store used to store arbitrary secrets within the configured physical storage for Vault. This backend can be run in one of two modes; v1 stores a single value for a key and v2 allows a configurable number of versions for each key.

The benefits of non-versioned kv is reduced storage size for each key since no additional metadata or history is stored. Additionally, requests going to a backend configured this way will be more performant because for any given request there will be fewer storage calls and no locking.

When running v2 of the kv backend a key can retain 10 versions by default. The older versions’ metadata and data can be retrieved. Additionally, Check-and-Set operations can be used to avoid overwriting data. When a version is deleted the underlying data is not removed, rather it is marked as deleted. Deleted versions can be undeleted. To permanently remove a version’s data the destroy command can be used.

For testing, we will create a path called spiffe-test backed for use with the Vault v2 kv secrets engine.

Use vault secrets enable to create the path:

/ $ vault secrets enable -version=2 -path="spiffe-test" kv

Success! Enabled the kv secrets engine at: spiffe-test/

/ $

Now that you have the path created, you can place secrets into Vault. This is usually done via a request made to the path, but within the Vault server (and using the vault CLI tool) you can use the kv put command to insert a key with a value under the path.

The key should include the path that it is nested under, so for the key01 example we want to place it under the name of spiffe-test/key01:

/ $ vault kv put spiffe-test/key01 value=notnull

===== Secret Path =====
spiffe-test/data/key01

======= Metadata =======
Key                Value
---                -----
created_time       2024-03-08T19:18:49.441806768Z
custom_metadata    <nil>
deletion_time      n/a
destroyed          false
version            1

/ $

The output shows the metadata and path of the secret stored in Vault.

The kv get command under the Vault CLI tool issues a request to retrieve the key. These commands are using the credentials and permissions granted by the vault login command you ran earlier in the container.

Try to retrieve the key01 secret that you just put in:

/ $ vault kv get spiffe-test/key01

===== Secret Path =====
spiffe-test/data/key01

======= Metadata =======
Key                Value
---                -----
created_time       2024-03-08T19:18:49.441806768Z
custom_metadata    <nil>
deletion_time      n/a
destroyed          false
version            1

==== Data ====
Key      Value
---      -----
value    notnull

/ $

Success! Vault now has the ability to hold key-value secrets, and the ability to allow users to retrieve them.

Enable Vault OIDC

In the previous step we used the internal Vault user Root Token to interact with Vault.

For external users and workloads, we need to be able to control what they can and cannot access. Zero Trust requires us to authenticate everyone and everything and Least Privilege models require we give each party the minimum privileges possible. In this step, we will set up a Java WebToken (JWT) based authentication and authorization scheme that will allow Vault to identify users and their capabilities. Fortunately, in a prior blog, we configured SPIRE so that each of our workloads (Pods) can acquire identity securely. Our SPIRE solution supports the OIDC interface and JWT which we can use for Vault integration.

Back in the Vault container shell, enable the JWT authentication method with vault auth enable:

/ $ vault auth enable jwt

Success! Enabled jwt auth method at: jwt/

/ $

Requests that need authentication and authorization will now be sent to Vault’s jwt/ endpoint.

Now, it’s time to set up an OIDC provider. We want Vault to be able to accept SPIRE JWT tokens, so in order to do so, we must ensure that Vault can contact SPIRE whenever it receives a request bearing a SPIRE-issued token. This is done by configuring Vault to authenticate a user (or user role) against some kind of provider. In this case, SPIRE.

Before you can proceed with this section, we have to go back to SPIRE and actually configure the OIDC discovery URL. This is most easily done in another terminal session so we can return to where we are leaving off in our Vault container.

Update the SPIRE Configuration

In this section we will set up OIDC Federation between our SPIRE server and a Vault server with the goal of allowing a workload with a SVID to authenticate against a Vault server by presenting its JWT-SVID. Using SVIDs removes the need for another authentication method (like username/password auth) with Vault.

The SPIRE OIDC Discovery Provider is a helper container that provides a minimal implementation of the OIDC discovery document for use with JSON Web Key Set (JWKS) for JSON Web Token (JWT) validation. Our SPIRE deployment will need to be updated in order to enable OIDC federation.

Edit the SPIRE server configmap to include the settings for enabling the OIDC functionality. We will change or add the following settings:

  • Add the federation configuration block – allows a trust domain to authenticate identities issued by other SPIFFE authorities, allowing workloads in one trust domain to securely authenticate workloads in a foreign trust domain (more on federation here)
    • This lets us run our Vault server outside our Kubernetes cluster if we want to
  • Add the jwt_issuer configuration that is set to the DNS name we will use
  • Add the spire-oidc Service Account to the allow list

Make a backup of the original file, then make the changes (we diff the file just below so if you want to see the diff, scroll down past the configmap):

$ cp spire-server-configmap.yaml no-oidc-spire-server-configmap.yaml

$ nano spire-server-configmap.yaml && cat $_

apiVersion: v1
kind: ConfigMap
metadata:
  name: spire-server
  namespace: spire
data:
  server.conf: |
    server {
      bind_address = "0.0.0.0"
      bind_port = "8081"
      socket_path = "/tmp/spire-server/private/api.sock"
      trust_domain = "frsca.rx-m.net"
      data_dir = "/run/spire/data"
      log_level = "DEBUG"
      # Add the federation block below
      federation {
        bundle_endpoint {
          address = "0.0.0.0"
          port = 8443
        }
      }
      ca_key_type = "rsa-2048"

      # Creates the iss claim in JWT-SVIDs.
      # NOTE: Change this with the FQDN of the Discovery Provider that you will configure in DNS
      jwt_issuer = "oidc-discovery.rx-m.net"

      ca_subject = {
        country = ["US"],
        organization = ["SPIFFE"],
        common_name = "",
      }
    }

    plugins {
      DataStore "sql" {
        plugin_data {
          database_type = "sqlite3"
          connection_string = "/run/spire/data/datastore.sqlite3"
        }
      }

      NodeAttestor "k8s_sat" {
        plugin_data {
          clusters = {
            # NOTE: Change this to your cluster name
            "frsca.rx-m.net" = {
              use_token_review_api_validation = true
              # Add the spire-oidc service account to the allow list
              service_account_allow_list = ["spire:spire-agent","spire:spire-oidc"]
            }
          }
        }
      }

      KeyManager "disk" {
        plugin_data {
          keys_path = "/run/spire/data/keys.json"
        }
      }

      Notifier "k8sbundle" {
        plugin_data {
        }
      }
    }

    health_checks {
      listener_enabled = true
      bind_address = "0.0.0.0"
      bind_port = "8080"
      live_path = "/live"
      ready_path = "/ready"

$

Diffing the original configmap with this new one reveals the changes and lets us double check our work!

$ diff no-oidc-spire-server-configmap.yaml spire-server-configmap.yaml

14a15,21
>       # Add the federation block below
>       federation {
>         bundle_endpoint {
>           address = "0.0.0.0"
>           port = 8443
>         }
>       }
16a24,26
>       # NOTE: Change this with the FQDN of the Discovery Provider that you will configure in DNS
>       jwt_issuer = "oidc-discovery.rx-m.net"
> 
38c48,49
<               service_account_allow_list = ["spire:spire-agent"]
---
>               # Add the spire-oidc service account to the allow list
>               service_account_allow_list = ["spire:spire-agent","spire:spire-oidc"]

$

Apply the new configuration:

$ kubectl apply -f spire-server-configmap.yaml

configmap/spire-server configured

$

Create a new configmap for the OIDC configuration:

$ kubectl apply -f spire-oidc-configmap.yaml && cat $_

apiVersion: v1
kind: ConfigMap
metadata:
  name: oidc-discovery-provider
  namespace: spire
data:
  oidc-discovery-provider.conf: |
    log_level = "INFO"
    # NOTE: Change this with the FQDN of the Discovery Provider that you will configure in DNS
    domains = ["oidc-discovery.rx-m.net"]
    acme {
        directory_url = "https://acme-v02.api.letsencrypt.org/directory"
        cache_dir = "/run/spire"
        tos_accepted = true
        # NOTE: Change this to your email
        email = "[email protected]"
    }
    server_api {
      address = "unix:///tmp/spire-server/private/api.sock"
    }
    health_checks {}

$

Now update the StatefulSet to use the new configmap (create a backup first!):

$ cp spire-server-statefulset.yaml no-oidc-spire-server-statefulset.yaml

$ nano spire-server-statefulset.yaml

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: spire-server
  namespace: spire
  labels:
    app: spire-server
spec:
  replicas: 1
  selector:
    matchLabels:
      app: spire-server
  serviceName: spire-server
  template:
    metadata:
      namespace: spire
      labels:
        app: spire-server
    spec:
      serviceAccountName: spire-server
      shareProcessNamespace: true
      containers:
        - name: spire-server
          image: ghcr.io/spiffe/spire-server:1.8.7
          args:
            - -config
            - /run/spire/config/server.conf
          ports:
            - containerPort: 8081
          volumeMounts:
            - name: spire-config
              mountPath: /run/spire/config
              readOnly: true
            - name: spire-data
              mountPath: /run/spire/data
              readOnly: false
            - name: spire-server-socket
              mountPath: /tmp/spire-server/private
              readOnly: false
          livenessProbe:
            httpGet:
              path: /live
              port: 8080
            failureThreshold: 2
            initialDelaySeconds: 15
            periodSeconds: 60
            timeoutSeconds: 3
          readinessProbe:
            httpGet:
              path: /ready
              port: 8080
            initialDelaySeconds: 5
            periodSeconds: 5

          # Add this container which implements the OIDC functionality
        - name: spire-oidc
          image: ghcr.io/spiffe/oidc-discovery-provider:1.8.7
          args:
          - -config
          - /run/spire/oidc/config/oidc-discovery-provider.conf
          ports:
          - containerPort: 443
            name: spire-oidc-port
          volumeMounts:
          - name: spire-server-socket
            mountPath: /tmp/spire-server/private
            readOnly: true
          - name: spire-oidc-config
            mountPath: /run/spire/oidc/config/
            readOnly: true
          - name: spire-data
            mountPath: /run/spire/data
            readOnly: false
          readinessProbe:
            httpGet:
              path: /ready
              port: 8008
            failureThreshold: 5
            initialDelaySeconds: 5
            periodSeconds: 5
            timeoutSeconds: 3
      volumes:
        - name: spire-config
          configMap:
            name: spire-server
        - name: spire-server-socket
          hostPath:
            path: /run/spire/sockets/server
            type: DirectoryOrCreate

          # Add this volume for the OIDC configmap
        - name: spire-oidc-config
          configMap:
            name: oidc-discovery-provider
  volumeClaimTemplates:
    - metadata:
        name: spire-data
        namespace: spire
      spec:
        accessModes:
          - ReadWriteOnce
        resources:
          requests:
            storage: 1Gi

$

Once again diff the updated manifest with the backed up original:

$ diff no-oidc-spire-server-statefulset.yaml spire-server-statefulset.yaml 

53a54,80
>           # Add this container which implements the OIDC functionality
>         - name: spire-oidc
>           image: ghcr.io/spiffe/oidc-discovery-provider:1.8.7
>           args:
>           - -config
>           - /run/spire/oidc/config/oidc-discovery-provider.conf
>           ports:
>           - containerPort: 443
>             name: spire-oidc-port
>           volumeMounts:
>           - name: spire-server-socket
>             mountPath: /tmp/spire-server/private
>             readOnly: true
>           - name: spire-oidc-config
>             mountPath: /run/spire/oidc/config/
>             readOnly: true
>           - name: spire-data
>             mountPath: /run/spire/data
>             readOnly: false
>           readinessProbe:
>             httpGet:
>               path: /ready
>               port: 8008
>             failureThreshold: 5
>             initialDelaySeconds: 5
>             periodSeconds: 5
>             timeoutSeconds: 3
61a89,92
>           # Add this volume for the OIDC configmap
>         - name: spire-oidc-config
>           configMap:
>             name: oidc-discovery-provider

$

Apply the updated manifest to trigger a rolling update:

$ kubectl apply -f spire-server-statefulset.yaml

statefulset.apps/spire-server configured

$

We want the SPIRE OIDC container to be reachable from outside our cluster by an external Vault server. To achieve this we create a Kubernetes LoadBalancer service:

$ kubectl apply -f spire-server-oidc-service.yaml && cat $_

apiVersion: v1
kind: Service
metadata:
  name: spire-oidc
  namespace: spire
spec:
  selector:
    app: spire-server
  ports:
    - name: https
      port: 443
      targetPort: spire-oidc-port
  type: LoadBalancer

$ kubectl -n spire get svc

NAME           TYPE           CLUSTER-IP       EXTERNAL-IP                                                              PORT(S)         AGE
spire-oidc     LoadBalancer   100.65.14.96     aaae60b4818ae4c9ebb1319fecad51c2-491207616.us-east-1.elb.amazonaws.com   443:32099/TCP   7s
spire-server   ClusterIP      100.69.180.182   <none>                                                                   8081/TCP        78m

$

Make note of the value under the EXTERNAL-IP column for the spire-oidc service. In the next step we will use the value from the EXTERNAL-IP (an IP address or DNS name depending on your environment) to create a DNS record that matches the OIDC DNS name we set in our SPIRE configmaps.

Configure DNS for the OIDC Discovery IP Address

One extra step needs to be taken outside Kubernetes. We need to register the public DNS record mentioned in our SPIRE server and OIDC configmaps so that it will resolve to the EXTERNAL-IP of our spire-oidc LoadBalancer service, in our case the AWS ELB. Looking back at our configmaps, note the oidc-discovery.rx-m.net entry in the spire-server configmap:

$ cat spire-server-configmap.yaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: spire-server
  namespace: spire
data:
  server.conf: |

...

      # NOTE: Change this with the FQDN of the Discovery Provider that you will configure in DNS
      jwt_issuer = "oidc-discovery.rx-m.net"

...

And again in the oidc-discovery-provider configmap:

$ cat spire-oidc-configmap.yaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: oidc-discovery-provider
  namespace: spire
data:
  oidc-discovery-provider.conf: |
    log_level = "INFO"
    # NOTE: Change this with the FQDN of the Discovery Provider that you will configure in DNS
    domains = ["oidc-discovery.rx-m.net"]

...

This DNS entry is not required if we deploy Vault on our Kubernetes cluster but we are creating it anyway so that we have the flexibility to deploy Vault inside or outside of our K8s cluster later on.

In our case, we created a DNS entry in AWS Route53 with the following settings:

  • Record name: oidc-discovery.rx-m.net
  • Record type: A
  • Alias to Application and Classic Load Balancer
  • Region: us-east-1
  • Target: dualstack.aaae60b4818ae4c9ebb1319fecad51c2-491207616.us-east-1.elb.amazonaws.com (the service external IP)

As with any change to DNS, it will take minutes or hours for the new A record to propagate to DNS servers. Negative DNS query results will be cached, causing headaches, so to be safe, wait an hour to test the DNS change after creating the A record.

Going to https://oidc-discovery.rx-m.net/.well-known/openid-configuration in a browser shows us that the DNS entry is working:

{
  "issuer": "https://oidc-discovery.rx-m.net",
  "jwks_uri": "https://oidc-discovery.rx-m.net/keys",
  "authorization_endpoint": "",
  "response_types_supported": [
    "id_token"
  ],
  "subject_types_supported": [],
  "id_token_signing_alg_values_supported": [
    "RS256",
    "ES256",
    "ES384"
  ]
}

The OIDC discovery URL you’ve provisioned will be what Vault references.

Register a Workload to Access a Secret

Before you return to your Vault exec session, you need to assign a SPIFFE identity to whichever workload you want to access a secret. Here we create a SpiffeID for the default Service Account in the default Namespace:

$ kubectl exec -n spire spire-server-0 -- /opt/spire/bin/spire-server entry create \
-spiffeID spiffe://frsca.rx-m.net/ns/default/sa/default           # Customize for the SA you will use \
-parentID spiffe://frsca.rx-m.net/ns/spire/sa/spire-agent \
-selector k8s:ns:default           # Customize for the NS of the SA you will use \
-selector k8s:sa:default           # Customize for the SA you will use

$

You will be using this spiffeID value later (in the example above spiffe://frsca.rx-m.net/ns/default/sa/default), so keep it somewhere accessible!

Return to your Vault pod exec session.

Once there, use vault write to add a new configuration line that associates any requests bearing an OIDC token to a role called Dev:

/ $ vault write auth/jwt/config oidc_discovery_url=https://oidc-discovery.rx-m.net default_role="dev"

Success! Data written to: auth/jwt/config

/ $

The value of oidc_discovery_url should be the OIDC Discovery URL you prepared earlier.

With the Dev role defined, you can now establish permissions. Permissions in Vault are provided using policies, which mention a path and define certain capabilities.

Say we want to provide access to the key01 secret created earlier to our new SVID. The policy will be formed like this:

/ $ vault policy write my-dev-policy - <<EOH
path "spiffe-test/data/key01" {
   capabilities = ["read"]
}
EOH

Success! Uploaded policy: my-dev-policy

/ $

Here, the my-dev-policy specified that secrets (or a secret) at the path of spiffe-test/data/key01 can be read. Make sure this path matches the path output when you inserted the key.

After establishing the policy, associate it with the expected workload subject. In this case, a Kubernetes Service Account that has a Spiffe ID associated with it (which you prepared earlier before returning to Vault):

/ $ vault write auth/jwt/role/dev role_type=jwt user_claim=sub bound_audiences=TESTING bound_subject=spiffe://frsca.rx-m.net/ns/default/sa/default token_ttl=24h token_policies=my-dev-policy

Success! Data written to: auth/jwt/role/dev

/ $

You may now exit the Vault pod exec session.

Testing Vault-SPIRE OIDC

With our SPIRE/Vault integration in place, your workload needs to perform the following steps to successfully retrieve a secret from Vault:

  • Acquire a SPIRE JWT
  • Log in to Vault using the SPIRE JWT and acquire a Vault client token
  • Request the secret data from Vault using the Vault client as an authentication header

Retrieve the token from SPIRE:

$ kubectl exec -n default client-7b64c4fc54-zj5rj -- /opt/spire/bin/spire-agent api fetch jwt  -audience TESTING  -socketPath /run/spire/sockets/agent.sock

token(spiffe://frsca.rx-m.net/ns/default/sa/default):        eyJhbGciOiJSUz…
bundle(spiffe://frsca.rx-m.net):
        {
    "keys": [
        {
            "kty": "RSA",
            "kid": "6mX5ko4dpcOW5iz0pXp1E1wkLromni0R",
            "n": "vtq7SbE7jfVoyNTz…w",
            "e": "AQAB"
        }
    ]
}

$

Copy the value returned in the token key (in the example eyJhbGciOiJSUz…) somewhere accessible; we will use it in a subsequent command.

Next, create a client pod that has curl or a similar tool:

kubectl run -it client --image docker.io/alpine:latest --rm

/ #

Add curl to the alpine container:

/ # apk update ; apk add curl

Prepare a payload.json file that states your role, the auth type, and the SPIRE token we retrieved at the beginning of this section:

/ # vi payload.json

{"role": "dev", "jwt": "eyJhbGciOiJSUz…"}

/ #

Login to vault using your token which returns the Vault response, including the Vault token:

$ curl -X POST -d @payload.json http://vault.vault:8200/v1/auth/jwt/login

{
  "request_id": "d73f6b02-bab9-e2e5-c5b8-3c10b8558d8f",
  "lease_id": "",
  "renewable": false,
  "lease_duration": 0,
  "data": null,
  "wrap_info": null,
  "warnings": null,
  "auth": {
    "client_token": "hvs.CAES…",
    "policies": [
      "default",
      "my-dev-policy"
    ],
    "token_policies": [
      "default",
      "my-dev-policy"
    ],
    "metadata": {
      "role": "dev"
    },
    "lease_duration": 86400,
    "renewable": true,
    "entity_id": "52261a2e-65ee-3b06-15de-7d3aa3d4a198",
    "token_type": "service",
    "orphan": true,
    "mfa_requirement": null,
    "num_uses": 0
  }
}

$

Take the value of client_token (hvs.CAES… in the example above) and pass it as the Header value of X-Vault-Token, targeting the path on the Vault server for the secret we want to retrieve:

 $ curl -H "X-Vault-Token: hvs.CAES…" http://vault.vault:8200/v1/spiffe-test/data/key01

{
  "request_id": "257c808f-681a-3cbe-ade3-1ac42e3c6ca4",
  "lease_id": "",
  "renewable": false,
  "lease_duration": 0,
  "data": {
    "data": {
      "value": "notnull"
    },
    "metadata": {
      "created_time": "2024-03-08T19:18:49.441806768Z",
      "custom_metadata": null,
      "deletion_time": "",
      "destroyed": false,
      "version": 1
    }
  },
  "wrap_info": null,
  "warnings": null,
  "auth": null
}

$

Vault allowed our client to use SPIRE credentials to retrieve the secret! 

Conclusion

Now that we have a working integration between SPIRE and Vault we can use it for any workload that needs to securely access secrets. In our next blog, we will grant this capability to Tekton so it can sign OCI artifacts with Vault secrets, stay tuned!

Secret Link