Kubernetes & Vault

Two of my favorite pieces of technology have a marriage made in heaven with the vault-secrets-operator from @rico_berger. It allows, in a GitOps manner, the syncing of Vault secrets into Kubernetes secrets. While syncing between one secrets store and another sounds like a waste of bits it can make a lot of sense in some situations. Where I work we run multiple Kubernetes clusters across several regions, both on-premise and in AWS. Getting secrets securely to all of the necessary applications & platforms can be difficult without a good toolset. Vault is an incredible tool that provides a simple, yet powerful, secure method of storing and managing secrets. Enabling Kubernetes applications to authenticate to Vault via a ServiceAccount even allows for secure bootstrapping of the vault-secrets-operator.

Let’s take a common scenario where a group of applications need a password to connect to the database. You could write a library that each of the applications use to authenticate to Vault, read the appropriate secret, use it, and be aware of when the secret changes. That assumes that all of the applications use the same programming language and that you can modify the source code. It’s repetitive, time consuming and it’s not declarative. Instead we can use the vault-secrets-operator and consume these secrets in a Kubernetes native method.

I like Helm because of how easy it makes templating Kubernetes yaml resources. While you could have Helm automatically apply charts to the cluster, you could also use the helm template command to have it render the yaml so that it can be saved and used in a GitOps workflow. Using Helm, with a cluster name variable, the VaultSecret can be created that declares which secret should be added to Kubernetes.

apiVersion: ricoberger.de/v1alpha1
kind: VaultSecret
metadata:
  name: myapp-db-creds
spec:
  path: secrets/myapp/{{ .Values.global.cluster_name }}/db-creds
  type: Opaque

After the VaultSecret has been applied to the cluster, a Kubernetes secret will exist with the values from Vault. The vault-secrets-operator keeps the secret in-sync so if it is modified in Vault, the resulting changes are automatically applied in Kubernetes. If you want to be extra precise, in the VaultSecret you could even define which version of the secret you want applied to the cluster/namespace. This is useful for when you want to stage rolling out the secret rotation or if you need to revert back to an older version of a secret.

At this point, the application manifest simply needs to reference the outcome and use the resulting secret in a Kubernetes native fashion. There’s no logic on the application side to test, validate & debug that it’s getting the correct secret. This manifest is the same no matter which cluster or environment you deploy it into.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp
spec:
  template:
    spec:
      containers:
      - name: myapp
        volumeMounts:
        - name: db-creds
          mountPath: "/etc/secrets/db-creds"
          readOnly: true
      volumes:
      - name: db-creds
        secret:
          secretName: myapp-db-creds

Personally, when I rotate a secret, I want the new value used immediately; I don’t want to require a deployment of the application. An app deployment seems like overkill to me for a simple secret rotation. There could be some debate as to whether or not this approach breaks the GitOps philosophy but I don’t believe it does. The benefits outweigh the concerns especially when dealing with large, slow to start applications.

To support the automatic reload of the secret, the application could accept a SIGHUP or provide a private endpoint (/sys/reload) to be called when the secret changes. Instead of writing a complicated process for watching the Kubernetes objects, you can run a sidecar container that watches for filesystem changes and sends a http request when the file(s) change. I frequently use the jimmidyson/configmap-reload container for this.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp
spec:
  template:
    spec:
      containers:
      - name: myapp
        volumeMounts:
        - name: db-creds
          mountPath: "/etc/secrets/db-creds"
          readOnly: true
      - name: config-reload
        image: jimmidyson/configmap-reload:v0.4.0
        args:
          - --volume-dir=/etc/secrets
          - --webhook-url=http://127.0.0.1:8080/sys/reload
        volumeMounts:
          - name: db-creds
            mountPath: /etc/secrets
            readOnly: true
      volumes:
      - name: db-creds
        secret:
          secretName: myapp-db-creds

The setup of the vault-secrets-operator itself is pretty straight forward. The documentation in the project does a good job of explaining it in detail. But there were a couple of lessons I learned that are worth sharing.

If you are using Vault Namespaces, the authentication path wasn’t what I expected. I had to include the namespace in the path instead of using the VAULT_NAMESPACE environment variable.

env:
  - name: VAULT_KUBERNETES_PATH
    value: "mynamespace/auth/k8s-cluster1"

The other challenge I faced was due to running many Kubernetes clusters. Each cluster has its own apiserver and certificates which means that each cluster needs a different Kubernetes Auth backend in Vault. While that’s not a bad thing, it did lead to more work setting things up than I expected.