Secrets need to be secure in transit as well as in rest. Here’s what that looks like for Amazon EKS and HashiCorp.
Secrets Management in EKS
When you are creating an EKS cluster, AWS gives you the option to enable etcd encryption using KMS keys. You can simply create a KMS key, turn on “Enable envelope encryption”, and select the KMS key to encrypt the secrets in etcd.
If you create the cluster without etcd encryption, then you can turn on this setting by navigating to the cluster’s configuration settings. However, after you enable encryption for the cluster, you can not disable it.
If you are using a tool such as Velero to back up your EKS cluster, you should note that secrets in Velero backups are not encrypted. Instead, they are stored as Base64-encoded JSON objects. Velero can use these JSON objects to recreate the objects when restoring. Below is a secret that has been created as a JSON object by Velero.
Now let’s look at how a third-party secrets-management solution, such as HashiCorp, can be used to manage secrets in Kubernetes. HashiCorp Vault uses a tool called vault-k8s and uses a Kubernetes mutating admission webhook to inject secrets into pods by using init and sidecar containers. Let’s dig a bit deeper into this approach.
Mutating admission webhooks can alter the original configuration of a Kubernetes resource. In the case of HashiCorp Vault, the mutation admission webhook checks the configuration of pods that are created in the cluster. If one of these pods is using secrets that are stored in HashiCorp Vault, the pod’s configuration will be changed to allow HashiCorp Vault to inject secrets into the pod. The mutating admission webhook determines whether the pod is using secrets that are stored in HashiCorp Vault by checking several annotations that are defined in the pod configuration. So, if a request to create any pod comes with these annotations, the mutating admission webhook will change the configuration of the pod.
The webhook performs several changes to the pod’s configuration. First, it mounts a shared memory volume to the container. This memory volume contains secrets that need to be passed to the containers in the pod. Next, the webhook injects two containers to the pod specification: init and sidecar. Before the containers in the pod are started, the init container loads the secret to the memory volume and terminates. Then, the sidecar container (along with other containers in the pod) runs and ensures that the secrets that are mounted in the memory volume are up to date. Either container can be disabled by using annotations.
A developer or an administrator does not need to be concerned about how all these mutations happen in the back end. They need only to configure the secrets in the vault and put the required annotations in the pod specification. All mutations will be handled by the mutating admission webhook that is transparent to developers.
Creating and using secrets with HashiCorp Vault
Now, let’s see the process of creating and using secrets with HashiCorp Vault in action. To try out HashiCorp Vault, I am using a nice HashiCorp playground that you can find in this tutorial: https://learn.hashicorp.com/tutorials/vault/kubernetes-sidecar. You can start this environment by clicking the “Show Terminal” button. You can also follow the steps in the tutorial to set up a Kubernetes cluster by using minikube and to install the vault by using a Helm chart. This tutorial also covers configuration and several use cases of injecting secrets into pods.
With a successfully configured test environment, let’s look at how the vault is configured and how it is functioning. We’ll start with the pods created by Vault.
Two pods are created by the vault. The vault-0 pod is responsible for storing secrets and other configuration data, and the vault-agent-injector pod is responsible for injecting vault agent containers into pods that meet the annotation criteria.
When creating a pod that needs to consume secrets in the vault, we need to specify that annotations.vault-agent-injector will look at these annotations and will inject the vault agent containers to our pod. Now, let’s create a pod that uses these annotations. This pod specification is taken from samples provided in the HashiCorp tutorial. Note that you have to first configure the secret in the vault by following the tutorial up to step 6.
Each annotation that is defined in the specification instructs the vault mutating admission webhook following:
Agent-inject: Enables agent injection to the pod
Agent-inject-status: Updates the secret when it’s rotated
Role: The Kubernetes role that has permission to read the secret data that is defined in the vault
Agent-inject-secret-database-config.txt: Is the secret to be loaded to the pod. “internal/data/database/config” is the path to the secret in the vault.
Agent-inject-template-database-config.txt: Defines the template to render the secret. In this example, the secret is rendered as a DB connection string.
When a pod with this specification is deployed, the vault mutating admission webhook changes the pod specification according to the defined annotations. You can see these changes by issuing the following command to look at the specification of the running pod:
Kubectl get pods <pod_name> -o yaml
You can see that the pod has been configured to have several volume mounts to mount the secret. It also has configuration related to the vault that the admission webhook injected into it.
Next, you can see the configuration of the sidecar container that was injected to the pod specification. The sidecar container is responsible for ensuring that the secrets that are loaded into the containers are up to date. If the secret expires or rotates when the pod is running, this sidecar container will update the value in /vault/secrets.
After the sidecar container, you can see initContainers, which was injected by the admission webhook. The init container is responsible for loading the secret to /vault/secrets before the main container starts.
As you can see, the initial pod configuration has been modified by the mutating admission webhook. The mutating admission webhook has modified the configuration of the containers that it had and has introduced new sidecar containers, init containers, and several volume mounts. The webhook has handled all these tasks transparently to the developers and administrators. The developer or administrator needs only to specify the pod specification and the annotations to enable the secret injection from the vault.
In this blog post, we have explored how secrets management happens in EKS and third-party secrets-management solutions such as HashiCorp Vault. In my next blog post on secrets management, I’ll investigate concepts such as dynamic secrets, secrets rotation, and secrets expiration.
Boominda is a Senior Consultant at Security Compass with more than 8 years of experience in Information Security. Throughout his career he has worked across multiple domains in information security such as application security, network security, cloud security, container security, incident response, forensics and on implementing security solutions such as SIEMs, PKI, WAFs and EDR. He has also worked as a visiting lecturer in several degree programs teaching undergraduates and students persuing their Masters about the concepts of Application Security and the art of security testing. He holds a MSc in Information Security, BSc in IT along with certificates such as CISSP, OSCP, CSSLP and CKA.