As penetration testers, we closely examine container environments for issues with secrets management. Secrets management includes how sensitive information such as passwords, OAuth tokens, SSH keys, and private keys is stored in containers.

Containers need to store and access secrets for tasks such as communicating with other microservices, calling back-end databases, and accessing other resources. These secrets need to be secure in transit as well as in rest. For example, when you create a container image that needs to be uploaded to a docker registry, this image should not store any secrets in plaintext. Also, when you have a container running in a production environment, that container should not store secrets in plaintext.

Container-orchestration platforms include Kubernetes, Docker Swarm, and Red Hat OpenShift; managed Kubernetes providers include EKS, AKS, and GKE; and secrets-management solutions include HashiCorp Vault and AWS Secrets Manager. In this blog post, I’ll focus on secrets management of Kubernetes.

This is the first blog post in our container secrets management series. In Part 2, I’ll take a look at the container orchestration platforms EKS and HashiCorp, a third-party solution. In future blog posts, I’ll dig deeper into concepts such as dynamic secrets, secret rotation and expiration, and attack vectors and misconfigurations that may allow an attacker to steal secrets.

Secrets Management in Kubernetes

By default, Kubernetes secrets are stored as unencrypted Base64-encoded strings, which goes against the best practices of secrets management. Anyone with Kubernetes API access can read these secrets in plaintext. To prevent this, encryption at rest can be enabled for secrets, and API access can be limited by using RBAC rules. 

We’ll first look at how a secret is defined in Kubernetes. Let’s say you have an .htpasswd file that you need to use to enable HTTP Basic Authentication in an Nginx server. If you want to store this file as a secret in Kubernetes, you can use this kubectl command:

kubectl create secret generic nginx-htpasswd --from-file .htpasswd

If you want to view all configured secrets, including the nginx-htpasswd secret you just created, you can run this command:

kubectl get secrets

How pods access secrets

A secret can be used with a pod in three main ways:

• As a file in a volume mounted on one or more of containers
• As a container environment variable
• By the kubelet when it is pulling images for the pod

We’ll focus on the first two methods. Both are directly used by containers in a pod to access secrets.

Loading secrets as a file in a volume 

In this method, we mount a volume to the container and put the secret in a file. Below is a YAML definition of a pod. It will mount our nginx-htpasswd secret in the /etc/nginx/conf folder. Any user who has access to this folder will be able to read this file. 

apiVersion: v1 
kind: Pod 
metadata: 
  name: nginx 
spec: 
  containers: 
  - name: nginx 
    image: nginx
    ports: 
    - containerPort: 80 
    volumeMounts: 
      - name: htpasswd-volume 
      mountPath: /etc/nginx/conf 
  volumes: 
  - name: htpasswd-volume 
    secret: 
      secretName: nginx-htpasswd

Loading secrets as a container environment variable

In this method, we mount the secret as an environment variable in the container. Below is a YAML definition of a pod. It will mount our nginx-htpasswd secret as an env variable. Any user who has command-execution privileges will be able to read environment variables and see the value of the secret. 

apiVersion: v1
kind: Pod
metadata:
  name: nginx
spec:
  containers:
  - name: nginx
    image: nginx
    ports:
    - containerPort: 80
    env:
      - name: env-password
        valueFrom:
          secretKeyRef:
            name: nginx-htpasswd
            key: .htpasswd

Security-hardening guidelines such as CIS benchmarks recommend loading a secret as a file in a volume rather than as an environment variable. 

Now let’s look at how these secrets are stored in Kubernetes and how they are consumed by the pods. 

How secrets are stored in Kubernetes

To view information about the secret that you created, you can type this kubectl command: 

kubectl get secret nginx-htpasswd -o yaml

This command will output the secret’s details in YAML format, as you can see in the following image:

In the data section of this output, you can see the filename that we inputted earlier as the key, which is .htpasswd. The file content is the value of this key and is Base64-encoded. Base64-decoding this value reveals the content of the .htpasswd file:

echo dXNlcjokYXByMSQ0SVZoR2V0USQ2djlGWWtpSkRRYTY4enIzVmNlbHUwCg== | base64 -d

When you add this value as a secret in Kubernetes, it is stored in etcd. Etcd stores data related to Kubernetes objects in a binary file. By default, this file is located in /var/lib/etcd/member/snap/db. You can also find the location of etcd by using the

command to look at the value of the –data-dir variable in the etcd process.

Now let’s see how the htpasswd file is stored in etcd. Etcd database is a binary file and can be queried with the etcdctl tool. For example, if you want to see the created secret, you can issue this command:

ETCDCTL_API=3 etcdctl --endpoints 127.0.0.1:2379  --cert=/etc/kubernetes/pki/etcd/server.crt   --key=/etc/kubernetes/pki/etcd/server.key   --cacert=/etc/kubernetes/pki/etcd/ca.crt   get /registry/secrets/default/nginx-htpasswd

However, to query etcd by using etcdctl, you need to have access to etcd key and crt files. So, without using the etcd tool, I’m going to use the strings tool to interact with the etcd db. In the command below, I used the strings tool to list all the strings in the etcd. I’m going to search for the “apr1” string, which is part of the data in the .htpasswd file.

Interestingly, we got the value of the secret in plaintext format from the db file. Now, let’s expand our criteria a bit to get the before and after lines of the matched string.

The output shows that the secrets are stored in plaintext. 

Now let’s create two more secrets and see whether we get the same results. I used this command:

kubectl create secret generic test-db-secret --from-literal=username=myuser --from-literal=password=mypass

Now let’s search for these secrets in etcd. This time, I’m going to search for the “Opaque” string and the two lines above the matched string.

In the second secret, I got one key-value pair. Let’s increase the number of lines above the matched string to four.

Now, we have both secrets in plaintext. Using the same method, I can also search for service-account tokens, which are another type of secret stored in etcd:

strings db | grep -B 1 -A 1 "service-account-token"

Likewise, we can search for and obtain secrets from etcd in plaintext if we have access to the etcd db. Kubernetes provides a way to encrypt secrets at rest (you can find more information at https://kubernetes.io/docs/tasks/administer-cluster/encrypt-data/). Let’s try it. 

Implementing encryption in etcd

You can make a file with EncryptionConfiguration in the /etc/kubernetes/pki folder (i.e. /etc/kubernetes/pki/encryptConfig.yml) in the control plane node:

Next, you need to update the –encryption-provider-config parameter in kube-apiserver to point to EncryptionConfiguration in the /etc/kubernetes/manifests/kube-apiserver.yaml file. After the file is updated, the pod that is running kube-apiserver will automatically restart with the new configuration.

Now let’s try to create some secrets and see how the encryption is working. Use the below command to create a new secret.

kubectl create secret generic my-encrypted-secret --from-literal=password=secretPassword

You can view the generated secret by using kubectl get command like below

Kubectl get secret my-encrypted-secreted -o yaml

As shown in the above screenshot, the value of the password is available in a Base64-encoded format. But let’s see how the value is stored in the etcd database. For this, I’m first going to try the strings command that we used earlier.

The strings command did not return the value of the encrypted string. So, let’s try to grep the secret by its name.

Based on the output, it looks like the secret is now in etcd in an encrypted format. Let’s verify this observation by using the etcdctl tool:

ETCDCTL_API=3 etcdctl --endpoints 127.0.0.1:2379  --cert=/etc/kubernetes/pki/etcd/server.crt   --key=/etc/kubernetes/pki/etcd/server.key   --cacert=/etc/kubernetes/pki/etcd/ca.crt   get /registry/secrets/default/my-encrypted-secret

With the above output, we can see that the secret is now encrypted. You should note the following about the process of encrypting etcd: 

• The encryption key that we used is stored in the EncryptionConfiguration yaml file. Therefore, if an attacker compromises the control-plane node that stores this yaml file, they will be able to access the key that was used for the encryption. 
• The secrets that you created before enabling encryption will not be encrypted. You will have to delete and recreate these secrets to be able to enable encryption on them. 
• Encrypting etcd will not prevent someone who has access to secrets from obtaining them in plaintext. You need to ensure that RBAC is properly enforced throughout the Kubernetes cluster.

In this blog post, we explored how secrets management happens in core Kubernetes. In my next blog post on secrets management, I’ll explore EKS, and the third-party secrets-management solution HashiCorp Vault.

Advisory Labs