Attacking & Defending Container Secrets Part 1: Kubernetes Secrets Management
Written By: Boominda Anushka
Share this post
As penetration testers, we closely examine container environments for issues with secrets management. Secrets management includes how sensitive information such as passwords, OAuth tokens, SSH keys, and private keys is stored in containers.
Containers need to store and access secrets for tasks such as communicating with other microservices, calling back-end databases, and accessing other resources. These secrets need to be secure in transit as well as in rest. For example, when you create a container image that needs to be uploaded to a docker registry, this image should not store any secrets in plaintext. Also, when you have a container running in a production environment, that container should not store secrets in plaintext.
Container-orchestration platforms include Kubernetes, Docker Swarm, and Red Hat OpenShift; managed Kubernetes providers include EKS, AKS, and GKE; and secrets-management solutions include HashiCorp Vault and AWS Secrets Manager. In this blog post, I’ll focus on secrets management of Kubernetes.
This is the first blog post in our container secrets management series. In Part 2, I’ll take a look at the container orchestration platforms EKS and HashiCorp, a third-party solution. In future blog posts, I’ll dig deeper into concepts such as dynamic secrets, secret rotation and expiration, and attack vectors and misconfigurations that may allow an attacker to steal secrets.
Secrets Management in Kubernetes
By default, Kubernetes secrets are stored as unencrypted Base64-encoded strings, which goes against the best practices of secrets management. Anyone with Kubernetes API access can read these secrets in plaintext. To prevent this, encryption at rest can be enabled for secrets, and API access can be limited by using RBAC rules.
We’ll first look at how a secret is defined in Kubernetes. Let’s say you have an .htpasswd file that you need to use to enable HTTP Basic Authentication in an Nginx server. If you want to store this file as a secret in Kubernetes, you can use this kubectl command:
If you want to view all configured secrets, including the nginx-htpasswd secret you just created, you can run this command:
kubectl get secrets
How pods access secrets
A secret can be used with a pod in three main ways:
As a file in a volume mounted on one or more of containers
As a container environment variable
By the kubelet when it is pulling images for the pod
We’ll focus on the first two methods. Both are directly used by containers in a pod to access secrets.
Loading secrets as a file in a volume
In this method, we mount a volume to the container and put the secret in a file. Below is a YAML definition of a pod. It will mount our nginx-htpasswd secret in the /etc/nginx/conf folder. Any user who has access to this folder will be able to read this file.
Loading secrets as a container environment variable
In this method, we mount the secret as an environment variable in the container. Below is a YAML definition of a pod. It will mount our nginx-htpasswd secret as an env variable. Any user who has command-execution privileges will be able to read environment variables and see the value of the secret.
Security-hardening guidelines such as CIS benchmarks recommend loading a secret as a file in a volume rather than as an environment variable.
Now let’s look at how these secrets are stored in Kubernetes and how they are consumed by the pods.
How secrets are stored in Kubernetes
To view information about the secret that you created, you can type this kubectl command:
kubectl get secret nginx-htpasswd -o yaml
This command will output the secret’s details in YAML format, as you can see in the following image:
In the data section of this output, you can see the filename that we inputted earlier as the key, which is .htpasswd. The file content is the value of this key and is Base64-encoded. Base64-decoding this value reveals the content of the .htpasswd file:
When you add this value as a secret in Kubernetes, it is stored in etcd. Etcd stores data related to Kubernetes objects in a binary file. By default, this file is located in /var/lib/etcd/member/snap/db. You can also find the location of etcd by using the
command to look at the value of the –data-dir variable in the etcd process.
Now let’s see how the htpasswd file is stored in etcd. Etcd database is a binary file and can be queried with the etcdctl tool. For example, if you want to see the created secret, you can issue this command:
ETCDCTL_API=3 etcdctl --endpoints 127.0.0.1:2379 --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key --cacert=/etc/kubernetes/pki/etcd/ca.crt get /registry/secrets/default/nginx-htpasswd
However, to query etcd by using etcdctl, you need to have access to etcd key and crt files. So, without using the etcd tool, I’m going to use the strings tool to interact with the etcd db. In the command below, I used the strings tool to list all the strings in the etcd. I’m going to search for the “apr1” string, which is part of the data in the .htpasswd file.
Interestingly, we got the value of the secret in plaintext format from the db file. Now, let’s expand our criteria a bit to get the before and after lines of the matched string.
The output shows that the secrets are stored in plaintext.
Now let’s create two more secrets and see whether we get the same results. I used this command:
You can make a file with EncryptionConfiguration in the /etc/kubernetes/pki folder (i.e. /etc/kubernetes/pki/encryptConfig.yml) in the control plane node:
Next, you need to update the –encryption-provider-config parameter in kube-apiserver to point to EncryptionConfiguration in the /etc/kubernetes/manifests/kube-apiserver.yaml file. After the file is updated, the pod that is running kube-apiserver will automatically restart with the new configuration.
Now let’s try to create some secrets and see how the encryption is working. Use the below command to create a new secret.
You can view the generated secret by using kubectl get command like below
Kubectl get secret my-encrypted-secreted -o yaml
As shown in the above screenshot, the value of the password is available in a Base64-encoded format. But let’s see how the value is stored in the etcd database. For this, I’m first going to try the strings command that we used earlier.
The strings command did not return the value of the encrypted string. So, let’s try to grep the secret by its name.
Based on the output, it looks like the secret is now in etcd in an encrypted format. Let’s verify this observation by using the etcdctl tool:
ETCDCTL_API=3 etcdctl --endpoints 127.0.0.1:2379 --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key --cacert=/etc/kubernetes/pki/etcd/ca.crt get /registry/secrets/default/my-encrypted-secret
With the above output, we can see that the secret is now encrypted. You should note the following about the process of encrypting etcd:
The encryption key that we used is stored in the EncryptionConfiguration yaml file. Therefore, if an attacker compromises the control-plane node that stores this yaml file, they will be able to access the key that was used for the encryption.
The secrets that you created before enabling encryption will not be encrypted. You will have to delete and recreate these secrets to be able to enable encryption on them.
Encrypting etcd will not prevent someone who has access to secrets from obtaining them in plaintext. You need to ensure that RBAC is properly enforced throughout the Kubernetes cluster.
In this blog post, we explored how secrets management happens in core Kubernetes. In my next blog post on secrets management, I’ll explore EKS, and the third-party secrets-management solution HashiCorp Vault.
Boominda is a Senior Consultant at Security Compass with more than 8 years of experience in Information Security. Throughout his career he has worked across multiple domains in information security such as application security, network security, cloud security, container security, incident response, forensics and on implementing security solutions such as SIEMs, PKI, WAFs and EDR. He has also worked as a visiting lecturer in several degree programs teaching undergraduates and students persuing their Masters about the concepts of Application Security and the art of security testing. He holds a MSc in Information Security, BSc in IT along with certificates such as CISSP, OSCP, CSSLP and CKA.