Keep your Kubernetes secrets in git with Kubesec 🔒

Estimated read time: 5 minutes

I will introduce you to a tool named kubesec. It allows to store kubernetes secrets in git but only a few will be able to read them: you, your team and the CI 💪.

The main benefits of using git for everything, from dev to operation, is to be able to follow the evolution of something, find how and why something has changed (thanks to conventional-changelog 🇫🇷) and ability to revert it when we want.

But secrets, like passwords, api keys or sensitive information, should not be stored in the repository mainly because git, by design, doesn’t allow file restriction access. What if we can now store them in git, leveraging the benefit of the versioning system without exposing our data to everyone ? This is the purpose of kubesec, a tool inspired by SOPS, dedicated to Kubernetes secrets.

Kubesec Installation

To do that, like many tools in the Kubernetes world, you can use brew (Linux & MacOs) or download it directly from Github. All the information is available on kubesec official installation guide.

GPG Configuration

To be able to store secrets in your git repository, you will need to encrypt them before committing them. To do so, kubesec relies on PGP keys and on your local gpg binary. You can also use keys stored in external system like Google KMS or Amazon KMS. In this article, we will focus only on local PGP keys.

We need to provide dedicated keys to our CI, so I advise you to create a new pair of keys following this pretty good documentation provided by Gitlab 🦊.

At the end, you should have at least one key of type S and one of type E:

$ gpg --list-secret-keys --keyid-format LONG ci@blog-kubesec.davinkevin.gitlab.com
sec   rsa4096/OAEFIJEFAOIJOEFA 2019-08-23 [SC] # Type 'S'
      OIPAEFFKJAJOIEA564FEAF4EA6G54EA65G4A654A
uid                 [ultimate] blog-kubesec <ci@blog-kubesec.davinkevin.gitlab.com>
ssb   rsa4096/OJA21FE56A456AAC 2019-08-23 [E] # Type 'E'

In this command result, you can see keys attached to my (fake) email ci@blog-kubesec.davinkevin.gitlab.com.

Encrypt a secret

We will work with a standard secret for this example coming from the kubernetes.io secret documentation:

$ cat secret.yaml
apiVersion: v1
kind: Secret
metadata:
  name: mysecret
type: Opaque
data:
  username: YWRtaW4=
  password: MWYyZDFlMmU2N2Rm

To encrypt this, you will have to type the following command:

$ kubesec encrypt secret.yaml > secret.enc.yaml && cat secret.enc.yaml
apiVersion: v1
data:
  password: 3sVfuE04hcXCwvhyMUjaCorqV875Am4A2k3W7ru7VWPDxDKzD+RQHSXJg7erZhqU.6C+t9OB/285GyOr8.FmG0yrZZ0/uIelLGa8/4mw==
  username: OPzpwkT8a/HVkbETAZ5twTFPsEFPoRjLBdy6ZyJjwfUh34+pnQUPIaccEGiMIBe8.xMoswOBSfsvv/PP9.utpc9ljz/xCthLk2mNTKow==
kind: Secret
metadata:
  name: mysecret
type: Opaque
# kubesec:v:3
# kubesec:pgp:4518B75D674DF44FE3A06E6ACD58CFC03A583C51:LS0tLS1CRUdJTiBQR1AgTUVTU0FHRS0tLS0tCgpoUUlNQTk3V1ZLWXZKTzBhQVEvL2UxZTJtUi9PZ3JFSVo3cjZuQVBTS0E5VzN3WW9mQnl4MXNqdGp1TjQ0WFBqCmJoM0ZxVDkrdWF0eVZKYVJYTG5hdkFLRThIblJZUXVoL002b25kamw4Nlk2WnFkZGFGbzVJV3VwVUVxYnFZOEUKODVuWkk0ekxqU0o2MFp4cmREWjBXdlRreU1TUU9NeitsMHd6eDBIdVFOMStRaDlDeU1TeGxjVFB3T0ExaXBzTApYRHNvNzN4MzdlOUs2Qi96ZkJTVTVMdE12TVh0U2pMK05rbHk1UHpqZjJUbVhNSUZpTnNOc0I2QTVwd1ZRdkJmCloyWU0yRU1xL1dvMTVEL2FwNXdKWVVxWlk2K0Rnb1NLQStILzJoL2paUGxGZ0pHTUFsTnBzb3o0cm9kOG5la0EKRUcwNmVYSzlweVM1anFiMzZDUDJNUEczU2J3TXR0RlNtaTBqL1F0MkNGVzdBT1JHbWdodzRxc3QzUkhKMDYrUgo4Uld1b1hldHFRU3FDeWNpeHdhVkdObk9hZnd5Tkk3VjV1UnpQTzQ0aHpTUFNHb3l6Q0pBRGl0S0VNTmp4andKCmNHalc1WFlISHlXRFFmS1A1dkRtRis5WkR4SGZDOWxvWDdCbkRDRVpuNVlHWXhyVUowUXVuanV2SXpBdlRxdGMKakhLMUxSNVBZUUtTbVZqckpSc2wxV2NscHFCMmtPZUszdGpIVmJiZ0QvOWdVdVJQOWFJMGJPMU5jMjdiRVVlNgowSUFGZS9rUUdRYUhRSlFHdWJaOW1udlMrTlJmUUZkODJ5TTJSbWNZMTJyVHBNZ01OaGNsZmorVm03U1l5cWpFCkZ1MzNTZ05KYVhNanRyc01JNUdZb1lJbVY1K3ZzZk92VVkrYnNQWnl1NFdRMWliU25WWWV4WGtUc1FzbTV0clMKNlFGbHB3dDZDeGpweEtiaVVaSVFDMTdlTUZ3eEFlWWpZeUtoc25EbWZQWU1VNnNnNnMxa015SGhBcm1pMHhqRgpiKzBEbVhpazlmUFFoSVM0Z3NqV1F0bEhXdlFGSDN0OGRmTThMUW5VR0xKelJ3Ti9FWkxWVDdnTHJ6ZlhDQ2dKCmsrbTc1VW9rMlNTTmlFYUlxMkZTTWlIYmFqYkhndHlxWDIycVZHVkxucitBNmF6SFEwRVkwU0h0eEtpa0d1cWsKcVlmWkEwL0VNOFVPTms5QmtvTStRM3I2dWJ3b0xBeGFDdUtHWnQrOHRFbDJlVFJXN3hoNWhNbDV4M0piaHRoTAprQlNrSXkyU1pwK1ZUa1NvdHRxNEhvc0V1Z1djckRrRGp4ZUdLS05RWTFaSzNEOUFBRDVPZTZPY0VVTDA2cFk2CisvNWV2WDc5RjdOL1ZCSzZFMkprOEN5STNtR1VBL2t0Q3hPSTZ1UGtTclA2ZEdrU3VFbThMKzdnWXJBNFVDL0sKODdJUW14TFIzeDFtTHZhT0dqR2dFdm9VNS9DbUF5dFhONVRGZVpjUmtzUVF0SW44Tml0TXQ1VzBzVlVWSGJIYgoyNVVzVTBoWklLMnZEa3lvcGNDL2tDc1FHYllNNHppNm5pMGJFUVduZzcxUTJOT2Rpa2VKQzIwT0pZa1dVa2xMCkd1a0FxY0ZYSkRPbVQzNW8xK2hIZjZKLzVJNUF0bmtzNXZGb1JkT0lIUjFnZnd4d0tzT1d5czBVMHlnNWI5SmcKVkJQWHNNVWFDWU52TmpmRytIU1JDdGlweGNlRGMzMXZ2b0lyZGVzb1pRV3Q4T1VUeUh0SnQ2aHEwWWNjKzJEMwpWbXg1eFMzODZsaXJhTFJyakY4NVVyZFVOeHQyTEVsTklBaTVXYjYwT0NEVXJOd3BjbkxiWlhrcXdob004ZEU2ClF4RmN1WlREM2RFd2MvdDRqbGM0S3V4Ujd5UGRING9RVDVyOWFLZXo4SGRXajFvRHg1TU12V1RCRlJjRWtENkkKcW1uYUM1bHpJZkMxa1BCQmQyWmpjcWZsOXd6cEh3QzZSUGRIQUREVjJOeVZaYmw4VDFvTjhEZS9GeVVSV3R2bwpFQlpsME51WXJBOGl4c0VYYnNLSWRIbFRyNndlYUZKVFdUamdyWDRTbm9jQUx3V2RmZi9PQzhQZGtkNmlxa1pwCmU3Kzkvcytpa2RnVkRpRHNUVU09Cj1xcjlTCi0tLS0tRU5EIFBHUCBNRVNTQUdFLS0tLS0K
# kubesec:mac:jY3fQeqNV/UOA7QC.4LS+pp8ItybVPneOPJ8S+A==

The following result should be stored in your git repository. In this example, we will name it secrets.enc.yaml.

Note: kubesec proposes many way of doing things. You can for example easily edit your k8s secrets with your favorite $EDITOR using the command line: kubesec edit -i secret.enc.yml.

If you want to test a decrypt operation, you can run the following command:

$ kubesec decrypt secret.enc.yaml
apiVersion: v1
data:
  username: YWRtaW4=
  password: MWYyZDFlMmU2N2Rm
kind: Secret
metadata:
  name: mysecret
type: Opaque

Note: Bonus, kubesec command used with the parameter -x can do the base64 decrypt for you, very helpful 👍.

Gitlab CI Configuration 🦊

This example shows how to configure Gitlab in order to decrypt secrets but this is totally compatible with other kinds of CI.

First, you need to export the private PGP key in order to add it as a Gitlab variable. You can do that by using the following command:

$ gpg --export-secret-key --armor > blog-kubesec.davinkevin.gitlab.com.private

After that, you will have access to the private key of your CI system. Be careful with it, this private key is the only element allowing decryption of your data, you should not loose it, nor share it widely 😅.

Now, you can copy the content of the private key in Gitlab, as PGP_PRIVATE_KEY with type file and define the following job:

decrypt:
  image: google/cloud-sdk # Because I use Google Kubernetes Engine to deploy ❤️
  before_script:
    - gpg --import $PGP_PRIVATE_KEY
    - curl -sSL https://github.com/shyiko/kubesec/releases/download/0.9.2/kubesec-0.9.2-linux-amd64 -o kubesec && chmod a+x kubesec && mv kubesec /usr/local/bin/
  script:
    - kubesec decrypt secret.enc.yaml -x

You can follow the result of my demo pipeline on dedicated Gitlab project

This is of course a proof of concept, but in a real project you would probably have the following configuration of your CI:

deploy:
  image: google/cloud-sdk
  before_script:
    - gpg --import $PGP_PRIVATE_KEY
    - curl -sSL https://github.com/shyiko/kubesec/releases/download/0.9.2/kubesec-0.9.2-linux-amd64 -o kubesec && chmod a+x kubesec && mv kubesec /usr/local/bin/
  script:
    - kubesec decrypt -i secret.yaml
    - kubectl apply -f *.yaml

Go further with kubesec 🚀

In this article, we chose to create and use the PGP private key of the CI to encrypt our secrets in the first step.

But in a real world case, each dev & ops should use its own PGP key to encrypt secrets. The private key of the CI should only be known… by the CI 👍.

Kubesec can manage multiple keys. You just have to specify extra keys during encryption step, like this:

kubesec encrypt --key:pgp:key_of_ci --key:pgp:key_of_alice --key:pgp:key_of_bob secret.yaml

At the end, the CI will be able to decrypt the secret thanks to its private key, same for each dev or ops.

Note: An issue has been raised about decrypt operation when you didn’t have access to the public key of others, I hope it will be fixed soon 🤞.