Kubernetes Secrets Encryption at Rest: What It Protects

Kubernetes encryption at rest protects one thing: the Secret data stored in etcd. It stops someone who gets etcd’s data, or a copy of it, from reading Secrets in plain text. It does not stop anyone the API server is willing to serve, and that group is larger than most teams assume: everyone with get, list or watch on Secrets, and everyone who can create a Pod in the namespace. Turn encryption on, but don’t count it as your Secrets access control.

What Kubernetes does by default

The Kubernetes documentation is blunt: “Kubernetes Secrets are, by default, stored unencrypted in the API server’s underlying data store (etcd).” The default provider is identity, which, in the docs’ words, “does not encrypt stored data and provides no additional confidentiality protection.”

The base64 you see in a Secret manifest doesn’t change that. The good-practices page says it plainly: “Base64 encoding is not an encryption method, it provides no additional confidentiality over plain text.” A Secret manifest committed to Git is a plain-text credential in Git.

Managed Kubernetes services run the control plane for you, and their defaults may differ from upstream. This article describes upstream behavior; check your provider’s documentation for what it encrypts by default and whether it supports envelope encryption with your own key.

Who can read a Secret

Before choosing an encryption provider, list who can read Secrets at all. Encryption at rest changes only the last two rows of this table.

The middle row surprises people. The Secret documentation says “anyone who is authorized to create a Pod in a namespace can use that access to read any Secret in that namespace; this includes indirect access such as the ability to create a Deployment.” Our Kubernetes RBAC audit covers how to find these grants in your bindings; this article stays on the storage side.

How encryption at rest is configured

You pass an EncryptionConfiguration file to kube-apiserver with --encryption-provider-config. A minimal example for Secrets with a local AES-CBC key, shaped like the one in the Kubernetes task page:

apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources:
      - secrets
    providers:
      - aescbc:
          keys:
            - name: key1
              secret: <BASE64-ENCODED 32-BYTE KEY>
      - identity: {} # lets the API server read Secrets written before encryption

Three rules from the documentation decide what this file does:

• The first provider encrypts. “The first provider in the list is used to encrypt resources written into the storage.” If identity is first, new writes are plain text even if other providers are listed.
• Later providers only decrypt. On reads, each matching provider is tried in order. The trailing identity is what lets the API server keep reading old plain-text Secrets during a migration. Once everything is rewritten, you can remove it.
• Order across rules matters too. The resources list is processed in order, so an earlier rule that names secrets with identity will win over a later *.* wildcard that encrypts. Wildcards are *. for the core group (which includes Secrets) and *.* for everything.

Which provider stops which attacker

The key-location column is the one that maps to threats. With a local key, the docs say encryption “protects against an etcd compromise, but it fails to protect against a host compromise,” because the key sits in the configuration file on the host. With KMS, Kubernetes encrypts data with a data encryption key and the KMS wraps that key, so an attacker “would need to compromise etcd and the third-party KMS provider.”

A local key is still worth having. It turns a leaked etcd snapshot or a mis-scoped etcd client certificate into ciphertext rather than credentials. Our view: use KMS v2 where you have a KMS to point at, and treat a local aescbc key as a stopgap you plan to replace.

Checking a configuration file

We wrote a short checker that reads an EncryptionConfiguration and flags the weak settings above. It reads only the file and does not contact a cluster. It needs Python 3 and PyYAML; we ran it with Python 3 and PyYAML 6.0.3 against three sample files. It is illustrative: it does not validate the whole schema, and it reads kms without an apiVersion as v1.

#!/usr/bin/env python3
"""Flag weak settings in a Kubernetes EncryptionConfiguration file.
Usage: python3 check_encryption_config.py /path/to/encryption-config.yaml
Reads the file only; it does not contact a cluster."""
import sys
import yaml

LOCAL_KEY = {"aescbc", "aesgcm", "secretbox"}

def covers_secrets(resources):
    return any(r in ("secrets", "*.", "*.*") for r in resources)

def main(path):
    with open(path) as f:
        cfg = yaml.safe_load(f)
    findings = []
    secrets_rule_seen = False
    for i, rule in enumerate(cfg.get("resources", [])):
        res = rule.get("resources", [])
        if not covers_secrets(res):
            continue
        # The first matching rule wins, so only the first rule covering secrets counts.
        if secrets_rule_seen:
            continue
        secrets_rule_seen = True
        providers = rule.get("providers", [])
        first = next(iter(providers[0])) if providers else None
        if first == "identity":
            findings.append(f"rule {i}: first provider is identity, so new Secrets are written in plain text")
        elif first in LOCAL_KEY:
            findings.append(f"rule {i}: first provider is {first}; key sits in this file on every control plane host")
            if first == "aescbc":
                findings.append(f"rule {i}: aescbc is not recommended by the Kubernetes docs (padding oracle)")
        elif first == "kms":
            ver = providers[0]["kms"].get("apiVersion", "v1")
            if ver != "v2":
                findings.append(f"rule {i}: KMS v1 is deprecated since Kubernetes v1.28; move to KMS v2")
        if any("identity" in p for p in providers[1:]):
            findings.append(f"rule {i}: identity fallback present; unencrypted Secrets can still be read (expected during migration)")
    if not secrets_rule_seen:
        findings.append("no rule covers secrets: Secrets are stored unencrypted")
    for f_ in findings:
        print("WARN", f_)
    if not findings:
        print("OK no weak settings found for secrets")
    return 1 if findings else 0

if __name__ == "__main__":
    sys.exit(main(sys.argv[1]))

Results on our samples: the aescbc-plus-identity file above produced three warnings and exit code 1; a file using kms with apiVersion: v2 printed OK and exited 0; a file that only configured ConfigMaps reported that no rule covers Secrets. The non-zero exit code lets you run it in CI against the configuration you deploy to control plane nodes.

Rolling it out without locking yourself out

The Kubernetes task page walks through the full procedure. The steps that most often go wrong:

1. Every control plane host needs the same file and key. The docs say to copy the key to every control plane host over a secure channel, and to restart each API server. An API server without the key can’t read Secrets another one encrypted.

2. Existing Secrets are not touched. Encryption applies on write. Rewrite everything once all API servers have the new configuration:

   kubectl get secrets --all-namespaces -o json | kubectl replace -f -
   

3. Verify in etcd, not through the API. The API always returns decrypted values, so kubectl get secret proves nothing. Read a Secret with etcdctl get /registry/secrets/<namespace>/<name> and check that the stored value starts with k8s:enc:<provider>:v1:<key-name>:, for example k8s:enc:aescbc:v1:key1:.

4. Rotate keys in two passes. The docs’ zero-downtime procedure: add the new key as the second entry on every control plane node and restart all API servers so each can decrypt with it; back the key up; then make it the first entry, restart again, rewrite all Secrets, and only then remove the old key. Losing every copy of a key means deleting the resources encrypted with it.

Controls that matter as much as encryption

Encryption at rest is one line in the Kubernetes Secrets good-practices list. The others address the readers in the first table:

• Restrict get, list and watch. The docs say to limit watch and list to “the most privileged, system-level components” and caution that list “implicitly lets the subject fetch the contents of the Secrets.”
• Separate namespaces by trust. Pod creation reaches every Secret in a namespace, so a namespace is the practical boundary. Enforce Pod Security Admission where workload creators are less trusted, and consider ValidatingAdmissionPolicy for rules Pod Security doesn’t cover.
• Audit Secret access at the Metadata level. The example audit policy in the Kubernetes docs logs Secrets and ConfigMaps at Metadata, which records who accessed what without writing values into your logs. The good-practices page suggests alerting on events such as one user reading many Secrets at once.
• Use short-lived credentials and external stores. The docs recommend short-lived Secrets and mention the Secrets Store CSI Driver for keeping values in an external store. The same reasoning drives least privilege in cloud IAM: a credential that expires in an hour is worth less to whoever reads it.

Limits of this article

This describes upstream Kubernetes as documented on kubernetes.io at the time of writing. We tested the checker against sample files only, not against a live cluster or a managed service. We have not tested the encryption rollout procedure ourselves; follow the Kubernetes task page and your distribution’s documentation, and rehearse on a non-production cluster first.

Sources and verification

KubernetesSecrets ManagementEncryptionEtcd