Kubernetes encryption at rest protects one thing: the Secret data stored in etcd. It stops someone who gets etcd’s data, or a copy of it, from reading Secrets in plain text. It does not stop anyone the API server is willing to serve, and that group is larger than most teams assume: everyone with get, list or watch on Secrets, and everyone who can create a Pod in the namespace. Turn encryption on, but don’t count it as your Secrets access control.

What Kubernetes does by default

The Kubernetes documentation is blunt: “Kubernetes Secrets are, by default, stored unencrypted in the API server’s underlying data store (etcd).” The default provider is identity, which, in the docs’ words, “does not encrypt stored data and provides no additional confidentiality protection.”

The base64 you see in a Secret manifest doesn’t change that. The good-practices page says it plainly: “Base64 encoding is not an encryption method, it provides no additional confidentiality over plain text.” A Secret manifest committed to Git is a plain-text credential in Git.

Managed Kubernetes services run the control plane for you, and their defaults may differ from upstream. This article describes upstream behavior; check your provider’s documentation for what it encrypts by default and whether it supports envelope encryption with your own key.

Who can read a Secret

Before choosing an encryption provider, list who can read Secrets at all. Encryption at rest changes only the last two rows of this table.

The middle row surprises people. The Secret documentation says “anyone who is authorized to create a Pod in a namespace can use that access to read any Secret in that namespace; this includes indirect access such as the ability to create a Deployment.” Our Kubernetes RBAC audit covers how to find these grants in your bindings; this article stays on the storage side.

How encryption at rest is configured

You pass an EncryptionConfiguration file to kube-apiserver with --encryption-provider-config. A minimal example for Secrets with a local AES-CBC key, shaped like the one in the Kubernetes task page:

apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources:
      - secrets
    providers:
      - aescbc:
          keys:
            - name: key1
              secret: <BASE64-ENCODED 32-BYTE KEY>
      - identity: {} # lets the API server read Secrets written before encryption

Three rules from the documentation decide what this file does:

• The first provider encrypts. “The first provider in the list is used to encrypt resources written into the storage.” If identity is first, new writes are plain text even if other providers are listed.
• Later providers only decrypt. On reads, each matching provider is tried in order. The trailing identity is what lets the API server keep reading old plain-text Secrets during a migration. Once everything is rewritten, you can remove it.
• Order across rules matters too. The resources list is processed in order, so an earlier rule that names secrets with identity will win over a later *.* wildcard that encrypts. Wildcards are *. for the core group (which includes Secrets) and *.* for everything.

Which provider stops which attacker

The key-location column is the one that maps to threats. With a local key, the docs say encryption “protects against an etcd compromise, but it fails to protect against a host compromise,” because the key sits in the configuration file on the host. With KMS, Kubernetes encrypts data with a data encryption key and the KMS wraps that key, so an attacker “would need to compromise etcd and the third-party KMS provider.”

A local key is still worth having. It turns a leaked etcd snapshot or a mis-scoped etcd client certificate into ciphertext rather than credentials. Our view: use KMS v2 where you have a KMS to point at, and treat a local aescbc key as a stopgap you plan to replace.

Checking a configuration file

We wrote a short checker that reads an EncryptionConfiguration and flags the weak settings above. It reads only the file and does not contact a cluster. It needs Python 3 and PyYAML; we ran it with Python 3 and PyYAML 6.0.3 against three sample files. It is illustrative: it does not validate the whole schema, and it reads kms without an apiVersion as v1.

#!/usr/bin/env python3
"""Flag weak settings in a Kubernetes EncryptionConfiguration file.
Usage: python3 check_encryption_config.py /path/to/encryption-config.yaml
Reads the file only; it does not contact a cluster."""
import sys
import yaml

LOCAL_KEY = {"aescbc", "aesgcm", "secretbox"}

def covers_secrets(resources):
    return any(r in ("secrets", "*.", "*.*") for r in resources)

def main(path):
    with open(path) as f:
        cfg = yaml.safe_load(f)
    findings = []
    secrets_rule_seen = False
    for i, rule in enumerate(cfg.get("resources", [])):
        res = rule.get("resources", [])
        if not covers_secrets(res):
            continue
        # The first matching rule wins, so only the first rule covering secrets counts.
        if secrets_rule_seen:
            continue
        secrets_rule_seen = True
        providers = rule.get("providers", [])
        first = next(iter(providers[0])) if providers else None
        if first == "identity":
            findings.append(f"rule {i}: first provider is identity, so new Secrets are written in plain text")
        elif first in LOCAL_KEY:
            findings.append(f"rule {i}: first provider is {first}; key sits in this file on every control plane host")
            if first == "aescbc":
                findings.append(f"rule {i}: aescbc is not recommended by the Kubernetes docs (padding oracle)")
        elif first == "kms":
            ver = providers[0]["kms"].get("apiVersion", "v1")
            if ver != "v2":
                findings.append(f"rule {i}: KMS v1 is deprecated since Kubernetes v1.28; move to KMS v2")
        if any("identity" in p for p in providers[1:]):
            findings.append(f"rule {i}: identity fallback present; unencrypted Secrets can still be read (expected during migration)")
    if not secrets_rule_seen:
        findings.append("no rule covers secrets: Secrets are stored unencrypted")
    for f_ in findings:
        print("WARN", f_)
    if not findings:
        print("OK no weak settings found for secrets")
    return 1 if findings else 0

if __name__ == "__main__":
    sys.exit(main(sys.argv[1]))

Results on our samples: the aescbc-plus-identity file above produced three warnings and exit code 1; a file using kms with apiVersion: v2 printed OK and exited 0; a file that only configured ConfigMaps reported that no rule covers Secrets. The non-zero exit code lets you run it in CI against the configuration you deploy to control plane nodes.

Rolling it out without locking yourself out

The Kubernetes task page walks through the full procedure. The steps that most often go wrong:

1. Every control plane host needs the same file and key. The docs say to copy the key to every control plane host over a secure channel, and to restart each API server. An API server without the key can’t read Secrets another one encrypted.

2. Existing Secrets are not touched. Encryption applies on write. Rewrite everything once all API servers have the new configuration:

   kubectl get secrets --all-namespaces -o json | kubectl replace -f -
   

3. Verify in etcd, not through the API. The API always returns decrypted values, so kubectl get secret proves nothing. Read a Secret with etcdctl get /registry/secrets/<namespace>/<name> and check that the stored value starts with k8s:enc:<provider>:v1:<key-name>:, for example k8s:enc:aescbc:v1:key1:.

4. Rotate keys in two passes. The docs’ zero-downtime procedure: add the new key as the second entry on every control plane node and restart all API servers so each can decrypt with it; back the key up; then make it the first entry, restart again, rewrite all Secrets, and only then remove the old key. Losing every copy of a key means deleting the resources encrypted with it.

Controls that matter as much as encryption

Encryption at rest is one line in the Kubernetes Secrets good-practices list. The others address the readers in the first table:

• Restrict get, list and watch. The docs say to limit watch and list to “the most privileged, system-level components” and caution that list “implicitly lets the subject fetch the contents of the Secrets.”
• Separate namespaces by trust. Pod creation reaches every Secret in a namespace, so a namespace is the practical boundary. Enforce Pod Security Admission where workload creators are less trusted, and consider ValidatingAdmissionPolicy for rules Pod Security doesn’t cover.
• Audit Secret access at the Metadata level. The example audit policy in the Kubernetes docs logs Secrets and ConfigMaps at Metadata, which records who accessed what without writing values into your logs. The good-practices page suggests alerting on events such as one user reading many Secrets at once.
• Use short-lived credentials and external stores. The docs recommend short-lived Secrets and mention the Secrets Store CSI Driver for keeping values in an external store. The same reasoning drives least privilege in cloud IAM: a credential that expires in an hour is worth less to whoever reads it.

Limits of this article

This describes upstream Kubernetes as documented on kubernetes.io at the time of writing. We tested the checker against sample files only, not against a live cluster or a managed service. We have not tested the encryption rollout procedure ourselves; follow the Kubernetes task page and your distribution’s documentation, and rehearse on a non-production cluster first.

Sources and verification

An API key committed to a public repository is not automatically compromised. The problem is that you usually cannot prove who copied it, when, or what they did with it. Scanners and bots index public GitHub commits within minutes of a push, so the window between “leaked” and “found by someone else” is often shorter than the window between “leaked” and “found by you.”

How keys actually get exposed

Most leaks don’t come from someone deliberately posting a credential. They come from ordinary developer workflow:

• A .env file committed because .gitignore was added after the first commit, not before it
• A key hardcoded during local testing and never removed before the branch was pushed
• A key embedded in client-side JavaScript because it was easier than building a backend proxy
• A key pasted into a CI log for debugging and left there, readable by anyone with log access
• A key shared in a support ticket, Slack thread, or screen-recorded demo

Client-side exposure deserves special attention because it isn’t a mistake in the traditional sense — a key embedded in a frontend bundle is, by definition, sent to every visitor’s browser. If a key must be used from the browser, it needs to be scoped so that exposure is expected and harmless, not treated as a leak when it inevitably surfaces.

Finding out you’re exposed

Don’t wait for a scanner to tell you. Check these sources directly:

• GitHub secret scanning — enabled by default on public repos for known token formats; push protection can block the commit before it lands if you turn it on
• Your own git history — a key removed from the current file is still in every prior commit unless you’ve rewritten history; git log -p and grep for the key pattern across the full history, not just HEAD
• CI/CD logs — check whether your pipeline prints environment variables or command arguments that include secrets, especially in debug or verbose modes
• Client-side bundles — search your built JS/CSS output for the key string directly; source maps can also re-expose something that was minified out of the readable bundle

Rotating without breaking production

Rotation fails when the old key is deactivated before every service using it has the new one. The order matters:

1. Issue a new key without deactivating the old one yet — most providers let both be valid simultaneously
2. Deploy the new key to every service, script, and integration that uses it, including ones that aren’t obviously “yours” — third-party tools, scheduled jobs, and one-off scripts are the most common things forgotten
3. Verify old-key usage has dropped to zero using the provider’s access logs or usage dashboard before deactivating anything; if usage doesn’t drop, something is still using the old key and you haven’t found it yet
4. Deactivate the old key, not just rotate the value in your own config — a key still valid on the provider’s side is still usable by whoever leaked it, regardless of what your app currently points at

Common mistakes

The most common rotation failure isn’t forgetting to rotate — it’s rotating the key in your secrets manager without confirming every consumer actually redeployed with the new value. A key that’s “rotated” in Vault but still cached in a running process’s memory, or in a service that reads secrets only at startup, means the old key is still functionally live until that service restarts.

The second most common mistake is treating rotation as the end of the incident. If the key had broad permissions, rotation stops future misuse but does nothing about anything already done with it during the exposure window — check the provider’s audit log for activity on that key specifically, not just your own application logs.

Detection and verification checklist

• Search full git history, not just current files, for the key pattern
• Check CI/CD logs for accidental secret printing
• Search built frontend bundles and source maps
• Confirm the provider’s own access log shows old-key usage before deactivating it
• Review what the leaked key had permission to do, and audit that specific window of activity
• Scope any client-exposed keys narrowly enough that exposure is expected, not catastrophic

Preventing the next one

Push protection (blocking commits containing recognizable secret patterns before they land) closes the most common exposure path at the point of failure, rather than relying on someone noticing afterward. Pair it with short-lived, scoped credentials where the provider supports them — a key with a narrow permission set and a defined expiry limits the damage of the next leak you don’t catch immediately, which, at some point, you won’t.

Secrets scanning is mature technology at this point — most CI pipelines can catch a hardcoded key before it merges. Yet leaked API keys remain one of the most common root causes in breach disclosures. The gap isn’t tooling. It’s three specific habits that scanners don’t catch.

Keys with no expiry

A scanner can tell you a key is exposed in a commit. It can’t tell you that the key was never going to expire anyway, so rotating it after the leak doesn’t actually close the window — the same key pattern gets reused in the next service, with the same no-expiry default. Default every new key to expire, and make the renewal request the normal path, not the exception.

Shared keys across environments

Using the same API key for staging and production is still common because it’s convenient — one less secret to manage. It also means a staging environment with weaker access controls becomes a direct path to production data. Environment-scoped keys should be a hard requirement in your provisioning process, not a “best practice” that gets skipped under deadline pressure.

Keys stored in places scanners don’t look

Secrets scanners are tuned for source code. They generally don’t scan Slack messages, shared docs, or ticket comments — all places engineers paste a key “just this once” while debugging. That key often outlives the debugging session by months. If your threat model assumes secrets only live in git, you’re missing the majority of where they actually leak from in practice.

Where to focus first

1. Audit for non-expiring keys before adding more scanning tools
2. Split shared staging/production keys this quarter, not next
3. Treat chat and docs as a secrets surface, with periodic manual sweeps

Scanning your repos is necessary. It was never sufficient.