Kubernetes audit log detections work because most attack techniques against a cluster have to go through kube-apiserver. Getting a shell in a Pod, starting a container that mounts the node’s filesystem, granting yourself cluster-admin, sweeping Secrets, minting a token: each is an API request, and each leaves an audit event with the identity, the verb, the resource and the result. The eight rules below turn those events into alerts.

We wrote them as jq filters so you can run them against a log file today, then ran them against kube-apiserver v1.34.1 while we played out a small attack: a stolen CI credential probing what it can do, a compromised administrator setting up persistence, and an anonymous caller. This article shows what each rule caught, what it would miss, and how to tune it before you port it to a SIEM.

The rules assume an audit policy that logs everything at Metadata or above and keeps request bodies for writes. If you do not have one yet, start with our Kubernetes audit policy guide; we used the same six-rule policy here.

The eight rules

The ATT&CK IDs and names are from MITRE’s STIX data (enterprise collection version 20.0). The mapping is ours: ATT&CK describes techniques, not audit fields.

The test

We ran kube-apiserver v1.34.1 and etcd 3.6.4 (the controller-runtime envtest binaries) with RBAC authorization, static bearer tokens for three users and the audit policy linked above, writing JSON lines to a file. There were no nodes, kubelets, controller manager or scheduler, so Pods were never scheduled. That keeps the log readable, but a real cluster’s log is far noisier; see the limits section.

The scenario, in order:

1. Benign: dev-bob, in the developers group, creates a Pod in dev, reads a Secret in his own namespace and runs kubectl exec.
2. Stolen CI token: ci-deployer, which can create Pods and exec in prod, runs kubectl auth can-i --list, tries to list Secrets, nodes, namespaces, ClusterRoles, ConfigMaps and ServiceAccounts cluster-wide, tries Secrets in kube-system, then tries to create a privileged Pod with hostPID, hostNetwork and a hostPath mount of /. The API server rejected that one (it was running with the default --allow-privileged=false), so the attacker dropped the privileged flags and created a Pod that only mounts / from the host. That one succeeded.
3. Compromised admin: alice binds cluster-admin to the prod/default service account, mints a 24-hour token for it with kubectl create token, reads kube-system Secrets while impersonating it, and attaches a debug container to a running Pod with kubectl debug.
4. Anonymous: an unauthenticated curl to /api/v1/namespaces/kube-system/secrets.

The resulting log had 574 events. 532 came from system:apiserver itself; the other 42 were ours (alice 18, ci-deployer 15, dev-bob 8, anonymous 1).

The rules as a script

Save this as detections.sh and run sh detections.sh audit.log. It needs jq 1.6 or later for IN; we ran it with jq 1.7.

#!/bin/sh
# Kubernetes audit log detections. Usage: ./detections.sh audit.log
# Input: kube-apiserver JSON-lines audit log (audit.k8s.io/v1 Event objects).
LOG="$1"

# Control-plane identities to ignore. Extend this for your distribution.
NOT_CP='(.user.username | IN("system:apiserver","system:kube-controller-manager","system:kube-scheduler") | not)'
# Count each request once: long-running calls such as exec also emit ResponseStarted.
DONE='.stage=="ResponseComplete"'

echo "D1 exec, attach and ephemeral containers"
jq -c "select($DONE)"' | select(.objectRef.resource=="pods"
      and (.objectRef.subresource | IN("exec","attach","ephemeralcontainers")))
  | {user:.user.username, sub:.objectRef.subresource, verb, ns:.objectRef.namespace,
     pod:.objectRef.name, code:.responseStatus.code, uri:.requestURI}' "$LOG"

echo "D2 pods requesting host access (allowed or rejected)"
jq -c 'select(.objectRef.resource=="pods" and .verb=="create" and .objectRef.subresource==null)
  | select(.requestObject.spec.hostPID or .requestObject.spec.hostNetwork or .requestObject.spec.hostIPC
      or any(.requestObject.spec.volumes[]?; .hostPath)
      or any(.requestObject.spec.containers[]?; .securityContext.privileged))
  | {user:.user.username, ns:.objectRef.namespace, pod:.objectRef.name, code:.responseStatus.code,
     hostPaths:[.requestObject.spec.volumes[]?.hostPath.path | select(.)]}' "$LOG"

echo "D3 bindings that grant cluster-admin"
jq -c "select($NOT_CP)"' | select(.objectRef.apiGroup=="rbac.authorization.k8s.io"
      and (.objectRef.resource | IN("clusterrolebindings","rolebindings"))
      and (.verb | IN("create","update","patch"))
      and .requestObject.roleRef.name=="cluster-admin")
  | {user:.user.username, binding:.objectRef.name, subjects:.requestObject.subjects,
     code:.responseStatus.code}' "$LOG"

echo "D4 Secret reads across all namespaces or in kube-system"
jq -c "select($NOT_CP)"' | select(.objectRef.resource=="secrets" and (.verb | IN("get","list","watch")))
  | select(.objectRef.namespace==null or .objectRef.namespace=="kube-system")
  | {user:.user.username, as:.impersonatedUser.username, verb, ns:.objectRef.namespace,
     code:.responseStatus.code}' "$LOG"

echo "D5 service account tokens minted outside the kubelet"
jq -c "select($NOT_CP)"' | select(.objectRef.resource=="serviceaccounts"
      and .objectRef.subresource=="token" and .verb=="create")
  | select(.user.username | startswith("system:node:") | not)
  | {user:.user.username, sa:(.objectRef.namespace + "/" + .objectRef.name),
     code:.responseStatus.code}' "$LOG"

echo "D6 impersonated requests"
jq -c 'select(.impersonatedUser)
  | {user:.user.username, as:.impersonatedUser.username, verb, res:.objectRef.resource,
     ns:.objectRef.namespace, code:.responseStatus.code}' "$LOG"

echo "D7a permission listing (kubectl auth can-i --list)"
jq -c 'select(.objectRef.resource=="selfsubjectrulesreviews")
  | {user:.user.username, ns:.objectRef.namespace, ip:.sourceIPs[0]}' "$LOG"

echo "D7b identities with 5 or more denied requests in the file"
jq -sc '[.[] | select(.annotations["authorization.k8s.io/decision"]=="forbid")]
  | group_by(.user.username)
  | map({user:.[0].user.username, denied:length,
         resources:([.[].objectRef.resource] | unique)})
  | map(select(.denied >= 5))' "$LOG"

echo "D8 anonymous requests"
jq -c 'select(.user.username=="system:anonymous")
  | {uri:.requestURI, ip:.sourceIPs[0], code:.responseStatus.code}' "$LOG"

What each rule caught

All 15 hits from the run, condensed:

Every step of both attack paths fired at least one rule. The only benign activity that fired was dev-bob’s exec, which is the false positive you would expect: D1 is a “who is in our workloads” feed, not a high-severity alert on its own.

D1: exec, attach and debug sessions

Two details matter. First, exec is a long-running request, and the Kubernetes docs say ResponseStarted “is only generated for long-running requests”. Our first version of the rule returned every exec event twice, once per stage; filtering on ResponseComplete fixed it. Second, one kubectl exec produced both a get and a create on pods/exec, so count sessions by auditID or by time window, not by event.

kubectl debug does not use exec at all. It sends a patch to the pods/ephemeralcontainers subresource, which is why the rule matches that subresource explicitly. Because it is a write, our policy logged it at Request level, and the event carried the debug container’s image (busybox:1.36). A debug container running an image you have never seen in the cluster is worth a look.

To tune: alert on exec into kube-system and other platform namespaces, exec by service accounts (people run exec, workloads rarely should), and exec whose requestURI contains shells or tools you do not expect. Route the rest to a lower-severity feed.

D2: Pods asking for the host

The fields come from the Pod Security Standards’ Baseline profile, which disallows host namespaces (spec.hostNetwork, spec.hostPID, spec.hostIPC), privileged containers and hostPath volumes. If you enforce those with Pod Security Admission, D2 tells you who keeps trying.

The useful finding from our run is that the rejected request was logged in full. The privileged Pod failed validation with HTTP 422 (“disallowed by cluster policy”), and the audit event still held the complete requestObject, including the hostPath of /. That is why D2 does not filter on success. An attacker’s first attempt is often the one that fails.

D2 needs Request level on Pod creates; at Metadata the spec is not there. It also only sees Pods. Deployments, DaemonSets, Jobs and CronJobs create Pods through controllers, so the Pod create event’s user is the controller’s identity. Add the same select on .requestObject.spec.template.spec for workload resources to see the person who submitted the template.

D3: bindings to cluster-admin

Without the control-plane exclusion, this rule fired on startup: system:apiserver creates the cluster-admin ClusterRoleBinding to system:masters when it bootstraps RBAC. After excluding it, the only hit was the persistence binding.

The binding name backup-sync is deliberately bland; do not rely on names. The rule matches roleRef.name, which our policy logs because RBAC writes are kept at RequestResponse. Extend it with the other roles that are as good as cluster-admin in your cluster, and with writes to ClusterRoles that add * verbs or escalate, bind or impersonate. Our RBAC permission audit lists them.

D4: Secret sweeps

A cluster-wide list has no namespace in objectRef, which is how D4 finds it. Our policy logs Secrets at Metadata, so none of these events contain Secret values, and the rule does not need them: who, which verb, which scope and the response code are enough.

The 403s are as interesting as the 200. Two of the four hits were the stolen CI token being denied, which tells you someone is trying the credential well outside its job. Do not exclude every system: user here. system:serviceaccount:... identities are system: users too: had the attacker used the minted token directly instead of impersonating, the read would have been logged as system:serviceaccount:prod:default and a blanket filter would have dropped it. That is why the script excludes control-plane components by exact name. Some distributions run controllers under kube-system service accounts; add those by name after you have looked at your own log.

D5: minted tokens

kubectl create token is a create on the serviceaccounts/token subresource. Our policy logs that subresource at Metadata, so the event recorded who asked and for which service account, but not the token. Keep it that way: at RequestResponse the response would contain a usable token.

The kubelet requests tokens on behalf of Pods that use projected service account tokens, so in a real cluster most of these events come from system:node:<name> identities. The rule drops those. What is left, a person or a workload minting a token for another service account, is rare and worth reviewing. Our service account token guide covers why bound tokens are still worth watching.

D6: impersonation

When a request uses impersonation, the audit event keeps the real caller in user and adds impersonatedUser. Here alice read kube-system Secrets as the service account she had just made cluster-admin. D4 fired on that read too, but only D6 tells you the request ran with the service account’s permissions, which is what links the read back to the binding created minutes earlier. Treat D6 as context: join it to every other rule so each alert shows both the real caller and the identity whose permissions were used.

D7: permission discovery

kubectl auth can-i --list sends a SelfSubjectRulesReview, which is logged as a create on selfsubjectrulesreviews. Developers run it too, so it is weak alone, but it is a strong signal from a CI or workload identity: our stolen CI token ran it first. The burst rule then found 8 denials for ci-deployer across 6 resource types.

The threshold of 5 per file is only for this demo. In a SIEM, count forbid decisions per identity over a window such as 10 minutes, and weight by the number of distinct resources: one identity hitting one forbidden resource repeatedly is usually a broken deployment; one identity hitting many is usually someone exploring.

D8: anonymous requests

kube-apiserver accepts anonymous requests by default (--anonymous-auth defaults to true; they arrive as system:anonymous in group system:unauthenticated). RBAC denied our anonymous call, but the event is still useful: anonymous requests for resource paths from outside your network mean the API server is reachable by people scanning for it. Any anonymous request that gets a 2xx on a resource path is an incident.

Porting the rules to a SIEM

The fields used here (user.username, impersonatedUser, objectRef.*, verb, stage, responseStatus.code, annotations, requestObject, requestURI, sourceIPs) are fields of the audit.k8s.io/v1 Event object, so the logic carries over to any backend that stores the events. Managed services may wrap each event in their own envelope and may run their own audit policy, which decides whether requestObject is present at all. Check one raw event from your provider before you trust a rule that depends on request bodies.

Three practical points:

• Deduplicate on auditID. It is the same across a request’s stages.
• Alert on attempts. Keep 403 and 422 results in every rule unless you have a reason to drop them.
• Join with identity context. Mapping user.username and sourceIPs to a person, a pipeline or a node is what turns these rules from noise into findings. Our incident logging guide covers what to keep alongside the audit log.

Limits of this article

We tested on a single kube-apiserver with no nodes, controllers or scheduler, so no Pod ever ran and exec returned 400 instead of opening a session. A production log has far more control-plane and controller traffic, and your exclusion list will need tuning against it. We used static tokens; OIDC, certificates and cloud IAM identities change what user.username looks like. Managed Kubernetes providers control their own audit policies, and some do not log request bodies, which disables D2, D3 and the image detail in D1. These rules detect API activity only: they do not see what happens inside a container or on a node, which needs runtime or host telemetry.

Sources and verification

A Kubernetes audit policy is the file that tells kube-apiserver which API requests to record and how much of each to keep. Without one, upstream kube-apiserver records nothing. A good policy logs everything at Metadata at minimum, keeps request bodies for changes that alter access, and never logs the contents of Secrets. Rule order is what makes that work: the first matching rule wins, and a broad rule placed too high will quietly write Secret values into your logs. We demonstrate that below.

This guide gives a six-rule policy we ran against kube-apiserver v1.34.1, shows what each rule captured, and ends with queries you can run against the log during an incident.

How kube-apiserver evaluates a policy

Every request produces audit events at one or more stages:

The policy is a list of rules. kube-apiserver compares each event with the rules in order, and the first rule that matches sets the event’s level:

Request and RequestResponse don’t apply to non-resource URLs such as /healthz. A rule can match on users, userGroups, verbs, resources (group, resource, subresource and resourceNames), namespaces and nonResourceURLs. A rule with no match fields matches everything, which is what you want for the last rule and exactly what you don’t want near the top.

The policy file is passed with --audit-policy-file. If the flag is omitted, no events are logged, and a policy with no rules is rejected.

A policy that answers incident questions

The policy below is built around the questions you get asked during an incident: who read this Secret, who exec’d into that Pod, who granted this binding, and was it allowed. It is a starting point, not a complete production policy; see the limits at the end.

apiVersion: audit.k8s.io/v1
kind: Policy
omitStages:
  - "RequestReceived"
rules:
  # 1. Drop health and version probes.
  - level: None
    nonResourceURLs: ["/healthz*", "/livez*", "/readyz*", "/version"]
  # 2. Sensitive payloads: who touched them, never the contents.
  - level: Metadata
    resources:
      - group: ""
        resources: ["secrets", "configmaps", "serviceaccounts/token"]
      - group: "authentication.k8s.io"
        resources: ["tokenreviews"]
  # 3. Interactive access into running workloads.
  - level: Metadata
    resources:
      - group: ""
        resources: ["pods/exec", "pods/attach", "pods/portforward", "pods/proxy", "nodes/proxy"]
  # 4. Changes to who can do what, and to admission control.
  - level: RequestResponse
    verbs: ["create", "update", "patch", "delete", "deletecollection"]
    resources:
      - group: "rbac.authorization.k8s.io"
      - group: "admissionregistration.k8s.io"
  # 5. Other writes: keep the submitted object.
  - level: Request
    verbs: ["create", "update", "patch", "delete", "deletecollection"]
  # 6. Everything else (reads, watches, non-resource URLs).
  - level: Metadata

Why each rule is where it is:

Pod Security Admission and our RBAC permission audit tell you what could happen. The audit log is how you find out what did.

What it recorded

We ran the policy on kube-apiserver v1.34.1 (the envtest build from the kubernetes-sigs/controller-tools releases) with etcd and RBAC authorization, then made requests as a system:masters user and as an unprivileged user alice. There were no nodes, so no Pod actually ran. Results:

Two details from the test are worth building detections around:

• Exec shows up under more than one verb. That single kubectl exec produced events with verb get and verb create for the exec subresource. Filter on objectRef.subresource == "exec", not on the verb.
• Impersonation keeps both identities. The authenticated user is in user and the identity they acted as is in impersonatedUser. If you only alert on user.username, impersonated actions look like they came from the admin’s own account.

Also keep in mind what the API reference warns about two fields: userAgent is reported by the client “and must not be trusted”, and in sourceIPs all entries except the last “can be arbitrarily set by the client”.

The rule-order mistake that leaks Secrets

We then moved rule 5 (Request for all writes) above rule 2, restarted kube-apiserver with that policy, and created a Secret with the value S3cretValue-456. The audit log now contained:

"data":{"key":"UzNjcmV0VmFsdWUtNDU2"}

That is the base64 encoding of S3cretValue-456. The broad write rule matched first, logged the request body at Request level, and the Metadata rule for Secrets was never reached. With the correct order, the equivalent test (a Secret holding S3cretValue-123) left no trace of either the plain or the encoded value in the log.

This is the most common way an audit log becomes a credential store. Your audit pipeline usually has wider read access than the cluster’s Secrets do, so treat a body-logging rule above your Secrets rule as a bug. Check it in code review with a simple test: create a dummy Secret in a test cluster and search the log for its encoded value.

If you already store Secrets encrypted at rest, remember that encryption protects etcd, not the audit log. And a Secret-based service account token leaked this way never expires.

Backends and volume

There are two backends: a log file and a webhook.

• Log backend: --audit-log-path sets the file (- means stdout). Rotation is controlled by --audit-log-maxage (days), --audit-log-maxbackup (files) and --audit-log-maxsize (megabytes). Batching is off by default for the log backend.
• Webhook backend: --audit-webhook-config-file points at a kubeconfig-format file for the remote endpoint. Batching is on by default, with a maximum of 400 events per batch and a 30-second maximum wait.
• Truncation of oversized events is off by default for both backends and has to be enabled explicitly.

The Kubernetes docs also note that auditing increases API server memory use, because some context is kept for each request, and that the increase depends on the policy. RequestResponse on a busy resource is where cost grows: the GCE reference policy, for instance, logs get, list and watch at Request level rather than RequestResponse because “Get responses can be large”.

Ship the log off the control plane. An attacker with control-plane access can edit a local file, and an incident log is only useful if it survives the incident.

Managed Kubernetes

On EKS, AKS and similar services the provider runs kube-apiserver, so you generally choose whether audit events are exported, not what the policy contains. Check your provider’s documentation for what its policy logs.

• Amazon EKS: control plane logs are not sent to CloudWatch Logs by default. Enable the audit log type per cluster, for example with aws eks update-cluster-config --logging. The audit stream is named kube-apiserver-audit-<id> in the /aws/eks/<cluster>/cluster log group. AWS notes delivery is best effort and standard CloudWatch Logs charges apply.
• Azure AKS: resource logs are not collected until you create a diagnostic setting. Microsoft’s guidance warns that kube-audit can be costly and suggests kube-audit-admin, which leaves out get and list events. That saves money but drops exactly the read events, such as Secret reads, that matter in a credential-theft investigation. Decide that trade-off deliberately.

Queries to keep ready

These run against a JSON-lines audit log with jq. We ran each one against the log from the test above.

# Who read or changed Secrets (excluding the API server itself)
jq -c 'select(.objectRef.resource=="secrets" and .user.username!="system:apiserver")
  | {ts:.requestReceivedTimestamp, user:.user.username, verb, ns:.objectRef.namespace,
     name:.objectRef.name, code:.responseStatus.code}' audit.log

# Every exec into a Pod, with the command line
jq -c 'select(.objectRef.subresource=="exec")
  | {ts:.requestReceivedTimestamp, user:.user.username, ns:.objectRef.namespace,
     pod:.objectRef.name, uri:.requestURI}' audit.log

# Denied requests, including impersonated ones
jq -c 'select(.annotations["authorization.k8s.io/decision"]=="forbid")
  | {user:.user.username, as:.impersonatedUser.username, verb,
     res:.objectRef.resource, ns:.objectRef.namespace}' audit.log

# RBAC writes by people rather than the control plane, with what they granted
jq -c 'select(.objectRef.apiGroup=="rbac.authorization.k8s.io"
       and (.verb|test("create|update|patch|delete"))
       and (.user.username|startswith("system:")|not))
  | {user:.user.username, verb, res:.objectRef.resource, name:.objectRef.name,
     subjects:.requestObject.subjects, role:.requestObject.roleRef.name}' audit.log

A burst of forbid events from one identity across many resources can be a sign that someone is enumerating what a stolen credential can do.

Limits of this article

We tested on a single kube-apiserver v1.34.1 with no nodes, no controller manager and static token authentication, so no workload actually ran and no kubelet or controller traffic was present. A real cluster produces far more events, and you will want extra None or Metadata rules for high-volume system traffic (the GCE policy is a good reference). The policy does not cover custom resources that hold credentials; add those to rule 2. Request level on Pod writes will log any credentials placed directly in environment variables. We did not test the webhook backend or any managed service.

Sources and verification

A Kubernetes service account token is only as safe as its lifetime. Tokens that Kubernetes issues through the TokenRequest API expire, carry an audience, and stop working the moment the Pod they were issued for is deleted. Secret-based tokens, the kubernetes.io/service-account-token type, never expire and stay valid until someone deletes the Secret. Kubernetes stopped creating those automatically in v1.24, but clusters that predate it and any team that created one by hand may still have them. Find them, find out who still uses them, move those callers to short-lived tokens, then delete the Secrets.

Two kinds of token

Both are signed JWTs that a client sends as Authorization: Bearer <token>. What differs is how long they live and what they are tied to.

The Kubernetes documentation states the trade-off without hedging: Secret-based tokens “don’t expire and don’t rotate”. A leaked bound token gives an attacker an hour, or until the Pod goes away. A leaked Secret-based token gives them that ServiceAccount’s permissions until someone notices. What those permissions allow is a separate question, and our Kubernetes RBAC audit covers the ones that let a ServiceAccount escalate.

How Kubernetes got here

Upgrading never removed existing token Secrets. A cluster created on v1.21 and upgraded every quarter since can still hold a never-expiring token for every ServiceAccount that existed back then.

The clean-up only covers half of the problem

From v1.29, the legacy token cleaner in kube-controller-manager checks every 24 hours. It marks an auto-generated token invalid if all of these are true:

• it is auto-generated, meaning the Secret and the ServiceAccount reference each other;
• no Pod currently mounts it;
• it has not been used for the clean-up period since it was created or last used.

Invalid tokens get a kubernetes.io/legacy-token-invalid-since label. If one stays unused for another period, the cleaner deletes it. Both periods default to one year and are set with --legacy-service-account-token-clean-up-period.

Two gaps follow from those rules. A token in active use is never invalidated, which is the point, but it also means the riskiest token, the one a CI system or an old script still uses every day, survives. And manually created token Secrets are never cleaned up, because they are not auto-generated. Kubernetes still lets you create them, and the docs ask you to “take into account the security implications” when you do.

The one-year Pod token

There is a less obvious long-lived token in most clusters: the default one in every Pod.

The ServiceAccount admission controller requests tokens with a lifetime of 3,607 seconds. In the Kubernetes source, --service-account-extend-token-expiration defaults to true, and the flag’s help text says that when it is enabled “admission injected tokens would be extended up to 1 year to prevent unexpected failure during transition, ignoring value of service-account-max-token-expiration.” The token also gets a warnafter claim at the original expiry.

The example claims in the Kubernetes documentation show exactly this: exp - iat is 31,536,000 seconds (365 days), and warnafter - iat is 3,607 seconds. We computed both from the published example.

That token is still bound to its Pod, so deleting the Pod invalidates it. But a token copied out of a running Pod, for example through a file-read bug in the application, can stay valid for a long time if the Pod is long-lived. Managed Kubernetes providers set kube-apiserver flags themselves, so check yours.

When a client uses such a token after warnafter, the API server accepts it but adds the audit annotation authentication.k8s.io/stale-token, naming the subject, and increments serviceaccount_stale_tokens_total. Those are the clients that read the token once at start-up and never reload it. Fix them first. The documentation’s advice is simple: reload the token file on a schedule, for example every five minutes. Once nothing is stale, you can consider turning the extension off.

Step 1: inventory Secret-based tokens

kubectl get secrets -A --field-selector type=kubernetes.io/service-account-token

Two labels tell you most of what you need. The API server sets kubernetes.io/legacy-token-last-used to the current date whenever a Secret-based token authenticates, both auto-generated and manual ones; the date is a day, not a timestamp. kubernetes.io/legacy-token-invalid-since appears once the cleaner has invalidated the token.

This short script turns the JSON into a table. It only reads stdin and changes nothing:

#!/usr/bin/env python3
"""List Secret-based service account tokens from `kubectl get secrets -A -o json`.

Usage: kubectl get secrets -A -o json | python3 sa_token_secrets.py
Read-only: it parses JSON on stdin and prints a table. Python 3.8+.
"""
import json
import sys

LAST_USED = "kubernetes.io/legacy-token-last-used"
INVALID_SINCE = "kubernetes.io/legacy-token-invalid-since"
SA_ANNOTATION = "kubernetes.io/service-account.name"

items = json.load(sys.stdin).get("items", [])
rows = []
for s in items:
    if s.get("type") != "kubernetes.io/service-account-token":
        continue
    md = s.get("metadata", {})
    labels = md.get("labels") or {}
    rows.append((
        md.get("namespace", ""),
        md.get("name", ""),
        (md.get("annotations") or {}).get(SA_ANNOTATION, "?"),
        md.get("creationTimestamp", ""),
        labels.get(LAST_USED, "-"),
        labels.get(INVALID_SINCE, "-"),
    ))

header = ("NAMESPACE", "SECRET", "SERVICEACCOUNT", "CREATED", "LAST-USED", "INVALID-SINCE")
for r in [header] + sorted(rows):
    print("\t".join(r))
print(f"\n{len(rows)} Secret-based service account token(s)", file=sys.stderr)

We ran it with Python 3.13 against a sample kubectl JSON document with two token Secrets and one ordinary Secret; it listed the two tokens with their labels and skipped the other. Not reading the token value is deliberate: the inventory does not need it.

Sort the output into three groups:

A missing last-used label does not prove a token was never used. Tracking only began when your cluster first ran a version with it enabled; the control plane records that date in the kube-system/kube-apiserver-legacy-service-account-token-tracking ConfigMap.

Step 2: find out who still uses them

The API server adds an audit annotation every time a Secret-based token authenticates:

Search your audit logs for those keys and read sourceIPs and userAgent from the same events to identify the caller. If you don’t have audit logs, the counters serviceaccount_legacy_auto_token_uses_total and serviceaccount_legacy_manual_token_uses_total at least tell you whether usage is going up or down. Clients using an auto-generated token also get an API warning asking them to use TokenRequest tokens instead, which sometimes shows up in CI logs.

Step 3: move each caller to a short-lived token

A projected token for a specific audience looks like this, adapted from the Kubernetes documentation example:

apiVersion: v1
kind: Pod
metadata:
  name: app
spec:
  serviceAccountName: build-robot
  containers:
  - name: app
    image: registry.example.com/app:1.4.2
    volumeMounts:
    - mountPath: /var/run/secrets/tokens
      name: vault-token
  volumes:
  - name: vault-token
    projected:
      sources:
      - serviceAccountToken:
          path: vault-token
          expirationSeconds: 7200
          audience: vault

A service that receives this token should reject any token whose audience is not its own. If you write that validation yourself, the Kubernetes project recommends the TokenReview API over OIDC discovery, because TokenReview fails immediately once the bound Pod is deleted, while plain JWT validation would keep accepting the token until it expires. Our guide to JWT security mistakes covers the audience and expiry checks that apply here too.

Step 4: delete, and stop new ones appearing

When a token Secret has no callers left, delete it:

kubectl -n <namespace> delete secret <token-secret-name>

If a caller you missed fails afterwards, you will see it in your audit logs and can issue a short-lived replacement. That is a better failure than an unexpired credential nobody owns.

To keep new ones from appearing, decide who may create Secrets of this type. RBAC can’t filter on a Secret’s type field, so use an admission policy for that rule; our walkthrough of ValidatingAdmissionPolicy with CEL shows the warn-first rollout pattern. Expressions along the lines of object.type != 'kubernetes.io/service-account-token', scoped to namespaces where nobody should create them, will do.

Reduce what any token can do

• Don’t mount tokens Pods don’t use. Set automountServiceAccountToken: false on the ServiceAccount or the Pod; the Pod setting wins if they disagree. Most application Pods never call the Kubernetes API.
• Give each workload its own ServiceAccount instead of sharing default, so a token’s permissions match one workload.
• Know how to revoke. There is no central record of issued bound tokens, so you can’t revoke one by ID. Delete the Pod to kill a Pod-bound token. To kill every token for a ServiceAccount, delete and recreate it; the new UID invalidates the tokens issued for it.
• Don’t mount projected tokens with subPath. A subPath mount never receives the refreshed file, so the application ends up with an expired token.

Common mistakes

• Treating an upgrade to v1.24 or later as the fix. It stopped new auto-generated tokens; it didn’t remove old ones.
• Waiting for the automatic clean-up. It ignores manual tokens and anything still in use, and takes up to two years by default.
• Copying a Pod’s token out for use elsewhere because it “lasts a year”. It is bound to that Pod and dies with it, and the API server counts it as stale after about an hour.
• Hunting for leaked tokens in Secrets but not in Git, CI variables and kubeconfig files on laptops, where most old Secret-based tokens end up.

Limits of this article

This guide follows the upstream Kubernetes documentation for v1.37 and the Kubernetes source on the main branch. Managed services (EKS, GKE, AKS and others) set kube-apiserver and kube-controller-manager flags themselves and may disable or change the token extension, the clean-up period or audit logging. Check your provider’s documentation before relying on any default described here. We tested the inventory script against sample JSON, not a live cluster.

Sources and verification

Kubernetes encryption at rest protects one thing: the Secret data stored in etcd. It stops someone who gets etcd’s data, or a copy of it, from reading Secrets in plain text. It does not stop anyone the API server is willing to serve, and that group is larger than most teams assume: everyone with get, list or watch on Secrets, and everyone who can create a Pod in the namespace. Turn encryption on, but don’t count it as your Secrets access control.

What Kubernetes does by default

The Kubernetes documentation is blunt: “Kubernetes Secrets are, by default, stored unencrypted in the API server’s underlying data store (etcd).” The default provider is identity, which, in the docs’ words, “does not encrypt stored data and provides no additional confidentiality protection.”

The base64 you see in a Secret manifest doesn’t change that. The good-practices page says it plainly: “Base64 encoding is not an encryption method, it provides no additional confidentiality over plain text.” A Secret manifest committed to Git is a plain-text credential in Git.

Managed Kubernetes services run the control plane for you, and their defaults may differ from upstream. This article describes upstream behavior; check your provider’s documentation for what it encrypts by default and whether it supports envelope encryption with your own key.

Who can read a Secret

Before choosing an encryption provider, list who can read Secrets at all. Encryption at rest changes only the last two rows of this table.

The middle row surprises people. The Secret documentation says “anyone who is authorized to create a Pod in a namespace can use that access to read any Secret in that namespace; this includes indirect access such as the ability to create a Deployment.” Our Kubernetes RBAC audit covers how to find these grants in your bindings; this article stays on the storage side.

How encryption at rest is configured

You pass an EncryptionConfiguration file to kube-apiserver with --encryption-provider-config. A minimal example for Secrets with a local AES-CBC key, shaped like the one in the Kubernetes task page:

apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources:
      - secrets
    providers:
      - aescbc:
          keys:
            - name: key1
              secret: <BASE64-ENCODED 32-BYTE KEY>
      - identity: {} # lets the API server read Secrets written before encryption

Three rules from the documentation decide what this file does:

• The first provider encrypts. “The first provider in the list is used to encrypt resources written into the storage.” If identity is first, new writes are plain text even if other providers are listed.
• Later providers only decrypt. On reads, each matching provider is tried in order. The trailing identity is what lets the API server keep reading old plain-text Secrets during a migration. Once everything is rewritten, you can remove it.
• Order across rules matters too. The resources list is processed in order, so an earlier rule that names secrets with identity will win over a later *.* wildcard that encrypts. Wildcards are *. for the core group (which includes Secrets) and *.* for everything.

Which provider stops which attacker

The key-location column is the one that maps to threats. With a local key, the docs say encryption “protects against an etcd compromise, but it fails to protect against a host compromise,” because the key sits in the configuration file on the host. With KMS, Kubernetes encrypts data with a data encryption key and the KMS wraps that key, so an attacker “would need to compromise etcd and the third-party KMS provider.”

A local key is still worth having. It turns a leaked etcd snapshot or a mis-scoped etcd client certificate into ciphertext rather than credentials. Our view: use KMS v2 where you have a KMS to point at, and treat a local aescbc key as a stopgap you plan to replace.

Checking a configuration file

We wrote a short checker that reads an EncryptionConfiguration and flags the weak settings above. It reads only the file and does not contact a cluster. It needs Python 3 and PyYAML; we ran it with Python 3 and PyYAML 6.0.3 against three sample files. It is illustrative: it does not validate the whole schema, and it reads kms without an apiVersion as v1.

#!/usr/bin/env python3
"""Flag weak settings in a Kubernetes EncryptionConfiguration file.
Usage: python3 check_encryption_config.py /path/to/encryption-config.yaml
Reads the file only; it does not contact a cluster."""
import sys
import yaml

LOCAL_KEY = {"aescbc", "aesgcm", "secretbox"}

def covers_secrets(resources):
    return any(r in ("secrets", "*.", "*.*") for r in resources)

def main(path):
    with open(path) as f:
        cfg = yaml.safe_load(f)
    findings = []
    secrets_rule_seen = False
    for i, rule in enumerate(cfg.get("resources", [])):
        res = rule.get("resources", [])
        if not covers_secrets(res):
            continue
        # The first matching rule wins, so only the first rule covering secrets counts.
        if secrets_rule_seen:
            continue
        secrets_rule_seen = True
        providers = rule.get("providers", [])
        first = next(iter(providers[0])) if providers else None
        if first == "identity":
            findings.append(f"rule {i}: first provider is identity, so new Secrets are written in plain text")
        elif first in LOCAL_KEY:
            findings.append(f"rule {i}: first provider is {first}; key sits in this file on every control plane host")
            if first == "aescbc":
                findings.append(f"rule {i}: aescbc is not recommended by the Kubernetes docs (padding oracle)")
        elif first == "kms":
            ver = providers[0]["kms"].get("apiVersion", "v1")
            if ver != "v2":
                findings.append(f"rule {i}: KMS v1 is deprecated since Kubernetes v1.28; move to KMS v2")
        if any("identity" in p for p in providers[1:]):
            findings.append(f"rule {i}: identity fallback present; unencrypted Secrets can still be read (expected during migration)")
    if not secrets_rule_seen:
        findings.append("no rule covers secrets: Secrets are stored unencrypted")
    for f_ in findings:
        print("WARN", f_)
    if not findings:
        print("OK no weak settings found for secrets")
    return 1 if findings else 0

if __name__ == "__main__":
    sys.exit(main(sys.argv[1]))

Results on our samples: the aescbc-plus-identity file above produced three warnings and exit code 1; a file using kms with apiVersion: v2 printed OK and exited 0; a file that only configured ConfigMaps reported that no rule covers Secrets. The non-zero exit code lets you run it in CI against the configuration you deploy to control plane nodes.

Rolling it out without locking yourself out

The Kubernetes task page walks through the full procedure. The steps that most often go wrong:

1. Every control plane host needs the same file and key. The docs say to copy the key to every control plane host over a secure channel, and to restart each API server. An API server without the key can’t read Secrets another one encrypted.

2. Existing Secrets are not touched. Encryption applies on write. Rewrite everything once all API servers have the new configuration:

   kubectl get secrets --all-namespaces -o json | kubectl replace -f -
   

3. Verify in etcd, not through the API. The API always returns decrypted values, so kubectl get secret proves nothing. Read a Secret with etcdctl get /registry/secrets/<namespace>/<name> and check that the stored value starts with k8s:enc:<provider>:v1:<key-name>:, for example k8s:enc:aescbc:v1:key1:.

4. Rotate keys in two passes. The docs’ zero-downtime procedure: add the new key as the second entry on every control plane node and restart all API servers so each can decrypt with it; back the key up; then make it the first entry, restart again, rewrite all Secrets, and only then remove the old key. Losing every copy of a key means deleting the resources encrypted with it.

Controls that matter as much as encryption

Encryption at rest is one line in the Kubernetes Secrets good-practices list. The others address the readers in the first table:

• Restrict get, list and watch. The docs say to limit watch and list to “the most privileged, system-level components” and caution that list “implicitly lets the subject fetch the contents of the Secrets.”
• Separate namespaces by trust. Pod creation reaches every Secret in a namespace, so a namespace is the practical boundary. Enforce Pod Security Admission where workload creators are less trusted, and consider ValidatingAdmissionPolicy for rules Pod Security doesn’t cover.
• Audit Secret access at the Metadata level. The example audit policy in the Kubernetes docs logs Secrets and ConfigMaps at Metadata, which records who accessed what without writing values into your logs. The good-practices page suggests alerting on events such as one user reading many Secrets at once.
• Use short-lived credentials and external stores. The docs recommend short-lived Secrets and mention the Secrets Store CSI Driver for keeping values in an external store. The same reasoning drives least privilege in cloud IAM: a credential that expires in an hour is worth less to whoever reads it.

Limits of this article

This describes upstream Kubernetes as documented on kubernetes.io at the time of writing. We tested the checker against sample files only, not against a live cluster or a managed service. We have not tested the encryption rollout procedure ourselves; follow the Kubernetes task page and your distribution’s documentation, and rehearse on a non-production cluster first.

Sources and verification

Kubernetes RBAC is usually audited by looking for cluster-admin bindings. That misses the permissions that don’t look like admin but lead there: a service account that can list Secrets, a CI identity that can create Pods, a monitoring agent with get on nodes/proxy. The Kubernetes project documents these escalation paths on its RBAC Good Practices page. This guide turns that list into an audit you can run.

Scope. Based on the Kubernetes v1.37 documentation pages “Role Based Access Control Good Practices”, “Using RBAC Authorization” and “Authorization”, read from the kubernetes/website repository on October 22, 2026. The audit script was tested with Python 3.13 on a sample JSON file we wrote; we did not run it against a live cluster.

The 11 escalation risks

The RBAC Good Practices page lists these under “privilege escalation risks”. The “why” column paraphrases the documentation.

Two points deserve emphasis.

Read is not always read-only. The documentation states plainly that get on nodes/proxy “is not a read-only permission”, because the kubelet APIs can be reached with websocket GET requests. Monitoring tools sometimes request it. If one of yours does, find out exactly why. There is often a narrower option. The kubelet maps /metrics, /stats and /logs to their own subresources (nodes/metrics, nodes/stats, nodes/log), so a collector that only reads those needs no nodes/proxy. On recent versions, kubelet fine-grained authorization (the KubeletFineGrainedAuthz feature, beta and on by default from v1.33, stable in v1.36) adds the same for /pods, /runningPods, /healthz and /configz, checking nodes/pods, nodes/healthz or nodes/configz before falling back to nodes/proxy.

Pod creation is a namespace-wide grant. Anyone who can create a Pod can choose which service account it runs as and which Secrets it mounts. The documentation concludes that “boundaries within a namespace should be considered weak” and recommends enforcing the Baseline or Restricted Pod Security Standard where you don’t fully trust the creator. See our Pod Security Admission rollout guide.

Two problems a RoleBinding review won’t catch

• system:masters. Members of this group bypass all RBAC checks, and that cannot be revoked by removing RoleBindings or ClusterRoleBindings. If an authorization webhook is in use, they bypass that too. Membership comes from the credential itself (for example a client certificate’s organization field), so look at how credentials are issued, not at bindings.
• Wildcards. * in resources or verbs grants rights not only to every resource type that exists now but to every type created in the future, for example by a new CRD (CustomResourceDefinition).

The documentation also warns about deleted users: if an attacker can create an account with the same name as a deleted user, they inherit any bindings that still name it. Remove bindings when you remove people.

How the built-in guardrails work

RBAC does have an anti-escalation check. Per the RBAC reference, you can only create or update a role if you already hold all of its permissions at the same scope, or have escalate on roles. You can only create or update a role binding if you already hold all of the referenced role’s permissions, or have bind on that role. That is why escalate and bind are on the list: they are the switches that turn the guardrail off.

The documentation shows a legitimate, narrow use of bind: letting a team lead grant only the built-in admin, edit and view ClusterRoles, by listing them in resourceNames. Without resourceNames, bind covers every role.

A script to find risky grants

The script below reads the JSON that kubectl produces for roles and bindings, and prints one line per subject per risky finding. It reports wildcards once instead of listing every risk they imply, and it skips cluster-scoped resources (such as nodes/proxy or persistentvolumes) when the grant comes through a RoleBinding, because the RBAC reference says those permissions only take effect through a ClusterRoleBinding.

Export the data first:

kubectl get clusterroles,clusterrolebindings,roles,rolebindings -A -o json > rbac.json
python3 rbac_audit.py rbac.json

import json, sys

# (resource, verbs, why) - from the Kubernetes "RBAC Good Practices" page
RISKY = [
    ("secrets", {"get", "list", "watch"}, "reads Secret contents"),
    ("pods", {"create"}, "workload creation: mounts Secrets, runs as any ServiceAccount in the namespace"),
    ("deployments", {"create", "update", "patch"}, "workload creation"),
    ("daemonsets", {"create", "update", "patch"}, "workload creation"),
    ("statefulsets", {"create", "update", "patch"}, "workload creation"),
    ("jobs", {"create"}, "workload creation"),
    ("cronjobs", {"create"}, "workload creation"),
    ("persistentvolumes", {"create"}, "hostPath volumes give node filesystem access"),
    ("nodes/proxy", {"get", "create"}, "kubelet API: exec into pods, bypasses audit and admission"),
    ("roles", {"escalate"}, "can write roles with rights it does not hold"),
    ("clusterroles", {"escalate"}, "can write roles with rights it does not hold"),
    ("roles", {"bind"}, "can bind roles with rights it does not hold"),
    ("clusterroles", {"bind"}, "can bind roles with rights it does not hold"),
    ("users", {"impersonate"}, "acts as other users"),
    ("groups", {"impersonate"}, "acts as other groups"),
    ("serviceaccounts", {"impersonate"}, "acts as service accounts"),
    ("certificatesigningrequests/approval", {"update"}, "with CSR create, can mint client certificates"),
    ("serviceaccounts/token", {"create"}, "issues tokens for existing service accounts"),
    ("validatingwebhookconfigurations", {"create", "update", "patch"}, "sees every admitted object"),
    ("mutatingwebhookconfigurations", {"create", "update", "patch"}, "sees and changes every admitted object"),
    ("namespaces", {"patch", "update"}, "can relabel namespaces (Pod Security, NetworkPolicy selectors)"),
]

# A RoleBinding cannot grant access to cluster-scoped resources.
CLUSTER_SCOPED = {"persistentvolumes", "nodes/proxy", "clusterroles",
                  "certificatesigningrequests/approval", "users", "groups",
                  "validatingwebhookconfigurations",
                  "mutatingwebhookconfigurations", "namespaces"}

def findings(rule, namespaced_binding):
    res, verbs = set(rule.get("resources", [])), set(rule.get("verbs", []))
    out = []
    if "*" in res or "*" in verbs:
        return [f"wildcard resources={sorted(res)} verbs={sorted(verbs)}: "
                "covers current and future resource types"]
    for r, risky_verbs, why in RISKY:
        if namespaced_binding and r in CLUSTER_SCOPED:
            continue
        if r in res and verbs & risky_verbs:
            out.append(f"{r} {sorted(verbs & risky_verbs)}: {why}")
    return out

def main(path):
    items = json.load(open(path))["items"]
    roles = {}
    for o in items:
        if o["kind"] in ("Role", "ClusterRole"):
            ns = o["metadata"].get("namespace", "")
            roles[(o["kind"], ns, o["metadata"]["name"])] = o.get("rules") or []
    for b in items:
        if b["kind"] not in ("RoleBinding", "ClusterRoleBinding"):
            continue
        ref = b["roleRef"]
        bns = b["metadata"].get("namespace", "")
        key = (ref["kind"], bns if ref["kind"] == "Role" else "", ref["name"])
        nsb = b["kind"] == "RoleBinding"
        hits = sorted({f for rule in roles.get(key, []) for f in findings(rule, nsb)})
        if not hits:
            continue
        scope = "cluster-wide" if b["kind"] == "ClusterRoleBinding" else f"namespace {bns}"
        for s in b.get("subjects") or []:
            who = f'{s["kind"]} {s.get("namespace", "") + "/" if s.get("namespace") else ""}{s["name"]}'
            for h in hits:
                print(f"{who} | {ref['kind']} {ref['name']} | {scope} | {h}")

if __name__ == "__main__":
    main(sys.argv[1])

We ran it on a sample file with five bindings: a CI service account with a ClusterRole that can patch Deployments and list Secrets cluster-wide; a namespaced “release” Role with Pod creation and bind; a read-only debug Role; a monitoring agent bound through a RoleBinding to a ClusterRole with get on nodes/proxy; and a user with a full wildcard ClusterRole. Output:

ServiceAccount build/ci | ClusterRole ci-deployer | cluster-wide | deployments ['patch']: workload creation
ServiceAccount build/ci | ClusterRole ci-deployer | cluster-wide | secrets ['list']: reads Secret contents
ServiceAccount payments/release-bot | Role release | namespace payments | pods ['create']: workload creation: mounts Secrets, runs as any ServiceAccount in the namespace
ServiceAccount payments/release-bot | Role release | namespace payments | roles ['bind']: can bind roles with rights it does not hold
User alice@example.com | ClusterRole platform-admin | cluster-wide | wildcard resources=['*'] verbs=['*']: covers current and future resource types

The read-only debug Role produced nothing, as intended, and the nodes/proxy grant was correctly skipped because a RoleBinding cannot grant a cluster-scoped resource. Had it been a ClusterRoleBinding, it would have been reported.

Known limits of the script. It ignores apiGroups (a CRD that reuses a name like jobs would be flagged), does not evaluate resourceNames restrictions, flags update and patch on workloads as well as create, does not cover every workload type (for example ReplicaSets), and does not see system:masters membership or permissions granted by authorizers other than RBAC. Treat its output as a review list, not a verdict.

Confirming a finding

For any subject the script flags, check what it can actually do. The Kubernetes authorization documentation shows kubectl auth can-i combined with impersonation:

kubectl auth can-i list secrets --namespace payments \
  --as system:serviceaccount:build:ci

You need impersonation rights to run that, which is itself item 7 on the list. Keep those rights with a small group of administrators.

Fixing what you find

From the Good Practices page, plus our notes:

• Prefer RoleBindings to ClusterRoleBindings, so rights stay inside one namespace.
• Replace wildcards with explicit resources and verbs.
• Give powerful service accounts to as few Pods as possible, and keep those Pods off nodes that run untrusted or internet-facing workloads, using taints and tolerations or affinity rules.
• Turn off token automounting where a workload doesn’t call the API, with automountServiceAccountToken: false on the ServiceAccount or Pod; the Pod setting takes precedence.
• Review bindings for system:unauthenticated, which give access to anyone who can reach the API server.
• Use resourceNames to limit bind and impersonate to specific roles or identities.
• Review periodically, and remove bindings for departed users and retired service accounts.

The same least-privilege thinking applies to cloud IAM; our guide to finding overprivileged AWS IAM users and roles covers that side. Namespace labels also drive network isolation, which is why item 11 matters for NetworkPolicy default deny.

Limits of this article

The list of risks is the Kubernetes project’s; it is not exhaustive for every add-on or CRD in your cluster, many of which define their own sensitive resources. The script was tested on sample data only. Managed Kubernetes services add their own identity layers (for example cloud IAM mappings) that are outside this article.

Sources and verification

Checked on October 22, 2026.

In a Kubernetes cluster with no NetworkPolicies, every pod can talk to every other pod and to the internet. A default-deny policy changes that for one namespace: nothing gets in or out unless a policy allows it. It is one of the most effective controls against lateral movement after a pod is compromised, and one of the easiest to get wrong, because it silently depends on your network plugin, breaks DNS, and needs allow rules on both ends of every connection.

Scope. This guide is based on the Kubernetes documentation for v1.37 (the “Network Policies” concept page and its example manifests in the kubernetes/website repository, read on October 22, 2026). The manifests below passed schema validation with kubeconform 0.6.7 in strict mode against the Kubernetes 1.34 schema. We did not apply them to a running cluster, and behavior can vary by network plugin, as the documentation itself notes.

How isolation works

The documentation describes two independent kinds of isolation for a pod, ingress and egress:

• By default a pod is non-isolated in both directions: all inbound and outbound connections are allowed.
• A pod becomes isolated for a direction when any NetworkPolicy both selects it and lists that direction (Ingress or Egress) in policyTypes.
• Once isolated, only allowed traffic passes. The allowed set is the union of what every applicable policy allows. Reply traffic for allowed connections is permitted automatically.
• Policies are additive and don’t conflict. There is no deny rule and no ordering. The documentation lists explicit deny as something the API cannot do.
• Both ends must agree. For a connection from pod A to pod B, A’s egress and B’s ingress must both allow it.
• Traffic from the pod’s own node is always allowed for ingress, and a pod cannot block access to itself.

Two practical consequences follow. First, the first policy that selects a pod for a direction is what isolates it, even if that policy was meant only to allow something. Second, “default deny” is not a special setting; it is a policy that selects all pods and allows nothing.

One more prerequisite: the documentation states that creating a NetworkPolicy without a network plugin that implements it “will have no effect”. Check that your CNI plugin enforces NetworkPolicy before you rely on any of this.

The default-deny manifests

The Kubernetes documentation provides three variants. This one denies both directions:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-all
  namespace: shop
spec:
  podSelector: {}
  policyTypes:
  - Ingress
  - Egress

podSelector: {} selects every pod in the namespace, including pods created later. The ingress-only and egress-only versions list only one entry in policyTypes.

A detail that trips people up: if you omit policyTypes, the documentation says Ingress is always set and Egress is set only if the policy has egress rules. A policy with no egress rules and no policyTypes therefore does not isolate egress. Write policyTypes explicitly.

Allow DNS in the same change

The documentation has a caution next to the egress example: a default-deny egress policy also blocks DNS. Add the allow before or with the deny:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-dns-egress
  namespace: shop
spec:
  podSelector: {}
  policyTypes:
  - Egress
  egress:
  - to:
    - namespaceSelector:
        matchLabels:
          kubernetes.io/metadata.name: kube-system
      podSelector:
        matchLabels:
          k8s-app: kube-dns
    ports:
    - protocol: UDP
      port: 53
    - protocol: TCP
      port: 53

The k8s-app: kube-dns label is the one the upstream documentation uses for CoreDNS, kept for backward compatibility with the older kube-dns. Managed clusters and node-local DNS caches can differ, so check with kubectl get pods -n kube-system --show-labels before applying. Allow TCP as well as UDP: DNS falls back to TCP for large responses.

Writing the allow rules

The selector indentation trap

The documentation highlights one YAML detail that changes meaning completely:

# ONE peer: pods labelled role=client IN namespaces labelled user=alice
- from:
  - namespaceSelector:
      matchLabels:
        user: alice
    podSelector:
      matchLabels:
        role: client

# TWO peers: role=client pods in THIS namespace, OR any pod in user=alice namespaces
- from:
  - namespaceSelector:
      matchLabels:
        user: alice
  - podSelector:
      matchLabels:
        role: client

The second form is far broader than it looks. The documentation suggests kubectl describe networkpolicy to see how Kubernetes interpreted a policy.

Both ends of an internal connection

With a namespace-wide default deny, a web to api call on port 9000 needs two policies:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: web-to-api-egress
  namespace: shop
spec:
  podSelector:
    matchLabels:
      app: web
  policyTypes:
  - Egress
  egress:
  - to:
    - podSelector:
        matchLabels:
          app: api
    ports:
    - protocol: TCP
      port: 9000
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: api-from-web
  namespace: shop
spec:
  podSelector:
    matchLabels:
      app: api
  policyTypes:
  - Ingress
  ingress:
  - from:
    - podSelector:
        matchLabels:
          app: web
    ports:
    - protocol: TCP
      port: 9000

Traffic from your ingress controller is another namespace-to-pod rule. Use the immutable kubernetes.io/metadata.name label, which the control plane sets on every namespace, rather than a label someone can edit. The example below assumes the controller runs in a namespace called ingress-nginx; adjust it to yours.

  ingress:
  - from:
    - namespaceSelector:
        matchLabels:
          kubernetes.io/metadata.name: ingress-nginx
    ports:
    - protocol: TCP
      port: 8080

Use ipBlock only for addresses outside the cluster

The documentation says ipBlock should be used for cluster-external addresses, because pod IPs are ephemeral. It also warns that source and destination rewriting by load balancers or Service implementations may happen before or after policy evaluation, so the source IP a policy sees for ingress could be a load balancer’s or node’s address. Test any ipBlock rule with your actual plugin.

What NetworkPolicy does not do

From the documentation’s own list, as of v1.37 the API cannot:

• Apply a default policy to all namespaces. You need a policy per namespace, or a third-party tool.
• Log blocked or allowed connections.
• Express explicit deny rules.
• Block traffic from the pod’s node, or loopback.
• Do anything TLS-related, force traffic through a gateway, or target Services by name.

It is defined for TCP, UDP and SCTP. For other protocols such as ICMP, the documentation says behavior is undefined and varies by plugin. Policy behavior for hostNetwork pods is also undefined; most plugins treat their traffic as node traffic.

A rollout order that avoids outages

These steps are our recommendations, not part of the Kubernetes documentation.

1. Confirm enforcement. Check that your CNI plugin enforces NetworkPolicy, and whether it supports endPort and SCTP if you need them.

2. Find namespaces with no policy at all. For example (illustrative, not run against a cluster):

   comm -23 \
     <(kubectl get ns -o jsonpath='{range .items[*]}{.metadata.name}{"\n"}{end}' | sort) \
     <(kubectl get netpol -A -o jsonpath='{range .items[*]}{.metadata.namespace}{"\n"}{end}' | sort -u)
   

3. Inventory real flows for one namespace: Services it exposes, what calls it, and every outbound dependency (databases, other namespaces, external APIs, metadata endpoints, DNS). Your plugin’s flow logs, if it has them, are the best source.

4. Apply the allow policies first, in complete sets. Remember that any policy selecting a pod isolates it in that direction. The DNS policy above selects every pod for egress, so on its own it already blocks all egress except DNS. Apply it together with every other egress allow rule for the namespace, and each pod’s ingress allows together. Start with low-risk namespaces.

5. Apply the default deny, ingress first, then egress. Egress is where unknown dependencies hide.

6. Test from inside the pods and watch error rates. The documentation notes that newly created pods may briefly start before policies are applied, and that a pod may start with no connectivity until allow rules arrive; make startup tolerant of that, for example with an init container that waits for dependencies.

7. Lock down namespace labels. Selectors depend on labels, and the Kubernetes RBAC good-practices page warns that users who can patch a Namespace can change labels to gain access an administrator did not intend. Our RBAC audit guide covers who holds that permission.

Changing policies under live traffic has one more caveat: the documentation says whether a change affects existing connections is implementation-defined.

NetworkPolicy pairs naturally with the admission controls we have covered. Pod Security Admission is also driven by namespace labels, and a ValidatingAdmissionPolicy can require that every new namespace gets a default-deny policy through your deployment tooling.

Limits of this article

We validated the manifests against the Kubernetes schema but did not apply them to a cluster or test them with a specific network plugin. The DNS label and the ingress-controller namespace are common defaults, not universal. Plugin-specific features such as cluster-wide policies, logging and layer 7 rules are outside this article.

Sources and verification

Checked on October 22, 2026.

A mutating admission policy answers a narrow question: when a request to create or update an object reaches the Kubernetes API server, should something in that object be changed before it is stored? Until recently the usual answer was a mutating webhook, which means running and securing a separate HTTPS service in the request path. MutatingAdmissionPolicy (MAP) moves the logic into the API server as declarative CEL expressions. This guide explains how it is built, what the documentation guarantees, and where you should still be careful.

Scope and method. This is research-based analysis of the Kubernetes v1.37 documentation, read on October 18, 2026: the Mutating Admission Policy page (last modified September 28, 2026), the MutatingAdmissionPolicy and MutatingAdmissionPolicyBinding API references (both last modified August 26, 2026), the Admission Controllers reference (last modified September 28, 2026) and the Dynamic Admission Control page. We did not run a Kubernetes cluster and did not test any policy against an API server. The only thing we ran ourselves is a check of generic JSON Patch behavior, described in the JSON Patch section and labelled there.

The short answer

• What it is. A declarative, in-process alternative to mutating admission webhooks. Mutations are written in the Common Expression Language (CEL), the small expression language Kubernetes uses for admission rules.
• Status. The Kubernetes documentation says it is stable since v1.36 and enabled by default, and first became available in v1.30.
• What you create. A MutatingAdmissionPolicy (the logic), a MutatingAdmissionPolicyBinding (the scope), and optionally a parameter resource (the values). At least a policy and a matching binding must exist for anything to happen.
• Where it runs. In the mutating phase of admission, before the validating phase.
• The catch. It is part of the control plane. The documentation says to write and deploy these policies “with great caution”.

The three resources

Two details are easy to miss. If a binding names a policy that does not exist, the API reference says the binding is “considered invalid and will be ignored”. If the policy declares a paramKind and the binding’s paramRef points to a resource that does not exist, the binding is treated as misconfigured and the policy’s failurePolicy applies. Also, per binding, the policy is evaluated once, or once per selected parameter object if parameters are used, and each evaluation has a runtime cost budget.

Which requests a policy sees

Matching is layered, and each layer can only narrow the previous one:

1. matchConstraints on the policy (required) selects resources and operations. The API reference says the operations CREATE, UPDATE and CONNECT are allowed and DELETE may not be matched.
2. matchResources on the binding narrows further. If it is unset, it does not constrain matching.
3. matchConditions on the policy are CEL expressions evaluated last. If any condition is false the policy is skipped; if none is false but one errors, failurePolicy decides. A policy may have at most 64 match conditions.

The documentation notes that namespaceObject is always null in match conditions and that variables cannot be used there, because match conditions are evaluated before the rest of the policy. To select by namespace labels, use a namespaceSelector in matchConstraints or in the binding’s matchResources.

One more limit that affects design: the documentation says that only apiVersion, kind, metadata.name, metadata.generateName and metadata.labels are accessible from the root of the object, and that no other metadata properties are accessible. A rule that depends on an annotation, for example, cannot read it through object. This is the documented behavior; we have not tested workarounds.

Two ways to express a mutation

Each entry in spec.mutations has a patchType, either ApplyConfiguration or JSONPatch.

ApplyConfiguration

The expression must evaluate to a CEL object created with Object{...}, and the result is merged using the server-side apply merge strategy. This is the Kubernetes documentation’s own example, which adds a mesh-proxy init container to new Pods that do not already have one:

apiVersion: admissionregistration.k8s.io/v1
kind: MutatingAdmissionPolicy
metadata:
  name: "sidecar-policy.example.com"
spec:
  paramKind:
    kind: Sidecar
    apiVersion: mutations.example.com/v1
  matchConstraints:
    resourceRules:
    - apiGroups:   [""]
      apiVersions: ["v1"]
      operations:  ["CREATE"]
      resources:   ["pods"]
  matchConditions:
    - name: does-not-already-have-sidecar
      expression: "!object.spec.initContainers.exists(ic, ic.name == \"mesh-proxy\")"
  failurePolicy: Fail
  reinvocationPolicy: IfNeeded
  mutations:
    - patchType: "ApplyConfiguration"
      applyConfiguration:
        expression: >
          Object{
            spec: Object.spec{
              initContainers: [
                Object.spec.initContainers{
                  name: "mesh-proxy",
                  image: "mesh/proxy:v1.0.0",
                  args: ["proxy", "sidecar"],
                  restartPolicy: "Always"
                }
              ]
            }
          }          

The documented restriction to remember: apply configurations may not modify atomic structs, maps or arrays, because of the risk of accidentally deleting values the configuration does not mention.

JSONPatch

The expression must return an array of JSONPatch{op, path, value} values, following the JSON Patch format. The documentation’s examples include a conditional change using a test operation before a replace, and a helper, jsonpatch.escapeKey(), for path keys that contain / or ~. It escapes ~ as ~0 and / as ~1, so a label such as example.com/environment can be addressed.

Read the documentation’s two sidecar examples side by side. The page says the JSON Patch version writes “the same mutation”, but the two examples are not identical: the apply-configuration example uses the image mesh/proxy:v1.0.0 and sets args, while the JSON Patch example uses the image string mesh-proxy/v1.0.0 and no args. Treat them as illustrations of syntax, not as a verified equivalence, and do not copy the image reference from either.

What we tested about JSON Patch

On October 18, 2026 we ran a generic JSON Patch library (Python jsonpatch 1.35, an implementation of the RFC 6902 patch format) on small sample documents. This does not run the Kubernetes API server’s implementation, so it shows how the format behaves, not how your cluster will.

The practical lesson is that a positional JSON Patch needs the parent path to exist and the key to be escaped. The documentation’s sidecar example uses a match condition that reads object.spec.initContainers; whether that expression behaves well when the field is absent is something we did not verify and you should test. The Kubernetes documentation we read does not say what the API server does when a JSON Patch fails to apply, so do not assume it is silently skipped.

Reinvocation: why your mutation must be safe to repeat

reinvocationPolicy is a required field on the policy, with two values:

• Never: the mutations are not called more than once per binding in a single admission evaluation.
• IfNeeded: they may be called more than once per binding for a single admission request, only when other mutations changed the object after this one ran. The API reference states there is no guarantee of order relative to other admission plugins, webhooks, bindings of this policy and other admission policies.

The reason is explained on the Dynamic Admission Control page: no single ordering of mutating plugins works for every case, so mutating plugins can be re-run so that they see changes made by others. That page also states that mutating webhooks must be idempotent, meaning able to process an object they have already modified. Our reading is that the same discipline applies to a policy using IfNeeded, which is why the documentation’s example begins with a match condition that skips Pods that already have the mesh-proxy container. For comparison, the webhook field reinvocationPolicy defaults to Never.

Failure behavior

failurePolicy accepts Fail or Ignore and defaults to Fail. The API reference says failures can come from CEL parse errors, type-check errors, runtime errors, and invalid or misconfigured policy definitions or bindings, such as a paramKind that refers to a non-existent kind. It does not define how validations that evaluate to false are handled; that is a validation concept, not a mutation one.

For a control-plane component, the difference matters. With Fail, an error in your expression can reject writes to every object you matched. With Ignore, the same error means the mutation is silently not applied, and anything that depended on it, such as a validating rule that expects the label to be there, may then reject the object or let a bad one through. The Admission Controllers page says that if any MutatingAdmissionPolicy fails, the request fails.

How it fits with other admission mechanisms

The Admission Controllers reference says admission runs in two phases: mutating controllers first, then validating ones, and a rejection in either phase rejects the whole request.

• With Pod Security Admission. The reference lists the PodSecurity plugin as validating. Pods are therefore checked after mutation. See our guide to Pod Security Admission.
• With ValidatingAdmissionPolicy. A mutating policy can set a value that a validating policy then checks. Our ValidatingAdmissionPolicy guide covers the validating side, which has Deny, Warn and Audit actions that the MAP binding does not.
• With image signature checks. Verifying cosign signatures at admission is done by a webhook such as Sigstore’s policy-controller; see our guide to verifying signed container images in Kubernetes.
• With webhooks. Built-in mutating plugins are re-run if a mutating webhook modifies an object, per the Dynamic Admission Control page.

Exempt kinds

Policies created through the REST API cannot mutate ValidatingAdmissionPolicies, their bindings, MutatingAdmissionPolicies or their bindings. The documentation says a policy loaded through manifest-based admission control can intercept those kinds. Separately, the non-persisted authentication and authorization review kinds (TokenReviews, SelfSubjectReviews, LocalSubjectAccessReviews, SelfSubjectAccessReviews, SelfSubjectRulesReviews and SubjectAccessReviews) are exempt from all admission policies, and the page says that starting with Kubernetes v1.37 admission webhooks also exclude them by default.

MAP, VAP or webhook: a decision guide

This table is our summary of the documented capabilities, not a benchmark. We did not measure latency or reliability differences between policies and webhooks, and we make no claim about them.

A cautious rollout

The MAP binding has no Warn or Audit action, so you cannot trial a policy in a non-enforcing mode the way you can with a validating binding. Our suggestions, all untested by us:

1. Start in a throwaway cluster on the Kubernetes minor version you run, because the feature became stable in v1.36 and your cluster may be older.
2. Scope the binding narrowly with matchResources and a namespaceSelector. The kubernetes.io/metadata.name label that the API server sets on namespaces, as covered in our ValidatingAdmissionPolicy guide, gives you a name-based selector without labelling anything.
3. Make the mutation idempotent and put the “already done” check in matchConditions.
4. Decide on failurePolicy deliberately. Use Ignore while developing so a typo does not block writes, then switch to Fail once you have shown the mutation applies.
5. Re-read the resulting object, and verify that your validating rules and Pod Security levels still accept it.

Limits of this article

• We did not run a Kubernetes cluster, so no policy in this article was tested against an API server.
• The JSON Patch results come from a generic Python library, not from Kubernetes.
• We did not find, in the pages we read, statements about what happens when a JSON Patch fails to apply, or about the ordering of several mutating policies; we have not claimed any.
• The Admission Controllers page still describes the MutatingAdmissionPolicy plugin as needing the mutatingadmissionpolicy feature gate and the admissionregistration.k8s.io/v1alpha1 API group, while its default-plugin list for 1.37 includes it and the MAP page says stable and enabled by default. We follow the latter two and flag the inconsistency.
• Documentation changes. Re-check the pages below for your Kubernetes version.

Sources and verification

Documentation read October 18, 2026 (site version v1.37).