Kubernetes audit log detections work because most attack techniques against a cluster have to go through kube-apiserver. Getting a shell in a Pod, starting a container that mounts the node’s filesystem, granting yourself cluster-admin, sweeping Secrets, minting a token: each is an API request, and each leaves an audit event with the identity, the verb, the resource and the result. The eight rules below turn those events into alerts.

We wrote them as jq filters so you can run them against a log file today, then ran them against kube-apiserver v1.34.1 while we played out a small attack: a stolen CI credential probing what it can do, a compromised administrator setting up persistence, and an anonymous caller. This article shows what each rule caught, what it would miss, and how to tune it before you port it to a SIEM.

The rules assume an audit policy that logs everything at Metadata or above and keeps request bodies for writes. If you do not have one yet, start with our Kubernetes audit policy guide; we used the same six-rule policy here.

The eight rules

The ATT&CK IDs and names are from MITRE’s STIX data (enterprise collection version 20.0). The mapping is ours: ATT&CK describes techniques, not audit fields.

The test

We ran kube-apiserver v1.34.1 and etcd 3.6.4 (the controller-runtime envtest binaries) with RBAC authorization, static bearer tokens for three users and the audit policy linked above, writing JSON lines to a file. There were no nodes, kubelets, controller manager or scheduler, so Pods were never scheduled. That keeps the log readable, but a real cluster’s log is far noisier; see the limits section.

The scenario, in order:

1. Benign: dev-bob, in the developers group, creates a Pod in dev, reads a Secret in his own namespace and runs kubectl exec.
2. Stolen CI token: ci-deployer, which can create Pods and exec in prod, runs kubectl auth can-i --list, tries to list Secrets, nodes, namespaces, ClusterRoles, ConfigMaps and ServiceAccounts cluster-wide, tries Secrets in kube-system, then tries to create a privileged Pod with hostPID, hostNetwork and a hostPath mount of /. The API server rejected that one (it was running with the default --allow-privileged=false), so the attacker dropped the privileged flags and created a Pod that only mounts / from the host. That one succeeded.
3. Compromised admin: alice binds cluster-admin to the prod/default service account, mints a 24-hour token for it with kubectl create token, reads kube-system Secrets while impersonating it, and attaches a debug container to a running Pod with kubectl debug.
4. Anonymous: an unauthenticated curl to /api/v1/namespaces/kube-system/secrets.

The resulting log had 574 events. 532 came from system:apiserver itself; the other 42 were ours (alice 18, ci-deployer 15, dev-bob 8, anonymous 1).

The rules as a script

Save this as detections.sh and run sh detections.sh audit.log. It needs jq 1.6 or later for IN; we ran it with jq 1.7.

#!/bin/sh
# Kubernetes audit log detections. Usage: ./detections.sh audit.log
# Input: kube-apiserver JSON-lines audit log (audit.k8s.io/v1 Event objects).
LOG="$1"

# Control-plane identities to ignore. Extend this for your distribution.
NOT_CP='(.user.username | IN("system:apiserver","system:kube-controller-manager","system:kube-scheduler") | not)'
# Count each request once: long-running calls such as exec also emit ResponseStarted.
DONE='.stage=="ResponseComplete"'

echo "D1 exec, attach and ephemeral containers"
jq -c "select($DONE)"' | select(.objectRef.resource=="pods"
      and (.objectRef.subresource | IN("exec","attach","ephemeralcontainers")))
  | {user:.user.username, sub:.objectRef.subresource, verb, ns:.objectRef.namespace,
     pod:.objectRef.name, code:.responseStatus.code, uri:.requestURI}' "$LOG"

echo "D2 pods requesting host access (allowed or rejected)"
jq -c 'select(.objectRef.resource=="pods" and .verb=="create" and .objectRef.subresource==null)
  | select(.requestObject.spec.hostPID or .requestObject.spec.hostNetwork or .requestObject.spec.hostIPC
      or any(.requestObject.spec.volumes[]?; .hostPath)
      or any(.requestObject.spec.containers[]?; .securityContext.privileged))
  | {user:.user.username, ns:.objectRef.namespace, pod:.objectRef.name, code:.responseStatus.code,
     hostPaths:[.requestObject.spec.volumes[]?.hostPath.path | select(.)]}' "$LOG"

echo "D3 bindings that grant cluster-admin"
jq -c "select($NOT_CP)"' | select(.objectRef.apiGroup=="rbac.authorization.k8s.io"
      and (.objectRef.resource | IN("clusterrolebindings","rolebindings"))
      and (.verb | IN("create","update","patch"))
      and .requestObject.roleRef.name=="cluster-admin")
  | {user:.user.username, binding:.objectRef.name, subjects:.requestObject.subjects,
     code:.responseStatus.code}' "$LOG"

echo "D4 Secret reads across all namespaces or in kube-system"
jq -c "select($NOT_CP)"' | select(.objectRef.resource=="secrets" and (.verb | IN("get","list","watch")))
  | select(.objectRef.namespace==null or .objectRef.namespace=="kube-system")
  | {user:.user.username, as:.impersonatedUser.username, verb, ns:.objectRef.namespace,
     code:.responseStatus.code}' "$LOG"

echo "D5 service account tokens minted outside the kubelet"
jq -c "select($NOT_CP)"' | select(.objectRef.resource=="serviceaccounts"
      and .objectRef.subresource=="token" and .verb=="create")
  | select(.user.username | startswith("system:node:") | not)
  | {user:.user.username, sa:(.objectRef.namespace + "/" + .objectRef.name),
     code:.responseStatus.code}' "$LOG"

echo "D6 impersonated requests"
jq -c 'select(.impersonatedUser)
  | {user:.user.username, as:.impersonatedUser.username, verb, res:.objectRef.resource,
     ns:.objectRef.namespace, code:.responseStatus.code}' "$LOG"

echo "D7a permission listing (kubectl auth can-i --list)"
jq -c 'select(.objectRef.resource=="selfsubjectrulesreviews")
  | {user:.user.username, ns:.objectRef.namespace, ip:.sourceIPs[0]}' "$LOG"

echo "D7b identities with 5 or more denied requests in the file"
jq -sc '[.[] | select(.annotations["authorization.k8s.io/decision"]=="forbid")]
  | group_by(.user.username)
  | map({user:.[0].user.username, denied:length,
         resources:([.[].objectRef.resource] | unique)})
  | map(select(.denied >= 5))' "$LOG"

echo "D8 anonymous requests"
jq -c 'select(.user.username=="system:anonymous")
  | {uri:.requestURI, ip:.sourceIPs[0], code:.responseStatus.code}' "$LOG"

What each rule caught

All 15 hits from the run, condensed:

Every step of both attack paths fired at least one rule. The only benign activity that fired was dev-bob’s exec, which is the false positive you would expect: D1 is a “who is in our workloads” feed, not a high-severity alert on its own.

D1: exec, attach and debug sessions

Two details matter. First, exec is a long-running request, and the Kubernetes docs say ResponseStarted “is only generated for long-running requests”. Our first version of the rule returned every exec event twice, once per stage; filtering on ResponseComplete fixed it. Second, one kubectl exec produced both a get and a create on pods/exec, so count sessions by auditID or by time window, not by event.

kubectl debug does not use exec at all. It sends a patch to the pods/ephemeralcontainers subresource, which is why the rule matches that subresource explicitly. Because it is a write, our policy logged it at Request level, and the event carried the debug container’s image (busybox:1.36). A debug container running an image you have never seen in the cluster is worth a look.

To tune: alert on exec into kube-system and other platform namespaces, exec by service accounts (people run exec, workloads rarely should), and exec whose requestURI contains shells or tools you do not expect. Route the rest to a lower-severity feed.

D2: Pods asking for the host

The fields come from the Pod Security Standards’ Baseline profile, which disallows host namespaces (spec.hostNetwork, spec.hostPID, spec.hostIPC), privileged containers and hostPath volumes. If you enforce those with Pod Security Admission, D2 tells you who keeps trying.

The useful finding from our run is that the rejected request was logged in full. The privileged Pod failed validation with HTTP 422 (“disallowed by cluster policy”), and the audit event still held the complete requestObject, including the hostPath of /. That is why D2 does not filter on success. An attacker’s first attempt is often the one that fails.

D2 needs Request level on Pod creates; at Metadata the spec is not there. It also only sees Pods. Deployments, DaemonSets, Jobs and CronJobs create Pods through controllers, so the Pod create event’s user is the controller’s identity. Add the same select on .requestObject.spec.template.spec for workload resources to see the person who submitted the template.

D3: bindings to cluster-admin

Without the control-plane exclusion, this rule fired on startup: system:apiserver creates the cluster-admin ClusterRoleBinding to system:masters when it bootstraps RBAC. After excluding it, the only hit was the persistence binding.

The binding name backup-sync is deliberately bland; do not rely on names. The rule matches roleRef.name, which our policy logs because RBAC writes are kept at RequestResponse. Extend it with the other roles that are as good as cluster-admin in your cluster, and with writes to ClusterRoles that add * verbs or escalate, bind or impersonate. Our RBAC permission audit lists them.

D4: Secret sweeps

A cluster-wide list has no namespace in objectRef, which is how D4 finds it. Our policy logs Secrets at Metadata, so none of these events contain Secret values, and the rule does not need them: who, which verb, which scope and the response code are enough.

The 403s are as interesting as the 200. Two of the four hits were the stolen CI token being denied, which tells you someone is trying the credential well outside its job. Do not exclude every system: user here. system:serviceaccount:... identities are system: users too: had the attacker used the minted token directly instead of impersonating, the read would have been logged as system:serviceaccount:prod:default and a blanket filter would have dropped it. That is why the script excludes control-plane components by exact name. Some distributions run controllers under kube-system service accounts; add those by name after you have looked at your own log.

D5: minted tokens

kubectl create token is a create on the serviceaccounts/token subresource. Our policy logs that subresource at Metadata, so the event recorded who asked and for which service account, but not the token. Keep it that way: at RequestResponse the response would contain a usable token.

The kubelet requests tokens on behalf of Pods that use projected service account tokens, so in a real cluster most of these events come from system:node:<name> identities. The rule drops those. What is left, a person or a workload minting a token for another service account, is rare and worth reviewing. Our service account token guide covers why bound tokens are still worth watching.

D6: impersonation

When a request uses impersonation, the audit event keeps the real caller in user and adds impersonatedUser. Here alice read kube-system Secrets as the service account she had just made cluster-admin. D4 fired on that read too, but only D6 tells you the request ran with the service account’s permissions, which is what links the read back to the binding created minutes earlier. Treat D6 as context: join it to every other rule so each alert shows both the real caller and the identity whose permissions were used.

D7: permission discovery

kubectl auth can-i --list sends a SelfSubjectRulesReview, which is logged as a create on selfsubjectrulesreviews. Developers run it too, so it is weak alone, but it is a strong signal from a CI or workload identity: our stolen CI token ran it first. The burst rule then found 8 denials for ci-deployer across 6 resource types.

The threshold of 5 per file is only for this demo. In a SIEM, count forbid decisions per identity over a window such as 10 minutes, and weight by the number of distinct resources: one identity hitting one forbidden resource repeatedly is usually a broken deployment; one identity hitting many is usually someone exploring.

D8: anonymous requests

kube-apiserver accepts anonymous requests by default (--anonymous-auth defaults to true; they arrive as system:anonymous in group system:unauthenticated). RBAC denied our anonymous call, but the event is still useful: anonymous requests for resource paths from outside your network mean the API server is reachable by people scanning for it. Any anonymous request that gets a 2xx on a resource path is an incident.

Porting the rules to a SIEM

The fields used here (user.username, impersonatedUser, objectRef.*, verb, stage, responseStatus.code, annotations, requestObject, requestURI, sourceIPs) are fields of the audit.k8s.io/v1 Event object, so the logic carries over to any backend that stores the events. Managed services may wrap each event in their own envelope and may run their own audit policy, which decides whether requestObject is present at all. Check one raw event from your provider before you trust a rule that depends on request bodies.

Three practical points:

• Deduplicate on auditID. It is the same across a request’s stages.
• Alert on attempts. Keep 403 and 422 results in every rule unless you have a reason to drop them.
• Join with identity context. Mapping user.username and sourceIPs to a person, a pipeline or a node is what turns these rules from noise into findings. Our incident logging guide covers what to keep alongside the audit log.

Limits of this article

We tested on a single kube-apiserver with no nodes, controllers or scheduler, so no Pod ever ran and exec returned 400 instead of opening a session. A production log has far more control-plane and controller traffic, and your exclusion list will need tuning against it. We used static tokens; OIDC, certificates and cloud IAM identities change what user.username looks like. Managed Kubernetes providers control their own audit policies, and some do not log request bodies, which disables D2, D3 and the image detail in D1. These rules detect API activity only: they do not see what happens inside a container or on a node, which needs runtime or host telemetry.

Sources and verification

A Kubernetes audit policy is the file that tells kube-apiserver which API requests to record and how much of each to keep. Without one, upstream kube-apiserver records nothing. A good policy logs everything at Metadata at minimum, keeps request bodies for changes that alter access, and never logs the contents of Secrets. Rule order is what makes that work: the first matching rule wins, and a broad rule placed too high will quietly write Secret values into your logs. We demonstrate that below.

This guide gives a six-rule policy we ran against kube-apiserver v1.34.1, shows what each rule captured, and ends with queries you can run against the log during an incident.

How kube-apiserver evaluates a policy

Every request produces audit events at one or more stages:

The policy is a list of rules. kube-apiserver compares each event with the rules in order, and the first rule that matches sets the event’s level:

Request and RequestResponse don’t apply to non-resource URLs such as /healthz. A rule can match on users, userGroups, verbs, resources (group, resource, subresource and resourceNames), namespaces and nonResourceURLs. A rule with no match fields matches everything, which is what you want for the last rule and exactly what you don’t want near the top.

The policy file is passed with --audit-policy-file. If the flag is omitted, no events are logged, and a policy with no rules is rejected.

A policy that answers incident questions

The policy below is built around the questions you get asked during an incident: who read this Secret, who exec’d into that Pod, who granted this binding, and was it allowed. It is a starting point, not a complete production policy; see the limits at the end.

apiVersion: audit.k8s.io/v1
kind: Policy
omitStages:
  - "RequestReceived"
rules:
  # 1. Drop health and version probes.
  - level: None
    nonResourceURLs: ["/healthz*", "/livez*", "/readyz*", "/version"]
  # 2. Sensitive payloads: who touched them, never the contents.
  - level: Metadata
    resources:
      - group: ""
        resources: ["secrets", "configmaps", "serviceaccounts/token"]
      - group: "authentication.k8s.io"
        resources: ["tokenreviews"]
  # 3. Interactive access into running workloads.
  - level: Metadata
    resources:
      - group: ""
        resources: ["pods/exec", "pods/attach", "pods/portforward", "pods/proxy", "nodes/proxy"]
  # 4. Changes to who can do what, and to admission control.
  - level: RequestResponse
    verbs: ["create", "update", "patch", "delete", "deletecollection"]
    resources:
      - group: "rbac.authorization.k8s.io"
      - group: "admissionregistration.k8s.io"
  # 5. Other writes: keep the submitted object.
  - level: Request
    verbs: ["create", "update", "patch", "delete", "deletecollection"]
  # 6. Everything else (reads, watches, non-resource URLs).
  - level: Metadata

Why each rule is where it is:

Pod Security Admission and our RBAC permission audit tell you what could happen. The audit log is how you find out what did.

What it recorded

We ran the policy on kube-apiserver v1.34.1 (the envtest build from the kubernetes-sigs/controller-tools releases) with etcd and RBAC authorization, then made requests as a system:masters user and as an unprivileged user alice. There were no nodes, so no Pod actually ran. Results:

Two details from the test are worth building detections around:

• Exec shows up under more than one verb. That single kubectl exec produced events with verb get and verb create for the exec subresource. Filter on objectRef.subresource == "exec", not on the verb.
• Impersonation keeps both identities. The authenticated user is in user and the identity they acted as is in impersonatedUser. If you only alert on user.username, impersonated actions look like they came from the admin’s own account.

Also keep in mind what the API reference warns about two fields: userAgent is reported by the client “and must not be trusted”, and in sourceIPs all entries except the last “can be arbitrarily set by the client”.

The rule-order mistake that leaks Secrets

We then moved rule 5 (Request for all writes) above rule 2, restarted kube-apiserver with that policy, and created a Secret with the value S3cretValue-456. The audit log now contained:

"data":{"key":"UzNjcmV0VmFsdWUtNDU2"}

That is the base64 encoding of S3cretValue-456. The broad write rule matched first, logged the request body at Request level, and the Metadata rule for Secrets was never reached. With the correct order, the equivalent test (a Secret holding S3cretValue-123) left no trace of either the plain or the encoded value in the log.

This is the most common way an audit log becomes a credential store. Your audit pipeline usually has wider read access than the cluster’s Secrets do, so treat a body-logging rule above your Secrets rule as a bug. Check it in code review with a simple test: create a dummy Secret in a test cluster and search the log for its encoded value.

If you already store Secrets encrypted at rest, remember that encryption protects etcd, not the audit log. And a Secret-based service account token leaked this way never expires.

Backends and volume

There are two backends: a log file and a webhook.

• Log backend: --audit-log-path sets the file (- means stdout). Rotation is controlled by --audit-log-maxage (days), --audit-log-maxbackup (files) and --audit-log-maxsize (megabytes). Batching is off by default for the log backend.
• Webhook backend: --audit-webhook-config-file points at a kubeconfig-format file for the remote endpoint. Batching is on by default, with a maximum of 400 events per batch and a 30-second maximum wait.
• Truncation of oversized events is off by default for both backends and has to be enabled explicitly.

The Kubernetes docs also note that auditing increases API server memory use, because some context is kept for each request, and that the increase depends on the policy. RequestResponse on a busy resource is where cost grows: the GCE reference policy, for instance, logs get, list and watch at Request level rather than RequestResponse because “Get responses can be large”.

Ship the log off the control plane. An attacker with control-plane access can edit a local file, and an incident log is only useful if it survives the incident.

Managed Kubernetes

On EKS, AKS and similar services the provider runs kube-apiserver, so you generally choose whether audit events are exported, not what the policy contains. Check your provider’s documentation for what its policy logs.

• Amazon EKS: control plane logs are not sent to CloudWatch Logs by default. Enable the audit log type per cluster, for example with aws eks update-cluster-config --logging. The audit stream is named kube-apiserver-audit-<id> in the /aws/eks/<cluster>/cluster log group. AWS notes delivery is best effort and standard CloudWatch Logs charges apply.
• Azure AKS: resource logs are not collected until you create a diagnostic setting. Microsoft’s guidance warns that kube-audit can be costly and suggests kube-audit-admin, which leaves out get and list events. That saves money but drops exactly the read events, such as Secret reads, that matter in a credential-theft investigation. Decide that trade-off deliberately.

Queries to keep ready

These run against a JSON-lines audit log with jq. We ran each one against the log from the test above.

# Who read or changed Secrets (excluding the API server itself)
jq -c 'select(.objectRef.resource=="secrets" and .user.username!="system:apiserver")
  | {ts:.requestReceivedTimestamp, user:.user.username, verb, ns:.objectRef.namespace,
     name:.objectRef.name, code:.responseStatus.code}' audit.log

# Every exec into a Pod, with the command line
jq -c 'select(.objectRef.subresource=="exec")
  | {ts:.requestReceivedTimestamp, user:.user.username, ns:.objectRef.namespace,
     pod:.objectRef.name, uri:.requestURI}' audit.log

# Denied requests, including impersonated ones
jq -c 'select(.annotations["authorization.k8s.io/decision"]=="forbid")
  | {user:.user.username, as:.impersonatedUser.username, verb,
     res:.objectRef.resource, ns:.objectRef.namespace}' audit.log

# RBAC writes by people rather than the control plane, with what they granted
jq -c 'select(.objectRef.apiGroup=="rbac.authorization.k8s.io"
       and (.verb|test("create|update|patch|delete"))
       and (.user.username|startswith("system:")|not))
  | {user:.user.username, verb, res:.objectRef.resource, name:.objectRef.name,
     subjects:.requestObject.subjects, role:.requestObject.roleRef.name}' audit.log

A burst of forbid events from one identity across many resources can be a sign that someone is enumerating what a stolen credential can do.

Limits of this article

We tested on a single kube-apiserver v1.34.1 with no nodes, no controller manager and static token authentication, so no workload actually ran and no kubelet or controller traffic was present. A real cluster produces far more events, and you will want extra None or Metadata rules for high-volume system traffic (the GCE policy is a good reference). The policy does not cover custom resources that hold credentials; add those to rule 2. Request level on Pod writes will log any credentials placed directly in environment variables. We did not test the webhook backend or any managed service.

Sources and verification