Kubernetes Audit Policy: What to Log, at What Level

A Kubernetes audit policy is the file that tells kube-apiserver which API requests to record and how much of each to keep. Without one, upstream kube-apiserver records nothing. A good policy logs everything at Metadata at minimum, keeps request bodies for changes that alter access, and never logs the contents of Secrets. Rule order is what makes that work: the first matching rule wins, and a broad rule placed too high will quietly write Secret values into your logs. We demonstrate that below.

This guide gives a six-rule policy we ran against kube-apiserver v1.34.1, shows what each rule captured, and ends with queries you can run against the log during an incident.

How kube-apiserver evaluates a policy

Every request produces audit events at one or more stages:

The policy is a list of rules. kube-apiserver compares each event with the rules in order, and the first rule that matches sets the event’s level:

Request and RequestResponse don’t apply to non-resource URLs such as /healthz. A rule can match on users, userGroups, verbs, resources (group, resource, subresource and resourceNames), namespaces and nonResourceURLs. A rule with no match fields matches everything, which is what you want for the last rule and exactly what you don’t want near the top.

The policy file is passed with --audit-policy-file. If the flag is omitted, no events are logged, and a policy with no rules is rejected.

A policy that answers incident questions

The policy below is built around the questions you get asked during an incident: who read this Secret, who exec’d into that Pod, who granted this binding, and was it allowed. It is a starting point, not a complete production policy; see the limits at the end.

apiVersion: audit.k8s.io/v1
kind: Policy
omitStages:
  - "RequestReceived"
rules:
  # 1. Drop health and version probes.
  - level: None
    nonResourceURLs: ["/healthz*", "/livez*", "/readyz*", "/version"]
  # 2. Sensitive payloads: who touched them, never the contents.
  - level: Metadata
    resources:
      - group: ""
        resources: ["secrets", "configmaps", "serviceaccounts/token"]
      - group: "authentication.k8s.io"
        resources: ["tokenreviews"]
  # 3. Interactive access into running workloads.
  - level: Metadata
    resources:
      - group: ""
        resources: ["pods/exec", "pods/attach", "pods/portforward", "pods/proxy", "nodes/proxy"]
  # 4. Changes to who can do what, and to admission control.
  - level: RequestResponse
    verbs: ["create", "update", "patch", "delete", "deletecollection"]
    resources:
      - group: "rbac.authorization.k8s.io"
      - group: "admissionregistration.k8s.io"
  # 5. Other writes: keep the submitted object.
  - level: Request
    verbs: ["create", "update", "patch", "delete", "deletecollection"]
  # 6. Everything else (reads, watches, non-resource URLs).
  - level: Metadata

Why each rule is where it is:

Pod Security Admission and our RBAC permission audit tell you what could happen. The audit log is how you find out what did.

What it recorded

We ran the policy on kube-apiserver v1.34.1 (the envtest build from the kubernetes-sigs/controller-tools releases) with etcd and RBAC authorization, then made requests as a system:masters user and as an unprivileged user alice. There were no nodes, so no Pod actually ran. Results:

Two details from the test are worth building detections around:

• Exec shows up under more than one verb. That single kubectl exec produced events with verb get and verb create for the exec subresource. Filter on objectRef.subresource == "exec", not on the verb.
• Impersonation keeps both identities. The authenticated user is in user and the identity they acted as is in impersonatedUser. If you only alert on user.username, impersonated actions look like they came from the admin’s own account.

Also keep in mind what the API reference warns about two fields: userAgent is reported by the client “and must not be trusted”, and in sourceIPs all entries except the last “can be arbitrarily set by the client”.

The rule-order mistake that leaks Secrets

We then moved rule 5 (Request for all writes) above rule 2, restarted kube-apiserver with that policy, and created a Secret with the value S3cretValue-456. The audit log now contained:

"data":{"key":"UzNjcmV0VmFsdWUtNDU2"}

That is the base64 encoding of S3cretValue-456. The broad write rule matched first, logged the request body at Request level, and the Metadata rule for Secrets was never reached. With the correct order, the equivalent test (a Secret holding S3cretValue-123) left no trace of either the plain or the encoded value in the log.

This is the most common way an audit log becomes a credential store. Your audit pipeline usually has wider read access than the cluster’s Secrets do, so treat a body-logging rule above your Secrets rule as a bug. Check it in code review with a simple test: create a dummy Secret in a test cluster and search the log for its encoded value.

If you already store Secrets encrypted at rest, remember that encryption protects etcd, not the audit log. And a Secret-based service account token leaked this way never expires.

Backends and volume

There are two backends: a log file and a webhook.

• Log backend: --audit-log-path sets the file (- means stdout). Rotation is controlled by --audit-log-maxage (days), --audit-log-maxbackup (files) and --audit-log-maxsize (megabytes). Batching is off by default for the log backend.
• Webhook backend: --audit-webhook-config-file points at a kubeconfig-format file for the remote endpoint. Batching is on by default, with a maximum of 400 events per batch and a 30-second maximum wait.
• Truncation of oversized events is off by default for both backends and has to be enabled explicitly.

The Kubernetes docs also note that auditing increases API server memory use, because some context is kept for each request, and that the increase depends on the policy. RequestResponse on a busy resource is where cost grows: the GCE reference policy, for instance, logs get, list and watch at Request level rather than RequestResponse because “Get responses can be large”.

Ship the log off the control plane. An attacker with control-plane access can edit a local file, and an incident log is only useful if it survives the incident.

Managed Kubernetes

On EKS, AKS and similar services the provider runs kube-apiserver, so you generally choose whether audit events are exported, not what the policy contains. Check your provider’s documentation for what its policy logs.

• Amazon EKS: control plane logs are not sent to CloudWatch Logs by default. Enable the audit log type per cluster, for example with aws eks update-cluster-config --logging. The audit stream is named kube-apiserver-audit-<id> in the /aws/eks/<cluster>/cluster log group. AWS notes delivery is best effort and standard CloudWatch Logs charges apply.
• Azure AKS: resource logs are not collected until you create a diagnostic setting. Microsoft’s guidance warns that kube-audit can be costly and suggests kube-audit-admin, which leaves out get and list events. That saves money but drops exactly the read events, such as Secret reads, that matter in a credential-theft investigation. Decide that trade-off deliberately.

Queries to keep ready

These run against a JSON-lines audit log with jq. We ran each one against the log from the test above.

# Who read or changed Secrets (excluding the API server itself)
jq -c 'select(.objectRef.resource=="secrets" and .user.username!="system:apiserver")
  | {ts:.requestReceivedTimestamp, user:.user.username, verb, ns:.objectRef.namespace,
     name:.objectRef.name, code:.responseStatus.code}' audit.log

# Every exec into a Pod, with the command line
jq -c 'select(.objectRef.subresource=="exec")
  | {ts:.requestReceivedTimestamp, user:.user.username, ns:.objectRef.namespace,
     pod:.objectRef.name, uri:.requestURI}' audit.log

# Denied requests, including impersonated ones
jq -c 'select(.annotations["authorization.k8s.io/decision"]=="forbid")
  | {user:.user.username, as:.impersonatedUser.username, verb,
     res:.objectRef.resource, ns:.objectRef.namespace}' audit.log

# RBAC writes by people rather than the control plane, with what they granted
jq -c 'select(.objectRef.apiGroup=="rbac.authorization.k8s.io"
       and (.verb|test("create|update|patch|delete"))
       and (.user.username|startswith("system:")|not))
  | {user:.user.username, verb, res:.objectRef.resource, name:.objectRef.name,
     subjects:.requestObject.subjects, role:.requestObject.roleRef.name}' audit.log

A burst of forbid events from one identity across many resources can be a sign that someone is enumerating what a stolen credential can do.

Limits of this article

We tested on a single kube-apiserver v1.34.1 with no nodes, no controller manager and static token authentication, so no workload actually ran and no kubelet or controller traffic was present. A real cluster produces far more events, and you will want extra None or Metadata rules for high-volume system traffic (the GCE policy is a good reference). The policy does not cover custom resources that hold credentials; add those to rule 2. Request level on Pod writes will log any credentials placed directly in environment variables. We did not test the webhook backend or any managed service.

Sources and verification

KubernetesAudit LoggingIncident ResponseDetection